Instructions — what the model was asked we need to add the ability to upload files via MCP otherwise we cant actually add new tasks or cases can we? we need to be able to give files that go into the input context. can you build that and give me the PR to send to my eng team Start from input/start-code.zip (EveryInc/checks at commit 18df437). input/instructions.txt preserves the original request and input/START_HERE.md explains the...
Read all we need to add the ability to upload files via MCP otherwise we cant actually add new tasks or cases can we? we need to be able to give files that go into the input context. can you build that and give me the PR to send to my eng team
Start from input/start-code.zip (EveryInc/checks at commit 18df437). input/instructions.txt preserves the original request and input/START_HERE.md explains the archive. Work in the extracted code/ tree. Give me the PR link, or a reviewable patch if authenticated GitHub access is unavailable.
4 of 7 checks failed ↓ ⇤ Checks beside the answer
The model left no answer file or screenshots.
4 failed fail
No AI-tell wordingai-wording every task
deliverable text contains none of the shared AI-tell phrases in bench/ai-tells.txt. Files byte-identical to something the task shipped (its inputs, the selected case, or a member of a shipped zip) are the brief's text, not the model's, and are not inspected. Task checks may add narrower contractual bans, but must not repeat this shared floor.
no readable deliverable text to inspect
Read all Show less fail
Focused diff sizefocused-diff-size this case
Is the final patch at most 15 changed files and 220 added lines? Nitesh's PR #33 changed 14 files and added 171 lines; the larger agent PR #35 changed 31 files and added 402 lines.
no final patch or changed code tree is available to measure
Read all Show less fail
Reject bad inputsthis case
Judge's reasoning
The tool does no path validation whatsoever — it only does `path = file["path"] or raise ArgumentError`, so `../../escape`, `prompt.md` (which overwrites `files["#{root}/prompt.md"]` set earlier in bundle.rb:96) and `eval/checks.rb` all attach successfully, and a missing/invalid `content` silently becomes `Base64.decode64("")` bytes; only the inherited `write_refusal` authorization check exists.
Read all Show less Rubric Do unauthorized uploads, paths outside the case workspace, paths that would shadow prompt.md or eval/, and non-file multipart values fail before attaching an input? FAIL if any such request reports success or writes bytes.
fail
Reviewable PR handoffthis case
Judge's reasoning
There is no PR link, no patch/diff file and not even a git repository in the delivered tree (`find . -name .git` returns nothing, no *.patch anywhere), and the only test added is a single case in test/integration/mcp_test.rb with no bundle round-trip or rejection coverage — and it cannot pass as written, since line 113 asserts the unsorted `["data/customers.csv", "assets/logo.png"]` against `kase.inputs.map(&:path).sort`.
Read all Show less Rubric Did the agent hand over a reviewable PR link, or a patch when GitHub access is unavailable, with focused tests that prove upload, bundle round trip and rejection behavior? FAIL if the response is only a plan or no working artifact is ready.
3 passed pass
Finished within 30 minutescompletion-within-30m this case
Did the agent finish successfully within 30 minutes of the runner start, without a timeout?
agent run took 3.1 minutes
Read all Show less pass
Usable MCP upload paththis case
Judge's reasoning
A new registered MCP tool `app/tools/mcp/upload_case_input_tool.rb` (`upload_case_input`, wired into config/initializers/mcp.rb alongside add_task/add_case) lets an editor/owner attach text and binary files as case inputs via base64 `{path, content}` entirely within MCP, no app upload needed.
Read all Show less Rubric Can an authorized editor or owner create a task or case through MCP and immediately attach text and binary files as case inputs using a returned inputs_url or an equally direct MCP workflow? FAIL if a person still must upload through the app or files only get pasted into the prompt.
pass
Input round tripthis case
Judge's reasoning
The tool stores bytes via `kase.inputs.attach(io: StringIO.new(content), ..., metadata: { path: })`, which the existing `config/initializers/attachment_paths.rb` surfaces as `#path` and `app/models/bundle.rb:97` writes to `"#{root}/#{f.path}"`, and a repeat path is purged (`existing&.purge`) before re-attaching so it replaces rather than duplicates.
Read all Show less Rubric Do uploaded file bytes and relative workspace paths survive into the case bundle that a model runs with, including replacement of a second upload at the same path? FAIL if a successful call leaves no usable input or duplicates an existing path.
judged by Claude