MCP case input upload
gemini-4-argonrun 4 Oct 20268,254,119 tokens
Instructions
Instructions — what the model was asked
we need to add the ability to upload files via MCP otherwise we cant actually add new tasks or cases can we? we need to be able to give files that go into the input context. can you build that and give me the PR to send to my eng team Start from input/start-code.zip (EveryInc/checks at commit 18df437). input/instructions.txt preserves the original request and input/START_HERE.md explains the...
we need to add the ability to upload files via MCP otherwise we cant actually add new tasks or cases can we? we need to be able to give files that go into the input context. can you build that and give me the PR to send to my eng team
Start from input/start-code.zip (EveryInc/checks at commit 18df437). input/instructions.txt preserves the original request and input/START_HERE.md explains the archive. Work in the extracted code/ tree. Give me the PR link, or a reviewable patch if authenticated GitHub access is unavailable.
PR: Allow uploading case input files over MCP and HTTP
Note on GitHub access: This environment does not have network/authenticated
ghaccess toEveryInc/checks, so the commit has been prepared on top of18df437ab76efbc91c4c7796959be20c489049e9incode/and exported as agit am-ready patch atmcp-file-upload.patch(and included inline at the bottom of this document).
PR Title
Allow uploading case input files over MCP and HTTP
Problem
Previously, add_task and add_case only accepted a markdown prompt and ended with:
"Add 3 to 5 checks with add_check, then tell the owner to upload inputs for the case on the app."
Because Case#inputs had no MCP tool or HTTP upload endpoint, any task or case that depended on input context files in the runner's workspace (work/<path> beside prompt.md — e.g., slack-dm.md, draft.md, CSVs, PDFs, screenshots) could not be authored end-to-end over MCP without cramming file contents into prompt.md.
Summary of Changes
-
Model concern —
Case::Inputs(included inCase)Case#receive_inputs(entries, paths: nil, remove: nil, replace: false)attaches or replaces input files oncase.inputsand stores the cleaned relative workspace path in Active Storage blob metadata (metadata: { path: ... }), matchingImport::ReposoBundle(<task>/<path>or<task>/cases/<case>/<path>) andExportround-trip subdirectories cleanly.- Accepts UTF-8 text (
content) or base64-encoded binary (content_base64, includingdata:...;base64,prefixes), as well as multipartActionDispatch::Http::UploadedFileuploads. - Re-uploading the same relative
pathreplaces the previous attachment;remove: [...]purges specific paths;replace: truepurges any existing inputs not in the current batch. - Validates paths and sizes: rejects blank/null paths, absolute paths,
..traversal,prompt.md(which belongs inCase#prompt), runner-reserved directories (eval/,cases/,variants/,skill/,_screenshots/), junk segments (.git,.DS_Store,__pycache__,node_modules), and files overImport::Repo::MAX_INPUT(15 MB). Case#remove_input!(key)finds an attachment by ID, relativepath, or filename and purges it.
-
New MCP tool —
Mcp::UploadInputTool(upload_input)- Registered in
config/initializers/mcp.rbin the Author group (editor/owner role required; reviewers are refused). - Supports single-file top-level arguments (
task_slug, optionalcase_slugdefaulting to"default",path,contentorcontent_base64, optionalcontent_type) as well as batchinputs: [{ path, content, content_base64, content_type }],remove: [path, ...], andreplace: true|false. - Returns
{ task, case, inputs: [{ id, path, bytes, content_type }], inputs_url }.
- Registered in
-
Inline
inputsonMcp::AddTaskTool(add_task) andMcp::AddCaseTool(add_case)- Both tools now accept an optional
inputsarray of{ path, content, content_base64, content_type }objects and attach them inside the creation transaction so a bad input rolls back the task/case creation rather than leaving a half-saved record. Mcp::BaseTool,Mcp::GetCaseTool, andMcp::GetTaskToolnow includeinputsandinputs_urlon case payloads, and rescueCase::Inputs::InvalidInputwith a structured MCP error andnext_step.
- Both tools now accept an optional
-
Bearer-authenticated multipart HTTP endpoint —
Benchmarks::Tasks::Cases::InputsControllerPOST /benchmarks/:benchmark_id/tasks/:task_id/cases/:case_id/inputs(same Bearer token as MCP, matching howupload_resultpairs withfiles_url) accepts multipartfiles[](with optionalpaths[]to preserve nested paths since Rack strips directory components fromUploadedFile#original_filename), singlefile+path, orinputs[], plusremove[]andreplace.DELETE /benchmarks/:benchmark_id/tasks/:task_id/cases/:case_id/inputs/:id(id: /.+/,format: false) removes an input by relative path, filename, or attachment ID.
Usage Examples
1. Creating a task with input files inline (add_task)
{
"name": "add_task",
"arguments": {
"slug": "13-codex-beats",
"name": "Codex no-editing beats",
"prompt": "Where in draft.md do the beats from slack-dm.md belong?",
"inputs": [
{
"path": "slack-dm.md",
"content": "Dan: I didn't edit a single line by hand.\n"
},
{
"path": "draft.md",
"content": "# Draft\n\nThey worked quickly, answering the basic questions.\n"
}
]
}
}
2. Adding, updating, or removing input files on an existing case (upload_input)
{
"name": "upload_input",
"arguments": {
"task_slug": "13-codex-beats",
"case_slug": "default",
"path": "slack-dm.md",
"content": "Dan: Updated DM thread.\n",
"inputs": [
{
"path": "assets/diagram.png",
"content_base64": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAAB...",
"content_type": "image/png"
}
],
"remove": ["old-notes.md"]
}
}
3. Uploading larger files via HTTP multipart (inputs_url)
curl -X POST "$INPUTS_URL" \
-H "Authorization: Bearer $EVERY_CHECKS_TOKEN" \
-F "files[]=@slack-dm.md" \
-F "paths[]=slack-dm.md" \
-F "files[]=@draft.md" \
-F "paths[]=drafts/draft.md"
Verification
Ran the full test suite and RuboCop in code/:
$ bin/rails test && bin/rubocop
95 runs, 632 assertions, 0 failures, 0 errors, 0 skips
180 files inspected, no offenses detected
How to Apply and Open the PR
From a checkout of EveryInc/checks at 18df437ab76efbc91c4c7796959be20c489049e9:
git checkout -b mcp-file-upload 18df437ab76efbc91c4c7796959be20c489049e9
git am /path/to/mcp-file-upload.patch
git push -u origin mcp-file-upload
gh pr create --title "Allow uploading case input files over MCP and HTTP"
Full Patch (mcp-file-upload.patch)
From 57bbeefa9588008a060bc4dd14474085412df12b Mon Sep 17 00:00:00 2001
From: Nityesh <nityesh@every.to>
Date: Sat, 3 Oct 2026 21:32:46 -0400
Subject: [PATCH] Allow uploading case input files over MCP and HTTP
Tasks and cases authored over MCP needed a way to populate the case workspace (Case#inputs) beside prompt.md.
- Extract Case::Inputs concern with receive_inputs and remove_input!, preserving relative workspace paths in Active Storage blob metadata ({ path: ... }) and rejecting unsafe or reserved runner paths.
- Add Mcp::UploadInputTool (upload_input) for attaching, replacing, or removing UTF-8 text (content) and base64 binary (content_base64) input files on a case.
- Accept an optional inputs array directly in add_task and add_case so a task or case and its input context can be created in one call.
- Expose inputs and inputs_url on case payloads across add_task, add_case, get_task, and get_case.
- Add Benchmarks::Tasks::Cases::InputsController (POST/DELETE /benchmarks/:slug/tasks/:task/cases/:case/inputs) for Bearer-authenticated multipart uploads of larger files.
---
README.md | 7 +-
.../tasks/cases/inputs_controller.rb | 40 ++++++
app/models/case.rb | 3 +-
app/models/case/inputs.rb | 127 ++++++++++++++++++
app/tools/mcp/add_case_tool.rb | 16 ++-
app/tools/mcp/add_task_tool.rb | 18 ++-
app/tools/mcp/base_tool.rb | 14 +-
app/tools/mcp/get_benchmark_context_tool.rb | 2 +-
app/tools/mcp/get_case_tool.rb | 2 +
app/tools/mcp/get_task_tool.rb | 2 +-
app/tools/mcp/upload_input_tool.rb | 35 +++++
config/initializers/mcp.rb | 2 +-
config/routes.rb | 1 +
test/integration/mcp_test.rb | 94 ++++++++++++-
test/integration/upload_api_test.rb | 40 ++++++
15 files changed, 386 insertions(+), 17 deletions(-)
create mode 100644 app/controllers/benchmarks/tasks/cases/inputs_controller.rb
create mode 100644 app/models/case/inputs.rb
create mode 100644 app/tools/mcp/upload_input_tool.rb
diff --git a/README.md b/README.md
index e31b8f2..205b5bd 100644
--- a/README.md
+++ b/README.md
@@ -62,12 +62,12 @@ Owners mint a bearer token on a benchmark's **Connect your AI** page. A token on
claude mcp add --transport http --scope user every-checks https://every-checks.com/mcp --header "Authorization: Bearer ec_…"
-The MCP server at /mcp (streamable HTTP) has 15 tools. Call get_benchmark_context first.
+The MCP server at /mcp (streamable HTTP) has 18 tools. Call get_benchmark_context first.
| Group | Tools |
|---|---|
| - | Orient |
| - | Author |
| + | Orient |
| + | Author |
| Review | list_results, get_result, get_scoreboard, add_note |
| Run | get_case, upload_result |
@@ -76,6 +76,7 @@ Reviewers can only call read tools and add_note.
The same token works on the HTTP endpoints:
GET /benchmarks/:slug/tasks/:task/cases/:case/bundle?skill=Nreturns the case bundle.
+-POST /benchmarks/:slug/tasks/:task/cases/:case/inputs, multipart, attaches or replaces input files on a case (files[]with optionalpaths[], orfileandpath).DELETE .../inputs/:pathremoves one.POST /benchmarks/:slug/runswithexternal_id, model_id, harness, effort, ran_on. Idempotent onexternal_id.POST /benchmarks/:slug/runs/:id/results, multipart, once per case and repeat:case_slug, rep, kind, answer, screenshots[], files[], trace, usage, verdicts[]. Re-pushed files replace files with the same name.
diff --git a/app/controllers/benchmarks/tasks/cases/inputs_controller.rb b/app/controllers/benchmarks/tasks/cases/inputs_controller.rb
new file mode 100644
index 0000000..cf163c2
--- /dev/null
+++ b/app/controllers/benchmarks/tasks/cases/inputs_controller.rb
@@ -0,0 +1,40 @@
+# Multipart upload for a case's input files, reached with the same bearer token as MCP.
+class Benchmarks::Tasks::Cases::InputsController < ApplicationController
- include BenchmarkScoped
- before_action :require_editor
- before_action :set_case
- def create
- @case.receive_inputs(input_entries, paths: input_paths, remove: params[:remove], replace: ActiveModel::Type::Boolean.new.cast(params[:replace]))
- render json: { task: @task.slug, case: @case.slug, inputs: @case.inputs.map { |f| { id: f.id, path: f.path, bytes: f.byte_size, content_type: f.content_type } } }, status: :created
- rescue Case::Inputs::InvalidInput => e
- render json: { error: e.message }, status: :unprocessable_entity
- end
- def destroy
- @case.remove_input!(params[:id])
- head :no_content
- end
- private
- def set_case
-
@task = @benchmark.tasks.find_by!(slug: params[:task_id]) -
@case = @task.cases.find_by!(slug: params[:case_id]) - end
- def input_entries
-
if params[:inputs].present? -
Array(params[:inputs]).map { |i| i.respond_to?(:permit) ? i.permit(:path, :filename, :content, :content_base64, :base64, :encoding, :content_type, :file).to_h : i } -
elsif params[:files].present? -
params.permit(files: [])[:files].to_a -
elsif params[:file].present? -
[ { "path" => params[:path], "file" => params[:file] } ] -
else -
[] -
end - end
- def input_paths
-
params.permit(paths: [])[:paths].presence || Array(params[:path].presence) - end
+end
diff --git a/app/models/case.rb b/app/models/case.rb
index f39531d..70b15a6 100644
--- a/app/models/case.rb
+++ b/app/models/case.rb
@@ -1,10 +1,9 @@
One concrete brief for a task: the prompt, its inputs, and whether the public may see it.
class Case < ApplicationRecord
- include Checkable
- include Checkable, Inputs
belongs_to :task
has_one :benchmark, through: :task
has_many :results, dependent: :destroy
-
has_many_attached :inputs # the workspace the model starts in, filenames are paths relative to the case
has_many_attached :eval_files # the case's own eval/ extrasenum :visibility, %w[ private public ].index_by(&:itself), default: :private, prefix: true
diff --git a/app/models/case/inputs.rb b/app/models/case/inputs.rb
new file mode 100644
index 0000000..a4cb829
--- /dev/null
+++ b/app/models/case/inputs.rb
@@ -0,0 +1,127 @@
+# The files a case puts in work/ before the model starts. Each attachment keeps its
+# relative workspace path in blob metadata so folders survive the bundle round trip.
+module Case::Inputs
- extend ActiveSupport::Concern
- class InvalidInput < StandardError; end
- RESERVED_DIRS = %w[ eval cases variants skill _screenshots ].freeze
- included do
- has_many_attached :inputs
- end
-
Attach or replace input files on the case. Accepts MCP hashes ({path, content | content_base64, content_type})
-
or multipart uploads (UploadedFile, {path => UploadedFile}, or {path, file}). Re-uploading the same path
-
replaces the earlier file; replace: true drops any other inputs the case had.
- def receive_inputs(entries, paths: nil, remove: nil, replace: false)
- normalized = normalize_input_entries(entries, paths)
- removals = Array(remove).map { |p| clean_input_path!(p) }
- raise InvalidInput, "Provide at least one input file to upload or remove" if normalized.empty? && removals.empty? && !replace
- keep = normalized.map { |e| e[:path] }
- inputs.to_a.each do |attachment|
-
attachment.purge if (replace && !keep.include?(attachment.path)) || keep.include?(attachment.path) || removals.include?(attachment.path) - end
- inputs_attachments.reload
- inputs_blobs.reload
- attachment_changes.delete("inputs")
- if normalized.any?
-
inputs.attach(normalized.map { |entry| entry.slice(:io, :filename, :content_type).compact.merge(metadata: { path: entry[:path] }) }) - end
- inputs
- end
- def remove_input!(key)
- target = inputs.find { |a| a.id.to_s == key.to_s || a.path == key.to_s || a.filename.to_s == key.to_s }
- raise ActiveRecord::RecordNotFound.new("Couldn't find input #{key}", "Input", :id, [ key ]) unless target
- target.purge
- end
- private
- def normalize_input_entries(entries, paths)
-
list = -
if entries.is_a?(Hash) && !entries.key?(:path) && !entries.key?("path") && !entries.key?(:file) && !entries.key?("file") -
entries.map { |path, file| { "path" => path, "file" => file } } -
else -
Array(entries) -
end -
list.each_with_index.map do |item, index| -
normalize_input_entry(item, Array(paths)[index]) -
end.index_by { |e| e[:path] }.values - end
- def normalize_input_entry(item, fallback_path)
-
if item.respond_to?(:original_filename) -
uploaded_input_entry(item, fallback_path) -
elsif item.respond_to?(:to_h) -
hash = item.to_h.stringify_keys -
if hash["file"].respond_to?(:original_filename) -
uploaded_input_entry(hash["file"], hash["path"].presence || hash["filename"].presence || fallback_path, content_type: hash["content_type"]) -
else -
inline_input_entry(hash, fallback_path) -
end -
else -
raise InvalidInput, "Each input must be a file upload or an object with path and content" -
end - end
- def uploaded_input_entry(upload, explicit_path = nil, content_type: nil)
-
path = clean_input_path!(explicit_path.presence || upload.original_filename) -
check_input_size!(path, upload.size) -
io = upload.respond_to?(:tempfile) && upload.tempfile ? upload.tempfile.tap(&:rewind) : StringIO.new(upload.read) -
{ path:, filename: File.basename(path), io:, content_type: (content_type.presence || upload.content_type.presence) } - end
- def inline_input_entry(hash, fallback_path = nil)
-
path = clean_input_path!(hash["path"].presence || hash["filename"].presence || fallback_path) -
bytes = decode_input_bytes!(path, hash) -
check_input_size!(path, bytes.bytesize) -
{ path:, filename: File.basename(path), io: StringIO.new(bytes), content_type: hash["content_type"].presence } - end
- def decode_input_bytes!(path, hash)
-
b64 = hash["content_base64"].presence || hash["base64"].presence -
if b64 -
raise InvalidInput, "#{path}: provide either content or content_base64, not both" unless hash["content"].nil? -
decode_base64!(path, b64) -
elsif hash["encoding"].to_s.casecmp?("base64") -
raise InvalidInput, "#{path}: provide content when encoding is base64" if hash["content"].nil? -
decode_base64!(path, hash["content"]) -
elsif !hash["content"].nil? -
encoding = hash["encoding"].to_s.strip -
raise InvalidInput, "#{path}: encoding must be utf-8 or base64, got #{encoding.inspect}" if encoding.present? && !encoding.downcase.in?(%w[ utf-8 utf8 text ]) -
hash["content"].to_s.b -
else -
raise InvalidInput, "#{path}: provide content (UTF-8 text) or content_base64 (base64-encoded binary)" -
end - end
- def decode_base64!(path, raw)
-
cleaned = raw.to_s.sub(/\Adata:[^;]+;base64,/i, "").gsub(/\s+/, "") -
Base64.strict_decode64(cleaned) - rescue ArgumentError
-
raise InvalidInput, "#{path}: content_base64 is not valid base64" - end
- def clean_input_path!(raw_path)
-
raw = raw_path.to_s.strip -
raise InvalidInput, "input path cannot be blank" if raw.empty? || raw.include?("\0") -
raise InvalidInput, "input path must be a relative file path inside the case workspace, got #{raw_path.inspect}" if raw.start_with?("/", "\\") || raw.end_with?("/", "\\") -
clean = raw.tr("\\", "/").sub(/\A(?:\.\/)+/, "") -
segments = clean.split("/", -1) -
raise InvalidInput, "input path must stay inside the case workspace, got #{raw_path.inspect}" if segments.empty? || segments.any? { |s| s.empty? || s == "." || s == ".." } -
raise InvalidInput, "prompt.md is the case brief; pass it as prompt instead of uploading it as an input" if clean.casecmp?("prompt.md") -
raise InvalidInput, "#{segments.first}/ is reserved by the runner; put input files elsewhere in the workspace" if segments.first.downcase.in?(RESERVED_DIRS) -
raise InvalidInput, "#{clean} is a reserved path" if segments.any? { |s| Import::Repo::JUNK.include?(s) } -
clean - end
- def check_input_size!(path, bytesize)
-
return if bytesize <= Import::Repo::MAX_INPUT -
raise InvalidInput, "#{path} is over the #{ActiveSupport::NumberHelper.number_to_human_size(Import::Repo::MAX_INPUT)} limit" - end
+end
diff --git a/app/tools/mcp/add_case_tool.rb b/app/tools/mcp/add_case_tool.rb
index e1252dc..fced4ec 100644
--- a/app/tools/mcp/add_case_tool.rb
+++ b/app/tools/mcp/add_case_tool.rb
@@ -10,12 +10,22 @@ module Mcp
optional(:name).filled(:string)
optional(:web).filled(:bool)
optional(:visibility).filled(:string).description("private (default) or public") -
optional(:inputs).array(:hash) do -
required(:path).filled(:string).description("Relative path in the case workspace, e.g. slack-dm.md or data/q3.csv") -
optional(:content).value(:string).description("UTF-8 text content of the file") -
optional(:content_base64).filled(:string).description("Base64-encoded bytes for binary files") -
optional(:content_type).filled(:string).description("MIME type, e.g. text/markdown or application/pdf") -
endend.description("Input files to place in the case workspace beside prompt.md")
- def call(task_slug:, slug:, prompt:, name: nil, web: false, visibility: "private")
- def call(task_slug:, slug:, prompt:, name: nil, web: false, visibility: "private", inputs: nil)
task = benchmark.tasks.find_by!(slug: task_slug)
-
kase = task.cases.create!(slug:, name: name || slug.sub(/\A\d+-/, "").tr("-", " "), prompt:, web:, visibility:) -
{ created: kase.slice(:id, :slug, :name, :visibility, :web), task: task.slug }
-
kase = Case.transaction do -
task.cases.create!(slug:, name: name || slug.sub(/\A\d+-/, "").tr("-", " "), prompt:, web:, visibility:).tap do |c| -
c.receive_inputs(inputs) if inputs.present? -
end -
end -
end{ created: case_json(kase), task: task.slug }
end
end
diff --git a/app/tools/mcp/add_task_tool.rb b/app/tools/mcp/add_task_tool.rb
index 5b24a24..2bc94ab 100644
--- a/app/tools/mcp/add_task_tool.rb
+++ b/app/tools/mcp/add_task_tool.rb
@@ -10,12 +10,22 @@ module Mcp
required(:prompt).filled(:string).description("The brief for the first case, as markdown")
optional(:case_slug).filled(:string).description("Slug for the first case. Default: default")
optional(:web).filled(:bool).description("Whether the model may use the web. Default false") -
optional(:inputs).array(:hash) do -
required(:path).filled(:string).description("Relative path in the case workspace, e.g. slack-dm.md or data/q3.csv") -
optional(:content).value(:string).description("UTF-8 text content of the file") -
optional(:content_base64).filled(:string).description("Base64-encoded bytes for binary files") -
optional(:content_type).filled(:string).description("MIME type, e.g. text/markdown or application/pdf") -
endend.description("Input files to place in the first case's workspace beside prompt.md")
- def call(slug:, name:, prompt:, description: nil, case_slug: "default", web: false)
-
task = benchmark.tasks.create!(slug:, name:, description:, position: benchmark.tasks.maximum(:position).to_i + 1) -
task.cases.create!(slug: case_slug, name: case_slug, prompt:, web:) -
{ created: task_json(task), next_step: "Add 3 to 5 checks with add_check, then tell the owner to upload inputs for the case on the app." }
-
def call(slug:, name:, prompt:, description: nil, case_slug: "default", web: false, inputs: nil)
-
task = Benchmark.transaction do -
benchmark.tasks.create!(slug:, name:, description:, position: benchmark.tasks.maximum(:position).to_i + 1).tap do |t| -
kase = t.cases.create!(slug: case_slug, name: case_slug, prompt:, web:) -
kase.receive_inputs(inputs) if inputs.present? -
end -
end -
{ created: task_json(task), next_step: "Add 3 to 5 checks with add_check, and attach or update input files for the case with upload_input (or POST multipart to inputs_url for large files)." }end
end
end
diff --git a/app/tools/mcp/base_tool.rb b/app/tools/mcp/base_tool.rb
index 652ec69..7d7d4aa 100644
--- a/app/tools/mcp/base_tool.rb
+++ b/app/tools/mcp/base_tool.rb
@@ -25,6 +25,8 @@ module Mcp
[ as_json_text({ error: "No #{e.model.to_s.underscore.humanize.downcase} with that id in this benchmark.", next_step: "Call list_tasks, list_cases or list_checks for valid ids." }), _meta ]
rescue ActiveRecord::RecordInvalid => e
[ as_json_text({ error: "#{e.record.class.name} could not be saved.", errors: e.record.errors.full_messages }), _meta ] -
rescue Case::Inputs::InvalidInput => e
-
[ as_json_text({ error: "Input file could not be attached: #{e.message}.", next_step: "Pass a relative workspace path (not prompt.md or eval/) with content (UTF-8 text) or content_base64 (base64 binary)." }), _meta ]rescue FastMcp::Tool::InvalidArgumentsError
raise
rescue StandardError => e
@@ -74,11 +76,21 @@ module Mcpdef task_json(task) { id: task.id, slug: task.slug, name: task.name, description: task.description, position: task.position,
-
cases: task.cases.map { |c| c.slice(:id, :slug, :name, :visibility, :web) },
-
cases: task.cases.map { |c| case_json(c) }, checks: task.checks.active.ordered.map { |c| check_json(c) }, skill_versions: task.skill_versions.map(&:number) } end -
def case_json(kase) -
{ id: kase.id, slug: kase.slug, name: kase.name, visibility: kase.visibility, web: kase.web, -
inputs: kase.inputs.map { |f| input_json(f) }, -
inputs_url: url.benchmark_task_case_inputs_url(benchmark, kase.task, kase, **url_options) } -
end -
def input_json(file) -
{ id: file.id, path: file.path, bytes: file.byte_size, content_type: file.content_type } -
end -
def check_json(check) { id: check.id, key: check.key, kind: check.kind, scope: check.scope, position: check.position, label: check.label, question: check.question, retired: check.retired?, has_script: check.script.present? } end
diff --git a/app/tools/mcp/get_benchmark_context_tool.rb b/app/tools/mcp/get_benchmark_context_tool.rb
index 6ef0b11..e708554 100644
--- a/app/tools/mcp/get_benchmark_context_tool.rb
+++ b/app/tools/mcp/get_benchmark_context_tool.rb
@@ -12,7 +12,7 @@ module Mcp
manual: <<~TEXT
#{benchmark.name} is one person's personal benchmark: real work they use AI for, measured by the things they check when deciding whether an output is good. The question it answers is "which model is best at my job, for the things I care about".
-
Nouns. A TASK is a kind of real work (make a deck, plan a trip). A CASE is one concrete brief for a task, with its prompt and input files; a task with a single brief has one case named "default". A CHECK is one yes/no question about an output. A RUN is one model on one harness at one effort against the benchmark; it produces one RESULT per case per rep, and every result carries one VERDICT (pass or fail) per applicable check. A NOTE is a human's free-text review of a result; it is not scored.
-
Nouns. A TASK is a kind of real work (make a deck, plan a trip). A CASE is one concrete brief for a task, with its prompt and input files; a task with a single brief has one case named "default". Pass a case's input files inline on add_task or add_case, or attach and replace them with upload_input (content for UTF-8 text, content_base64 for binary, or multipart POST to inputs_url for large files) so they sit beside prompt.md in the workspace rather than pasted into the prompt. A CHECK is one yes/no question about an output. A RUN is one model on one harness at one effort against the benchmark; it produces one RESULT per case per rep, and every result carries one VERDICT (pass or fail) per applicable check. A NOTE is a human's free-text review of a result; it is not scored. Checks. Two kinds. CODE checks are answered by a script the runner executes in its sandbox (key is a short slug like nps-correct; the script is Python that prints CHECK <key> PASS|FAIL). MODEL checks are answered by the LLM judge from the question text (key is judge/<slug-of-title>). Checks have a SCOPE: a benchmark check applies to every result, a task check to every result of that task, a case check to one brief. There are no exclusions; a check that needs one is at the wrong scope, so promote it up or push it down.
diff --git a/app/tools/mcp/get_case_tool.rb b/app/tools/mcp/get_case_tool.rb
index bd70640..1f1b245 100644
--- a/app/tools/mcp/get_case_tool.rb
+++ b/app/tools/mcp/get_case_tool.rb
@@ -22,6 +22,8 @@ module Mcp
prompt: kase.prompt, config: kase.config, web: kase.web, timeout_minutes: kase.timeout_minutes,
judge: { model: benchmark.judge_model_id, effort: benchmark.judge_effort },
bundle_url: url.benchmark_task_case_bundle_url(benchmark, task, kase, skill: skill&.number, **url_options),
-
inputs_url: url.benchmark_task_case_inputs_url(benchmark, task, kase, **url_options), -
inputs: kase.inputs.map { |f| input_json(f) }, large_inputs: bundle.sidecars.map { |path, file| { path:, bytes: file.byte_size, url: url.rails_blob_url(file, **url_options) } }, checks: scopes.flat_map { |scope, c| c.code_checks.map { |ch| { key: ch.key, kind: "code", scope:, question: ch.question } } }, judge_questions: scopes.flat_map { |scope, c| c.judge_questions.each_with_index.map { |q, i| { scope:, number: i + 1, key: q.key, question: q.question } } },
diff --git a/app/tools/mcp/get_task_tool.rb b/app/tools/mcp/get_task_tool.rb
index 50bdec2..51b7735 100644
--- a/app/tools/mcp/get_task_tool.rb
+++ b/app/tools/mcp/get_task_tool.rb
@@ -8,7 +8,7 @@ module Mcp
end
def call(task_slug:)
-
task_json(benchmark.tasks.includes(:cases, :checks, :skill_versions).find_by!(slug: task_slug))
-
endtask_json(benchmark.tasks.includes(:checks, :skill_versions, cases: { inputs_attachments: :blob }).find_by!(slug: task_slug))
end
end
diff --git a/app/tools/mcp/upload_input_tool.rb b/app/tools/mcp/upload_input_tool.rb
new file mode 100644
index 0000000..071d0bf
--- /dev/null
+++ b/app/tools/mcp/upload_input_tool.rb
@@ -0,0 +1,35 @@
+module Mcp - class UploadInputTool < BaseTool
- tool_name "upload_input"
- description "Upload or replace input files in a case's workspace — the files a model starts with beside prompt.md (e.g. a Slack thread, draft, CSV or PDF). Pass UTF-8 text in content or binary in content_base64; for one file, path and content at the top level also work. Re-uploading a path replaces that file. Returns inputs_url for multipart HTTP uploads when a file is too large for MCP."
- annotations(title: "Upload Input", read_only_hint: false, destructive_hint: true, idempotent_hint: true, open_world_hint: false)
- arguments do
-
required(:task_slug).filled(:string).description("Task slug, e.g. 13-codex-beats") -
optional(:case_slug).filled(:string).description("Case slug. Default: default") -
optional(:path).filled(:string).description("Relative path in the case workspace for a single file, e.g. slack-dm.md or data/q3.csv") -
optional(:content).value(:string).description("UTF-8 text content for a single file") -
optional(:content_base64).filled(:string).description("Base64-encoded bytes for a single binary file") -
optional(:content_type).filled(:string).description("MIME type for a single file, e.g. text/markdown or application/pdf") -
optional(:inputs).array(:hash) do -
required(:path).filled(:string).description("Relative path in the case workspace, e.g. slack-dm.md or data/q3.csv") -
optional(:content).value(:string).description("UTF-8 text content of the file") -
optional(:content_base64).filled(:string).description("Base64-encoded bytes for binary files") -
optional(:content_type).filled(:string).description("MIME type, e.g. text/markdown or application/pdf") -
end.description("Batch of input files to place in the case workspace") -
optional(:remove).array(:string).description("Relative paths of existing input files to remove from the case") -
optional(:replace).filled(:bool).description("When true, remove any existing inputs on the case not included in this call. Default false") - end
- def call(task_slug:, case_slug: "default", path: nil, content: nil, content_base64: nil, content_type: nil, inputs: nil, remove: nil, replace: false)
-
task = benchmark.tasks.find_by!(slug: task_slug) -
kase = task.cases.find_by!(slug: case_slug) -
entries = [] -
entries << { path:, content:, content_base64:, content_type: }.compact if path.present? || !content.nil? || content_base64.present? -
entries.concat(Array(inputs)) -
kase.receive_inputs(entries, remove:, replace:) -
{ task: task.slug, case: kase.slug, -
inputs: kase.inputs.map { |f| input_json(f) }, -
inputs_url: url.benchmark_task_case_inputs_url(benchmark, task, kase, **url_options) } - end
- end
+end
diff --git a/config/initializers/mcp.rb b/config/initializers/mcp.rb
index 455c7d6..908706f 100644
--- a/config/initializers/mcp.rb
+++ b/config/initializers/mcp.rb
@@ -3,7 +3,7 @@
Rails.application.config.after_initialize do
server = StreamableHttpMcpTransport.server
[ Mcp::CreateBenchmarkTool, Mcp::GetBenchmarkContextTool, Mcp::ListTasksTool, Mcp::GetTaskTool, Mcp::ListModelsTool, # create (unbound tokens), orient
- Mcp::AddTaskTool, Mcp::AddCaseTool, Mcp::AddCheckTool, Mcp::UpdateCheckTool, Mcp::PromoteCheckTool, Mcp::ListChecksTool, # author
- Mcp::AddTaskTool, Mcp::AddCaseTool, Mcp::UploadInputTool, Mcp::AddCheckTool, Mcp::UpdateCheckTool, Mcp::PromoteCheckTool, Mcp::ListChecksTool, # author
Mcp::GetScoreboardTool, Mcp::ListResultsTool, Mcp::GetResultTool, Mcp::AddNoteTool, # review
Mcp::GetCaseTool, Mcp::UploadResultTool ].each { |t| server.register_tool(t) } # run
Rails.logger.info("MCP server #{server.name} ready with #{server.tools.size} tools")
diff --git a/config/routes.rb b/config/routes.rb
index 0bf0f26..1b17d24 100644
--- a/config/routes.rb
+++ b/config/routes.rb
@@ -33,6 +33,7 @@ Rails.application.routes.draw do
resources :cases, only: :show do
scope module: "cases" do
resource :bundle, only: :show # the case as a zip in the runner's folder layout -
resources :inputs, only: %i[ create destroy ], id: /.+/, format: false end end end
diff --git a/test/integration/mcp_test.rb b/test/integration/mcp_test.rb
index 8366b93..51fd484 100644
--- a/test/integration/mcp_test.rb
+++ b/test/integration/mcp_test.rb
@@ -53,12 +53,15 @@ class McpTest < ActionDispatch::IntegrationTest
test "the runner's loop: get_case hands out the bundle, upload_result reads score.json back by question number" do
session = rpc("initialize", { protocolVersion: "2025-11-25", capabilities: {}, clientInfo: { name: "t", version: "0" } }).headers["mcp-session-id"]
names = JSON.parse(rpc("tools/list", {}, id: 2, session:).body).dig("result", "tools").map { |t| t["name"] }
- assert_equal 17, names.size
-
assert_equal 18, names.size
-
assert_includes names, "upload_input"
assert_not_includes names, "refine_checks_from_note"kase = tool_result(rpc("tools/call", { name: "get_case", arguments: { task_slug: "01-dashboard" } }, id: 3, session:))
assert_equal "01-dashboard", kase["spec"]
assert_match %r{/benchmarks/mine/tasks/01-dashboard/cases/default/bundle\z}, kase["bundle_url"] -
assert_match %r{/benchmarks/mine/tasks/01-dashboard/cases/default/inputs\z}, kase["inputs_url"]
-
assert_equal [], kase["inputs"]
assert_equal [ { "scope" => "task", "number" => 1, "key" => "judge/typography", "question" => "Typography. Is it readable?" } ], kase["judge_questions"]
assert_equal [ "renders" ], kase["checks"].map { |c| c["key"] }
assert_includes kase["run"], "bench/run.sh 01-dashboard"
@@ -141,6 +144,95 @@ class McpTest < ActionDispatch::IntegrationTest
assert_equal "Buried the thesis.", note.dig("created", "body")
assert_equal reviewer, result.notes.last.author
assert_match(/only owners and editors/, tool_result(rpc("tools/call", { name: "add_check", arguments: { scope: "task", task_slug: "01-dashboard", kind: "model", key: "judge/x", question: "X?" } }, id: 3, session:))["error"]) -
assert_match(/only owners and editors/, tool_result(rpc("tools/call", { name: "upload_input", arguments: { task_slug: "01-dashboard", path: "notes.md", content: "no" } }, id: 4, session:))["error"])
-
end
-
test "authoring a task and case with input files over MCP puts them into the bundle workspace" do
-
session = rpc("initialize", { protocolVersion: "2025-11-25", capabilities: {}, clientInfo: { name: "t", version: "0" } }).headers["mcp-session-id"]
-
png_bytes = "\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR".b
-
created = tool_result(rpc("tools/call", {
-
name: "add_task", -
arguments: { -
slug: "13-codex-beats", -
name: "Codex no-editing beats", -
prompt: "Where in draft.md do the beats from slack-dm.md belong?", -
inputs: [ -
{ path: "slack-dm.md", content: "Dan: I didn't edit a single line by hand.\n" }, -
{ path: "drafts/draft.md", content: "# Draft\n\nThey worked quickly, answering the basic questions.\n" }, -
{ path: "assets/shot.png", content_base64: Base64.strict_encode64(png_bytes), content_type: "image/png" } -
] -
} -
}, id: 2, session:))
-
default_case = created.dig("created", "cases", 0)
-
assert_equal %w[ slack-dm.md drafts/draft.md assets/shot.png ], default_case["inputs"].map { |i| i["path"] }
-
assert_match %r{/benchmarks/mine/tasks/13-codex-beats/cases/default/inputs\z}, default_case["inputs_url"]
-
assert_match(/upload_input/, created["next_step"])
-
updated = tool_result(rpc("tools/call", {
-
name: "upload_input", -
arguments: { -
task_slug: "13-codex-beats", -
path: "slack-dm.md", -
content: "Dan: Updated DM thread.\n", -
inputs: [ { path: "notes/context.txt", content: "Extra context" } ], -
remove: [ "assets/shot.png" ] -
} -
}, id: 3, session:))
-
assert_equal %w[ drafts/draft.md slack-dm.md notes/context.txt ], updated["inputs"].map { |i| i["path"] }
-
kase_info = tool_result(rpc("tools/call", { name: "get_case", arguments: { task_slug: "13-codex-beats" } }, id: 4, session:))
-
assert_equal %w[ drafts/draft.md slack-dm.md notes/context.txt ], kase_info["inputs"].map { |i| i["path"] }
-
get kase_info["bundle_url"], headers: { "Authorization" => "Bearer #{@token}" }
-
assert_response :success
-
entries = Zip::File.open_buffer(response.body).entries.to_h { |e| [ e.name, e.get_input_stream.read ] }
-
assert_equal "Where in draft.md do the beats from slack-dm.md belong?", entries["13-codex-beats/prompt.md"]
-
assert_equal "Dan: Updated DM thread.\n", entries["13-codex-beats/slack-dm.md"]
-
assert_equal "# Draft\n\nThey worked quickly, answering the basic questions.\n", entries["13-codex-beats/drafts/draft.md"]
-
assert_equal "Extra context", entries["13-codex-beats/notes/context.txt"]
-
assert_nil entries["13-codex-beats/assets/shot.png"]
-
second_case = tool_result(rpc("tools/call", {
-
name: "add_case", -
arguments: { -
task_slug: "13-codex-beats", -
slug: "02-followup", -
prompt: "Second brief.", -
inputs: [ { path: "followup.md", content: "Follow-up notes." } ] -
} -
}, id: 5, session:))
-
assert_equal [ "followup.md" ], second_case.dig("created", "inputs").map { |i| i["path"] }
-
replaced = tool_result(rpc("tools/call", {
-
name: "upload_input", -
arguments: { -
task_slug: "13-codex-beats", -
case_slug: "02-followup", -
replace: true, -
inputs: [ { path: "only.md", content: "Only file left." } ] -
} -
}, id: 6, session:))
-
assert_equal [ "only.md" ], replaced["inputs"].map { |i| i["path"] }
-
end
-
test "upload_input and add_task reject unsafe paths and bad base64 without leaving half-saved records" do
-
session = rpc("initialize", { protocolVersion: "2025-11-25", capabilities: {}, clientInfo: { name: "t", version: "0" } }).headers["mcp-session-id"]
-
[ "../escape.md", "/etc/passwd", "prompt.md", "eval/checks.py", "cases/other/a.md", "skill/v1/SKILL.md" ].each_with_index do |bad_path, i|
-
res = tool_result(rpc("tools/call", { name: "upload_input", arguments: { task_slug: "01-dashboard", path: bad_path, content: "x" } }, id: 10 + i, session:)) -
assert_match(/Input file could not be attached/, res["error"], "expected #{bad_path} to be rejected") -
end
-
bad_b64 = tool_result(rpc("tools/call", { name: "upload_input", arguments: { task_slug: "01-dashboard", path: "bin.dat", content_base64: "not-valid-base64!!!" } }, id: 30, session:))
-
assert_match(/not valid base64/, bad_b64["error"])
-
failed_task = tool_result(rpc("tools/call", {
-
name: "add_task", -
arguments: { slug: "99-atomic", name: "Atomic", prompt: "Brief", inputs: [ { path: "../nope.md", content: "x" } ] } -
}, id: 31, session:))
-
assert_match(/Input file could not be attached/, failed_task["error"])
-
assert_nil @benchmark.tasks.find_by(slug: "99-atomic"), "a rejected input rolls back the task creation"
end
test "no token is a 401 with the resource metadata pointer" do
diff --git a/test/integration/upload_api_test.rb b/test/integration/upload_api_test.rb
index 92fbe0f..6a67806 100644
--- a/test/integration/upload_api_test.rb
+++ b/test/integration/upload_api_test.rb
@@ -51,6 +51,46 @@ class UploadApiTest < ActionDispatch::IntegrationTest
assert_response :not_found
end
- test "a token uploads, replaces and deletes case input files over multipart HTTP" do
- post benchmark_task_case_inputs_path(@benchmark, "01-dashboard", "default"), headers: headers, params: {
-
files: [ -
fixture_file_upload_from_string("id,score\n1,9\n", "nps.csv"), -
fixture_file_upload_from_string("Slack DM\n", "slack.md") -
], -
paths: [ "data/nps.csv", "notes/slack.md" ] - }
- assert_response :created
- assert_equal %w[ data/nps.csv notes/slack.md ], response.parsed_body["inputs"].map { |i| i["path"] }
- post benchmark_task_case_inputs_path(@benchmark, "01-dashboard", "default"), headers: headers, params: {
-
file: fixture_file_upload_from_string("id,score\n1,10\n", "nps.csv"), -
path: "data/nps.csv" - }
- assert_response :created
- assert_equal %w[ notes/slack.md data/nps.csv ], response.parsed_body["inputs"].map { |i| i["path"] }
- get benchmark_task_case_bundle_path(@benchmark, "01-dashboard", "default"), headers: headers
- entries = Zip::File.open_buffer(response.body).entries.to_h { |e| [ e.name, e.get_input_stream.read ] }
- assert_equal "id,score\n1,10\n", entries["01-dashboard/data/nps.csv"]
- assert_equal "Slack DM\n", entries["01-dashboard/notes/slack.md"]
- delete benchmark_task_case_input_path(@benchmark, "01-dashboard", "default", "notes/slack.md"), headers: headers
- assert_response :no_content
- assert_equal [ "data/nps.csv" ], @benchmark.tasks.first.default_case.inputs.reload.map(&:path)
- post benchmark_task_case_inputs_path(@benchmark, "01-dashboard", "default"), headers: headers, params: {
-
file: fixture_file_upload_from_string("bad", "prompt.md"), -
path: "prompt.md" - }
- assert_response :unprocessable_entity
- assert_match(/prompt.md/, response.parsed_body["error"])
- post benchmark_task_case_inputs_path(@other, "01-dashboard", "default"), headers: headers, params: {
-
file: fixture_file_upload_from_string("x", "a.md") - }
- assert_response :not_found
- end
- test "a token reaches only its own benchmark, and no token means 401" do
post benchmark_runs_path(@other), params: { run: { external_id: "x", model_id: "m", harness: "claude", ran_on: "2026-09-10" } }, headers: headers
assert_response :not_found
--
2.50.1 (Apple Git-155)