Instructions — what the model was asked build a typeform competitor where the backend is identical to typeform with structured questions and answers, but the frontend is purely an AI voice interview, with the AI interviewer aware of what structured data it needs to fill in for the user, but asking in a more natural conversational way. get it running including auth just ask me for the API keys for the services you need.
Read all build a typeform competitor where the backend is identical to typeform with structured questions and answers, but the frontend is purely an AI voice interview, with the AI interviewer aware of what structured data it needs to fill in for the user, but asking in a more natural conversational way. get it running including auth just ask me for the API keys for the services you need.
4 of 8 checks failed ↓ ⇤ Checks beside the answer
Notes 1 not scored
Quick test run, not yet like-for-like. Actual harness: Google Antigravity CLI (agy 1.2.16, model agy-demo = Gemini 4 Argon). "codex" is a stand-in label until the app has an Antigravity harness. Effort is agy's default: agy refuses --effort for this model. Sealed with Seatbelt + netgate (eval/ unreadable, verified before the run), but agy's Google login in the Keychain was readable inside the seal. Web was on for this task. Judged by claude-opus-5 on Mike's machine.
really-calls-api FAIL is ours, not the model's: no OpenRouter or OpenAI key was on the machine (none in the environment or ~/.config/ec-bench/openrouter.env), so the check's own log says it "cannot pass, and that will be our fault". Re-run with a key before reading anything into it.
4 Oct 2026
4 failed fail
No AI-tell wordingai-wording every task
deliverable text contains none of the shared AI-tell phrases in bench/ai-tells.txt. Files byte-identical to something the task shipped (its inputs, the selected case, or a member of a shipped zip) are the brief's text, not the model's, and are not inspected. Task checks may add narrower contractual bans, but must not repeat this shared floor.
shared AI-tell phrases: seamlessly
Read all Show less fail
Real API callreally-calls-api this task
driving the app through its screens produces a real outbound OpenAI request, by any of the five evidence classes below. FAIL when neither a builder nor an interview screen could be reached, so nothing that would call the API was ever performed.
fail
Clear landing page pitchthis task
Judge's reasoning
There is no landing page or hero headline at all — the root renders the app shell, and the only headline-level text is the wordmark 'Talkform' plus the badge 'AI VOICE TYPEFORM', which names the category/technology rather than promising the reader a benefit.
Read all Show less Rubric Landing page hook. Read `_screenshots/landing-copy.txt` and look at `landing.png` / `landing-full.png`. The reviewer's bar: "does it do a good job with like landing page copy? ... does it have a, a good hook basically" — the example he called out as beautiful reads "The form that fills itself in while you talk." FAIL if the headline has no hook: it names the category or the technology instead of promising the reader something ("AI Voice Form Builder", "Voice-powered forms with GPT-4", "Welcome to VoiceForm", a feature list where the headline should be). PASS if the headline states, in the reader's terms, what changes for them. The gold example is a bar, not a required phrase — do not reward imitation of it.
fail
Distinctive designthis task
Judge's reasoning
The UI is the default AI look the failure mode describes: a violet/indigo glowing orb on a near-black background, indigo gradient buttons and badges, emoji standing in for icons, and nav labels wrapping mid-phrase at full desktop width.
Read all Show less Rubric Design is not slop. Look at `landing.png`, `landing-full.png`, `builder.png`, `interview.png`. The reviewer's words: "design-wise, it comes out as like purple slop like this." FAIL if the interface is that default AI look — purple/violet/indigo gradients or glow on a near-black background, generic hero-plus-three-cards with emoji icons, unconsidered spacing and type. PASS if the design reads as deliberately made, in any palette, at a standard someone could put in front of users ("you could launch this as a product tomorrow").
4 passed pass
App startsboots this task
the app installs, starts, and serves HTTP on a port
pass
Authentication existsauth-exists this task
a signup or login screen carrying credential fields is reachable from the landing page or the usual routes. FAIL without a browser.
pass
Screenshots capturedscreenshots this task
all three of landing.png, builder.png and interview.png were captured. FAIL without a browser.
pass
Complete user journeythis task
Judge's reasoning
The builder edits typed Typeform fields (10 types, refs, required flags, choices), the interview sequences questions from that form's fields via determineNextField, and answers are extracted into canonical Typeform answer objects keyed by field.ref and persisted in answers_json separately from the transcript.
Read all Show less Rubric The loop is coherent. The prompt asks for a backend "identical to typeform with structured questions and answers" and a frontend that is "purely an AI voice interview, with the AI interviewer aware of what structured data it needs to fill in." Look at `builder.png` and `interview.png`, then read the schema and the interview code to confirm what the screenshots imply. FAIL if any link in the chain is missing: the builder does not let a creator define typed structured questions, or the interview is not driven by a specific form's questions, or the answers are not written back to those questions as structured fields (a free transcript saved with no extraction counts as missing). PASS if all three links exist and connect.
judged by Claude