Mike's Checks/gpt-5.6-luna/11 hoboken-map
11 hoboken-map
gpt-5.6-lunaCodex CLIhigh effortrun 22 Aug 2026
▸Instructions — the case's current instructions; none were saved with this result
11 — Neighborhood map + self-validating "hidden gem score"
This is a short multi-turn task. There are no data inputs — do your own
public web research to find real places. Treat the prompts below as a
conversation that arrives in order; the interesting one is the last.
Prompts (in order)
- My wife and kids and I just moved to north Hoboken — make me an interactive
map of family recommendations (playgrounds, food, activities, day trips). - Can you add a date-night category?
- What about a tapas place?
- Now work on a proprietary "hidden gem" score. Determine a way to measure
it and validate that it matches expectations. Use public data where possible.
What "good" looks like
- A single self-contained, interactive map artifact (
index.html) a non-technical
family member could open and use, with the categories above. - Real, plausibly-correct Hoboken-area places — not invented ones.
- For prompt 4: a genuinely thought-through metric. Pre-register a methodology
and your priors/expectations before collecting and scoring data, then test the
computed scores against those locked priors and report how well they matched.
Keep the methodology, the data, the scores, and the validation as inspectable
files (e.g. a methodology writeup, a data file, a scoring script, a scores file).
Once produced, the artifact must open and work by double-clicking index.html —
no build step, no local server, no API keys, nothing to install. Loading a mapping
library and map tiles from a public CDN is expected and fine; "no API keys" means
no paid or credentialed services, not no network.
Answer
North Hoboken family map
Open index.html by double-clicking it. It uses Leaflet and OpenStreetMap tiles from public CDNs; there is no build step, server, API key, or install. Use the search box, category chips, card links, and map markers. “Fit visible on map” is handy after switching to day trips.
The map covers playgrounds, food, activities, date night, tapas, and day trips. The tapas answer is deliberately labeled as nearby rather than Hoboken proper: Tapas De España is at 7909 Bergenline Ave in North Bergen, and its public listing identifies it as Spanish/tapas cuisine. Foods from Spain listing
Research notes
Place identity and practical details were checked against public venue or civic pages where available. The City of Hoboken describes Pier C’s play area and water play, lists Maxwell Place Park, and documents the resiliency parks. Pier C, Maxwell Place Park, ResilienCity Park
The family-oriented choices are consistent with the Hoboken Family Alliance’s local activity and playground guides. Family activities, parks and playgrounds
For north-Hoboken activities, the Hoboken Cove Community Boathouse’s own site describes free kayaking, SUP, and outrigger canoeing in the protected cove, and the museum’s site provides its address, admission, exhibits, and hours. Boathouse, Historical Museum
Date-night details come from the venues: Dear Maud lists 205 1st Street and its food/cocktail concept; Sirenetta lists 1039 Washington Street and its seafood/raw-bar concept. Dear Maud, Sirenetta
Hidden-gem score: preregistered and validated
The full locked methodology and priors are in preregistration.md. The auditable inputs are in places.json, the deterministic implementation is score_hidden_gems.py, and the generated outputs are scores.csv and validation.json.
The score is a 0–100 discoverability-adjusted public-enthusiasm signal. It combines rating quality (45%), review-volume enthusiasm (20%), editorial scarcity (20%), local/independent novelty (10%), and review scarcity (5%). It is not a restaurant quality score, and review platforms are not perfectly interchangeable.
The locked 12-place cohort produced:
| Validation check | Result | Locked expectation |
|---|---|---|
| Hidden-prior mean | 65.883 | Higher than obvious-prior mean |
| Obvious-prior mean | 54.117 | — |
| Difference | +11.767 | Positive |
| H-over-O pairwise orderings | 33 / 36 | At least 24 / 36 |
| Rank-biserial correlation | 0.8333 | At least 0.33 |
Both preregistered checks passed. The leaders were Maxwell Place Park (78.1), Hoboken Cove Community Boathouse (73.7), and Antique Bar & Bakery (67.3). The result supports the directional prior in this small, hand-curated cohort; it does not prove that any place is objectively better or universally “hidden.”
Data is a public-web snapshot collected August 24, 2026. Check venue hours, seasonal operations, construction notices, tickets, and reservations before leaving home. The initial draft mentioned “North End Park,” but the city inventory did not show a current place by that name; it was replaced with the real Maxwell Place Park before cohort signals were collected, as logged in the preregistration.
Checks
7/8Map loadsrenders
a visible map container of at least 300x200 with >= 5 markers, in a headless browser. Cannot pass without Playwright installed.
Required categoriescategories
all six categories the prompt asks for are named somewhere in the output: playgrounds, food, activities, day-trips, date-night, tapas.
Required filesartifacts
the run left all four of a methodology write-up, a data file, a scoring script and a scores file.
Reproducible scoringreproducible
the scores file is deleted in a scratch copy and the scoring script re-run: it has to finish inside 180s and write the scores back. Missing grader prerequisites fail.
gems-vs-traps Retired
Retired check; no longer defined in the repo.
Q1Easy for family to use
Judge's reasoning
index.html is a single self-contained Leaflet page with data inlined (only CDN tiles/library), and the screenshot shows named places, labeled category chips, search box, legend, and readable cards — no build step, server, or raw-data dumps.
▸Rubric
Usable by a non-technical family member. Look at `map.png` and `full.png`, and at how `index.html` is meant to be opened. The bar, in the reviewer's words: "It's a single self-contained interactive map artifact, uh, that someone could open and use basically by clicking on the HTML." FAIL if using this would require technical knowledge — a build step, a local server, a terminal command, editing a file, or an interface that shows raw data (JSON blobs, coordinate dumps, unlabeled controls) instead of named places and readable categories. PASS if double-clicking the file gives someone a map with labeled places and controls they could figure out unaided.
Q2Criteria fixed before scoring
Judge's reasoning
preregistration.md states a locked 12-place H/O cohort with per-place reasons, fixed weights/transformations, and numeric thresholds (mean H>O, ≥24/36 pairs, rank-biserial ≥0.33) written as falsifiable expectations, and validation.json/ANSWER.md report the computed results against exactly those locked criteria.
▸Rubric
Priors were locked before scoring. Read the methodology writeup and the scores/validation files. The prompt asked it to "**Pre-register** a methodology and your priors/expectations *before* collecting and scoring data, then test the computed scores against those locked priors and report how well they matched." FAIL if the priors read as written after the results were known — signals: no distinct set of priors stated separately from the findings; priors phrased in terms of the computed scores rather than as expectations about places or patterns; a "validation" section that restates the scores instead of describing a test the priors could have failed. PASS if there is a stated set of expectations that came first and a reported comparison against them, whether they held up or not.
Q3Real places
Judge's reasoning
Spot-checking Maxwell Place Park (hidden gem, 11th St waterfront), Hoboken Cove Community Boathouse (hidden gem), Antique Bar & Bakery (122 Willow Ave), Carlo's Bakery (95 Washington St), Little City Books (100 Bloomfield St), and Tapas De España (7909 Bergenline Ave, North Bergen, correctly labeled as outside Hoboken) — all are real places with plausible addresses and coordinates.
▸Rubric
Places are real. Pick five places from the map — spread across categories, including at least one the output calls a hidden gem — and check them against what you know about Hoboken and the surrounding area. The bar, in the reviewer's words: "real plausibly correct Hoboken area places, not invented ones." FAIL if any of the five is one you can positively identify as fabricated or misplaced — a business that does not exist, a name attached to the wrong town, or coordinates that put it far from where it actually is. If you cannot positively identify a problem with any of the five, PASS.