Mike's Checks
Checks

Mike's Checks/claude-fable-5/11 hoboken-map

11 hoboken-map

claude-fable-5Claude Codehigh effortrun 22 Aug 2026

Compare models
6/8
checks passed
75%
▸Instructions — the case's current instructions; none were saved with this result

11 — Neighborhood map + self-validating "hidden gem score"

This is a short multi-turn task. There are no data inputs — do your own
public web research to find real places. Treat the prompts below as a
conversation that arrives in order; the interesting one is the last.

Prompts (in order)

  1. My wife and kids and I just moved to north Hoboken — make me an interactive
    map of family recommendations (playgrounds, food, activities, day trips).
  2. Can you add a date-night category?
  3. What about a tapas place?
  4. Now work on a proprietary "hidden gem" score. Determine a way to measure
    it and validate that it matches expectations. Use public data where possible.

What "good" looks like

  • A single self-contained, interactive map artifact (index.html) a non-technical
    family member could open and use, with the categories above.
  • Real, plausibly-correct Hoboken-area places — not invented ones.
  • For prompt 4: a genuinely thought-through metric. Pre-register a methodology
    and your priors/expectations before collecting and scoring data, then test the
    computed scores against those locked priors and report how well they matched.
    Keep the methodology, the data, the scores, and the validation as inspectable
    files (e.g. a methodology writeup, a data file, a scoring script, a scores file).

Once produced, the artifact must open and work by double-clicking index.html —
no build step, no local server, no API keys, nothing to install. Loading a mapping
library and map tiles from a public CDN is expected and fine; "no API keys" means
no paid or credentialed services, not no network.

Answer

_screenshots_full.png
ANSWER.md the model's output · 9.74 KBdownload

North Hoboken family map — answers to the four prompts

Everything lives in this folder; double‑click index.html for the map. README.md lists every file.


1. "Make me an interactive map of family recommendations"

index.html is a single self‑contained Leaflet map (OpenStreetMap tiles from a public CDN, no keys, no server). Home base 🏠 is pinned at roughly 14th & Washington; every card shows distance from there, and a "within ½ mile of home" switch narrows the list to your immediate blocks. 89 real places, all verified against public sources in Aug 2026 and geocoded with OpenStreetMap:

  • Playgrounds & parks (11) — your nearest are Maxwell Place Park (gated playgrounds, little beach, boathouse bathrooms), Elysian Park, Harborside Park (quiet, tucked behind the north end), 1600 Park, and the new ResilienCity Park at 12th & Madison (splash pads). Plus Columbus, Church Square, Stevens, Madison Street, Pier C (the showpiece waterfront playground) and Weehawken Waterfront Park just over the viaduct.
  • Family food (25) — Elysian Cafe (kids‑eat‑free Tuesdays — confirm when you go), Pilsener Haus, Fiore's mozzarella, Leo's Grandevous, Benny Tudino's giant slices, Keming (BYOB soup dumplings at 1006 Washington), Dozzino (BYOB pizza, backyard fire pit), Apulia, Margherita's, Choc O Pain and Toastique by the Tea Building, bwè kafe's north‑end café, Black Rail Coffee, Lisa's and Vito's delis, Dom's bread, Biggie's clams, Van Leeuwen / Jun's for ice cream, Tony Boloney's, Muteki ramen, La Isla, Carlo's, Anthony David's.
  • Activities (21) — free kayaking at the Hoboken Cove Community Boathouse (late May–Sept), Historical Museum Friday storytime, Fire Department Museum (Sunday storytime, truck to climb on), Mile Square Theatre (kids' shows, uptown), Urban Jungle Play (indoor, 1140 Maxwell Ln), Gravity Vault climbing, the Uptown Farmers Market (Sat 9–1, 13th & Hudson), Pier 13, the skate park, waterfront walkway, Little City Books (both locations), Symposia, Monroe Center classes, Castle Point lookout, the 14th St ferry, Pier A, Hoboken Terminal, and the library — note the main library is closed for renovation Aug 13 – Oct 5 2026; kids' programs moved to the Grand St branch and parks.
  • Day trips (17) — Liberty State Park + Liberty Science Center + Statue/Ellis ferry (all by light rail), Turtle Back Zoo, Van Saun Park zoo, American Dream (NJT bus 85 from the terminal), Sandy Hook, Storm King, Bear Mountain, State Line Lookout, Bronx Zoo, AMNH, the Intrepid (via the 14th St ferry), Duke Farms, Medieval Times, Flat Rock Brook, Dia Beacon. Toggling the category zooms the map out.

Places found to be closed during research were left out on purpose: La Casa (tapas), Grand Vin, Orale (Hoboken), Schnackenberg's, Ubu, Field Station: Dinosaurs.

2. "Add a date‑night category"

Added (19 places, pink pins): Lola's, Bin 14, Augustino's (8 tables, cash only, phone reservations), Barbès (live jazz/flamenco on Park Ave), Antique Bar & Bakery, Halifax at the W (best skyline dining room), Dino & Harry's, Amanda's, Zack's, Sorellina, Saku, Stingray Lounge, Sirenetta, Olivia's, Seven Valleys, The Cuban, Apulia, plus two non‑dinner options: a couples' cooking class at Hudson Table (1403 Clinton) and a grown‑up play at Mile Square Theatre. Six of them are within a ten‑minute walk of 14th & Washington.

3. "What about a tapas place?"

Lola's Tapas & Wine Bar, 153 14th St — literally on your block, 20+ years old, ~80 regional Spanish plates, good sangria, tiled room with a fountain. Early seating works with kids; late is the date. The other true Spanish tapas bar in town, La Casa, closed in 2026, so the runner‑up is Bin 14 (1314 Washington) — Italian‑style small plates with 70 wines by the glass, "tapas in spirit". If you want a second Spanish option it means a drive (e.g. Bonita by the River in Edgewater) — not on the map because it's outside the brief.

4. The proprietary "Hidden Gem Score" — and whether it matched expectations

What it measures

A hidden gem is a place people who go love, few people go to or write about, and you wouldn't stumble on. So:

HGS = 100 × Quality × Obscurity, where Quality is a Bayesian‑shrunk rating (3.5★→0, 5★→1, prior weight 10 reviews) and Obscurity = 0.5·(1 − review‑count percentile vs. every rated Hoboken venue) + 0.25·(press: not in six fixed "best of" lists, no famous Wikipedia article) + 0.25·(location: off Washington St/waterfront corridors, far from the PATH). Places with < 10 public reviews get no score ("unknown ≠ verified gem"). Full spec: hidden_gem/METHODOLOGY.md.

Pre‑registration

Before collecting a single rating I wrote the methodology, assigned every in‑scope place an expected tier (G = gem / M / N = famous), wrote nine pass/fail predictions and a verdict rule, recorded third‑party "hidden gem" labels for a hold‑out test, disclosed the handful of numbers I had already glimpsed while testing which sites were fetchable, and hashed the two files with a timestamp (PREREGISTRATION_LOCK.txt).

Data (all public, no keys)

TripAdvisor's public list pages for Hoboken — the main restaurant list, its category sub‑lists, and the attractions sub‑lists — giving 157 rated restaurants and 30 rated attractions as the baseline population; Wikimedia's pageviews API for the six places with their own article; six fixed listicles; OpenStreetMap coordinates. Yelp, Google and TripAdvisor detail pages all refuse anonymous fetches (TripAdvisor's list pages are the one key‑free source that returns rating + review count for the whole town). Raw HTML is cached in hidden_gem/data/raw/.

Results (hidden_gem/scores.csv)

31 places scored, 7 had too few reviews, 31 had no TripAdvisor rating at all — including every park: TripAdvisor has no ratings for any Hoboken playground, which is the single biggest limitation.

Rank Place HGS Why
1 Lisa's Italian Deli 36.7 4.6★ from only 43 reviews, Park Ave, 0.9 mi from PATH
2 Apulia 30.1 4.4★/18, in no tracked list
3 Fiore's 26.7 4.8★ is the best quality score in town; famous, but on Adams St
4 Lola's 26.2 solid rating, modest visitor count, 14th St
5 Castle Point Lookout (Stevens) 25.8 4.6★/12 — "press" zeroed by the university's Wikipedia traffic
6 Hudson Table 25.7 4.1★/11, Clinton St uptown
7–10 Vito's, Antique Bar & Bakery, Augustino's, Zack's 22–19
…
27–31 Elysian Cafe 6.3, Hoboken Terminal 5.8, Halifax 5.3, La Isla 2.1, Carlo's Bake Shop 0.8 famous, on the corridors, heavily reviewed

Validation against the locked priors (hidden_gem/VALIDATION.md)

Verdict by the pre‑registered rule: does NOT fully match expectations — 4 of 9 predictions passed.

Prediction Result
P1 Carlo's in the bottom 3 PASS (31 of 31)
P2 Fiore's below median FAIL (26.7 vs median 17.3)
P3 Dozzino top‑5 restaurant FAIL (9th of 26)
P4 tier medians monotone N < M < G PASS (11.4 < 18.6 < 21.5)
P5 Spearman ρ(score, prior tier) ≥ 0.50 PASS (ρ = 0.563, n = 31)
P6 tertile accuracy ≥ 55 % FAIL (45 %, chance 33 %)
P7 Pier A & Pier C in bottom third of attractions FAIL (Pier A 4th of 5; Pier C has no data)
P8 coverage ≥ 70 % restaurants / ≥ 40 % attractions FAIL (62 % / 19 %)
P9 third‑party‑labelled gems score higher PASS (20.7 vs 15.7, but n = 3)

So: the score orders places in the direction I expected (P4, P5, P9) and nails the famous end (P1), but it is not accurate enough place‑by‑place (P6), and two named "gem" calls missed.

What the data disagreed with, and why (post‑hoc, labelled as such — explore_posthoc.py)

  • Obscurity alone agrees with my priors better than the full score (ρ = 0.76 vs 0.56). The quality term is negatively related to my tiers (ρ = −0.23): TripAdvisor's visitor crowd rates locals' favourites like Dozzino (3.9★/21) and Barbès (3.8★/26) lower than the local consensus, and rates tourist‑facing places (Amanda's 4.5, Pier A 4.7) higher. When I said "hidden gem" I was mostly encoding obscurity; the metric also insists on a verified rating, and TripAdvisor is the wrong jury for that.
  • Fiore's is the honest disagreement: the metric says a 4.8★ place on a side street with only 175 TripAdvisor reviews is gem‑like even though every Hoboken local knows it. My prior conflated "famous to locals" with "famous to visitors".
  • Coverage is the real failure: no park has a public rating, and most small cafés fall under the 10‑review gate. The score is trustworthy for restaurants/bars with ≥ 10 reviews and silent elsewhere — the map says "no public data" rather than guessing.

What I changed after the lock (all disclosed in hidden_gem/AMENDMENTS.md)

Only collection/parsing: a broken list‑row regex, switching to curl with 8‑second pacing and retry on TripAdvisor's bot‑challenge page, adding the category sub‑lists and a detail‑page fallback (that is what made Carlo's scorable), and cross‑kind name matching. The formula, weights, gate and predictions are byte‑identical to the locked file. The first run's report is kept (VALIDATION_run1_incomplete_baseline.md); it reached the same verdict.

If you want a second iteration

A new pre‑registration would (a) drop or down‑weight the TripAdvisor rating and use it only as a floor, (b) add a locals' review source if one becomes fetchable without keys, and (c) score parks from a different public signal (e.g. OpenStreetMap amenity completeness or city permit/event counts). I have not done any of that, because tuning after seeing the scores is exactly what the pre‑registration was there to prevent.

Checks

6/8
Script checks 3/5answered by a program
fail

Reproducible scoringreproducible

the scores file is deleted in a scratch copy and the scoring script re-run: it has to finish inside 180s and write the scores back. Missing grader prerequisites fail.

pass

Map loadsrenders

a visible map container of at least 300x200 with >= 5 markers, in a headless browser. Cannot pass without Playwright installed.

pass

Required categoriescategories

all six categories the prompt asks for are named somewhere in the output: playgrounds, food, activities, day-trips, date-night, tapas.

pass

Required filesartifacts

the run left all four of a methodology write-up, a data file, a scoring script and a scores file.

fail

gems-vs-traps Retired

Retired check; no longer defined in the repo.

Judge checks 3/3judged by Claude
pass

Q1Easy for family to use

Judge's reasoning

index.html is a single self-contained Leaflet page loading only public CDN JS/CSS and OSM tiles, and the screenshot shows labeled category pills, named place cards with plain-English descriptions, search, and gem-score toggles — no build step, server, or raw data dumps.

▸Rubric

Usable by a non-technical family member. Look at `map.png` and `full.png`, and at how `index.html` is meant to be opened. The bar, in the reviewer's words: "It's a single self-contained interactive map artifact, uh, that someone could open and use basically by clicking on the HTML." FAIL if using this would require technical knowledge — a build step, a local server, a terminal command, editing a file, or an interface that shows raw data (JSON blobs, coordinate dumps, unlabeled controls) instead of named places and readable categories. PASS if double-clicking the file gives someone a map with labeled places and controls they could figure out unaided.

pass

Q2Criteria fixed before scoring

Judge's reasoning

priors.json assigns G/M/N tiers to every place and METHODOLOGY.md fixes formula plus nine pass/fail predictions, both SHA-256-locked in PREREGISTRATION_LOCK.txt before collection (hashes verify), and VALIDATION.md reports an honest 4-of-9 miss with disagreements listed rather than restating the scores.

▸Rubric

Priors were locked before scoring. Read the methodology writeup and the scores/validation files. The prompt asked it to "**Pre-register** a methodology and your priors/expectations *before* collecting and scoring data, then test the computed scores against those locked priors and report how well they matched." FAIL if the priors read as written after the results were known — signals: no distinct set of priors stated separately from the findings; priors phrased in terms of the computed scores rather than as expectations about places or patterns; a "validation" section that restates the scores instead of describing a test the priors could have failed. PASS if there is a stated set of expectations that came first and a reported comparison against them, whether they held up or not.

pass

Q3Real places

Judge's reasoning

Spot-checked Lisa's Italian Deli (901 Park Ave, top-ranked gem), Fiore's House of Quality (414 Adams St), Carlo's Bake Shop (95 Washington St), Church Square Park (400 Garden St), and Liberty Science Center (Jersey City) — all real with addresses and coordinates matching their actual locations.

▸Rubric

Places are real. Pick five places from the map — spread across categories, including at least one the output calls a hidden gem — and check them against what you know about Hoboken and the surrounding area. The bar, in the reviewer's words: "real plausibly correct Hoboken area places, not invented ones." FAIL if any of the five is one you can positively identify as fabricated or misplaced — a business that does not exist, a name attached to the wrong town, or coordinates that put it far from where it actually is. If you cannot positively identify a problem with any of the five, PASS.