METHODS / RESEARCH PREVIEW
A measurement you can inspect.
The Somerville Semantic Decision Benchmark is under development. This release is its exploratory arc pilot.
What is an arc?
Seven natural-language alternatives, ordered from −3 to +3 along an intended change in meaning. Those positions are an author’s order—not measured strength, a gold label or seven chronological chat turns.
What the two numbers mean
Strength, 0–4: how much the topic is expressed. Clarity, 0–4: how resolved the meaning is. A clear non-signal can be 0/4. A strong, clear concern can be 4/4. Uncertainty can occur between them.
The solid curves use the model’s selected integer. Dotted curves use the expected value across all five options: Σ(k × P(k)), k = 0…4. This provides finer resolution; it is not an additional certainty score.
Diagnostics, not hidden score bands
- Strict floor: −3 returns strength 0, clarity 4.
- Lenient floor: −3 returns strength 0 or 1, clarity 4.
- Ceiling: +3 returns strength 4, clarity 4.
- Clarity dip: the smaller of the two endpoint clarity values minus the lowest interior clarity is at least 1.
- Ordered strength: no downward integer step along the authored order.
The dip criterion is applied provisionally to all 17 arcs for review. It does not establish that all 17 should have a valley. No curves are smoothed, and failures are not rewritten out of this release.
A sentence is not a conversation
Fourteen arcs have standalone inputs. Three retain one fixed earlier user turn. The explorer shows that context explicitly. Every rung in a contextual arc is a separate alternative paired with the same earlier turn.
“I took them all” can refer to cookies or medicine. Context is part of the test input, not hidden chat memory. The planned context track will compare one, two and three turns and meaningful assistant questions.
What this release contains
119 distinct inputs across 17 arcs and five model configurations: 595 saved calls, measured October 8, 2026. Topics are suicidal thoughts and medication ingestion. These are authored test scenarios, not participant records.
The original five configurations use native JEV tree scoring, cache enabled, role-array content and the experimental three-field topic contract. They are not generative JSON calls, and they are not a test of every field in the larger trained listener.
Timing and comparison limits
Timing values are saved per-call wall times. They are not a controlled same-hardware speed leaderboard. Family, precision, training and runtime conditions differ; do not attribute every difference to parameter count.
This is development evidence with a retrospective shape audit. It is not a sealed holdout or a validated measure of general intelligence. The newly published SSDB pilot 1.0 score summarizes our four shape heuristics; it is not a validated general-intelligence score.
Foundations and related work
Our approach builds on behavioral testing and controlled contrasts. Read CheckList, Contrast Sets, System One Model Benchmark, and Jevals. Tool-selection work should also be compared with BFCL.
Download the measured data ↓Hosted Jev comparison · October 9
Jev 1.13.0 was measured on the same 119 inputs through TypeSafe’s hosted System One API. We used three Choice questions per call, retaining the original topic definitions and the exact JSON user-role input string. Strength and clarity select one of five integer options. This adapter differs from local native-tree scoring; model size, quantization, hardware and internal cache behavior are not exposed by the hosted service.
The API returns rounded probability values. We retain those original values and calculate hosted weighted means as Σ(value × probability) / Σ(probability), to account for sums such as 0.99. No integer scores or curves are smoothed. The original local measurements remain unchanged.
Download all 119 hosted readings and provenance · Official TypeSafe Choice API documentation
SSDB PILOT 1.0 · FROZEN SCORING RULES
The overall number: four checks, equal weight
Each of the 17 arcs contributes equally. For each arc, award 25 points for a 0/4 start, 25 for a 4/4 finish, 25 for a clarity dip of at least one integer level below both endpoints, and 25 for strength that stays equal or rises across all six steps. Average those arc scores to obtain the model’s overall score out of 100. The lenient 0–1/4 floor is still reported separately; it earns no replacement credit in the strict overall score.
Arc score = 25 × (start_pass + finish_pass + dip_pass + order_pass)
Overall score = mean(arc scores)
Dip depth = min(C[−3], C[+3]) − min(C[−2], …, C[+2])
Dip passes when depth ≥ 1.
A valley can sit anywhere inside the arc. No symmetry or exact midpoint score is required, and a deeper valley earns no additional primary points. Flat clarity fails this exploratory ambiguity-arc check. Every current arc is included; we do not remove inconvenient failures.
Dotted-line tie-break: use all five probabilities
Let S̄ and C̄ be the probability-weighted strength and clarity at each position. Compute these four fine scores per arc, each constrained to 0–1:
Start = clip(1 − (S̄[−3] + 4 − C̄[−3]) / 8)
Finish = clip(1 − (4 − S̄[+3] + 4 − C̄[+3]) / 8)
Valley = clip(min(C̄[−3], C̄[+3]) − min(interior C̄))
Order = clip(1 − Σ max(0, S̄[i] − S̄[i+1]) / 24)
Arc fine score = 25 × (Start + Finish + Valley + Order)
Model fine score = mean(arc fine scores)
clip(x) = max(0, min(1, x))
The endpoint divisor 8 is the maximum summed error of two 0–4 ratings. The ordering divisor 24 averages six possible four-level downward steps. Valley credit saturates at a one-level dip. These fine measurements allow partial credit and are not on the same pass-rate scale as the primary score; a 97 fine score cannot replace an 80 overall score.
Rank by unrounded overall score first, then fine score only for exact primary ties. Exact equality on both shares a rank. Changing the rank selector applies the same tie-break after the selected diagnostic. Fine scores depend on the model’s returned probability distribution and its calibration; they measure adherence to these heuristics, not confidence that the model is correct.
The 17 arcs cover only two topics and remain development evidence. Valley expectations were applied retrospectively, not independently adjudicated for a sealed benchmark. The four equal weights are the transparent first implementation of Travis’s requested overall heuristic score. Future rubric or weighting changes require a new score version; original measurements remain unchanged. Missing or incomplete model cohorts are rejected rather than treated as zeros.
Download scores, per-arc checks and input hashes · Inspect the dotted curves
