1 - Quick Start
Build the report, validate the data, and open it locally.
Requires python3, bash, git, and (for deploys) the gh CLI. Building
and serving the Hugo/Docsy site additionally requires hugo (extended) and
node/npm.
Build the report
# Build the dashboard from the JSON data inputs
bash src/scripts/build.sh
# Validate JSON + rebuild + sanity-check the artifact
bash src/scripts/release-check.sh
# Open the current report directly, without the documentation site
xdg-open dist/dashboard.html
Build and serve the documentation site
# Serve the Hugo/Docsy site locally at http://localhost:1313/, embedding a
# freshly built report
bash docs/scripts/serve-docs.sh
# Produce a production build under docs/public/
bash docs/scripts/build-docs.sh
docs/scripts/build-docs.sh and docs/scripts/serve-docs.sh both rebuild
dist/dashboard.html via src/scripts/build.sh and copy it to
static/report/dashboard.html before invoking Hugo, so the embedded report in
Signal Room always reflects the current data.
Deploy
bash src/scripts/release-check.sh
bash docs/scripts/deploy-docs.sh
GitHub Pages is fed from the root of the gh-pages branch by a local script.
No GitHub Actions workflow is used or permitted. See
Project Status for the deployment
architecture decision.
Publish an archived revision
Run the deployment from the release commit or tag, then set DOCS_VERSION to
the revision name. The snapshot is published below that path without replacing
the live site:
git switch --detach v0.4.0
DOCS_VERSION=v0.4.0 bash docs/scripts/deploy-docs.sh
Archived revisions show a support banner and link readers back to main. Add
the version to params.versions in docs/hugo.yaml in the subsequent live-site
change so it appears in the Revisions picker and on the
Revision index.
2 - Project Status
Lifecycle, support, and what is and is not covered by this report.
Lifecycle: active. The market roster and published dashboard are actively
maintained. Historical snapshots, when published, remain available for
provenance, but only the current dashboard and latest tagged release are
supported.
For the live revision, archived-snapshot policy, and release links, see
Revisions.
What this is
Signal Room is inspectable decision support: a manually rebuilt, single static
dashboard that tracks model rosters, provider-native reasoning configurations,
speed evidence, agent harnesses, and self-hosting economics, with evidence
classes and source links kept visible throughout.
What this is not
- Not exhaustive. Pricing changes, and vendor-reported claims and
heterogeneous benchmarks are not universally comparable.
- Not a recommendation engine for every workload – treat its outputs as a
starting point, not a substitute for your own evaluation.
- Not dynamically updated. There is no runtime CDN dependency or live data
fetch; GitHub Pages serves a manually rebuilt static artifact.
Deployment architecture
GitHub Pages is fed from the root of the gh-pages branch by a local build
and deploy of this Hugo/Docsy site; no GitHub Actions workflow is used or
permitted. See
DEC-20260727_0742-ClearSpring,
which supersedes
DEC-20260517_1455-DeftLynx.
Release process
Releases follow the repository’s phased release process defined in
AGENTS.md.
Do not cut a version directly before the release gates have been evaluated.
3 - Data Methodology
Formulas, evidence classes, chart encodings, and the repeatable update procedure behind the Signal Room report.
The dashboard has three canonical data inputs:
data/market-state.json preserves the original report schema and backward
compatibility.data/model-roster-v2.json contains the curated provider roster, model
configuration levels, availability, pricing labels, and speed evidence.data/report-metrics.json contains normalized metrics, chart encodings,
current economics, hardware fit, and decision-support data.
The build validates all three files and merges the latter two under
marketData.model_roster and marketData.report_metrics. Do not fetch data
at runtime: GitHub Pages remains a single, manually rebuilt static artifact.
Evidence classes
Every speed or hardware value should be one of:
measured: reproduced in the project’s pinned harness.vendor: copied from an official primary source with its test context.vendor-relative: a vendor comparison without an absolute measurement.planning_estimate: useful for capacity planning but not a measured fact.
Unknown values remain JSON null. Never turn an absent number into zero. A
hardware combination that cannot run uses fit_status: "unsupported" and a
reason. This replaces ambiguous empty GB tok/s cells.
Benchmark evidence and reference quality
Each benchmark definition has a max_value, version, unit, and direction.
For the current all-higher-is-better register, individual cells may be
normalized for color intensity:
normalized(model, benchmark) = 100 * value / max_value
max_value is the documented maximum for that benchmark version or comparison
set. Values from different benchmark versions must not share a column. Missing
results stay null; the dashboard does not infer zero or manufacture a benchmark
composite from sparse, non-overlapping evaluations.
Do not merge a vendor launch table with an independent leaderboard into one
ranking unless the model version, agent harness, dataset split, scoring rules,
and evaluation date are demonstrably the same. Treat both as useful evidence:
the former documents the vendor’s measured configuration, while the latter can
provide an independent public comparison. The July 2026 GPT-5.6 launch table
and Scale’s public SWE-bench Pro board are intentionally shown as separate
sources for this reason.
The July 31 DeepSeek V4-Flash-0731 update is handled the same way. Its
Terminal-Bench 2.1 and Agents’ Last Exam values are shown with the vendor’s
max-effort, DeepSeek Harness minimal-mode caveat. NL2Repo, Cybergym,
Toolathlon, Automation Bench, and the internal DSBench sets remain narrative
evidence until their versions, harnesses, and public reproducibility are
sufficient for a shared comparison column.
Each model instead has a curated quality_vs_fable anchor. Prefer a published
multi-benchmark comparison against Fable 5. A historical normalized score or a
shared SWE-Pro version may be used as a clearly documented fallback. The Kimi
K3 anchor, for example, is the geometric mean of the 14 K3/Fable 5 ratios in
Moonshot’s launch comparison.
Quality relative to the selected reference is computed at render time:
quality_vs_reference = 100 * quality_vs_fable(model)
/ quality_vs_fable(reference)
It may exceed 100 and must not be capped.
Speed evidence and score
Keep TTFT, output tokens per second, and end-to-end task time separate. The
current cross-provider chart uses an end-to-end task-speed index because
comparable token-throughput measurements are unavailable:
task_speed_index = 100 * reference_task_time / model_task_time
speed_score = 100 * sqrt(task_speed_index / max_task_speed_index)
Thus Fable 5 is 100 in the cited comparison. A task-speed index of 300 means
the model completed that workload at three times the rate; it is dimensionless
and does not mean 300 tok/s. Provider-documented output throughput is displayed
separately in tok/s with its evidence class. The square root reduces the visual
dominance of extreme relative task-rate claims.
Replace relative indices with local measurements after recording model,
effort, provider, service tier, region, date, cache state, input/output token
counts, and workload. Historical points that do not come from a frozen test
must retain their history_note caveat.
Cost and burn
The workload price blend is:
blended_price = 0.30 * input_price + 0.70 * output_price
This represents an output-heavy agent workload. It excludes tools, long-
context surcharges, cache writes, batch discounts, and hidden reasoning
tokens. Those must be modeled separately when applicable.
Cost is converted to a higher-is-better score with log normalization:
cost_score = 10 + 90 * ln(max_price / price)
/ ln(max_price / min_price)
Recompute min_price and max_price over visible, eligible models.
Burn uses the original report’s selectable-reference design:
cache_factor = (1 - cache_hit_rate)
+ cache_hit_rate * cache_read_ratio
absolute_burn = base_blended_price * effort_factor * cache_factor
display_burn = absolute_burn / reference_medium_absolute_burn
The GPT-5.6 and Fable effort factors are planning estimates until a controlled
run measures token use. Fable has always-on adaptive thinking, so its effort
setting is a behavioral control, not a published token multiplier.
Quality, speed, and cost views
The bubble chart uses speed score on x, cost efficiency on y, quality versus
the selected reference for bubble area, and provider for color. The heat map
uses the same three values without a synthetic overall compound. A model enters
either view only when all three values are present. Bubble radius is
proportional to the square root of quality, so area represents magnitude.
Labels sit in edge rails with connector lines to avoid clipping dense clusters.
Capability compound
The six editorial axes are coding, reasoning and architecture, knowledge and
research, communication and documentation, multimodal, and agentic work. They
are rubric assessments, not benchmark results.
Balanced mode assigns every axis weight 1. Selecting a focus assigns that
axis weight 2.5 and leaves the other five at 1:
capability = sum(axis_score * axis_weight) / sum(axis_weight)
The radar compares two chosen models on the raw six-axis scores. The BCG-style
chart uses the mean of quality versus reference, speed, and cost on x and the
focus-weighted capability composite on y. It requires all three market values.
Its 50 lines are orientation guides, not targets.
Hardware fit
Hardware throughput depends on quantization, backend, batch size, context,
prompt length, interconnect, power limit, and speculative decoding. Every
measured update should store those dimensions. Until then, values marked
planning_estimate may support filtering and pastel speed bands but must be
labelled as estimates in the UI.
Recommended display bands are computed per visible table, using the 25th,
50th, and 75th percentiles of non-null throughput. Do not give unsupported or
unknown cells a color implying performance.
Hosting price history
Hosting prices live under report_metrics.hosting_prices. Each offer identifies
the provider, GPU SKU, GPU count, service tier, region basis, currency, billing
unit, evidence date, and source. Observations are append-only:
offers[].history += {date, value, source_n}
Never replace an old observation when a price changes. If the GPU, service
tier, region basis, or currency changes, start a new series instead of joining
incompatible prices. Two observations produce a mini trendline; one observation
is shown as a dated baseline. Marketplace and calculator-only offers remain
null until a reproducible snapshot is available.
For fixed monthly plans, the report derives a comparison-only hourly value:
normalized_hourly = published_monthly_price / 730
monthly_equivalent = published_hourly_price * 730
The dashboard preserves provider-native currencies and does not compare EUR,
CHF, and USD as if they were equal. The normalization excludes tax, discounts,
storage, egress, public IPs, support, utilization, and serving operations. It is
not a quote and not an effective token price. A defensible cost per million
tokens requires a measured workload:
effective_cost_per_million_tokens = total_observed_cost
/ observed_output_tokens * 1,000,000
Record model, quantization, framework, concurrency, prompt/output mix, uptime,
and all infrastructure charges with that measurement.
Update procedure
- Research official provider pages and versioned benchmark publications.
- Update source URLs and
accessed dates. - Update raw prices, benchmarks, context, and evidence before derived values.
- Recompute benchmark maxima, Fable quality anchors, scores, and chart domains.
- Review reference eligibility and burn factors.
- Review action queue, routing, hardware fit, and changelog.
- Validate every JSON file with
python3 -m json.tool or jq empty. - Run the local build and release checks, then deploy manually. Do not use a
GitHub Actions workflow.
Primary sources for this refresh:
4 - Research Data Policy
Source rights, attribution, permitted use, retention, verification, and privacy review for the data behind Signal Room.
This policy applies to market facts, benchmark observations, pricing,
availability, source links, and historical snapshots used by the Signal Room.
Source rights and collection
Use sources that are publicly accessible and lawful to consult. Prefer primary
sources such as official model cards, documentation, pricing pages, release
notes, repositories, and benchmark publications. Do not bypass access controls,
paywalls, authentication, rate limits, or technical restrictions.
Store only the facts and short summaries needed for market analysis. Do not
mirror articles, proprietary datasets, benchmark submissions, or other
copyrighted source material. A public URL does not imply permission to copy the
underlying work.
Attribution and evidence
Every externally verifiable claim must retain a canonical source URL through
the repository’s source or citation fields. Record the observation date and,
when relevant, the source’s publication or release date. Prefer stable,
first-party URLs; use independent benchmark sources for claims that the vendor
cannot establish itself.
Label evidence as measured, vendor-reported, derived, estimated, or unknown.
Do not combine measurements with incompatible conditions as though they were
directly comparable. Calculated values must identify their formula and inputs.
Permitted use
Collected facts and links are used for research, comparison, summarization, and
decision support. They must remain within the permissions and terms of their
original sources. The repository’s MIT License covers original code and text;
it does not relicense third-party trademarks, source publications, benchmark
data, or linked materials.
Retention and corrections
Keep the current normalized facts required by the dashboard and dated archive
snapshots needed to explain material changes. Retain citation links and
changelog entries so published conclusions remain auditable. Do not retain raw
scrapes, authentication material, private exports, or unnecessary personal
data.
Correct demonstrably inaccurate data in the current dataset and preserve a
short correction record when the change affects a published conclusion. Honor
valid removal requests for material that should not have been collected while
keeping a non-sensitive audit note when appropriate.
Update verification
For each refresh:
- compare the current value with the most recent primary source;
- confirm the product or model identifier, unit, currency, region, and date;
- verify consequential comparative claims against an independent source when
one is available;
- preserve unknown values instead of filling gaps with unsupported estimates;
- validate every citation and record the material change in the archive or
changelog;
- run
bash src/scripts/release-check.sh; and - inspect the rendered dashboard for misleading presentation, unsafe links,
personal data, secrets, and accidental source-text reproduction.
Privacy and security review
The delivered data was reviewed on 2026-07-21. It contains public model,
provider, tool, benchmark, and infrastructure information, not customer or
individual-level records. No credentials, private keys, authenticated URLs,
private datasets, or sensitive personal information were identified. The
project deployment email is intentionally public and is not research data.
Repeat this review whenever a new data class, ingestion mechanism, binary
asset, or non-public source is proposed. Security concerns should follow
SECURITY.md.