# Data Methodology > Formulas, evidence classes, chart encodings, and the repeatable update procedure behind the Signal Room report. --- LLMS index: [llms.txt](/ai-market-research/llms.txt) --- The dashboard has three canonical data inputs: - `data/market-state.json` preserves the original report schema and backward compatibility. - `data/model-roster-v2.json` contains the curated provider roster, model configuration levels, availability, pricing labels, and speed evidence. - `data/report-metrics.json` contains normalized metrics, chart encodings, current economics, hardware fit, and decision-support data. The build validates all three files and merges the latter two under `marketData.model_roster` and `marketData.report_metrics`. Do not fetch data at runtime: GitHub Pages remains a single, manually rebuilt static artifact. ## Evidence classes Every speed or hardware value should be one of: 1. `measured`: reproduced in the project's pinned harness. 2. `vendor`: copied from an official primary source with its test context. 3. `vendor-relative`: a vendor comparison without an absolute measurement. 4. `planning_estimate`: useful for capacity planning but not a measured fact. Unknown values remain JSON `null`. Never turn an absent number into zero. A hardware combination that cannot run uses `fit_status: "unsupported"` and a reason. This replaces ambiguous empty `GB tok/s` cells. ## Benchmark evidence and reference quality Each benchmark definition has a `max_value`, version, unit, and direction. For the current all-higher-is-better register, individual cells may be normalized for color intensity: ```text normalized(model, benchmark) = 100 * value / max_value ``` `max_value` is the documented maximum for that benchmark version or comparison set. Values from different benchmark versions must not share a column. Missing results stay null; the dashboard does not infer zero or manufacture a benchmark composite from sparse, non-overlapping evaluations. Do not merge a vendor launch table with an independent leaderboard into one ranking unless the model version, agent harness, dataset split, scoring rules, and evaluation date are demonstrably the same. Treat both as useful evidence: the former documents the vendor's measured configuration, while the latter can provide an independent public comparison. The July 2026 GPT-5.6 launch table and Scale's public SWE-bench Pro board are intentionally shown as separate sources for this reason. The July 31 DeepSeek V4-Flash-0731 update is handled the same way. Its Terminal-Bench 2.1 and Agents' Last Exam values are shown with the vendor's max-effort, DeepSeek Harness minimal-mode caveat. NL2Repo, Cybergym, Toolathlon, Automation Bench, and the internal DSBench sets remain narrative evidence until their versions, harnesses, and public reproducibility are sufficient for a shared comparison column. Each model instead has a curated `quality_vs_fable` anchor. Prefer a published multi-benchmark comparison against Fable 5. A historical normalized score or a shared SWE-Pro version may be used as a clearly documented fallback. The Kimi K3 anchor, for example, is the geometric mean of the 14 K3/Fable 5 ratios in Moonshot's launch comparison. Quality relative to the selected reference is computed at render time: ```text quality_vs_reference = 100 * quality_vs_fable(model) / quality_vs_fable(reference) ``` It may exceed 100 and must not be capped. ## Speed evidence and score Keep TTFT, output tokens per second, and end-to-end task time separate. The current cross-provider chart uses an end-to-end task-speed index because comparable token-throughput measurements are unavailable: ```text task_speed_index = 100 * reference_task_time / model_task_time speed_score = 100 * sqrt(task_speed_index / max_task_speed_index) ``` Thus Fable 5 is 100 in the cited comparison. A task-speed index of 300 means the model completed that workload at three times the rate; it is dimensionless and does not mean 300 tok/s. Provider-documented output throughput is displayed separately in tok/s with its evidence class. The square root reduces the visual dominance of extreme relative task-rate claims. Replace relative indices with local measurements after recording model, effort, provider, service tier, region, date, cache state, input/output token counts, and workload. Historical points that do not come from a frozen test must retain their `history_note` caveat. ## Cost and burn The workload price blend is: ```text blended_price = 0.30 * input_price + 0.70 * output_price ``` This represents an output-heavy agent workload. It excludes tools, long- context surcharges, cache writes, batch discounts, and hidden reasoning tokens. Those must be modeled separately when applicable. Cost is converted to a higher-is-better score with log normalization: ```text cost_score = 10 + 90 * ln(max_price / price) / ln(max_price / min_price) ``` Recompute `min_price` and `max_price` over visible, eligible models. Burn uses the original report's selectable-reference design: ```text cache_factor = (1 - cache_hit_rate) + cache_hit_rate * cache_read_ratio absolute_burn = base_blended_price * effort_factor * cache_factor display_burn = absolute_burn / reference_medium_absolute_burn ``` The GPT-5.6 and Fable effort factors are planning estimates until a controlled run measures token use. Fable has always-on adaptive thinking, so its effort setting is a behavioral control, not a published token multiplier. ## Quality, speed, and cost views The bubble chart uses speed score on x, cost efficiency on y, quality versus the selected reference for bubble area, and provider for color. The heat map uses the same three values without a synthetic overall compound. A model enters either view only when all three values are present. Bubble radius is proportional to the square root of quality, so area represents magnitude. Labels sit in edge rails with connector lines to avoid clipping dense clusters. ## Capability compound The six editorial axes are coding, reasoning and architecture, knowledge and research, communication and documentation, multimodal, and agentic work. They are rubric assessments, not benchmark results. Balanced mode assigns every axis weight 1. Selecting a focus assigns that axis weight 2.5 and leaves the other five at 1: ```text capability = sum(axis_score * axis_weight) / sum(axis_weight) ``` The radar compares two chosen models on the raw six-axis scores. The BCG-style chart uses the mean of quality versus reference, speed, and cost on x and the focus-weighted capability composite on y. It requires all three market values. Its 50 lines are orientation guides, not targets. ## Hardware fit Hardware throughput depends on quantization, backend, batch size, context, prompt length, interconnect, power limit, and speculative decoding. Every measured update should store those dimensions. Until then, values marked `planning_estimate` may support filtering and pastel speed bands but must be labelled as estimates in the UI. Recommended display bands are computed per visible table, using the 25th, 50th, and 75th percentiles of non-null throughput. Do not give unsupported or unknown cells a color implying performance. ## Hosting price history Hosting prices live under `report_metrics.hosting_prices`. Each offer identifies the provider, GPU SKU, GPU count, service tier, region basis, currency, billing unit, evidence date, and source. Observations are append-only: ```text offers[].history += {date, value, source_n} ``` Never replace an old observation when a price changes. If the GPU, service tier, region basis, or currency changes, start a new series instead of joining incompatible prices. Two observations produce a mini trendline; one observation is shown as a dated baseline. Marketplace and calculator-only offers remain null until a reproducible snapshot is available. For fixed monthly plans, the report derives a comparison-only hourly value: ```text normalized_hourly = published_monthly_price / 730 monthly_equivalent = published_hourly_price * 730 ``` The dashboard preserves provider-native currencies and does not compare EUR, CHF, and USD as if they were equal. The normalization excludes tax, discounts, storage, egress, public IPs, support, utilization, and serving operations. It is not a quote and not an effective token price. A defensible cost per million tokens requires a measured workload: ```text effective_cost_per_million_tokens = total_observed_cost / observed_output_tokens * 1,000,000 ``` Record model, quantization, framework, concurrency, prompt/output mix, uptime, and all infrastructure charges with that measurement. ## Update procedure 1. Research official provider pages and versioned benchmark publications. 2. Update source URLs and `accessed` dates. 3. Update raw prices, benchmarks, context, and evidence before derived values. 4. Recompute benchmark maxima, Fable quality anchors, scores, and chart domains. 5. Review reference eligibility and burn factors. 6. Review action queue, routing, hardware fit, and changelog. 7. Validate every JSON file with `python3 -m json.tool` or `jq empty`. 8. Run the local build and release checks, then deploy manually. Do not use a GitHub Actions workflow. Primary sources for this refresh: - - - - - - - - - - - - - - - - - - - -