Method · updated 27 October 2026
How I verify an AI claim
These are the working checks, not a code of ethics. Each one is an action I can be caught skipping, which is the point. When a piece is wrong, the correction log is where the process changes.
-
01
Read two official pages, line by line
Action: open the announcement and the documentation — or the model card, the pricing page, the filing. Read both completely, including tables, locale lists and footnotes. Quote the exact line. If they disagree, the documentation describes behaviour and the announcement describes intent; say which one the claim uses.
Why: the most common error I have made is reading the prose and skipping the table beside it. That is how a language list became “Chinese was not mentioned”.
-
02
Give every number a source and a date
Action: for each figure, record who published it, on what date, and what it counts. Prices, parameter counts, shipments and benchmark scores all move. If I cannot fill all three fields, the number does not ship — even when it is the headline.
Why: an undated number becomes wrong silently, and the reader has no way to tell when.
-
03
Label vendor-reported benchmarks at the claim
Action: write “vendor-reported” next to the score and name who ran the test. If a table mixes vendor and third-party results, add a column for the runner and keep them separate. Do not average across runners.
Why: a benchmark measures the benchmark too. Who ran it, on which version, with which prompt and decoding settings, is part of the number.
-
04
Separate preprint from peer review
Action: name the venue and the review status. “A paper” is not evidence of acceptance. State whether the code, data and evaluation harness are available, and whether the result was reproduced outside the authors’ lab.
Why: the distance between a preprint and a reviewed result is the difference between a claim and a finding.
-
05
Print conflicting numbers; never average them
Action: print both numbers, both dates and both definitions. Add one line on what would settle it — a filing, a later document, a direct measurement. If nothing settles it yet, label the conflict open.
Why: an average invents a third number that nobody published, and hides which source said what.
-
06
Read the Chinese original
Action: for Chinese hardware and model releases, open the Chinese report, name the outlet, and translate the sentence myself. Use an English aggregator only to find the original — never as the citation. When the original is a vendor post, say that too.
Why: rewrites drop qualifiers, and qualifiers are where the caveats live.
-
07
Treat a promise as a promise, not a shipment
Action: record promised dates — “full weights by 4 November”, “GA in the coming weeks” — in the watch list, with the date. On the date, check and publish the result. Until then the verb is “says it will”, not “has shipped”.
Why: most of the gap between an announcement and a product is time, and time is checkable.
-
08
Publish corrections with a date
Action: keep the original wording visible, append the correction, and state what was wrong, what changed and what is still open. If the error came from a method rather than a typo, add the process change to this page.
Why: a correction that only lives in a diff is not a correction. The reader who saw the first version needs to see the second.
What I will not publish
- Forecasts written as if they already happened.
- Funding or acquisition rumours stated as fact.
- A number without a source and a date.
- Paid placement, affiliate links or sponsored posts.
- Screenshots or documents that expose private user data.
- Unnamed client work or confidential material.
Corrections policy
Email superapp.devin@gmail.com with the claim, the page and what you think is wrong. Every correction that changes a published claim gets an entry in the correction log with a date and the new wording. If I cannot verify a correction, I will say that too, rather than silently ignoring it.
The correction log currently has one entry. That is a small number because the archive is small, not because the method is finished.
“The difference is method, not volume.”
Back to the notebook