The document was the Rail Accident Investigation Branch’s report into the October 2021 collision at Salisbury Tunnel Junction. XDA gave the same 113-page PDF and eight questions to paid versions of ChatGPT, Claude, and Google’s NotebookLM, covering retrieval, lay summaries, chronology, causal analysis, cross-document evidence synthesis, governance, counterfactual reasoning, and a chart-reading task.
One key result can be checked independently against the primary record. XDA reported that Claude chose double variable-rate sanding as the intervention most likely to prevent the crash, while ChatGPT identified earlier braking at Broken Cross bridge. The RAIB report supports ChatGPT’s reading: its analysis says the collision would probably have been avoided had the brakes been applied at Broken Cross bridge under the adhesion conditions that evening. It treats braking at the fallen tree, 250 meters farther on, as much less certain: the train would likely have passed the signal at danger, although the collision might have been avoided or occurred at lower speed.
That is a real miss in a high-stakes question. It is also precisely the kind of miss that makes document AI unsuitable as an unreviewed source of record for incident investigations, policy decisions, compliance work, or technical change approval.
The checked answer favors ChatGPT, but it does not validate the whole ranking
XDA’s author says ChatGPT returned the collision time as 6:42:57 p.m., while the other tools rounded it to about 6:43 p.m. The report’s evidence also supports XDA’s account that ChatGPT assembled a multi-part explanation for why the train could not stop: onboard recorder information, wheel-slide-protection activity, rail contamination testing, brake testing, and the driver’s braking point all belong in that analysis.
The RAIB’s own findings show why this is a meaningful test rather than a trivia contest. The accident was not explained by one sentence in an executive summary. The investigation found very low wheel-to-rail adhesion, late braking, and wider operational and infrastructure conditions that made the failure possible. A tool that retrieves one relevant paragraph can sound authoritative while leaving out the relationship between these causes.
ChatGPT’s reported strength, then, was synthesis: tracking a conclusion across evidence spread through a technical report. That is useful for admins working through a vendor postmortem, a long Microsoft advisory, a regulatory filing, or a software incident report. The practical benefit is speed in locating the relevant record and framing the questions that need human review.
It is not a license to replace that review. The Salisbury example shows that even when a chatbot names several pieces of evidence, the user must still test whether it selected the decisive evidence and represented the conclusion with the same degree of certainty as the source.
Claude’s document limit complicates the chart test
The 113-page file created a significant comparison problem that XDA did not appear to discuss. Anthropic’s current documentation says that Claude models capable of visual PDF analysis can examine charts, graphics, and images only in PDFs under 100 pages. For PDFs over 100 pages, Claude processes text rather than images.
That limitation matters because one of XDA’s questions asked every tool to inspect Figure 41 and compare the Ellipse work bank with a red dotted mean-average line on the accident date. Claude may still have returned the right answer if the chart’s labels, surrounding explanation, or accessible document text supplied enough information. The RAIB report does describe the chart and identifies the accident-date marker. But a correct response on that question cannot demonstrate Claude’s visual comprehension of a 113-page PDF under Anthropic’s stated processing rules.
OpenAI’s own file-upload documentation contains a related constraint. Outside ChatGPT Enterprise’s visual-retrieval feature for PDFs, uploaded documents are handled through text-based retrieval, with embedded images discarded. A user asking consumer ChatGPT to interpret an unlabeled graph, architecture diagram, scanned table, screenshot, or page-layout detail should therefore not assume the model saw what a human sees in a PDF viewer.
This is the part of the test that should change workflow decisions. “It answered the chart question” is weaker than “it correctly interpreted the chart’s visual data.” Those are different claims, and the second needs proof: either a documented multimodal processing path or a separate image upload and validation against the chart.
NotebookLM’s citations remain its operational advantage
XDA gave NotebookLM credit for the feature that matters most when the output must survive scrutiny: source-linked citations that can be opened or previewed beside the answer. Google’s NotebookLM documentation describes answers as being grounded in selected uploaded sources and explains that its citations can take a reader directly to the supporting text or source context.
That does not make NotebookLM infallible. A citation can support only part of an answer, point to evidence that does not justify the model’s inference, or omit conflicting material elsewhere in the document. Still, it gives the reader a much faster way to perform the essential check that Claude’s polished but incorrect answer required: show me the passage, and show me whether it says what the assistant claims.
For this class of task, citation usability may be more valuable than a marginal advantage in prose quality. If an administrator needs to brief leadership after reading a 90-page outage report, the best answer is not merely concise. It needs a traceable path back to exact source language, page context, diagrams, and assumptions.
ChatGPT can be asked to quote or identify supporting passages, but that is a user-imposed discipline rather than the citation-first interface XDA describes in NotebookLM. For factual extraction from one or more supplied documents, keeping the evidence visible should be the default.
The test needs reproducibility before it becomes a buying guide
XDA’s reported test has value because it includes a falsifiable factual question and one answer that can be compared with the RAIB’s analysis. But it is a single-document evaluation, performed at a particular time, with unspecified model selections and no published output transcript. That is enough to reveal a failure mode; it is not enough to establish durable performance rankings for ChatGPT, Claude, and NotebookLM.
Model selection is especially important. “Premium version” does not identify the actual model, whether a tool invoked a research or reasoning mode, whether document-analysis settings changed, how much context was preserved, or whether answers were regenerated after initial failures. Prompt wording also matters. A five-sentence lay explanation tests instruction following as much as it tests source comprehension, while a counterfactual question tests whether the assistant can distinguish a report’s stated conclusion from a plausible alternative intervention.
A sound in-house comparison should retain the original PDF, record the exact model and plan, save every prompt and response, and use answer keys that include both correct conclusions and the source passages that establish them. It should also include deliberate traps: facts found only late in the report, similar-but-not-identical recommendations, figures whose answer cannot be extracted from nearby prose, and questions where the correct answer is “the document does not establish this.”
For IT departments, a simple acceptance rule is more useful than a leaderboard:
- Treat AI-generated summaries as drafts until a reviewer checks the cited passage in the original document.
- Require page or section references for factual claims, recommendations, numerical values, and causal conclusions.
- Test charts, scans, tables, and diagrams separately from text extraction, especially when the PDF exceeds a provider’s visual-processing limits.
- Keep uploaded incident reports, contracts, customer records, and security material within an approved service and retention policy.
XDA’s test supports the conclusion that ChatGPT can produce stronger synthesis than its competitors on this particular report, and the RAIB record backs its key finding on the Broken Cross bridge scenario. The larger lesson is less flattering to every tool involved: the answer that sounds clearest can still bypass the decisive evidence, while the answer that is easiest to verify is often the one safest to put into a real workflow.