The News Pipeline AI Setup I Use Isn’t The One That Scored Best

🔍 Read the full analysis: The News Pipeline AI Setup I Use Isn’t The One That Scored Best on ThorstenMeyerAI.com

Publisher Thorsten Meyer says a test of 71 AI model configurations for writing news articles found that GPT-6 Astra produced the highest-rated work, but its reported cost led him to use GPT-6 Luna in production instead. The results, published October 13, also led Meyer to remove two fallback models after his judges flagged invented quotes or invalid output.

Meyer tested configurations between September 8 and October 13, using the production prompt and real news sources from his publishing workflow. Each source was turned into an English and a German article, then evaluated by two AI judges: GPT-6.1 Sol and Claude Opus 5.5. The judges rated source fidelity, fabrication, clarity about what was known, news value, structure and polish; German articles also received a language-quality score.

In the initial test, Meyer reports that GPT-6 Astra scored 4.00 out of 5 in both languages, with a source-fidelity score of 5.00. In a follow-up run across 12 articles, it averaged 3.93, with a reported 95% bootstrap interval of 3.83 to 4.02. GPT-6 Luna averaged 3.67 in that run, with a reported interval of 3.59 to 3.76. Meyer says Astra cost about $0.12 per article, compared with about $0.0012 for Luna, and took about 50 seconds rather than 17.

Meyer says Luna remained his production choice because it was consistently usable at a much lower cost. He reports that it scored 4.71 out of 5 on source fidelity in the follow-up run and that all its articles passed his pipeline’s output-validity check. He also says raising Luna’s reasoning setting produced no dependable improvement: the highest setting cost about 12 times as much and took about 12 times as long as the no-reasoning setting, without a clear score gain.

Cost and Accuracy in the Publishing Pipeline

The test highlights a practical decision for publishers using AI at scale: the highest-rated model may not be the most suitable choice when each article’s price and production speed matter. In Meyer’s reported figures, Astra’s stronger scores came with a per-article cost about 100 times Luna’s. Whether that premium is worthwhile depends on a publisher’s quality threshold, volume and tolerance for errors; Meyer’s results do not establish a universal best model.

The fallback findings also point to a separate risk. A primary model may perform well while a backup quietly introduces unsupported material or fails to produce usable output. Meyer says one fallback generated invented quotations and another produced no valid articles in its test. For readers and publishers, such errors matter because fabricated attributions can make unsupported statements appear to come from real people.

The results also caution against assuming that performance transfers from one task to another. Meyer reports that Claude Opus 5.5, which he describes as his preferred coding model, scored below Luna in his article-writing test. The comparison supports testing models on the actual work they are expected to do rather than relying only on broad rankings, though it remains one publisher’s evaluation.

How Meyer Tested Article Quality

The evaluation used two source texts: a 6,340-character product filing and a 619-character political news item. Meyer says the shorter item better reflected much of his usual news workflow, where source material may be limited. The strongest configurations then completed a robustness run producing 10 additional articles, mostly from short sources.

An article counted only if it yielded valid output after the same repair step used in production. Meyer says the two judges saw the original source and scored each article across several criteria. He also set a rule that score gaps below about 0.3 points in the two-article initial test should be treated as noise, relying more on the follow-up run for rankings.

According to Meyer, the test led him on October 7 to remove GPT-4.1 nano and locally run gpt-oss-120b from the news route. He reports that GPT-4.1 nano scored 1.62 in English and 1.68 in German, with source fidelity at the bottom of the scale. The local model produced no valid articles in four attempts. Meyer says Luna became the production model after those changes.

„The right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.“

— Thorsten Meyer, in his October 13 article

Limits of the Reported Test

The figures are Meyer’s own reported results; the source material does not describe an independent replication or external audit. The test used a limited set of source texts, and the follow-up run covered 12 articles for the leading configurations. The results therefore show how the models performed under this particular prompt, source selection, judging setup and output-validation process, not how they will perform for every publisher or topic.

The source does not provide the full article outputs, detailed judge-by-judge scores, or enough information to independently assess how the judges’ evaluations were reconciled. It also does not state whether model versions or pricing could change after the test period. Meyer’s description of one configuration as clearly ahead is his conclusion from the reported intervals and robustness run; readers cannot verify that conclusion from the summary alone.

It is also unclear from the supplied material exactly how Luna is configured as a production route after October 7. Meyer says Luna runs first, but the final description of the route is incomplete. The article does not establish whether there are replacement fallbacks, nor how often the production system encounters failures or requires repair.

Production Use and Further Testing

Meyer says GPT-6 Luna remains in production after the October 7 routing changes, while Astra remains the top-scoring option in his evaluation but is too expensive for his news workflow. The immediate next step described by the report is continued use of Luna; it does not announce a new benchmark date or a planned switch to another model.

For the reported findings to guide other publishers, the key next evidence would be repeated testing on a broader set of sources and ongoing checks of the live pipeline, particularly its fallback behavior. Meyer’s article does not say whether he plans those tests. Model prices, versions and performance may change, so the reported scores and costs describe his evaluation period rather than a guarantee of future results.

Key Questions

Which model scored highest in Meyer’s test?

Meyer reports that GPT-6 Astra scored 4.00 out of 5 in both English and German in the initial test. Across a 12-article follow-up run, it averaged 3.93.

Why did he choose GPT-6 Luna for production?

Meyer says Luna was much cheaper and faster while still producing consistently usable articles. He reports an approximate cost of $0.0012 per article for Luna versus $0.12 for Astra, with reported times of 17 and 50 seconds respectively.

Did higher reasoning settings improve Luna’s articles?

Meyer says the differences across six reasoning settings were within the test’s noise. The maximum setting cost about 12 times as much and took about 12 times as long as the no-reasoning setting, without a reliable quality gain.

What happened to the two fallback models?

Meyer says GPT-4.1 nano produced articles with invented quotations, while locally run gpt-oss-120b produced no valid articles in four attempts. He reports removing both from the news route on October 7.

Can the results be applied to other publishers?

Not directly. The scores reflect one publisher’s prompt, sources, validation process and AI judges. Other publishers would need to test models against their own content and production requirements; the results have not been independently replicated in the supplied material.

Source: ThorstenMeyerAI.com