🔍 Read the full analysis: We Tried Weight-Space Programming At Home: Claude Edited A Local Model, GPT-6 Astra Checked The Work, And The Legal Score Stayed At Zero on ThorstenMeyerAI.com
TL;DR
A home experiment used Claude-written code to edit a local AI model’s weights and reduce looping. On the 10 published Lexassay legal benchmark tasks, the edited model answered all 10, compared with 4 on the same setup before editing, but its legal score remained zero. The author says the result reflects improved completion, not improved legal knowledge.
A home experiment reported by ThorstenMeyerAI.com found that code written during a Claude Opus 5.5 session could reduce looping in a local AI model, helping it answer 10 of 10 published Lexassay tasks on one test setup, compared with 4 of 10 before the edit. The model’s legal score stayed at zero, however, showing that finishing more answers did not make them legally accurate.
The experiment used Ornith 1.5 35B-A3B, a local mixture-of-experts model whose publisher describes it as designed for coding and agent tasks. The author says the model had failed to finish two of the 10 published legal benchmark tasks, reaching a limit of 62,000 output tokens while repeating itself. In the final portions of those transcripts, only 2% to 13% of lines were distinct, according to the report.
Claude Opus 5.5 wrote code that ran the model, measured a recurring internal activation direction and removed that direction at half strength from the output weights of 27 of 40 layers. The report describes the approach as directional ablation, a known technique. It was not the same method as the MIT researchers’ weight-space programming demonstration: no AI read the model’s weights as text, and the experiment used no training or new legal data.
On the same setup, the edited model completed 10 tasks, up from 4. Its content remained weak: it met 7 of 50 checked criteria, compared with 3 before, and the 10 edited answers included 55 wrong or invented citations, the report says. The author also reports that changing the unchanged model to sampling settings recommended on its model card raised completions to 9 of 10 on its original setup, without editing the weights.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single „loop neuron“.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completed Answers, Same Legal Score
The results illustrate a distinction that matters for anyone evaluating AI tools: completion is not correctness. A model that stops looping may be more usable in a narrow operational sense, but that does not establish that it has gained the knowledge needed to answer specialized questions reliably.
For legal work, the reported citation errors are a serious limitation. Lexassay checks cited laws and court decisions against primary sources, and the edited model’s score remained zero. The experiment therefore does not support using this edit to improve legal advice or research. The source explicitly describes the article as an AI experiment, not legal advice.
The test also highlights how basic sampling choices can affect observed performance. The unchanged model answered 9 of 10 tasks after its settings were changed, while the same model on another setup answered 4 of 10. That makes it difficult to attribute the full completion difference to the weight edit alone.
Popular right now
Ninja AF181 Air Fryer XL, 6.5 Qt, 6-in-1, Dark Grey
Dyson V8 Cyclone Cordless Vacuum
KitchenAid Classic Series 4.5-Quart Tilt-Head Stand Mixer, Onyx Black
As an affiliate, we earn on qualifying purchases.
The MIT Demonstration Behind the Test
The experiment followed a report by four MIT CSAIL researchers—Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab—on what they called “weight-space programming.” In a Qwen model case described in the source, GPT-6 Astra identified a direction associated with looping and edited the model’s weights. The reported maths score rose from 7.9% to 48.1%. The MIT method involved reasoning directly about weights, unlike the home experiment’s run-and-measure approach.
The MIT authors’ stated limitation shaped the home test: an edit may remove an obstacle hiding an ability, but it cannot add an ability the model lacks. The author applied that idea to Lexassay, a benchmark of questions on German, US and EU law. The benchmark has 15 questions; 10 tasks are published and five remain hidden. The figures in the report refer only to the published tasks, and the runs and grading materials are public on Lexassay.com.
The author says Ornith was selected because it showed the looping problem, while other local models either did not loop or were unsuitable for the test. The report says the benchmark’s law-focused model scored 8 out of 100 in an earlier run, while Ornith scored 1 out of 100. Those results provide context for the model’s weak legal performance; they do not establish how it would perform in other settings.
„The model finishes is not the answer is right.“
— Thorsten Meyer, author of the experiment
Limits of the Home Experiment
The results come from one run per setting and a small development set, so they do not show whether the completion change would repeat reliably. The report says the loop detector changed during the experiment, and the three graders came from the same model family. Claude agents also conducted the blind grading, creating a potential conflict that the author sought to address with an independent review by GPT-6 Astra.
That review found real faults in the code, figures and claims, according to the author. The report does not detail those faults in the supplied material or explain how each was resolved. It also cautions that two local setups were not directly comparable: the unchanged model answered 8 of 10 in a published run but only 4 of 10 on the setup used for the weight-edit comparison.
The reported results cover only the 10 published tasks, not the five hidden questions or all 15 questions in the benchmark. It remains unclear whether the edited model’s completion behavior would hold across repeated runs, other prompts, different hardware or a broader set of legal tasks.
Public Runs and Further Testing
The experiment’s answers, grades and citation checks are available through Lexassay.com, alongside the original run, the model-card sampling run, the edited-model run and an edit control. The author also points readers to a method page describing the procedure. Those materials allow readers to inspect the specific outputs and compare the reported setups.
The next useful step would be repeated tests under matched settings, with a stable loop detector and independent evaluation of both completion and factual accuracy. Testing the five hidden benchmark tasks could also help show whether a change reflects general improvement rather than performance on the published development set. The supplied report does not announce a schedule for such work or claim that additional experiments have been completed.
Key Questions
What did the weight edit change?
Code written during a Claude Opus 5.5 session measured a recurring internal direction in Ornith 1.5 35B-A3B and removed it at half strength from the output weights of 27 of 40 layers. The report calls this directional ablation, not direct AI reasoning over the weights.
Did the edit make the model better at law?
Not according to the reported legal score. The model completed more tasks on the comparison setup, but its score remained zero. The edited answers met 7 of 50 checked criteria and contained 55 wrong or invented citations, the report says.
How many benchmark tasks were tested?
The reported figures cover 10 published Lexassay tasks. The benchmark has 15 questions in total, with five held back; the article says its figures refer to the published tasks.
Was this the same method used in the MIT study?
No. The home test used code to run the model, measure an internal direction and alter weights. The MIT demonstration described in the source involved GPT-6 Astra reasoning directly over another model’s weights. The report presents the home test as a simpler, related experiment.
Can the results be treated as reliable evidence of improvement?
The author cautions that the test had one run per setting, a small development set and setup differences that complicate comparison. The results show what happened in this experiment, but do not establish that the change will repeat or improve performance on other legal questions.
Source: ThorstenMeyerAI.com
