In a debate study, rhetoric tricks pulled people toward whichever AI used them. Score claims, not speeches

On October 4, researchers posted a study of a proposed check: let two AIs argue opposite sides and have a person pick the winner. When one side used rhetorical tricks, people’s verdicts moved toward that side, whether it was arguing for the truth or against it. The fix this post offers is to judge the claims under an argument instead of how well it reads. That is what a decision map’s separate parts for evidence, claims and kill tests are for.

What the study did

The paper, “AI Safety via Debate is Compromised by Cognitive Biases,” is a preprint on arXiv by Gefei Liu, Sonya Rashkovan, Sophia Lloyd George, Isaac Sheidlower and Serena Booth.

• The setup. The authors built 68 model-written debates about detective mysteries where the real culprit was known. One side argued for the culprit, the other for an innocent suspect.
• The tricks. They used four. Anchoring is the pull of whatever comes first. Fallacy oversight means missing a flawed step, such as a straw man or a false choice. Pro-jargon is treating technical-sounding words as evidence of rigor. Verbosity means favoring the longer answer.
• The key control. Each trick was given to the true side in some debates and to the false side in others. That separates “the trick persuaded people” from “the trick happened to sit on the right side.”
• The people. 400 people on Prolific finished the task. 31 were excluded for failing a story check or reading too fast, which left 369. Each judged one debate, and picking the true culprit earned a $0.50 bonus.

What it found, and what it did not

With all four tricks pooled, the trick shifted judgments toward the side using it by 8.6 percentage points (95% confidence interval, 2.9 to 14.2). The authors say this shift “favors the manipulated side rather than correctness.”

Three limits matter as much as the headline:

1. No single trick was proven on its own. All four estimates leaned the same way, from 7.1 points for fallacies to 11.1 for verbosity, but none was statistically significant alone. The authors call the individual effects uncertain.
2. These were detective stories in a lab. The authors say effects may differ elsewhere and that real tricks may be subtler, or combined.
3. The models were told to use the tricks. The study does not show how often models would learn to do this on their own.

So the finding is narrow and useful: a person judging an argument can be moved by how it is presented, even with the other side there to respond. The authors suggest that judges assess evidentiary support, logical validity and responses to counterarguments before deciding. They also say such fixes still need testing.

Where a map helps

The Of One method starts from a rule that fits this finding: the answer is a rendering, never the source of truth. The skill classifies “each material assertion” as evidence, a claim, an unknown, an assumption and so on, and keeps them separate from the prose a reader sees. Its validator asks whether each strong claim lists support, contradiction or gaps, a confidence basis and a failure mode. Each of the four tricks has less to work with once an argument is broken into those parts:

• Verbosity adds words, not rows. The method’s movement economy says to delete any sentence that does none of ten jobs, such as ground, claim, test or warn.
• Jargon cannot fill an empty evidence field.
• A fallacy shows up as a claim linked to evidence that does not support it.
• Anchoring depends on order. A table can be read in any order.

The skill also requires an adversarial challenge pass, and it requires human review for high-severity dissent. Its question runtime blocks release while a contradiction that could change the decision is still open. The disagreement itself goes to a person, not the better speech.

The claims-table test

Run this once this week on a real call where an AI made a case to you: a vendor choice, a pricing change, a plan, or a pro-and-con it wrote for you.

1. Pull out the claims. For each side, make a table with one row per claim and three columns: the claim, the evidence it points to, and the result that would prove it wrong. You can ask the AI to fill it, but check each evidence cell against the source yourself.
2. Apply one rule. A row with no evidence and no way to prove it wrong scores zero, however well it was argued.
3. Run four quick checks.
   • Order: read the second side’s table first. Does your lean change?
   • Length: count rows with real evidence, not words.
   • Jargon: rewrite each claim in plain words. If nothing is left, cross it out.
   • Logic: for every “so” or “therefore,” write what connects the evidence to the claim. If you cannot, mark the row weak.
4. Decide from the tables. Then compare with what you leaned toward after reading the prose. If the two differ, write down which row changed your mind. That gap is the finding.

This test is not from the paper. It applies the authors’ suggestion and the method’s claim checks to your own decisions. It does not promise the right answer. It shows what your answer rests on.

What to do this week

• Pick one AI-backed decision from the last month and run the claims-table test on it.
• When you ask an AI to argue both sides, ask for the claims table too, not only the essay.
• Never let a missing evidence cell pass because the paragraph around it was confident.
• For a call you cannot undo, have a named person sign off on the tables, not the speeches.

If you want time with Utlyze running this method on a decision of yours, that is a consult: $400 an hour, in 5-hour blocks.