Most multi-agent frameworks tell you to give each agent a persona. CrewAI makes you define role, backstory, and goal. The intuition is obvious: a “Senior Compliance Officer with 15 years of experience” should produce better compliance reviews than a generic AI assistant.
Research says otherwise for accuracy tasks. But accuracy tasks (multiple choice, math) aren’t what consulting agents do. They produce professional deliverables — policies, gap analyses, reports. Maybe personas help there?
I ran two experiments to find out. Same frontier models (Gemini 3.1 Pro, Claude Opus 4.6, GPT-5.4), same CrewAI pipeline, same tasks. Only variable: Run A had generic agents, Run B had full expert personas. Outputs were blind-evaluated by Opus — the judge didn’t know which was which.
Experiment 1: Policy Document
Task: draft an AI risk tiering policy for a tier-1 bank.
Result: Run A (no persona) scored 29/30. Run B (persona) scored 26/30. The no-persona run was more precise, more self-contained, had better regulatory citations. Personas hurt for structured output.
Experiment 2: Gap Analysis
Task: regulatory gap analysis across five jurisdictions.
Result: Run B (persona) scored 28/30. Run A (no persona) scored 25/30. The persona run named specific institutional systems, cited internal document versions, quantified current state with precise numbers. The judge praised the “institutional depth.”
So personas help for judgment tasks? I almost published that conclusion.
Then I checked the facts.
The persona run named “Amy chatbot” — that’s real, HSBC launched it in 2017. It cited “Group AI Governance Standard v2.1” — no public trace. Fabricated. It stated “627 active use cases” — the prompt said “600+.” False precision. “Azure OpenAI pilot” — plausible but unverifiable.
The persona made the model mix real institutional knowledge with confident fabrication. The blind judge couldn’t tell the difference. It scored fabricated specificity as “institutional depth” and gave Run B three extra points for it.
The finding
Persona prompting generates the exact type of hallucination that LLM-as-judge evaluation rewards.
Here’s the mechanism:
- Personas prompt the model to simulate domain expertise
- Part of that simulation is generating specific institutional details (system names, document versions, precise numbers)
- Some of those details are real (from training data), some are fabricated
- LLM judges evaluate outputs on dimensions like “completeness,” “depth,” and “actionability”
- Fabricated institutional details score well on all three dimensions
- The judge literally cannot distinguish real from fake institutional knowledge
This isn’t the same as general LLM hallucination. General hallucination is well-studied. This is persona-amplified fabrication that specifically exploits automated evaluation. The persona doesn’t just make the model hallucinate more — it makes it hallucinate in ways that look like quality to another LLM.
Why this matters
Everyone is building LLM-as-judge evaluation into their agent pipelines. It’s the standard approach — run the output through a rubric-scoring LLM, check the scores, ship if above threshold. The entire evals ecosystem assumes the judge can detect quality.
But if persona prompting systematically inflates judge scores by generating rewarded fabrication, then:
- Persona-equipped pipelines will always score higher in automated evaluation — not because they’re better, but because they hallucinate in ways the judge rewards
- A/B tests using LLM-as-judge will systematically prefer persona configurations — the evaluation itself is biased
- Production systems with personas will ship more confident fabrication — because the quality gate can’t catch it
The practical rule
If your evaluation uses LLM-as-judge: don’t use personas. The combination is adversarial — personas generate exactly what judges reward, regardless of whether it’s true.
If a human domain expert reviews every output: personas may help for judgment-heavy tasks. Humans catch “Group AI Governance Standard v2.1” instantly. The persona’s real benefit — activating domain-relevant attention patterns — survives human review. Its fabrication doesn’t.
If you must use personas with automated evaluation: add a verification dimension to the rubric. Score “are institutional claims verifiable?” alongside completeness and depth. This at least forces the judge to be skeptical, though it may not fully solve the problem since the judge can’t actually verify claims either.
Caveats
This is two experiments with n=1 per condition. The effect could be noise. The judge (Opus) may have its own biases. CrewAI’s prompt template injects some framing even in the no-persona condition. The “enriched task prompt” alternative (same context, no persona framing) hasn’t been tested yet.
But the directional finding — personas amplify the type of hallucination that automated judges reward — is consistent across both experiments and mechanistically plausible. It’s worth testing before you ship a persona-equipped pipeline with LLM-as-judge evaluation into production.
The frameworks won’t tell you this. They sell personas as a feature.