AI in Chambers

Where It Helps, Where It Fails, and How to Tell

R. Polk Wagner

FJC National Symposium for U.S. Court of Appeals Judges, October 8, 2026

AI-assisted · author-reviewed

→What these AI tools can do ... and how they can fail
What is new: AI agents
What chambers are doing
Where AI helps in appellate work
What law schools are doing

In Three Years, AI Went from a C+ to Near the Top of the Class

2023: ChatGPT passed four law school finals, at about a C+
2023: GPT-4 passed a simulated bar exam
2026: Contracts professors, reading blind, preferred AI answers to other professors’ answers about 75% of the time
One caution: legal research AI tools were wrong or unsupported by the cited source in 17 to 33% of answers!

The question is no longer whether these tools can do legal work. They can.

Choi et al., 71 J. Legal Educ. 387 (2023); Katz et al. (2024) and Martínez (2025); Salinas et al. (2026); Magesh et al., 22 J. Empirical Legal Stud. 216 (2025).

In Spring ’26, on Real PennLaw Finals, AI Landed at or Near the Top of the Class

10/10

of the ten graded exams at or above the class median

7/10

of the ten graded exams in the top tenth of the class

2

exams where it outscored every enrolled student

Wagner, Can AI Ace Your Exam? (draft, June 2026). Eleven Penn Carey Law finals, Spring 2026; one model, Claude Opus 4.7; graded on the class curve by the faculty who wrote them. Two exams were not graded blind; one, graded by the author, is one of the two where AI outscored every student. Ten of the eleven were returned with grades.

Search Retrieves. An AI Model Generates.

Current AI tools can do both, when used well.

A Fabricated Citation Is a Predictable Failure

Retrieve to confirm existence. Read to confirm support.

Bias Can Enter at Multiple Stages

Carefully consider the entire chain, not one fluent answer.

Grounding AI Work Dramatically Changes the Risk Profile

Grounding narrows risk. It does not resolve all questions.

✓What these AI tools can do ... and how they can fail
→What is new: AI agents
What chambers are doing
Where AI helps in appellate work
What law schools are doing

In 2026 Three Big Changes Matter

More of the record
- Current AI tools can work with much longer records or source documents
- Having the entire record as context dramatically improves their power
More reasoning
- Extra computation before answering: fewer hallucinations, better writing
More agency
- Agents search, read, choose next steps, and act through connected systems
- They can read and create files, search databases, act independently

Current AI Tools Can Operate in Three Modes

The third mode is the new one: the tool acts. This is real power.

AI Agents Now Ship Inside Mainstream Products

Product	What it does
Claude Cowork* (Anthropic)	Works across your files and tools
ChatGPT Work (OpenAI)	Runs multi-step projects across your apps
Copilot Cowork (Microsoft)	Acts across Outlook, Teams, and Excel
Claude Code and Codex	Coding agents that read, write, and run files

Examples, not endorsements. Check what your court has approved.

*Being folded into the main Claude app since September 2026. Product descriptions from vendor announcements and press coverage, 2026.

With Agents, You Give a Goal, Not a Prompt

task-for-the-agent.txt

<goal>
  Procedural history of this appeal
<sources>
  Docket and joint appendix only
<limits>
  Read only. Cite the record.
<done when>
  Every entry included or flagged
<stop and ask>
  If anything is missing

You state the outcome, not the steps
The agent plans the work and carries it out
Sources, limits, and a stopping point are the new prompt

The skill has moved from phrasing a question to delegating a task.

Best Practice: AI Agents Work Between Checkpoints

Important: A trace given by the AI is not judgment, even if it appears to be.

Each Agent Strength Carries Its Own Risk

An agent	The strength	The risk
Takes many steps	Covers the whole record	An early error carries forward
Chooses its own steps	Needs far less direction	You see results, not choices
Reads what it finds	Gathers its own sources	May obey text inside a document
Acts in other systems	Finishes the task	Acts before anyone checks

Five Controls for an AI Agent in Chambers

Give the agent a bounded task and a defined set of sources
Grant read-only access unless the task needs more
Require approval before any action that cannot be undone (deleting!)
Check the output against the sources, not against the trace
Use only tools you approve and understand!

Delegate the steps. Keep the checkpoints.

Capability Is Not Authority

More capability, and more plausibility, does not make generated text authoritative. That is the human’s job.

✓What these AI tools can do ... and how they can fail
✓What is new: AI agents
→What chambers are doing
Where AI helps in appellate work
What law schools are doing

Penn Law Runs an AI Testbed for Judicial Chambers

PennLaw-Judiciary AI Testbed. Counts as of September 2026. Participation terms, including confidentiality, are set by a memorandum of understanding.

Our Chambers Report Most Value in Bounded, Sourced Tasks

Procedural history and factual background from uploaded pleadings
Summaries of the parties’ arguments across briefs
Charts and timelines assembled from the record (visuals!)
Proofreading and citation-format checks on drafts
Plain-English rewrites of remarks and voir dire questions

Each task starts from a document in view, so the output can be checked against it.

PennLaw-Judiciary AI Testbed. Chambers reports, fall 2024 to October 2026; chambers are always de-identified.

Where Our Chambers Report Failure

Case law recalled from model memory
- An unmodified AI tool is not the right tool for this!
Full opinion drafts in a single pass, even with the briefs uploaded
- Better success: use chunks, section-by-section or even paragraph-by-paragraph
Answers about a long uploaded record, with invented quotations
- Be careful of ‘context window’ issues
Legal analysis is often shallow or over-broad
- Clerks and judges are still better

Learning how to use these tools well is the key to success.

PennLaw-Judiciary AI Testbed. Chambers reports, fall 2024 to October 2026; chambers de-identified. Some items come from a single chambers.

A Recurring Failure We’ve Observed

A real case, with an invented quotation.

Reported by three chambers since fall 2025
One chambers saw it even with a deep-research mode
Confirming that the case exists does not catch the invented quotation

Our Chambers Are Also Seeing AI in the Filings

Most visible in pro se filings; some chambers also point to TRO and injunction motions
Fabricated citations are only one sign
Another sign: the elements of a legal test reworded, where a lawyer would quote them directly
One chambers has a clerk verify by reading, and sets a hearing when in doubt

These reports come mostly from district courts. Expect similar issues in the record on appeal.

PennLaw-Judiciary AI Testbed. Chambers reports, fall 2024 to October 2026; chambers de-identified. Some items come from a single chambers.

AI Detectors Have Improved, but a Score Is Not Proof

The best detector tested was near-perfect on unedited text
- Pangram, in a 2025 University of Chicago study of 1,992 passages
Results on edited text conflict
- That study found Pangram robust to a “humanizer”; a 2026 study found under 4% flagged
Earlier detectors flagged 61% of essays by non-native English writers (2023)
A panel of five practiced readers matched the best detector: 1 error in 300 articles
None of these studies tested legal filings

Treat a detector score as a reason to ask, never as a finding.

Jabarian & Imas, NBER Working Paper 34223 (2025); Liang et al., Patterns (2023); Russell et al., ACL (2025); Karr et al., arXiv:2608.11256 (2026).

Up Next for the AI Testbed: Hands-On Training

Sessions for chambers, using sample briefs
- Summarizing and extracting
- Drafting from sources
- Failure points and a verification routine
Interested chambers: talk to me
Findings are public at judges.pennai.law

✓What these AI tools can do ... and how they can fail
✓What is new: AI agents
✓What chambers are doing
→Where AI helps in appellate work
What law schools are doing

A Useful First Screen

AI in Appellate Work: Good Use Cases

Procedural history from the district court docket, with record cites
An issue map: what each brief argues and where the parties agree/disagree
Appendix pages that address a point, with cites you can click to open
Candidate questions for oral argument, drawn from the briefs
Proofreading and cite-format checks on a draft opinion

Bounded, sourced, and checkable. The judgment stays with you.

AI in Appellate Work: Don’t Do These

A decisional draft or bench memo written from model memory alone
Open-ended research: find me a case that says X
Questions about a long record that is not fully in view
Anything you will not be able to fully check

If no one will read the source, the tool should not have the task.

How to Learn: Use These Tools

Start where error is cheap and no protected data is at risk.

A professional-level account, ideally court-supplied
Public or synthetic material as test cases
A task whose output you can check carefully
Never a free commercial version of any AI tool

✓What these AI tools can do ... and how they can fail
✓What is new: AI agents
✓What chambers are doing
✓Where AI helps in appellate work
→What law schools are doing

Penn Began with First-Year Skills

2023

AI lessons and exercises entered first-year Legal Practice Skills

2025–26

1Ls, writing fellows, and TAs received secure ChatGPT EDU accounts

2026

In spring, 2Ls, 3Ls, and LLM students received Harvey access

Penn Carey Law, “Forging the Future,” Jan. 30, 2026. Dates, not quantities.

Penn Carey Law’s AI Tools, Fall 2026

Tool	Who has it	What it is for
Claude, with Claude Code	Every 1L; faculty and staff	General assistant and agent
PennChat, Copilot Chat	Everyone	Secure chat for University data
Harvey, Legora	2Ls, 3Ls, LLM students, faculty	Legal research and drafting
Westlaw AI, Lexis+ AI	All students and faculty	AI inside legal research

Penn Carey Law AI resources guide, updated Oct. 6, 2026. Access terms vary by tool and data sensitivity.

What Penn Teaches Alongside the Tools

1Ls: AI is optional; course rules come first
- The test for any use: who is doing the work?
Upper-level students: the legal platforms they will meet in practice
Faculty: shared tools for exams and class prep, and regular drop-in sessions
No detector score is treated as proof of misconduct

Access comes with instruction, not instead of it.

Penn Carey Law AI resources guide (updated Oct. 6, 2026) and 1L guide.

What This Means for Your Chambers

Some new clerks are arriving having learned alongside these tools.

Ask what they used, and how they checked it
Set chambers practice before your clerks’ habits do

The Penn Carey Law AI Project: Research, Build, Teach

The Judicial AI Field Guide: judges.pennai.law

Current AI tools offer real benefits for judges. But nothing about these tools changes legal judgment, or responsibility for it.

R. Polk Wagner

pwagner@law.upenn.edu

judges.pennai.law

AI-assisted · author-reviewed