How Accurate and How Safe Is Astra for Law?

When OpenAI launched Astra for Law on 17 September 2026, I wanted to know two things: how accurate its legal answers are, and what happens to client data. The basics are already covered in Yoni Marcus's article Claude, Gemini or Astra: Who's Actually Winning Legal AI?, so this article focuses on those two questions.1

It covers four points:

1. What Astra for Law is
2. An example OpenAI published to show its accuracy
3. How its base model performed on an independent benchmark
4. How it handles client data

Astra for Law is built on GPT-6 Astra, OpenAI's general-purpose model, so it is not a new model. OpenAI connected it to a search index of US case law, statutes, regulations, court rules and administrative decisions, with new sources added daily, and gave it instructions for legal analysis and writing (OpenAI).1 A general LLM relies on what it learned in training or on a web search, while Astra for Law can look up the specific US rule or case that applies.

OpenAI gave the same legal question to two models, Claude Fable 5.1 (Anthropic's model) and Astra for Law, and published both answers side by side.1

The scenario, in simple terms:

• A supplier agreed to make a retailer's products. Before signing, the retailer said comparable suppliers had received about $8 million a year in orders.
• The written 5-year contract guaranteed only $2 million a year, and any extra orders were up to the retailer.
• Before signing, the supplier knew the retailer was changing how it split orders among its suppliers.
• Orders barely went above $2 million, and internal records suggested the real historical average was much lower than $8 million.

OpenAI asked both models to find "good law" (valid law that courts can still rely on) with similar facts, and to analyse the supplier's claim.

Claude Fable 5.1 relied on Paragon Metals Holdings v. Smith,7 a fraud case about the sale of a company. It cited a 2025 decision from a lower Delaware court. In July 2026, the Delaware Supreme Court reversed the key part of that decision, so the argument rested on a holding courts can no longer rely on.

Astra for Law found Oliver Wyman, Inc. v. Eielson,8 a 2017 New York federal case. There, sellers were told partners had earned about $1 million a year on average and accepted a lower guaranteed amount, and the court still allowed their fraud claim to go forward. Astra for Law concluded: "Our client has a viable claim based on false history, not a right to $8 million in annual orders."1

OpenAI chose this example itself. It shows what the tool can do, not how often it does it.

Vals AI,3 an independent company, tests AI models on legal research questions in eight areas of US law. Astra for Law itself has not been tested by Vals yet, but its base model, GPT-6 Astra, has.

It answered 39.42% of all questions fully correctly and ranked 28th of 75 models when checked on 9 October 2026. The best models reached 55.29%.3

Source: Vals AI Legal Research Bench, leaderboard updated 7 October 2026, checked 9 October 2026. Scores rounded.

OpenAI also ran its own test on 200 questions from a separate, private set of Vals questions, with both models at their highest reasoning setting:1

Because the questions differ, these numbers cannot be compared directly with the Vals scores above, and they have not been independently confirmed.

Astra for Law is initially offered to selected law firms in the United States through Trusted Access in ChatGPT and Codex, with API access coming soon (OpenAI Help Center).12 OpenAI has put these measures in place:

• No training by default. OpenAI does not use data from its business products, such as ChatGPT Enterprise, to train its models (OpenAI).4
• No routine human review. OpenAI says its staff access ChatGPT Enterprise chats only to handle incidents, to recover a user's chats with their permission, or when the law requires it.4
• Zero Data Retention (ZDR). Through its API, OpenAI offers ZDR, which means it does not keep prompts or answers.5 Astra for Law is not available through the API yet, so lawyers use it in ChatGPT or Codex, where the firm's own data settings apply.
• Ethical walls. OpenAI is working with Latham & Watkins to design ethical walls (information barriers between conflicting matters inside one firm) and other controls for firms.1

Two points are not covered by these measures:

• Metadata. Think of a book: even if no one reads the text, the cover still shows the title and the author. In the same way, OpenAI's documentation5 says it keeps some information about how its business products are used, such as account and usage data, separately from the content of chats.
• Court orders. In May 2025, in New York Times v. OpenAI, a court ordered OpenAI to keep ChatGPT and API chats that would normally be deleted.69 ChatGPT Enterprise, ChatGPT Edu and Zero Data Retention customers were not affected. The order to keep new chats ended in September 2025, but OpenAI still keeps a limited set of chats saved between April and September 2025 for the case.6

Astra for Law adds a US legal search index to a general model. On an independent benchmark, its base model answered about two in five legal research questions fully correctly. The stronger results so far, the example and the 54.0% score, both come from OpenAI itself and have not been independently confirmed. On client data, OpenAI offers several protections, but they do not cover metadata or court orders. Once Vals tests Astra for Law itself, we can compare it fairly with GPT-6 Astra and see how much the legal search index really helps.

A longer version of this article, with more detail on how Astra for Law works and practical points for law firms, is available on my LinkedIn.