Project Panama

Scanning and Book Destruction at Anthropic

The image shows rows of categorised books stored in a massive facility. This photo was published as part of a prominent investigative report uncovering “Project Panama,” an internal initiative by artificial intelligence start-up Anthropic. Photo origin: unsealed court filings

Book burning. Book pulping. Book vandalising. It’s all the fashion and, dare one say it, the rage. Libraries are carting them off to the dump. Repositories of memory are being shredded in favour of supposedly more useful digital formats. And now, Anthropic’s hungering for books, not as sources of enlightened knowledge so much as blue raw data for their Language Learning Models, is there for all to see. Acquire the books in question. Give service providers the task of severing their spines. Employ scanners to process the information. Dispatch the paper to be pulped and recycled (awfully good of them). The result: a private research library able to nourish Claude, the company’s premier LLM.

Last month, an investigation by 404 Media took note of the purchasing habits of AI companies as regards old books—whatever that means—identifying ISBNdb as an instrumental broker in the field. The company, according to its own description, “gathers data from various public sources like libraries and merchants to compile a vast collection of unique book data searchable by ISBN, title, author or publisher.” At present, it boasts 111,978,817 searchable books and offers clients somewhere between 1,000 to 1 million books per engagement. Older print publications are advertised as the purer sort, uncontaminated by presence of generative-AI. “Print books from the pre-LLM era are structurally guaranteed to be free of this contamination,” states an article published by the company. It remains unclear whether ISBNdb’s client list is crowded by those of the destructive scanning persuasion, though the company is not oblivious to the problem.

The same goes for other book sellers as to whether their activities are feeding the AI maw of physical pulverisation. The Canadian company Zoom Books specialises in acquiring nonfiction and academic titles from the 1970s in bulk, storing titles in European warehouses (Germany, in particular), before shipping them off to the US and Canada for destructive scanning. The practice struck Thomas Koch, press spokesperson for the German Publishers and Booksellers Association, as particularly reprehensible, though he admitted a note of caution about the extent this was taking place : “It appears to be yet another example of AI companies using vast quantities of copyright-protected works to train their language models, without consent and without payment.”

Anthropic’s activities came to light in filed documents in the northern California District Court case of Bartz v Anthropic PBC, involving the authors Andrea Bartz, Charles Graebner and Kirk Wallace Johnson. All alleged copyright infringements against the company. Known as “Project Panama,” this extract-and-pulp mission was led by Tom Turvey, a previous employee of Google who had made strides in creating such partnerships as Google Books. As the introduction to the court order notes, the project was intended to create “a central library of “all the books in the world” to retain “forever.”” The library would enable the AI firm to select “various sets and subsets of digitized books to train various large language models under development to power its AI services.”

The company’s appetite for book purchases proved voracious. Instead of brokering agreements with publishers to license copies for purposes of training AI (this approach, derided as “legal/practice/business slog,” was initially explored, if only tentatively), “Turvey and his team emailed major book distributors and retailers about bulk purchasing their print copies for the AI firm’s “research library”.” Millions of dollars were expended on printed books, even those in used condition. Retained service providers then went about their work: stripping the book bindings, cutting the pages to size, scanning the books into digital form. Paper originals were discarded. “Each print copy book resulted in a PDF copy containing images of the scanned pages with machine-readable text (including front and back cover scans for softcover books).” The company also made a point of harnessing pirated digital books, which eventually led the company to reach a $1.5 billion out-of-court settlement. These included five million copies from LibGen, two million items from Pirate Library Mirror and approximately 183,000 from Books3.

The court describes how Anthropic “came to value most highly for its data mixes books like the ones” written by the plaintiff authors “because of the creative expressions they contained.” Those using the Claude LLM wanted it “to write as accurately and compellingly” as the authors, using “well-curated facts, well-organized analyses, and captivating fictional narratives.”

On 13 April 2024, an internal memorandum was circulated in the company which let the mad cat of destruction out of the bag. “Project Panama is our effort to destructively scan all of the books in the world,” is the lavishly ambitious claim heading the message by Turvey. The memorandum in question urges discretion. “Why use a codename? We use “soft codename” because we don’t want to be known that we are working on this. This document is available to all Anthropic employees, but you should avoid talking about it in public areas and the fact that we are working on this should not be shared with anyone outside Anthropic.”

In an email to Snopes, a spokesperson for Anthropic issued something of a qualification, claiming that, “None of our data acquisition programs buy and destroy “rare” or “antiquarian” books.” The company has made a distinction between “rare and valuable,” a formulation most shoddy.

In all this excitement of vandalism, it was easy to forget that the plaintiffs had failed to convince Judge William Alsup that copyright violations had taken place. “Anthropic’s LLMs have not reproduced to the public a given work’s creative elements, nor even one author’s identifiable expressive style.” Claude “outputted grammar, composition, and style that the underlying LLM distilled from thousands of works.” But copyright did not cover ““method[s] of operation, concept[s], [or] principle[s]” “illustrated” [ ] or embodied in [a] work.” The copyrighted works, in being used to train LLMs in generating “new text was quintessentially transformative.”

On the issue of the assembled “research library,” the court found no issue with Anthropic’s conversion of each book copy’s format from print form to digital, effectively turning a blind eye to the eventual destruction of their physical existence. “Anthropic purchased its print copies fair and square.” But the pirated copies were something else. “The person who copies the textbook from a pirate site has infringed already, full stop.” Anthropic’s claim that using copies for a central library was a case of fair use was untenable.

In all, a melancholic day for books, an undeserved if partial victory for Anthropic, and the offering of a bleak, wearying prospect that the creative faculties of the writer risk being turned into the blind and dumb receptiveness of the slothful consumer. The means of language production, as Kathryn James, rare book librarian of Yale University’s Lillian Goldman Law Library at Yale University appositely remarks, has been ceded.