Are Robots.txt Instructions Legally Binding?–Ziff Davis v. OpenAI

Robots.txt files express a website’s preferences for robot access. Despite their venerability, there are not many cases discussing the legal implications of robots.txt files and robot exclusion headers.[FN] As a result, we still don’t know if they are legally effective instructions. Are robots obligated to check and honor robots.txt? Or are websites expected to do more than put up a “no-trespassing” sign if they really don’t want robots? The legal answer today remains: 🤷‍♂️

[FN: Last night, I searched Westlaw ALLCASES and came up with 17 hits for “robots.txt” or “robot exclusion,” including some well-known cases like LinkedIn v. hiQ, Field v. Google, and Bidder’s Edge v. eBay, plus some less-heralded cases like Tamburo v. Dwoskin and AP v. Meltwater.]

* * *

This is a lawsuit over OpenAI’s automated scraping to build its ChatGPT model. Ziff Davis deployed a robots.txt file that “allegedly instructed GPTBot not to scrape Ziff Davis’s websites,” but the GPTBot allegedly ignored the instruction. Ziff Davis argued that OpenAI ignoring the robots.txt instruction violated 17 USC 1201(a) (the anti-circumvention restrictions) because OpenAI’s scraper circumvents “a technological measure that effectively controls access to a work.” The court disagrees and dismisses the claim.

The court explains:

Robots.txt files instructing web crawlers to refrain from scraping certain content do not “effectively control” access to that content any more than a sign requesting that visitors “keep off the grass” effectively controls access to a lawn. On Ziff Davis’s own telling, robots.txt directives are merely requests and do not effectively control access to copyrighted works. A web crawler need not “appl[y] . . . information, or a process or a treatment,” in order to gain access to web content on pages that include robots.txt directives; it may access the content without taking any affirmative step other than impertinently disregarding the request embodied in the robots.txt files. The FAC therefore fails to allege that robots.txt files are a “technological measure that effectively controls access” to Ziff Davis’s copyrighted works, and the DMCA section 1201(a) claim fails for this reason.

The court adds that: “At most, Ziff Davis alleges that OpenAI disregarded the instructions that were contained in robots.txt files. This is not ‘circumvention’ under the DMCA.”

* * *

This opinion doesn’t answer the most important questions about robots.txt because it only addresses 1201. Using 1201(a) as an anti-scraping law is not intuitive. Indeed, it feels overreaching to me, such that I doubt the plaintiffs expected the claim to succeed in court.

This opinion turns on the specific statutory language of 1201, which limits its applicability to other doctrines. That’s unfortunate, because we could use more clarity about if and when robots.txt can effectively limit access to websites.

That issue is central to trespass to chattels claims, including CFAA claims. For example, the Supreme Court indicated in Van Buren that website access could be delimited by whether technological “gates” are up or down (the opinion didn’t clarify if the delimiting occurs when gates are up or gates are down…details…). This 1201 opinion casts a little doubt on robots.txt acting as a technological “gate” for TTC purposes. After all, the court expressly says that robots.txt don’t control access to sites and says they are as technologically effective at preventing access as a “keep off the grass” sign. If I were a scraper, I would cite the decision in favor of treating robots.txt instructions as legally irrelevant.

Case Citation: Ziff Davis v. OpenAI, Inc., 2025 WL 3635559 (S.D.N.Y. Dec. 15, 2025)