OpenAI has announced that it will honor a shared signal in robots.txt, the file websites use to communicate preferences about how their content may be accessed and used. The signal is intended to give publishers and website operators a simpler way to express their position on the use of their material for foundational Generative AI training across multiple participating providers.
Under the current approach, operators may need to identify individual crawlers and maintain separate rules for each provider. A common signal could reduce that administrative work by allowing the same preference to be communicated more broadly.
OpenAI said it plans to implement the signal in the coming months and will provide further details when that work is complete. The new option will not replace the crawler controls already available through robots.txt. According to the company, those existing controls will continue to apply and be respected.
The announcement builds on work by the Internet Engineering Task Force’s AI Preferences working group. OpenAI said it is joining other participants in announcing the shared signal and will continue adapting its practices as discussions about standards, controls and fairness develop. The company also encouraged other AI providers to honor the signal.
Sources: OpenAI News.