The robots.txt AI crawling setting proposed by Kevin Kelley of Anthropic will not apply to artificial intelligence models that perform exclusively non-generative tasks