Meta Platforms stated it will clarify how its web crawlers operate and affirm that content owners can use robots.txt files to set crawling preferences