Check captures, robots.txt history, block fingerprints and a live CCBot probe for any domain, free, straight from Common Crawl's own archives.
This is a curated summary. The full story was reported by Search Engine Journal.
Read the full story at Search Engine Journal


