# Robots.txt file # All robots will spider the domain User-agent: * Disallow: # Disallow archive.org (Wayback Machine) User-agent: ia_archiver Disallow: /