Commit History

Author SHA1 Message Date
  Zed 6c4c691fe7 Add Crawl-delay to robots.txt 3 years ago
  Zed 138826fb4f Fix Twitterbot rule in robots.txt 4 years ago
  Zed dfb2519870 Explicitly allow Twitterbot to generate previews 4 years ago
  minus c543a1df8c Block search engines via robots.txt (#631) 4 years ago