# robots.txt 
User-Agent: *
Disallow: /css/
Disallow: /html/
Disallow: /img/
Disallow: /js/
Disallow: /media/
Disallow: /txt/