User-agent: *
Disallow: /twitter/
Disallow: /temp/
Disallow: /awstats/
Disallow: /files/