Host sitemaps on S3?

[tanlup]

Hey guys,

I run a dynamic web service and I will start building static sitemaps for it pretty soon. The fact that my app lives in a multitude of servers doesn't make it easy to distribute frequently updated static files throughout the servers.

My idea was to host the files in AWS S3 and point my robots.txt sitemap directive there. I'll use a sitemap index so, every other sitemap will be hosted on S3 as well.

I could dynamically mirror the content from the files in S3 through my app, but that would be a little more resource intensive than just serving the static files from a common place.

Any ideas? Thanks!

[CoreyNorthcutt]

My general take on this sort of scenario is first to eliminate all the redundant hostnames with round-robin DNS, through adding extra server power with software-based load-balancing in the interim with a solution like InterWorx, and breaking out database servers.   If you do that, you should have a nice little server cluster that's crazy efficient.and scalable.  You can add a CDN to the mix if you like as well.  With all of that, SEO should work the same way as on a single server.

Sitemaps can then be generated dynamically really easily (in under 25 lines of code, most of the time).

If you just want a way to mirror static files, you'll want to look at rsync.

And finally, as for S3, my personal opinion is to stay away.  I'm an SEO, but I also spent 7 years building a hosting company.  Those solutions sound great in their marketing, but are scientifically less reliable than standard hosting, and you can verify that via public uptime tracking sites like HyperSpin.