Banning Bots
One neat feature of a site built entirely on Simpledb is that it’s easy to keep track of the per page load database cost in dollars.
For most of my pages, that cost is fairly low, but still on the order of a penny per thousand hits.
Thinking back to swik.net, we were hit by bots (mostly the googlebot), over a million times a day. Even at 1 cent per thousand hits, that would add up to thousands of dollars over the course of a year.
Which might be fine for googlebot since that drives a lot of traffic, but what about the other bots? Should I let them use up my traffic without sending their share of visitors my way?
Idealistically I’d say everyone should have the chance to be the next Google, but if its on my dime I don’t really see the advantage to not banning all bots except for google and yahoo.
