This is our Amazon EC2 admin console.
Marc coded this up for our servers. It’s pretty cool!
If you run your servers on EC2, all they give you is some command line tools. There are a bevy of them, but it can still be kind of a pain to manage servers en masse. This tool is a lot more convenient, and it makes rolling out new code a snap. We go from staging to production with 1 click.
The admin setup does a lot more behind the scenes. On a new instance bootup, instances must report to the DNS for round-robin-ing, find all other instances and add them to the memcache server pool, download and setup the server software, and fetch the latest production scripts from SVN.
We can also manage which servers are in the DNS pool.
Lately we’ve had a bug where Apache wigs out for some unknown reason we are debugging: server daemons still are listed in processes, but they are either unresponsive and use 0.0 CPU or use up 100% CPU and are still unresponsive, even with 0 hits to the server. I used the DNS tool to cut out one of these unresponsive servers from production, and then check on what was going wrong. No luck though, sometimes these things are tricky to trace :D
The last nice thing about our ec2 admin is that it reacts to instance load: if an instance comes under heavy load, it sends out an email, and then sends out an email later if the instance comes down to normal load. It can also respond automatically to 0.0 load, which can be another indicator of trouble.
Even though this setup works really well, I’m still waiting to see what amazon rolls out as their official admin console for EC2, hopefully it will do some similar things to ours.
