I for one wish we had just stuck with RCS or SCCS :).

Paul Lindner One sysadmin to another: how to you roll out changes in an environment the size you have? Or can you not say?

Google is custom stuff to the hilt, so can't say much there. In past jobs machine management involved

- set master PXE server with bootable linux installer du-jour, DHCP, ipmi or serial console

- install cfengine, puppet or the latest hot product being talked about at LISA to manage configs.

- put custom packages onto central yum/apt host.

- push stuff out using custom software/scripts or something like zookeeper. I think there are some commercial products in this space too.

Gets a lot more complicated with multiple DCs and machines with disk state (unless you use something that supports that, like voldemort, or other nosql solutions)