TWC:Sysadmin info

I was bored and decided to put together this page of handy info for our active sysadmins, currently GED and me. I (Simetrical) started this page on September 20, 2008, and I cynically speculate that it will get updated approximately never after that, so I've tried to keep things relatively nonspecific as to exact versions and stuff, and to provide verification procedures where possible. But take it all with a grain of salt.

If you don't have at least shell access to the TWC server, this page is almost certainly useless to you. If you at least know what shell access is, you're probably capable of understanding at least some of it. Maybe you'll find it interesting, good for you.

Hardware

We currently have one dedicated server. We own none of the hardware, we rent it from The Planet on a monthly basis. It's their "Xeon 3060 - RAID" model with some adjustments. 8 GB RAM is the only difference from stock, I think. Since we didn't buy the hardware, the exact models and stuff are mostly from orbit.theplanet.com, the magical control panel they give you, but I don't totally trust it. Some of it is of course obvious from running the right commands (cat /proc/cpuinfo, free -m, . . .).

• "Dell Single socket 1067FSB - Quad Core Capable PowerEdge 840" motherboard
• One dual-core Intel Xeon Conroe 3060 at 2.40 GHz
• 8 GB RAM, DDR2 667 ECC
• Four 250 GB, 7200 RPM SATA drives (Western Digital WD2500JD) in RAID 10, for 500 GB usable disk space (they say it's SATA II, but peak throughput from hdparm -t is just a hair under 1.5 Gbps, so I wonder . . . but it doesn't matter, anyway, since our disk throughput is orders of magnitude less)
• "3ware 4 channel SATA2 w/ battery backup 9550SX-4LP" RAID controller
• 2500 GB monthly bandwidth
• Some indeterminate-speed network uplink, probably 100 Mbps? It might matter if we get another server.

Software

Linux

The "L" in "LAMP". We use Linux because, as Simetrical will tell you, it is both technically and morally superior to Windows in every conceivable way. (Some people who currently have root access might hold different opinions.) In fact, the old server (loki) ran Linux, and it was all Simetrical knew how to administer when he was picking out the new server (odin), so it was a fairly pragmatic choice even if he wasn't a penguin-hugging open-source hippie.

The exact version is Red Hat Enterprise Linux 5.0. The exacter version, from /etc/redhat-release, is "Red Hat Enterprise Linux Server release 5.1 (Tikanga)". uname -a gives "Linux odin.twcenter.net 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux".

Red Hat sucks and I wish we had used Ubuntu. The packages are paleolithic here. 2.6.18 was released in September 2006. I want large argument lists, dammit! But The Planet doesn't offer Ubuntu as an option, and I had enough trouble repartitioning over SSH without having to try installing a new OS too . . .

lighttpd

The "A" in "LAMP". Yes, this is fairly weird, alphabetically speaking, but nobody wants to try to pronounce "LLMP". We used to use Apache, but it used way too much memory. We switched in March 2008, saved 1.5G of RAM, got perceptibly faster page load times, and never looked back. Fiddling around with random stuff for fun can pay off sometimes.

lighttpd's config file is at /etc/lighttpd/lighttpd.conf (history). Documentation is at the lighttpd website (config file docs are the most useful). The binary is at /usr/sbin/lighttpd.conf. lighttpd runs as user lighttpd. Access logs are at /var/log/lighttpd/access.log*, automagically rotated by logrotate. The error log is at /var/log/lighttpd-error, apparently because when I set it up I was feeling pig-headed about silly things like file naming conventions. A handy web-based server status thing is available in the secret stuff thread in the Tech Cathedral, for those with access there (it gives IP addresses of all current connections, so not fit for public consumption).

To restart lighttpd in our current setup, for instance due to new lighttpd.conf or php.ini, I use this:

killall php-cgi && service lighttpd restart 2>/dev/null

This 1) kills the FastCGI processes (probably not necessary if you don't need to restart FastCGI too), 2) restarts lighttpd using the system restart scripts, and 3) redirects stderr for (2) to /dev/null because FastCGI seems to spam your terminal with PHP warnings forever after if you don't.

Restarting lighttpd usually takes a few seconds, so no perceptible downtime for most users, but it will shut down any active connections, so anyone doing a big upload/download will get an error or bogus file or something. So don't get too trigger-happy. If you're feeling really kind, you could check the server status and see if anyone's downloading anything large, but personally I don't bother. Restarts of lighttpd are fairly rare anyway.

lighttpd usually uses a few percent CPU and a few hundred megs of memory, in my experience. It runs on a single thread and doesn't run any scripts itself, so this is pretty reasonable. I wonder what the few hundred megs are for, actually, but thinking back to Apache with mod_php I'm not even going to bother looking into it.