Server migrations (or, “How I spent the last month of my life”)
So, the last couple of weeks I’ve been pretty preoccupied by this whole server migration process, but it’s about wrapped up and I’m feeling pretty good about it overall, so I’ll go ahead and tell the tale now. But it might be boring and technical, I dunno :p.
About … the middle of April, the PHSI folks split off from SEAS lab, and by the end of June SEAS lab had to pretty much be out of the lab itself for the duration of the summer. Because I’m leaving (since it became obvious that a full-time offer wasn’t forthcoming), we wanted to hand over the server administration for PHSI’s servers to other people, so they’d be supported. After fishing for quotes and service overviews from ECN and ITaP, we decided to go with ITaP for the biosecurity e-course server administration, while ECN’s taking over desktop support.
Well, that’s all fine and dandy, right? So at the meetings to hammer out the technical details of how things would go down and how the servers were set up, I offered to change the OS from what it was to something more standard before handing them over. I figured, “well, I know my way around Gentoo, am intimately familiar with my setups, and am comfortable enough with RHEL that I could have these things up and running over the course of a weekend or so, where they don’t really know how things are rigged” … hence offering to do the job. But, they politely declined, volunteering to do so themselves instead. “Fine, fine” I thought … “just more free time for me.”
So, we handed the servers over around the second week in May, dropped them into the machine room, and that’s the last time I’ll ever physically see those machines. But not the end of the story.
So, a couple of weeks go by, a couple scripts break, they do a couple small mostly-inconsequential changes (adding a pre-logon warning banner), and I have them switch from the servers emailing me when stuff broke to emailing them when stuff broke. All was well, and I thought that’d be the end of it.
Then, around the last week in May, I get an email, saying basically “want some hours this summer? We’d like some help migrating this stuff.” So I said “sure, I’m not doing much else” (because I wasn’t). Over the course of the first 2 weeks of June, one of the ITaP sysadmins and I hammered out a migration strategy, and started rolling with it. Finally, the last week in June, we copied one of the webservers off to a VM image, had him install RHEL on it and hand it over to me to do the rest on that system.
So, over the first week in July, I installed, configured, tweaked and tested the site all migrated to RHEL, dropped it back in the rotation, and waited. And we knocked out the other two servers the second week of this month, with a bit less fumbling on all our parts. There’s still some outstanding issues with the setup (a charset mismatch that wasn’t manifesting itself, and the lack of pdflib and php-pecl-pdflib packages for rhel5), but other than that things are great.
I think between email logistics and working out details and data transfers and various other shufflings, this whole migration probably took like … 50-60 hours of work. Maybe a bit more. But doing a complete system reinstall and redeploy on an entirely different platform with negligible downtime (miscellaneous snafus aside) … that’s pretty good!
This project felt pretty good. There were a lot of little threads to pull together. We stayed on the same PHP version, but the move from Apache 2.0 to 2.2 had a couple threads to pull (mainly the renaming of mod_access), the php.ini config was a bit different, apache setup had to pretty much entirely change, maintenance and deploy scripts changed substantially, new ipsec setups (rhel ipsec config is WAYYY more straightforward than gentoo), cron jobs, user accounts, auth settings, firewall rules, mail forward configs. This project felt like sort of a combination confidence-booster and capstone piece in my Purdue sysadminning life. I’m ready to move on and conquer bigger and better things now.
Well, after I tell the folks who’re actually taking over the administration how everything actually works, that is 
.