The AWS Import Saga, or “Why does this computer shit have to be so hard?”

Category: [Beyond Nerddom, whining]

I’ve got a very large data set that we’re trying to move into AWS, and we did the math to realize that at the speeds we’re getting from our network connection it’ll take at least 4-5 months of transfer time to get the whole data set over to S3.

Amazon has a nice feature called Import/Export, that lets you take advantage of the old adage “never underestimate the bandwidth of a station wagon full of backup tapes”, but updated to modern times. Basically you ship them hard drives, and they suck the data off of them directly into S3.

So we’ve got a couple 3TB disks lying around, and figure every 3TB we transfer is about 4 weeks of data transfer time we save. Sounds great, right?

Well, not so fast. First thing to know is that those 3TB USB drives, unlike most other large drives, have a 4K sector size instead of the usual 512. That’s to make them compatible with older systems that don’t understand how to count past 4 billion sectors.

Of course, there’s an implicit “windows” there… using 4K sectors works with older versions of windows apparently, but not with older versions of Linux. Specifically RHEL5, which is where a bunch of the data lives.

Well, not a problem, just build a new parted and use the partprobe utility that ships with it… it’s a userspace tools problem on rhel5, not a kernel problem. At least for up-to-date rhel5… all bets are off for 5.5 and below.

But wait, the data’s considered sensitive, and we’d really like to encrypt it in transit. Fortunately import/export has the solution, and that solution is Truecrypt. Of course, Truecrypt has been dead since the end of May, but it’s still the only solution that Import/Export supports.

Sigh. Ok. Download truecrypt 7.1 for windows, start formatting up the disk ntfs (because my ntfs driver should work with Linux too). Realize that the process is going to take 2 solid days in un-interruptible mode. Try again by making an ntfs volume then telling to convert. 3 days, but can be paused.

I don’t even know if the ntfs3g driver will play nicely on Linux with Truecrypt, so … open a support case with AWS asking for guidance. They say that even though it’s not listed, truecrypt+ext4 is supported. Ok. Native Linux it is. Plug the USB drive into my RHEL6 box and let’s go.

Install Truecrypt 7.1 on my RHEL6 workstation? Check. Point it at /dev/sdb1? Check. Tell it to go? DENIED. Apparently on kernels less than 2.6.33 and devices over 2TB, you have to disable the “kernel cryptographic services” integration. RHEL6, of course, runs 2.6.32…

OK. Go through the truecrypt setup options, disable kernel cryptographic services integration. Yes, I know it might slow me down, but I’m on a USB2 port anyway, how much slower are we talking?

I never got to find out, because apparently for 4K sector sizes, you must use kernel cryptographic services. But for 2TB drives, you can’t use rhel6 kernel cryptographic services.

Ok. So I’m going to try something else. Install Fedora 20 on a kvm guest, pass through the whole USB device to that guest, and run truecrypt there. Fetch the iso, install qemu-kvm, libvirtd, virt-manager, get the iso to local machine, set up, install fedora …. le sigh… it’s the gnome3 spin. Intolerable, and unusable in my virt-manager-over-vnc-over-vpn environment. OK. Install updates, mate and lxde.

584 updates. And now we play the waiting game …

I feel like a hornet trapped inside a window. I’m strong, I’m formidable, I’m a bit angry, and I just can’t quite seem to figure out that the thing I’m bouncing off is a pane of glass and that I’ll just need to find a different way out. Maybe the fact that I keep bouncing, keep trying new approaches is that whole tenacity thing. Or maybe I’m just a dumb bug for not seeing that a foot to the left the window is open and I’m just completely missing it.

Ok, so 584 updates and 200ish other package installs later (or, an hour and change), reboot into a desktop that has buttons I can actually click in a guest. Run truecrypt installer. Attach USB host device of the disk I’m trying to work with. Truecrypt, format the partition, quick format (because I did a zero-pass earlier and don’t care about disclosing how much data is on the disk). Format ext4. Password. Check. Formatting….. hanging…. hanging ….. success!

Mount the volume. now let’s see how bad it is. DD out a 4 gig file from /dev/zero onto the mounted volume, wrap in ‘time’, pump through ‘pv’ for a status. And …. pv isn’t showing up. Hanging. Hanging. Hanging. Waiting. Oh, output! 480 megs? Ok. Hanging….. another 400 megs? Ok…. hanging…. another 300?

Yeah, so after those hoops, the “run it in a vm with usb passthrough” approach works. But gods is it awful….

OK. So when I just dump data onto a LUKS-encrypted device natively on my linux workstation, I get about 30MB/sec. Which is close to line speed of usb2, if less than half the expected write speed of the disk (it’s usb3 and if I had a port that fast this disk would be glad to suck down data at 60+). Adding KVM overhead for usb passthrough, truecrypt, and whatever else fedora might be bringing to the mix, I’m seeing … between 1 and 2 MB/sec. At this rate, it’d be as fast to just pump the data straight to S3 over the network.

Why’s this shit gotta be so hard?

Oppenheimer comment on science and the scientist

Category: [unelaborated]

Via https://www.youtube.com/watch?v=NuMdjjWgHVM , in turn via reddit /r/physics.

(edit: youtube link rot has set in, see transcript here)

“It isn’t the layman that’s ignorant. It’s everybody that’s ignorant. The scientist may know a little patch of something, and if he’s a humane and intelligent and curious guy, he’ll know a few spots from other people’s work. He may even be able to read a book. But his condition is the condition of everyone, which is that almost everything that’s known to man he doesn’t know anything about at all, or knows it only in a very sketchy way. And that’s because it’s gotten a bit complicated.

The problem of a coherent civilization is the problem of living with ignorance and not being frustrated by it, so that you find occasionally a man knows two things, and that intersection may be of a great event in the history of ideas. Occasionally a man may think that something is relevant or exciting which no one before thought concerned him professionally; that may change the history of the world.

And these are the connections, these virtual connections, these casual and occasional connections, which make the only kind of coherence we have. That and affection. That and respect. That and, I suppose, a kind of humanity. ”

The ever-delayed 2012 in review

Category: [life and times]

It’s been a whole year since I last posted a blog update. It’s probably gone into semi-private mode now, what with people not reading it ever. I don’t blame you, dear non-reader… I never come here either :p.

Ok, so no self-flogging over not posting lots of my inner workings on a blog. Instead, I’ll just share what 2012 was like, looking back.

I pretty much focused on my work through all of 2012. I did almost nothing with my personal life.

On the financial front, I crossed a couple big milestones. Got a nice pay raise to start the year out, easily sprinted through the “paying off car” goal (and thus am completely debt-free) and really started working hard toward the “saving for a house” goal. End of the year I got a so-so raise, better than average for my center but maybe not quite in line with my output. But I’m equanimous: I’ve had a couple good years for that, going so-so for a year only really makes room to spread it around to some other folks that haven’t had it as good. I’ve made good progress — I’m up to about a 10% down payment on the type of houses I’m looking at. Not enough to warrant moving yet, but it’s great progress from the nothing+debt a year ago, and my living situation is still pretty acceptable. It’s nice to watch the options open up though…

Based on the pay raises and whatnot, my charity goals were a little ambitious. Despite giving more than ever before, it still landed right around the 2.1% mark.

On the health front, I managed to keep the brakes firmly engaged and didn’t roll further down the hill, but didn’t really get very far back up it either. Stable and slightly improved is better than sliding out of control though. The medical front is largely unchanged. I did manage to screw my back up standing on the mall back in March, and I’ve been nursing that back a bit, but no other news to report there.

I’ve largely neglected the relics of my past, in particular the old forum community. I’ve tried to make it clear that I don’t really want to spend any more time on that at all, but I don’t see anyone to hand the keys over to either. Without a clear technical successor, I don’t feel like I can just walk away from it, but having a lot of trouble finding motivation to do even basic stuff like software upgrades.

I partly moved away from wine in 2012. Had some more good bottles, but the lack of drinking buddies to split bottles with is burdensome — if I drink the whole bottle I feel awful the day after, if I don’t I feel like I’m wasting wine (and money!). In its place, I’ve moved toward whiskeys, trying out a couple good Irish and Scotch whiskeys and making friends with a few varieties of Bourbon. The nice thing about that is that a bottle of bourbon lasts for-frigging-ever, so I don’t have to push it to enjoy a dram now and then. I still have yet to break the “scotch old enough to order scotch” barrier, but I’ve had whiskey old enough to drive. So far I’m gravitating toward Knob Creek, Redbreast and Highland Park.

My job has been intense, though. I’ve continued to learn a lot, we hired a few new people (and lost a few). I’m getting pushed into the role “cloud advocate” and doing more interesting architecture-type stuff. Lots of work with our security folks, some work on a cloud research project that’s been pretty fascinating, and big core changes afoot in our server room.

On the reading front, probably the biggest chunk of reading material consumed was again Audible audiobooks. Highlights included two more Stephen King globs (The Stand and Salem’s Lot), Moby Dick, some newer Steven Pinker, and a documentary about James Garfield. As usual, spent some time wrapped up in ;login: and SciAm, and also got around to joining ACM and ripping through some of Communications of the ACM. Dense stuff. Ripped through some stuff on ITIL, Puppet, config management, project management.

I’ve been entertaining the thought of grad school, maybe chasing an MS in Information Assurance. Have to see whether I get started on that road in 2013…

I didn’t take the same sort of big flashy vacation I did in 2011, but instead went back to the midwest for a week or so to visit with various collections of family and friends. It’s been a long time, and lots of stuff has changed, but lots hasn’t. It was a whirlwind trip, and I ended up catching up with less than half the people I wanted to, but totally worth it. Also gave me a reason to be in the car listening to audiobooks for like 28 hours, which is a bonus — driving home from southern Indiana to about half of Moby Dick was a pretty amazing experience. Especially the part that involved driving through Hoosier National Forest in the late morning hours with virtually no traffic.

Goals for 2013: stay the course, roll over the old goals.

2011 wrapup, 2012 goals (terse version)

Category: [life and times]

It’s sad that my once-a-year blog post has degenerated to what could show up as 5 tweets. I’m throwing more of my day-to-day brain-sauce at twitter lately. But mirroring last year’s one blog post, here’s my annual post for 2012.

Charity: 2011: 2.1%.  2012: 2.3% is a small but reasonable step up.
Weight:2011: started 370, ended ~380. Not the right direction.  2012: Keeping last year’s 340 goal, which would be on average a bit less than a pound a week. Possible.
Exercise:2011: nothing significant until December’s treadmill acquisition.  2012: goal of 5 hours a week of some kind of decent exercise. Treadmill is definitely helping toward that goal.
Medical: 2011: Dentist, eye doctor taken care of. Still no doctor.  2012: find a doctor, get rid of some cysts.
Romance: 2011: nibbles, no hook-setting.  2012: not really any goals still.
Job: 2011: crazy busy, but good work, good pay raises. 2012: more of the same.
Vacation: 2011: STS-135, Bitches! Also other Florida attractions. 2012: thinking about a few days in NYC or a road trip back home. We’ll see.
Finance: 2011: was a “grind” year, no real milestones. 2012: will pay off car completely, should be about 25-30% of the way to a down payment on a house.
Forums: 2011: upgraded to vb4, no cms. 2012: maybe do the cms thing, or find something else to put on the front page.
Reading : 2011: few issues of sciam/login, lots of audiobooks. 2012: more sciam/login, some ITIL stuff on deck, lots more audiobooks.
Property: 2011: monitor upgrade, treadmill. 2012: nothing planned.
Wine: 2011: good for synthetic corks, couple good reds, couple good whites. 2012: more red, more white, more exploring, more unnatural corks!

2010 wrapup, 2011 goals

Category: [life and times]

Time for a terse 2010 update, roughly two weeks later than just about anyone else would do it :p.

Charity: 2010: 2% target, met. 2011 target: 2.5%.
Size: 2010 peak: 372. 2011 goal: stabilize at 340.
Exercise: 2010 goals: abject failure. 2011 goals: just make a dent.
Medical 2010: abject failure. 2011 goal: not that.
Romance 2010: First dates = 2. Second dates = 0. Second dates declined by me: 2. 2011: no goals, I’m not digging myself enough to dig the ladies who dig me.
Job 2010: great. 2011: dark scary things coming, gonna do my best to keep it good though.
Education 2010: nil. 2011: take something! Anything!
Work Travel 2010: Minneapolis in February (solo, which was new). Louisville in the springtime. 2011: no plans.
Vacation travel 2010: Gettysburg, PA. Luray VA, Shenandoah National Park/Skyline Drive. 2011: take a week for myself, go somewhere!
Finance 2010: Student Loans gone (!). Car loan down to ~12k. Positive net worth (w/ retirement included). 2011: car loan sub-4k, positive net worth (w/o retirement/investments), big start on savings for a house.
Forums 2010: bought vb license, upgraded 3.5->3.8. 2011: upgrade to 4.x w/integrated cms for front page. Stretch: update tracker interfaces.
Property 2010: major purchases: PC upgrade, office furniture. 2011 goal: living room furniture that doesn’t suck.
Wines in 2010: lots of reds. Ravenswood Shiraz 2007 was the go-to. Who knows what 2011 will hold?