The AWS Import Saga, or “Why does this computer shit have to be so hard?”

I’ve got a very large data set that we’re trying to move into AWS, and we did the math to realize that at the speeds we’re getting from our network connection it’ll take at least 4-5 months of transfer time to get the whole data set over to S3.

Amazon has a nice feature called Import/Export, that lets you take advantage of the old adage “never underestimate the bandwidth of a station wagon full of backup tapes”, but updated to modern times. Basically you ship them hard drives, and they suck the data off of them directly into S3.

So we’ve got a couple 3TB disks lying around, and figure every 3TB we transfer is about 4 weeks of data transfer time we save. Sounds great, right?

Well, not so fast. First thing to know is that those 3TB USB drives, unlike most other large drives, have a 4K sector size instead of the usual 512. That’s to make them compatible with older systems that don’t understand how to count past 4 billion sectors.

Of course, there’s an implicit “windows” there… using 4K sectors works with older versions of windows apparently, but not with older versions of Linux. Specifically RHEL5, which is where a bunch of the data lives.

Well, not a problem, just build a new parted and use the partprobe utility that ships with it… it’s a userspace tools problem on rhel5, not a kernel problem. At least for up-to-date rhel5… all bets are off for 5.5 and below.

But wait, the data’s considered sensitive, and we’d really like to encrypt it in transit. Fortunately import/export has the solution, and that solution is Truecrypt. Of course, Truecrypt has been dead since the end of May, but it’s still the only solution that Import/Export supports.

Sigh. Ok. Download truecrypt 7.1 for windows, start formatting up the disk ntfs (because my ntfs driver should work with Linux too). Realize that the process is going to take 2 solid days in un-interruptible mode. Try again by making an ntfs volume then telling to convert. 3 days, but can be paused.

I don’t even know if the ntfs3g driver will play nicely on Linux with Truecrypt, so … open a support case with AWS asking for guidance. They say that even though it’s not listed, truecrypt+ext4 is supported. Ok. Native Linux it is. Plug the USB drive into my RHEL6 box and let’s go.

Install Truecrypt 7.1 on my RHEL6 workstation? Check. Point it at /dev/sdb1? Check. Tell it to go? DENIED. Apparently on kernels less than 2.6.33 and devices over 2TB, you have to disable the “kernel cryptographic services” integration. RHEL6, of course, runs 2.6.32…

OK. Go through the truecrypt setup options, disable kernel cryptographic services integration. Yes, I know it might slow me down, but I’m on a USB2 port anyway, how much slower are we talking?

I never got to find out, because apparently for 4K sector sizes, you must use kernel cryptographic services. But for 2TB drives, you can’t use rhel6 kernel cryptographic services.

Ok. So I’m going to try something else. Install Fedora 20 on a kvm guest, pass through the whole USB device to that guest, and run truecrypt there. Fetch the iso, install qemu-kvm, libvirtd, virt-manager, get the iso to local machine, set up, install fedora …. le sigh… it’s the gnome3 spin. Intolerable, and unusable in my virt-manager-over-vnc-over-vpn environment. OK. Install updates, mate and lxde.

584 updates. And now we play the waiting game …

I feel like a hornet trapped inside a window. I’m strong, I’m formidable, I’m a bit angry, and I just can’t quite seem to figure out that the thing I’m bouncing off is a pane of glass and that I’ll just need to find a different way out. Maybe the fact that I keep bouncing, keep trying new approaches is that whole tenacity thing. Or maybe I’m just a dumb bug for not seeing that a foot to the left the window is open and I’m just completely missing it.

Ok, so 584 updates and 200ish other package installs later (or, an hour and change), reboot into a desktop that has buttons I can actually click in a guest. Run truecrypt installer. Attach USB host device of the disk I’m trying to work with. Truecrypt, format the partition, quick format (because I did a zero-pass earlier and don’t care about disclosing how much data is on the disk). Format ext4. Password. Check. Formatting….. hanging…. hanging ….. success!

Mount the volume. now let’s see how bad it is. DD out a 4 gig file from /dev/zero onto the mounted volume, wrap in ‘time’, pump through ‘pv’ for a status. And …. pv isn’t showing up. Hanging. Hanging. Hanging. Waiting. Oh, output! 480 megs? Ok. Hanging….. another 400 megs? Ok…. hanging…. another 300?

Yeah, so after those hoops, the “run it in a vm with usb passthrough” approach works. But gods is it awful….

OK. So when I just dump data onto a LUKS-encrypted device natively on my linux workstation, I get about 30MB/sec. Which is close to line speed of usb2, if less than half the expected write speed of the disk (it’s usb3 and if I had a port that fast this disk would be glad to suck down data at 60+). Adding KVM overhead for usb passthrough, truecrypt, and whatever else fedora might be bringing to the mix, I’m seeing … between 1 and 2 MB/sec. At this rate, it’d be as fast to just pump the data straight to S3 over the network.

Why’s this shit gotta be so hard?

Why does calendaring suck so much?

Ok, so here’s the story.

At work, we’ve got exchange calendars. They do everything from meeting and room scheduling to events and outings and all that stuff, with meeting invites carrying along the necessary info to make stuff work in general. It’s pretty slick, but it only works with outlook and outlook web access.

Which is all well and great, except that it means to use it I’ve got to have our OWA page sitting in a browser window open all the time. No reminders, nothing else. See, I’m a Linux sysadmin, and I don’t even have a windows box on my desk. No windows, no outlook.

Now, most places you can get away with doing stuff like that by running Evolution and using the owa interface to interface with exchange calendars and it works, but with our company, there’s some tweak or customization to the login page that causes exchange to simply not work with it. I’m not sure on the details, but it’s a bit of a problem either way. So that route’s out. Also, evolution blows a little bit as a mail client.

So, I thought what I’d do was I’d run thunderbird for email and not worry about calendaring. Which worked great for me for the first three months I was here, until I started having more than one meeting a week and had to actually schedule and keep track of things for myself appropriately. As my daily and weekly complexity levels rose, I needed a calendar. So I turned to Lightning, the thunderbird calendar plugin.

And you know, Lightning was working ok. Not any really huge problems with it except that the calendar was only on the local system, not synced with exchange or anything else. Of course, that meant I couldn’t check the calendar from owa at home, or anywhere else… it was only any good to me on my local system at work. Which wasn’t really great for me, because I keep a slightly odd schedule and need the calendar to reference to remind me what I’m doing tomorrow. So I was still maintaining two calendars — the OWA one and the lightning one, so I could have web-accessibility and still have pop-up reminders and such.

But maintaining two calendars manually is a pain, so I sought out a different solution. The obvious answer was to tap Google Calendars inside of lightning using the lightning GDATA provider to replace the local lightning-native calendar.

And that’s when the headaches really started. You see, Lightning’s gdata provider does something weird, or maybe it’s just lightning itself. Some meeting invitations work perfectly in lightning, directly. They just go into the google calendar, they work, and things are great. But then others, you try to accept them and they crap all over the place. For example, the “accept” button will show up, you click it and nothing happens. Or you click accept and thunderbird freezes. Or you click “accept” and it adds it to the local calendar, or if you don’t have a local calendar pops up a box asking you to pick the calendar to add it to but not listing the google one. But you can still drag-and-drop such invites into the calendar. Just that if you do, it decides to send new invitations to everyone on the attendees list, because rather than interpreting that as “accept the invitation” or “add this event to the calendar” it interprets it at “take this event, add it to the calendar like it’s your own, and send out appropriate invites as specified in the invitation”.

This … you see, this is annoying as hell. The only way I can convince the google calendar to accept the invite without spamming everyone on the list is to basically duplicate the event manually, for myself only. Now instead of maintaining two calendars by clicking “accept” in lightning then “accept” in owa, I’m maintaining two calendars by clicking “accept” in lightning, getting annoyed when it doesn’t work, manually recreating the meeting info, then clicking “accept” in owa. Calling it “retarded” would be offensive to retarded people everywhere.

So now I’m pretty far down the rabbit hole. Most of the events that I’m tracking are in my google calendar, but that means they don’t get the exchange-pushed updates to things like locations and cancellation statuses. And that’s not all of the events that I’m keeping track of, just most of them. And none of this even begins to address things like tying into my phone.

My basic desire is to have a single, central calendar, which I can see from my cell phone, from my mail client on any system I use, and from a web interface. I want to be able to click “accept” and accept invitations. I want to be able to do this without having an extra system just sitting there being my own personal calendar sync system. And honestly, if lightning could understand all the invitations I get and send them all to the gdata provider, I’d have close enough to what I want. Or if lightning could just talk to exchange directly.

The war on sudafed, and the evils of trees

Basically, for those that don’t know, nonflowering trees pollinate differently than most other plants (eg: flowers): rather than insect-mediated pollination (where bugs fly flower-to-flower), a lot of trees do wind-mediated pollination — which means basically from the middle of April through the early part of June, they dump tons of pollen into the air. It gets into everything, and when it’s really heavy it coats the surface of my car with a fine yellow powder. Seriously, I wish I were kidding…

Anyway, it turns out I’m allergic to some kind of tree pollen. I don’t know what kind, but around here it doesn’t matter too much because of the diversity in people’s yards and in the local parks.

Back when I was at Purdue, I had bad days a couple times a year, where I’d be sitting in class with the window open and I’d suffer a bit… or something like that. But it was always controllable — pop a couple sudafed and the problems went away completely.

But now I’m living something like two FDA hardiness zones south of where Purdue is (West Laf is 5a/5b, Rockville’s 7a). And apparently whatever I’m allergic to is a lot more aggressive around here.

For the last two weeks I’ve been on Zyrtec, trying to fight it off. And you know, for the first week of that, it was working great, keeping the symptoms down and all. But the last couple days, the trees really kicked it into high gear and completely obliterated the effects of the Zyrtec. So when I went into work today, I felt totally miserable, and decided I needed to go get some Sudafed as a stronger anti-allergy booster. And of course, I went to the local Safeway which is about 300 feet away from where I work, because it seemed like the logical place.

The thing about Sudafed (pseudoephedrine hcl, for those in the know) is that it’s both very effective as a decongestant and useful as a chemical precursor to methamphetamine (ie: crystal meth). So with meth becoming a more significant problem in the last decade or so, pseudoephedrine is being displaced by phenylephrine.

Phenylephrine, while possibly more effective than a placebo in studies, is significantly less effective than pseudoephedrine for most people. And it doesn’t do jack freaking squat for me, in particular. And what’s worse, some stores don’t carry the real stuff at all anymore. That Safeway, which lacks a pharmacy, is unfortunately one such store.

Fortunately, I managed to find some real sudafed at the Target pharmacy, which I left work early specifically to go after (since I was pretty much incapacitated by the allergies all day and didn’t get anything significant done anyway). While I got the meds I needed (and an air filter for my room to hopefully drop the pollen count in my sleeping/living space), it was definitely more of a hassle than it should have been.

I’m coming to appreciate bees.

At the crisis point again

So I’ve come once again to the question: what do I want to do with my life?

Seriously, I don’t know. I’m considering my current job and my level of satisfaction with it, my other options in my career field, and room for personal growth. I feel like being a sysadmin at this point is somewhat a surrender to the mediocrity that stalks me, threatening to sap away those things I most value about myself.

Maybe spending my life in the land of computers is the wrong choice. Or maybe just in IT. But would making a different choice require me to essentially become a different person, rejecting the me that will be on this path for the me that might be on another? Or is continuing down the path I’m on a rejection of the me that should be on that path but isn’t?

So do I boldly take the next step down this path, or turn around and walk a new one, or do I just let myself slide?

amazing

It’s really pretty amazing how easy it is to shift from optimistic enthusiasm to abject and somewhat inconsolable depression.

I think my future plans are somewhat coming to a head at this point. I’ve got until the 10th of this month to decide whether to renew my lease or not. If I don’t, I should probably plan on leaving the area. I’m not sure if I’m going to or not, yet.

My roommate is still totally useless. I don’t care about his existence at all at this point. So I’m quite sure I’m not going to renew the lease with him.

Do I want to go to grad school? Yeah, I do, but I don’t know what in, and I don’t know where. I’ve been halfassed enough about college as a whole that I am not going to be a particularly attractive grad student. Should I be pursuing the GRE’s? I probably already should have taken them, at this point.

Do I want to pursue a sysadmin job? From what the boss was saying, it sounds like it’s not very promising that I get a full-time position here, regardless. They’re asking me to try to find my replacement so they can start learning the ropes. I wonder if it’s even possible to find, especially at what they’re paying here. Can I find a similar adminny job somewhere else? Or am I going to be dragged into programming?

Could I handle a programming job? I haven’t written meaningful code (other than shellscripts) since 448. 448 was fun, but I wasn’t great at the coding part of it… could I write code in a professional environment?

You know, I woke up this morning, pretty much oblivious to these things. Here I am now, after a couple hours of thought and interaction, and I’m totally entrenched in what amounts to a major existential funk. I’m not happy with the status of my own existence, and that’s all there is to it.

… Maybe I should be happy about all of this? Maybe I really need a change in scenery, an external change could induce an internal one? Maybe I’ve just got anxiety about my poor self-assessment and uncertain future… I dunno.

Either way, I’m kinda down.

Am I wrong?

OK, this entry comes with a prepended perspective check. It’s an entry about sandwiches. That’s all. I know in the global sense, or even in the sort of problems in life that people deal with, this doesn’t amount to a hill of beans, and is really not a big deal. And it’s probably not going to change my behavior patterns either. I know that people are starving, that people are homeless, and that my ability to bitch about bad service at a subway is a great luxury that many people in the world don’t have. So don’t even bother pointing that shit out, I already know. That said …

When I go to the local subway to get a sandwich, is it wrong to expect that the sandwich is the way I ask for it? That I don’t get shortchanged? That it’s not so poorly assembled that it falls apart before it even gets wrapped up?

So here’s the story.

A couple days ago, I go in to subway, and this one guy (henceforth “black guy”) is working there. I tell him what I want, it’s pretty simple … I ask for a footlong club on wheat. He reaches for a loaf that’s sitting on the counter that’s been sliced completely apart (so that instead of forming a nice pouch, it’s just a loose top). I ask him to give me a different bun, so it can hold some toppings, and he is like “ahh aight”, and takes a minute or so to lazily turn around and get a different bun. Builds the meat halfassedly. I tell him what veggies I want, and he puts them down … on about 2/3 of the sandwich. Leaves about a third of the sandwich bare.

Also overdoes it on the salt and pepper, makes for a generally not-so-good sandwich. Also takes him about 2 minutes to ring the thing up and give me change … this is at a time when there’s no other customers at all.

So I go in again yesterday, during the superbowl (so again, no other customers). This girl (henceforth “latino girl”) is working.

I tell her what I want, she does fine on that, a bit lazy on the veggies (pretty uneven, not as heavy as I wanted). Then she goes to ring me up. It’s like $6.03, I give her two fives and a dime, she hands me back 7 cents. I’m like “umm, what?”. She’s like “huh?”. “Umm, I gave you two fives”. “Yeah, I gave you your change”. “Umm, no, the bill was six bucks, I gave you ten. look in your hands”. She looks, “oh, umm … wait … ” … thinks about it for half a minute or so, and gives me the rest of my change. Craptastic.

And I’m in there again today, because I’m a glutton for punishment I guess. Latino girl starts putting the veggies on my sandwich (and again, lazy and uneven). Then “middle-aged white woman” takes over. I ask for spicy mustard, she hoses the thing down with regular yellow mustard.

Now, I dunno if you know, but regular yellow mustard has a significantly different taste than spicy. And it’s not something I particularly enjoy on a chicken sandwich. So I pointed out that I said spicy. She realizes what she did, goes back to the beginning of the line and remakes my sandwich. This isn’t a big deal, just one of those sort of careless things that happens sometimes. But as she’s explaining to the girl at the front of the line (henceforth “the blonde bitch”) how she (in her words) “ruined” my sandwich and is making me another one, the blonde bitch says “oh, yeah, he’s not a very nice customer”.

I’m thinking … ok, so in the last 5 days I’ve eaten here three times, and had to put up with two halfassed sandwiches, someone trying to shortchange me 4 bucks, someone carelessly wrecking an otherwise great looking sandwich, and you’re saying I’m “not a very nice customer”? What … the … fuck.

Seriously. What the fuck? I mean … none of this is a really big deal, it’s just indicative of carelessness or laziness, of people not taking pride in their work, whatever. But I’m not a nice customer because I keep coming back despite the consistently shitty service? Jeez …

courses, introspection

First up: Updated course plan, hot off the OpenOffice 1.1 HTML Exporter. Assuming that I pass compilers, and that I pass the other class too. I’m signing up for classes a bit earlier, to see if I can get myself wedged into phil 303.

Second up: lately I’ve been introspecting a bit, thinking about what I want to do, what I’m doing now, etc. Also thinking about my hobbies, and whether they’ve been of any value to me, and whether they’ll be fo any value to continue. And I’m still not sure, no conclusions have been reached.

(introspective kwak follows)
Continue reading courses, introspection