At work, we’ve got exchange calendars. They do everything from meeting and room scheduling to events and outings and all that stuff, with meeting invites carrying along the necessary info to make stuff work in general. It’s pretty slick, but it only works with outlook and outlook web access.

Which is all well and great, except that it means to use it I’ve got to have our OWA page sitting in a browser window open all the time. No reminders, nothing else. See, I’m a Linux sysadmin, and I don’t even have a windows box on my desk. No windows, no outlook.

Now, most places you can get away with doing stuff like that by running Evolution and using the owa interface to interface with exchange calendars and it works, but with our company, there’s some tweak or customization to the login page that causes exchange to simply not work with it. I’m not sure on the details, but it’s a bit of a problem either way. So that route’s out. Also, evolution blows a little bit as a mail client.

So, I thought what I’d do was I’d run thunderbird for email and not worry about calendaring. Which worked great for me for the first three months I was here, until I started having more than one meeting a week and had to actually schedule and keep track of things for myself appropriately. As my daily and weekly complexity levels rose, I needed a calendar. So I turned to Lightning, the thunderbird calendar plugin.

And you know, Lightning was working ok. Not any really huge problems with it except that the calendar was only on the local system, not synced with exchange or anything else. Of course, that meant I couldn’t check the calendar from owa at home, or anywhere else… it was only any good to me on my local system at work. Which wasn’t really great for me, because I keep a slightly odd schedule and need the calendar to reference to remind me what I’m doing tomorrow. So I was still maintaining two calendars — the OWA one and the lightning one, so I could have web-accessibility and still have pop-up reminders and such.

But maintaining two calendars manually is a pain, so I sought out a different solution. The obvious answer was to tap Google Calendars inside of lightning using the lightning GDATA provider to replace the local lightning-native calendar.

And that’s when the headaches really started. You see, Lightning’s gdata provider does something weird, or maybe it’s just lightning itself. Some meeting invitations work perfectly in lightning, directly. They just go into the google calendar, they work, and things are great. But then others, you try to accept them and they crap all over the place. For example, the “accept” button will show up, you click it and nothing happens. Or you click accept and thunderbird freezes. Or you click “accept” and it adds it to the local calendar, or if you don’t have a local calendar pops up a box asking you to pick the calendar to add it to but not listing the google one. But you can still drag-and-drop such invites into the calendar. Just that if you do, it decides to send new invitations to everyone on the attendees list, because rather than interpreting that as “accept the invitation” or “add this event to the calendar” it interprets it at “take this event, add it to the calendar like it’s your own, and send out appropriate invites as specified in the invitation”.

This … you see, this is annoying as hell. The only way I can convince the google calendar to accept the invite without spamming everyone on the list is to basically duplicate the event manually, for myself only. Now instead of maintaining two calendars by clicking “accept” in lightning then “accept” in owa, I’m maintaining two calendars by clicking “accept” in lightning, getting annoyed when it doesn’t work, manually recreating the meeting info, then clicking “accept” in owa. Calling it “retarded” would be offensive to retarded people everywhere.

So now I’m pretty far down the rabbit hole. Most of the events that I’m tracking are in my google calendar, but that means they don’t get the exchange-pushed updates to things like locations and cancellation statuses. And that’s not all of the events that I’m keeping track of, just most of them. And none of this even begins to address things like tying into my phone.

My basic desire is to have a single, central calendar, which I can see from my cell phone, from my mail client on any system I use, and from a web interface. I want to be able to click “accept” and accept invitations. I want to be able to do this without having an extra system just sitting there being my own personal calendar sync system. And honestly, if lightning could understand all the invitations I get and send them all to the gdata provider, I’d have close enough to what I want. Or if lightning could just talk to exchange directly.

So Apache 2.2 has a couple of “stable” mpm’s, namely prefork and worker.

Prefork is the old tried-and-true method, where the server spawns $StartServers httpd processes, and on-demand starts additional up to $MaxClients. Each subprocess handles $MaxRequestsPerChild requests, then dies and is replaced as needed.

Worker, on the other hand, starts $StartServers httpd processes, and on each process runs up to $ThreadsPerChild. Each thread serves requests (the same as processes on prefork), and when any given process’s child threads hit $MaxRequestsPerChild, the process kills its idle threads, unpools its working threads and waits for them to finish, then dies.

Sounds cool, right? Threads are lighter than processes, so having 5 processes running 20 threads each sounds better than having 100 processes. I got caught up in that and decided to try it on a large-ish site (namely gotwoot, and the 20-odd other sites hosted on that box).

Well, it turns out Worker isn’t as stable as I was hoping. At least for me… one of the sites we host is a huge sender of fairly large files and file streams. When apache processes under mpm-worker try to die, they wait until all the threads are done sending… but if the children are sending for hours, it’s going to take hours for these otherwise-defunct processes to die. If we were talking about just one process taking that long to die, it wouldn’t be a big deal because it wouldn’t interact with anything else. But somehow in that interaction mode, various modules start behaving badly.

So I was running mod_php (only using threadsafe modules) on mpm-worker, and every couple of days I’d see random problems. Sometimes it was zero-byte replies from php, other times it was php segfaulting, still other times it was apache itself dropping empty page replies.

I got sick of that, so I switched over to mpm-prefork and php on fastcgi. Things seem to be better now… it gives me user-mode php, and because of that APC maintains a per-site cache. It’s running fcgi processes on demand, which is also cool ’cause if a site doesn’t get traffic, it doesn’t keep a running php process.

Overall, my system load is a bit lower and things just “feel” more stable with prefork+fcgi. In the next couple days I should actually _see_ whether it’s more stable or not… but either way, I guess there’s value in feelings too :p.

Detection: looked in /proc/net/ip_conntrack. Noticed that the connections listed were mostly SYN_SENT from some src, and “UNREPLIED” status. Many source ip’s. Verified that the tracker was functional (by temporarily firewalling off all inbound syn traffic to it except my own ip addy). Saw lots of packets getting dropped.

Mitigation attempt 1: tuned /proc/sys/net/ip_conntrack_max and /proc/sys/net/ipv4/netfilter/{many settings} in /etc/sysctl.conf, turning up maxes and turning down timeouts. This should more aggressively expire lost tcp connections. Also added a rule: iptables -t raw -A OUTPUT -p tcp --sport $TRACKER_PORT -j NOTRACK in an attempt to further lower the conntrack burden. Result: I no longer kept getting as many conntrack “table full” errors, but the tracker was still not talking to anyone.

Mitigation attempt 2: added two more firewall rules:
iptables -I INPUT 1 -p tcp --dport $TRACKER_PORT --syn -m hashlimit --hashlimit 10/min --hashlimit-burst 15 --hashlimit-name torrenthash --hashlimit-htable-size 2048 --hashlimit-htable-max 65536 --hashlimit-mode srcip -j ACCEPT
followed by
iptables -I INPUT 2 -p tcp --dport $TRACKER_PORT --syn -j DROP.
Result: the tracker appears to be talking again … I can see the web interface on it, and get peers on torrents hosted there without the help of DHT. It’s good times.

I imagine this technique could be used for quite a bit more than just protecting a tracker, so I suppose it’d be great to have it written down somewhere

So, you start a local firefox session, then ssh to a machine with a -X (enabling x11 forwarding). Then on the remote machine, you run firefox. You get … another locally-running firefox.

So you close both locally-running firefox sessions (and any others you might have) and invoke firefox on the remote machine. Now you get an X11-forwarded firefox running on the remote box (the expected behavior). And then you run firefox on the local machine, and you get …. another remotely-running firefox window.

Apparently, Xorg doesn’t differentiate between remotely-running windows and locally-running ones, and firefox catches any requests for a new X window named firefox, and instead of letting another copy be run, just makes a new window on the same firefox instance.

A little googling shows that the environment variable MOZ_NO_REMOTE controls this behavior — set it to 1 and firefox doesn’t lurk under the surface intercepting other instances that try to run.

So basically, for the last couple years I lived almost exclusively in gentoo for all my linux needs. Sounds good, right? A single distro, albeit a source-based one. I got used to a lot of gentoo-isms. I had my hands in RHEL a bit, I poked a bsd or two a bit, but gentoo was definitely my area of expertise, and my home.

Now I’m not so sure.

See, at work, I run an ubuntu box for my desktop. I’m the only one there on ubuntu, so I’m kinda the odd-man out in that regard (the other sysadmins are on fedora). We’ve also got servers that are fedora, so it’d make sense to be there… but no~oo, I had to install ubuntu instead.

Regardless, the majority of the server functions at work are solaris boxes. So I’m simultaneously getting more comfortable with Ubuntu, getting more comfortable with Fedora, and learning a shitlot about Solaris.

Which is confusing.

See, Gentoo’s got /etc/conf.d. Everything that’s distro-specific is controlled out of there. Things like … default behaviors, network configs, what xdm tool should be called (eg: gdm, kdm, xdm), what options to pass iptables and where to save it, what options to pass in init scripts, etc. It gives a lot of flexibility in a single place, and it’s very clean.

But nobody else does that. At all.

Ubuntu’s got it’s configs strewn all over /etc. Fedora and RHEL shove a lot of, but not all of, their stuff in /etc/sysconfig. Solaris … hell, I still don’t have any idea for half of that stuff … if it’s not in SMF, it’s probably somewhere in /etc, or maybe /var/sadm, or possibly in some random db2 file or something.

But it gets worse. Mainly because of package managers. I am finding myself typing “aptitude search” when I want to find a package on fedora or gentoo, and typing “eix” when I want to find things in ubuntu. I have to remind myself “oh wait, this is ${DISTRO}, not ${OTHERDISTRO}” all the time. This is only exacerbated by the fact that I’ve been building a new fileserver at home, and out of my distros of choice (ubuntu and gentoo), only gentoo’s install cd worked cleanly on the new hardware. On the bright side, at least solaris doesn’t have a sane auto-updating package manager to work with at all, so there’s one less thing to think about.

So yeah, that’s my life these days :p