Yesterday I tried the music thing again, using an at-job ( at(1p) in the man pages) and mpg123, and it worked great. But pre-selecting a track is sorta lame, so I thought “maybe I can do a shuffle thing, using my old radio scripts” — until I realized that that’d require both work and more metadata-parsing, which I don’t feel like doing ’cause I need to wake up in 7 hours. So I thought “radio stream!”

So I grabbed a PLS from one of the most salient radio stations I could think of, and tossed mpg123 at it. Of course, it didn’t work. So I tried tossing the stream url itself at it, and it didn’t work because the stream’s aac+. Bleh.

Then I tried mplayer … ’cause mplayer’s natively command-line. And it worked from the console, so I thought things would be good. Then I put it in a test at-job, and it didn’t fire off, because while mplayer’s natively a console app it’s also natively interactive. So that was out, unless I launched it in screen… but that would just get complicated.

Then I remembered. mpg123 is a very unixy tool (as opposed to linuxy). So I built mpg321, and tried with that. SUCCESS!

So there you have it: if you want to listen to radio streams from at-jobs, remember mpg321!

And in 7 hours and 5 minutes, my at job will go off after an initial alarm clock ring, and I’ll drag myself up and go fight traffic down to my new job!

So Apache 2.2 has a couple of “stable” mpm’s, namely prefork and worker.

Prefork is the old tried-and-true method, where the server spawns $StartServers httpd processes, and on-demand starts additional up to $MaxClients. Each subprocess handles $MaxRequestsPerChild requests, then dies and is replaced as needed.

Worker, on the other hand, starts $StartServers httpd processes, and on each process runs up to $ThreadsPerChild. Each thread serves requests (the same as processes on prefork), and when any given process’s child threads hit $MaxRequestsPerChild, the process kills its idle threads, unpools its working threads and waits for them to finish, then dies.

Sounds cool, right? Threads are lighter than processes, so having 5 processes running 20 threads each sounds better than having 100 processes. I got caught up in that and decided to try it on a large-ish site (namely gotwoot, and the 20-odd other sites hosted on that box).

Well, it turns out Worker isn’t as stable as I was hoping. At least for me… one of the sites we host is a huge sender of fairly large files and file streams. When apache processes under mpm-worker try to die, they wait until all the threads are done sending… but if the children are sending for hours, it’s going to take hours for these otherwise-defunct processes to die. If we were talking about just one process taking that long to die, it wouldn’t be a big deal because it wouldn’t interact with anything else. But somehow in that interaction mode, various modules start behaving badly.

So I was running mod_php (only using threadsafe modules) on mpm-worker, and every couple of days I’d see random problems. Sometimes it was zero-byte replies from php, other times it was php segfaulting, still other times it was apache itself dropping empty page replies.

I got sick of that, so I switched over to mpm-prefork and php on fastcgi. Things seem to be better now… it gives me user-mode php, and because of that APC maintains a per-site cache. It’s running fcgi processes on demand, which is also cool ’cause if a site doesn’t get traffic, it doesn’t keep a running php process.

Overall, my system load is a bit lower and things just “feel” more stable with prefork+fcgi. In the next couple days I should actually _see_ whether it’s more stable or not… but either way, I guess there’s value in feelings too :p.

Detection: looked in /proc/net/ip_conntrack. Noticed that the connections listed were mostly SYN_SENT from some src, and “UNREPLIED” status. Many source ip’s. Verified that the tracker was functional (by temporarily firewalling off all inbound syn traffic to it except my own ip addy). Saw lots of packets getting dropped.

Mitigation attempt 1: tuned /proc/sys/net/ip_conntrack_max and /proc/sys/net/ipv4/netfilter/{many settings} in /etc/sysctl.conf, turning up maxes and turning down timeouts. This should more aggressively expire lost tcp connections. Also added a rule: iptables -t raw -A OUTPUT -p tcp --sport $TRACKER_PORT -j NOTRACK in an attempt to further lower the conntrack burden. Result: I no longer kept getting as many conntrack “table full” errors, but the tracker was still not talking to anyone.

Mitigation attempt 2: added two more firewall rules:
iptables -I INPUT 1 -p tcp --dport $TRACKER_PORT --syn -m hashlimit --hashlimit 10/min --hashlimit-burst 15 --hashlimit-name torrenthash --hashlimit-htable-size 2048 --hashlimit-htable-max 65536 --hashlimit-mode srcip -j ACCEPT
followed by
iptables -I INPUT 2 -p tcp --dport $TRACKER_PORT --syn -j DROP.
Result: the tracker appears to be talking again … I can see the web interface on it, and get peers on torrents hosted there without the help of DHT. It’s good times.

I imagine this technique could be used for quite a bit more than just protecting a tracker, so I suppose it’d be great to have it written down somewhere

So, you start a local firefox session, then ssh to a machine with a -X (enabling x11 forwarding). Then on the remote machine, you run firefox. You get … another locally-running firefox.

So you close both locally-running firefox sessions (and any others you might have) and invoke firefox on the remote machine. Now you get an X11-forwarded firefox running on the remote box (the expected behavior). And then you run firefox on the local machine, and you get …. another remotely-running firefox window.

Apparently, Xorg doesn’t differentiate between remotely-running windows and locally-running ones, and firefox catches any requests for a new X window named firefox, and instead of letting another copy be run, just makes a new window on the same firefox instance.

A little googling shows that the environment variable MOZ_NO_REMOTE controls this behavior — set it to 1 and firefox doesn’t lurk under the surface intercepting other instances that try to run.

So basically, for the last couple years I lived almost exclusively in gentoo for all my linux needs. Sounds good, right? A single distro, albeit a source-based one. I got used to a lot of gentoo-isms. I had my hands in RHEL a bit, I poked a bsd or two a bit, but gentoo was definitely my area of expertise, and my home.

Now I’m not so sure.

See, at work, I run an ubuntu box for my desktop. I’m the only one there on ubuntu, so I’m kinda the odd-man out in that regard (the other sysadmins are on fedora). We’ve also got servers that are fedora, so it’d make sense to be there… but no~oo, I had to install ubuntu instead.

Regardless, the majority of the server functions at work are solaris boxes. So I’m simultaneously getting more comfortable with Ubuntu, getting more comfortable with Fedora, and learning a shitlot about Solaris.

Which is confusing.

See, Gentoo’s got /etc/conf.d. Everything that’s distro-specific is controlled out of there. Things like … default behaviors, network configs, what xdm tool should be called (eg: gdm, kdm, xdm), what options to pass iptables and where to save it, what options to pass in init scripts, etc. It gives a lot of flexibility in a single place, and it’s very clean.

But nobody else does that. At all.

Ubuntu’s got it’s configs strewn all over /etc. Fedora and RHEL shove a lot of, but not all of, their stuff in /etc/sysconfig. Solaris … hell, I still don’t have any idea for half of that stuff … if it’s not in SMF, it’s probably somewhere in /etc, or maybe /var/sadm, or possibly in some random db2 file or something.

But it gets worse. Mainly because of package managers. I am finding myself typing “aptitude search” when I want to find a package on fedora or gentoo, and typing “eix” when I want to find things in ubuntu. I have to remind myself “oh wait, this is ${DISTRO}, not ${OTHERDISTRO}” all the time. This is only exacerbated by the fact that I’ve been building a new fileserver at home, and out of my distros of choice (ubuntu and gentoo), only gentoo’s install cd worked cleanly on the new hardware. On the bright side, at least solaris doesn’t have a sane auto-updating package manager to work with at all, so there’s one less thing to think about.

So yeah, that’s my life these days :p