UPDATE: 2012: I’VE STOPPED DOING THIS NOW, MY NEW MACBOOK SEEMS TO HAVE ENOUGH MEMORY AND FLASH DRIVE SPEED FOR IT NOT TO BE A PROBLEM THAT NEEDS FIXING.
I have a MacBookAir; it has 2Gb of RAM, a dual-core 1.6GHz processor, and a bunch of applications on it.
The performance, especially when Firefox is running, or when it’s shutting down, can be dreadful.
I used to wonder why this was; part of it is a thermal issue – when the processor gets hot the operating system (a) clocks it down, so it goes slower and (b) generates lots of idle-loops to try and cool the processor, further degrading power.
If you trawl around the Apple support forums you’ll find a bunch of people talking about “kernel task 100%” which is the operating system either (1) running the aforementioned idle loops, or (2) logging warning messages to (somewhere) about how hot it is, all of a sudden.
Further if you go to the Apple Genius Bar in Regent St, as I recently did for a repair, at least one of the guys there will tell you that the MacBookAir has a known problem which causes the “kernel task 100%” issue, and which can be fixed “by swapping a cable”… though this won’t make your machine any cooler or faster, I suppose it will help performance by not making the machine degrade itself when it gets hot.
But the above is only half of the story.
The other half of the story became evident to me when I started using dtrace (specifically “/usr/bin/iosnoop”) to work out why @adriana872‘s similar MacBookAir was being abominably slow; the culprits seemed to be Firefox and Skype, and I was blaming the performance problems on the overheating issues above…
…but I did wonder why it was that “iosnoop” reported “Firefox” to be writing to/from /var/vm/swapfile0 (and others) when it was exiting? That’s right. You’re killing an application, and it’s causing the machine to page virtual memory, really really heavily.
What’s going on?
So I started piecing together what I already knew, and looked at it in terms of what I was trying to achieve. The actual culprit is /sbin/dynamic_pager; in case you haven’t looked at the OSX VM subsystem:
- OSX does not use fixed raw swap partitions; in fact, to get it to use one seems to be such heavy magic that I’ve been put off investigating them. It’s certainly not as easy as poking /etc/fstab.
- Instead, OSX uses dynamically allocated swapfiles on a journalled filesystem, ie: “root”. I don’t know whether there’s an ioctl() that an be used to switch-off journalling for swap devices, but damn this strikes me as a recipe for vile performance, especially on a fragmented disk when the kernel suddenly decides it has to pull-together a few hundred megabytes for scratch space.
So far so bad, but here’s the kicker: the swapfiles are allocated, grown and freed dynamically, too; so the first time you need swap space, it allocates 64M (yes, MEGABYTES) of swap; then the next file is ALSO 64 megabytes, and then the next ones double to 128M, 256M, etc… and when you stop needing that disk space it immediately expends heat and valuable processor time – scarce resources on a netbook-class machine – trying to give you that 256 megabyte file, back.
09:15:14 luther:vm $ ls -la
drwxr-xr-x 6 root wheel 204 1 Mar 09:15 .
drwxr-xr-x 26 root wheel 884 8 Nov 20:47 ..
-rw------- 1 root wheel 67108864 28 Feb 23:28 swapfile0
-rw------- 1 root wheel 67108864 1 Mar 09:19 swapfile1
-rw------- 1 root wheel 134217728 1 Mar 09:19 swapfile2
-rw------- 1 root wheel 268435456 1 Mar 09:19 swapfile3
[firefox explanation elided – see comments, below]
I am certain that somebody thought this exponential-backoff-inspired grow-and-reclaim strategy was a good idea, but it doesn’t make any sense any more. It probably stopped making sense in around 2005 and certainly makes no sense at all today.
The issues are:
- 64M? You Kick-Over Into Paging For A Measly 64 MEGABYTES? Twice? What The Hell Are You Guys Thinking? Firefox Mallocs More Than That For Some Of Its Plugins!
- (less rantingly:) If you’re going to take a bite of the harddisk then at least make it worthwhile. Disk is cheap and it has been for some time.
- And what’s with this dynamic reclamation for small swapfiles?It’s bad enough that you’re allocating tiny amounts of virtual memory on-disk but then you insist on shuffling piddling amounts of data between files in order to “release” that disk space back to the user! Frankly if I need 256Mb of disk back then (a) I will empty the trash or (b) I will buy an external disk.The OSX strategy makes especially no sense in the Netbook world – We are running with a “small” amount of memory, where “small” = 2Gb, but are likely to be firing-off a small number of very fat applications. What sense is there in creating and reclaiming 64Mb swapfiles when the applications eat 250+Mb apiece?
- UPDATE: worst case scenario: Quit Firefox and you’ll be trying to free (say) a 512Mb file by cloning any live data into one of several smaller files. Fragmentation-Thrash City, on a system that is probably shutting down, or where the user is trying otherwise to shut down the laptop prior to shoving it in his/her briefcase and get the hell out of the office.
Following Brooks’ strategy of “take no half measures” I decided to address this, and last night had the opportunity to re-install my MacBookAir from backup, so I took the chance to do some major surgery.
The steps I have taken to try and bring some sanity to my system, are:
- The disk is now partitioned in two.The first partition (8Gb, near away from the spindle, fast) is called “/Volumes/scratch” and will be used for swapfiles and for temporary files which are not to be backed up. Under SnowLeopard it is no longer possible to make this a UFS filesystem (fast, non-journalled) but at least the partition is unlikely to get fragmented, and it is easily erased/reformatted. I reckon that if you need more than 8G of swap on a MacBookAir then you’re doing something wrong; on bigger systems I will use a bigger partition.The second partition is of course the remainder of the disk, and constitutes my “root” / install target.
- The system should be switchable-off; so I’ve set it so that OSX reverts to default behaviour unless the file “/USE_ALTERNATE_SWAP” exists; if something goes wrong I can remove that file and reboot.
- The “dynamic_pager” daemon is now wrapped in a launcher-script which enforces a new paging strategy:
- Every swapfile is to be 1G
- Do not allocate a swapfile unless less-than 200M of space remains
- Do not release/reclaim swapfiles unless 2G of swapspace (ie: 2 whole files) are vacant
I’ve wanted to do this for a while; the only thing missing was the method of getting this strategy installed, and I finally found it this morning at a wonderful blog-post on superuser.com – although the guys there are only trying to address the fragmentation issue.
So I wrote the wrapper and installed it (by the same means described in the blog posting) this morning – and I set about trying to thrash my MacBookAir; so far I have gotten it to create 3Gb of swapfiles and it’s not given them back yet – but I don’t mind because that’s what the “scratch” partition is for.
If it gets up to 7Gb and doesn’t give it back, then I will worry, but the manpage for dynamic_pager seems to back up my method.
If you want to see the script I am using, it’s on Google Code.
If you want to try this out, on your head be it. It’s only had 4h of testing, so far.
We can’t fix the Air’s low-speed CPU and we can’t address the heat-retention issue (much) – but we can stop it wasting time fiddling with small changes to virtual memory.
UPDATE: Oh, and to answer the obvious question: performance on the Air seems a lot better; I simultaneously ran up a VirtualBox Ubuntu with 1G of core (ie: half of physmem, + overhead) – plus Firefox with 60 tabs, Safari, Chrome, FinalCutExpress, iPhoto, Tweetdeck and an Adobe AIR updater. It ran slowly, but with almost no beachball-of-death experiences. FinalCut took the longest to launch, but it always does…