Addressing The Outmoded Swapping And Paging Strategy in OSX?

UPDATE: 2012: I’VE STOPPED DOING THIS NOW, MY NEW MACBOOK SEEMS TO HAVE ENOUGH MEMORY AND FLASH DRIVE SPEED FOR IT NOT TO BE A PROBLEM THAT NEEDS FIXING.

I have a MacBookAir; it has 2Gb of RAM, a dual-core 1.6GHz processor, and a bunch of applications on it.

The performance, especially when Firefox is running, or when it’s shutting down, can be dreadful.

I used to wonder why this was; part of it is a thermal issue – when the processor gets hot the operating system (a) clocks it down, so it goes slower and (b) generates lots of idle-loops to try and cool the processor, further degrading power.

If you trawl around the Apple support forums you’ll find a bunch of people talking about “kernel task 100%” which is the operating system either (1) running the aforementioned idle loops, or (2) logging warning messages to (somewhere) about how hot it is, all of a sudden.

Further if you go to the Apple Genius Bar in Regent St, as I recently did for a repair, at least one of the guys there will tell you that the MacBookAir has a known problem which causes the “kernel task 100%” issue, and which can be fixed “by swapping a cable”… though this won’t make your machine any cooler or faster, I suppose it will help performance by not making the machine degrade itself when it gets hot.

But the above is only half of the story.

The other half of the story became evident to me when I started using dtrace (specifically “/usr/bin/iosnoop”) to work out why @adriana872‘s similar MacBookAir was being abominably slow; the culprits seemed to be Firefox and Skype, and I was blaming the performance problems on the overheating issues above…

…but I did wonder why it was that “iosnoop” reported “Firefox” to be writing to/from /var/vm/swapfile0 (and others) when it was exiting? That’s right. You’re killing an application, and it’s causing the machine to page virtual memory, really really heavily.

What’s going on?

So I started piecing together what I already knew, and looked at it in terms of what I was trying to achieve. The actual culprit is /sbin/dynamic_pager; in case you haven’t looked at the OSX VM subsystem:

  • OSX does not use fixed raw swap partitions; in fact, to get it to use one seems to be such heavy magic that I’ve been put off investigating them. It’s certainly not as easy as poking /etc/fstab.

 

  • Instead, OSX uses dynamically allocated swapfiles on a journalled filesystem, ie: “root”. I don’t know whether there’s an ioctl() that an be used to switch-off journalling for swap devices, but damn this strikes me as a recipe for vile performance, especially on a fragmented disk when the kernel suddenly decides it has to pull-together a few hundred megabytes for scratch space.

 

So far so bad, but here’s the kicker: the swapfiles are allocated, grown and freed dynamically, too; so the first time you need swap space, it allocates 64M (yes, MEGABYTES) of swap; then the next file is ALSO 64 megabytes, and then the next ones double to 128M, 256M, etc… and when you stop needing that disk space it immediately expends heat and valuable processor time – scarce resources on a netbook-class machine – trying to give you that 256 megabyte file, back.


09:15:14 luther:vm $ ls -la
total 1048576
drwxr-xr-x 6 root wheel 204 1 Mar 09:15 .
drwxr-xr-x 26 root wheel 884 8 Nov 20:47 ..
-rw------- 1 root wheel 67108864 28 Feb 23:28 swapfile0
-rw------- 1 root wheel 67108864 1 Mar 09:19 swapfile1
-rw------- 1 root wheel 134217728 1 Mar 09:19 swapfile2
-rw------- 1 root wheel 268435456 1 Mar 09:19 swapfile3

[firefox explanation elided – see comments, below]

I am certain that somebody thought this exponential-backoff-inspired grow-and-reclaim strategy was a good idea, but it doesn’t make any sense any more. It probably stopped making sense in around 2005 and certainly makes no sense at all today.

The issues are:

  • 64M? You Kick-Over Into Paging For A Measly 64 MEGABYTES? Twice? What The Hell Are You Guys Thinking? Firefox Mallocs More Than That For Some Of Its Plugins!

 

  • (less rantingly:) If you’re going to take a bite of the harddisk then at least make it worthwhile. Disk is cheap and it has been for some time.

 

  • And what’s with this dynamic reclamation for small swapfiles?It’s bad enough that you’re allocating tiny amounts of virtual memory on-disk but then you insist on shuffling piddling amounts of data between files in order to “release” that disk space back to the user! Frankly if I need 256Mb of disk back then (a) I will empty the trash or (b) I will buy an external disk.The OSX strategy makes especially no sense in the Netbook world – We are running with a “small” amount of memory, where “small” = 2Gb, but are likely to be firing-off a small number of very fat applications. What sense is there in creating and reclaiming 64Mb swapfiles when the applications eat 250+Mb apiece?
  • UPDATE: worst case scenario: Quit Firefox and you’ll be trying to free (say) a 512Mb file by cloning any live data into one of several smaller files. Fragmentation-Thrash City, on a system that is probably shutting down, or where the user is trying otherwise to shut down the laptop prior to shoving it in his/her briefcase and get the hell out of the office.

 

Following Brooks’ strategy of “take no half measures” I decided to address this, and last night had the opportunity to re-install my MacBookAir from backup, so I took the chance to do some major surgery.

The steps I have taken to try and bring some sanity to my system, are:

  1. The disk is now partitioned in two.The first partition (8Gb, near away from the spindle, fast) is called “/Volumes/scratch” and will be used for swapfiles and for temporary files which are not to be backed up. Under SnowLeopard it is no longer possible to make this a UFS filesystem (fast, non-journalled) but at least the partition is unlikely to get fragmented, and it is easily erased/reformatted. I reckon that if you need more than 8G of swap on a MacBookAir then you’re doing something wrong; on bigger systems I will use a bigger partition.The second partition is of course the remainder of the disk, and constitutes my “root” / install target.

 

  1. The system should be switchable-off; so I’ve set it so that OSX reverts to default behaviour unless the file “/USE_ALTERNATE_SWAP” exists; if something goes wrong I can remove that file and reboot.

 

  1. The “dynamic_pager” daemon is now wrapped in a launcher-script which enforces a new paging strategy:
    • Every swapfile is to be 1G
    • Do not allocate a swapfile unless less-than 200M of space remains
    • Do not release/reclaim swapfiles unless 2G of swapspace (ie: 2 whole files) are vacant

I’ve wanted to do this for a while; the only thing missing was the method of getting this strategy installed, and I finally found it this morning at a wonderful blog-post on superuser.com – although the guys there are only trying to address the fragmentation issue.

So I wrote the wrapper and installed it (by the same means described in the blog posting) this morning – and I set about trying to thrash my MacBookAir; so far I have gotten it to create 3Gb of swapfiles and it’s not given them back yet – but I don’t mind because that’s what the “scratch” partition is for.

If it gets up to 7Gb and doesn’t give it back, then I will worry, but the manpage for dynamic_pager seems to back up my method.

If you want to see the script I am using, it’s on Google Code.

Links:

If you want to try this out, on your head be it. It’s only had 4h of testing, so far.

We can’t fix the Air’s low-speed CPU and we can’t address the heat-retention issue (much) – but we can stop it wasting time fiddling with small changes to virtual memory.

UPDATE: Oh, and to answer the obvious question: performance on the Air seems a lot better; I simultaneously ran up a VirtualBox Ubuntu with 1G of core (ie: half of physmem, + overhead) – plus Firefox with 60 tabs, Safari, Chrome, FinalCutExpress, iPhoto, Tweetdeck and an Adobe AIR updater. It ran slowly, but with almost no beachball-of-death experiences. FinalCut took the longest to launch, but it always does…

32 thoughts on “Addressing The Outmoded Swapping And Paging Strategy in OSX?

  1. Daniel Barlow

    Firefox/Mozilla has always taken an age to shutdown on Linux too: once upon a time I was told by a Mozilla hacker that it was because it uses a refcounting “gc”, and spends a long time trying to map in almost every swapped page at exit time so it could write 0 into random objects’ reference counts.

    Don’t know if this is still true. It feels truthy

    Reply
    1. alecm Post author

      Dan: That also would fit some of the FF behaviour I’ve seen, but I can’t tell if it’s true or not. It definitely pages heavily and randomly around the half-dozen swapfiles (mumble mumble fragmentation performance nightmare) prior to exit, and shortly afterwards the files are gone, so I can’t be sure if it’s (a) doing what you say above, (b) electively releasing pages back to the kernel or (c) that some solaris-door-like mechanism means that the dynamic-pager is using the Firefox timeslot to free up the VM that Firefox has consumed.

      The latter would be cute but is least-likely; the former two will definitely benefit from an unfragmented VM space, so I think my method is a win all round.

      Reply
  2. alecm Post author

    BUGFIX POSTED:
    the script needs to manually erase the old swapfiles on reboot, because otherwise it will not write-over them.

    Reply
  3. William

    Oh! How cool! I have the same problems. Er, is there a one-click download version of the fix for dummies I can buy for £2.99? Until that day….

    Reply
    1. alecm Post author

      Ah, the joy of open source.

      Integration for the general public will require a geek for some time to come.

      At the moment it’s alpha-test. :-)

      Reply
  4. Mark J Musante

    Lines 73 & 76 should probably have quotes around the pathnames, in case the scratch volume’s name contains whitespace.

    Reply
  5. alecm Post author

    UPDATE:

    Well it seems like dynamic_pager, isn’t.

    As I imply above, am/was more than happy to hard-code a sizable swap-partition onto my MacBook, but I was rather hoping – given the nature of the solution outlined – that dynamic_pager would release and delete swapfiles when it was finished with them.

    However: it doesn’t appear to be doing so.

    I’ve been filling my MacBookAir’s RAM by use of a simple perl-script:

    #!/usr/bin/perl
    $size = 2048; # 2Gb
    $mb = 1024 ** 2;
    print length ($d = “!” x ($size * $mb)), “\n”;
    exit 0;

    …and it’s now up to 5Gb of swapfiles, but now that I have killed the scripts there is only 120*Mb* of swap actually in use; from this I infer that the files are not being reclaimed/deleted when dynamic_pager is executed in this fashion.

    However: It’s not a great hassle. The space at least appears to be being *reused* – ie: it is not a memory leak situation – and the maximum extent it could fill is constrained by the size of scratch partition that I have allocated.

    Anyone who can shed light on dynamic_pager, is welcome. The only source code I can find is severely out of date:

    http://www.opensource.apple.com/source/system_cmds/system_cmds-279.6.1/dynamic_pager.tproj/dynamic_pager.c

    …that must date back to 10.3 or something, since the filenames have since changed…

    Reply
  6. alecm Post author


    Just rebooted with the latest (r6) script; curiously the pager has created two swapfiles already (swapfile0 and swapfile1) – upon a fresh reboot.

    I actually don’t mine if it does this because it means I have 2G of swap pre-allocated and the system won’t have to fight to get ahold of it; the behaviour also echoes the “64Mb + 64Mb” behaviour of the standard configuration, so I suspect it reflects the underlying system.

    Performance is pretty good so far, but I’ll do a followup post in a few days to confirm my findings.

    FIXED: this was due to a bug in the script with HWT set too low.

    Everybody experimenting with this should be using at least revision 8 of the script.

    Reply
  7. Dan D.

    Nitpick: a partition at the beginning of the drive is FARTHEST from the spindle in order to make it fast.

    Reply
  8. Pingback: links for 2010-03-05 « Blarney Fellow

  9. bcl

    Thanks! One thing to note, you need to create /Volumes/scratch/.vm/ directory manually, otherwise the script will default to normal operation.

    Reply
  10. alecm Post author

    installation notes:

    (place the script in /sbin)

    cd /System/Library/LaunchDaemons

    cp -p com.apple.dynamic_pager.plist{,_bak}

    plutil -convert xml1 com.apple.dynamic_pager.plist

    vi com.apple.dynamic_pager.plist

    plutil -convert binary1 com.apple.dynamic_pager.plist

    (make target swapfile directory)

    (check script permissions)

    touch /USE_ALTERNATE_SWAP

    Reply
  11. Dude

    I can’t get it to work correctly. There is now one swap file of 1GB created at startup. However, as soon as the system starts paging it creates a second one, even if it doesn’t need 2GB yet. It also never reclaims any of the allocated swap files no matter how much programs I close, i.e. ram gets freed.
    Is anybody seeing the sam symptoms and found a solution, e.g. by tweaking the different size values in the plist?

    Reply
    1. alecm Post author

      Hi Dude,

      Yes, I m still seeing this, and occasionally I try to investigate what is going on; I too am not seeing reclamation, and I _was_ seeing near-instant creation of two swapfiles although that’s less rapid after I fixed a bug in the script. I do not know what is going on with reclamation, but on the other hand I am seeing considerable benefit to performance from having the swapfiles set out clearly and in the large, so I’m not investigating all that often or much.

      Are you tight for disk space?

      Reply
  12. Dude

    Thanks for the quick response. The problems aren’t dramatic and I don’t care too much about 1, 2 or 4 GB of swap used.
    The only thing I’m actually interested in is to locate the swap files to another drive, something that wasn’t working for me any longer after the update to 10.6. (due to the binary format of the plists). And now I just wasn’t sure if I do the right thing seeing the described behavior.
    Can you recommend a good tool to compare the swap performance or is this purely how you perceive the speed of the system?

    Reply
    1. alecm Post author

      It’s a perception thing; when I get the spinning-beachball-of-death I fire up a new Terminal and do “sudo iosnoop” which shows you the amount of swap activity going on, and with the default strategy I experience a lot-more beachballs which go on for longer, whilst it tries to consolidate and remove swapfiles.

      Reply
  13. George

    Where do I put the “/USE_ALTERNATE_SWAP” file? Can this file be just a txt file with that name? Sorry, I’m new to this.

    I already moved the swap file to a dedicated partition and copied the “dynamic_pager_wrapper.sh.txt” to the /sbin directory.

    I’m doing something wrong here huh?

    Reply
    1. alecm Post author

      It goes in the root of your filesystem.

      Seriously, if you’re having problems following the instructions – for instance the script does not take any file suffix – then this may not be the right method for you.

      Reply
  14. Pingback: Ye Macce Threade

  15. alecm Post author

    Copied from http://www.head-case.org/forums/goredwings19s-computer-help-hotline/6452-ye-macce-threade-66.html#post410560

    For anybody who’s fairly technical but not into spending the cash for an SSD quite yet:

    A couple weeks ago I followed the advice in dropsafe : Addressing The Outmoded Swapping And Paging Strategy in OSX? to adjust some paging parameters and move paging to a separate (non-journaled) partitioned volume. The results have been extremely satisfying. In no way am I getting SSD-level results, but beachballs are now rare*, and things like quitting Firefox after a long session no longer take acres of forever while swap space is reclaimed.

    In other words I can now use my (non-unibody, maxed-out-at-4GB RAM) MacBook Pro all day (in my usual mode of lots of apps running and lots of tabs open) without wanting to punch something. Rage-free (and free) is nice, even if it’s not SSD-nice.

    A few caveats:

    1. This is a bit of work (and depending on your skillz, maybe a little risk) so it’s not worth doing unless you need it or love tinkering. My wife has an 8GB unibody MBP and it runs so smoothly it’s really not worth doing this on hers.

    2. Really should not be attempted unless you are unixy enough to understand the entire article and infer the (very slight) missing pieces, or have ready access to someone who is.

    3. Although Disk Utility seems to offer live repartitioning, it did not actually work the first time I tried it — it crashed my Mac (hard, but completely harmlessly) when I was trying to shrink the main partition. In my case the path of least resistance was to boot off my nightly SuperDuper clone and do the partitioning from there while I got some work done, so I didn’t pursue the live-partitioning thing further. (If live repartitioning doesn’t work for you and you don’t have a bootable external, you can always boot off your OS installation or upgrade DVD and go to the utilities from there — assuming you still have that DVD.)

    4. After you make your scratch volume, remember to exclude it from Time Machine and Spotlight explicitly so you don’t get pointless extra activity on it. OS X doesn’t automatically know to do this for you.

    Also, a tip: I made my scratch partition a little bigger than I expected to need, so that I could move some additional application-specific caches onto it. That was a good call. A few days later I moved the caches for Firefox and one of my video games onto the scratch drive, making an incremental but welcome difference in the general smoothness and non-spasticness of both apps.

    If I haven’t made you afraid/reluctant/bored by now, then it’s a good tweak that you might want to try.

    *Well, rare except when I do anything in Safari, which just seems ridiculously prone to them, but even in Safari they’re not nearly as long or plentiful as before.

    Reply
  16. Matthew Elvey

    Yes, indeed, if you’re having problems using the script, it might not be the best thing for you to try to implement.

    For example, you should know that
    “vi com.apple.dynamic_pager.plist”
    really means that you should open the file in a text editor of your choice and make changes similar to those described at http://is.gd/9qZIX , so it’ll run this script instead of running dynamic_pager directly.

    And you should know to chmod +x the script appropriately.

    OTOH, even if you generally know when & how to do such things, it still helps to have a list of ‘em, so I’ve just emailed such suggestions to Alec.

    Reply
  17. Steve Brown

    Oh My Goodness!

    Thanks for this, my MacBook has 4G and has been running like an absolute pig after a few hours. Mind you, I run to Eve Online clients and Netbeans IDE, as well as Safari, Mail.app and various other guff.

    But previously switching between apps would SBOD regularly and generally just kill the system for what seemed like hours. But of course it was only seconds.

    But now, switching between apps is instant – even though there’s still stuff swapped out I think the lack of having to clean up lots of swap files is really helping.

    And as for closing apps…. It’s like night and day. I can close an app and not have to go make a coffee!

    Thanks – I owe you a beer or two!

    Steve.

    P.S. This is on Lion, so it works on Lion too :)

    Reply
  18. Dan Pritts

    Thanks for this hint, i’ll give it a shot. My system has 4GB of RAM. Right now I have 919MB free (872 inactive), and 608MB swap in use.

    over the course of surfing around on this, both the inactive number and the swap-in-use number have been growing. WTF?

    Minor nit on “the fastest part of the disk”.

    Assume you are using the whole disk (not short stroking). In random I/O situations the fastest part of the disk is neither the inside nor the outside, it’s the middle.

    Right in the middle means your average seek time to that region is as low as is possible. Access time (seek + rotational latency) is much more important to your random i/o speed than the raw transfer rate of whatever portion of the disk you are looking at. You can’t control rotational latency but you can control the seek time.

    In this situation this is largely academic, you probably don’t want to split your mac into three partitions to accomplish this. However, if you had multiple partitions anyway (or an OS with a logical volume manager), putting swap in the middle would be best.

    Reply
  19. Mike Smith

    Finally got around to implementing this on my Mac as it slowly ground to a halt with the SBOD.
    Seems to be a big improvement so far.

    Reply
  20. Matthew Elvey

    This is working well for me on Mountain Lion 10.8.2. (I’d tried to up the paging file size beyond 1GB, but 10.8.2 doesn’t like that.)

    (I added this line before the exec (our alternate one on line 76) to help with troubleshooting:

    echo about to exec $PAGER -F “${ALT_PATH}”/swapfile -S $SIZE -H $HWT -L $LWT at `/bin/date` >> “${ALT_PATH}”/dynamic_pager_wrapper_isUsingThisDirectory.OSis`/usr/bin/uname -r`
    )

    Reply
  21. Pingback: How To Disable Mac OS X From Using Swap When There Still Is “Inactive” Memory? | Click & Find Answer !

  22. Romain Kang

    Thanks, Alec! Works great for my wife’s 2011 MacBook Air with 2 GB RAM. For those who haven’t already found it elsewhere, journaling can be disabled from a root shell:
    # diskutil disableJournal /Volumes/scratch [or whatever you’re calling it]

    Reply
  23. Christopher

    Has anyone tried just changing the launchctl plist (/System/Library/LaunchDaemons/com.apple.dynamic_pager.plist) for dynamic_pager to add the -S -H & -L options? That seems like it would be easier than creating a wrapper script for dynamic_pager. I haven’t tried this, because I’ve been reasonably content with the VM performance on my machine, but I thought I’d ask the question in case it simplifies this solution for folks who need it.

    Reply
  24. Matthew Elvey

    Chris, I’m glad I didn’t do that; I recently booted my system without my swap partition available, and it booted OK. It wouldn’t have if I’d simply edited apple’s .plist.

    Reply

Leave a Reply