What I didn't realise was that 32-bit Linux kernels have absolutely no NUMA support. The physical architecture of the machine is NUMA by design -- each processor has its own bank of memory -- but Linux can't be configured to recognise this unless it is compiled in 64-bit mode. So the 32-bit kernel is not efficient about allocating memory.
Debian supplies some 64-bit kernels but I need a more up to date one than 2.6.8 (for things such as the very wonderful k8-powernow for SMP systems). So I had no option but to compile my own 64-bit kernel. I downloaded 2.6.16 from kernel.org.
I first tried to figure out the options for doing this for a standalone compile while running a 32-bit kernel. The architecture must be specified on the command line:
make ARCH=x86_64 oldconfigThis gives access to the NUMA-related options. Great. But
make ARCH=x86_64doesn't work, because the default compiler is gcc-3.3, which can't cope with the -m64 switch in Debian sarge. This could be fixed in a number of ways, including changing the symlink in /usr/bin, but I chose to hack the definition of CC in the makefile to gcc-3.4 . I suspect the lib64gcc1 and lib64stdc++6 packages are useful to have at this point.
Next up, trying to build it the Debian way. Normally I would do
export CONCURRENCY_LEVEL=4 make-kpkg --revision=01 --initrd kernel-imagebut this won't work because it tries to do a make oldconfig with the old architecture.
make-kpkg --arch amd64 --initrd kernel-imageought to work but (a) tries to cross-compile (which we don't want -- fixable by makefile hacking again and (b) fails at the end with a complaint about the architecture. In the end I faked up the relevant stamp files so that it would just package the hand-built kernel. To do this, copy stamp.* from some earlier directory and touch stamp.* . You also need debian/buildinfo in your kernel working directory.
Install worked fine. 32-bit iptables doesn't work, but I have a 64-bit chroot for that. Quick install of the numactl stuff shows that everything is working as expected, including the very useful ability to tie processes to CPUs from the command line. Installed dchroot to let users play with pure64 environment (not that I have found any use for it so far in terms of speed).
One thing that struck me straight away was how much slower graphics were. X complained in the logs about not being able to do write-combining, and /proc/mtrr says there are no write-combining areas. I think it's plausible that this is because I enabled the option GART_IOMMU, which disables AGP support. This doesn't look necessary to me as the machine has only 2 Gb physical memory, so I turned it off again, and compiled AGP support back into the kernel. Now everything is fine.
Building a kernel the Debian way when the 64-bit kernel is running is still a hassle inside the 32-bit userland: the standard Linux make commands work based on the currently running kernel, but Debian thinks you want to build an i386 kernel anyway. The Debian package should be built inside the 64-bit chroot and installed in the 32-bit world using dpkg --force-architecture (kudos to the authors of dpkg for allowing this apparently lethally useless option to dpkg).