Gnome, limits.conf and 2.6 kernels

This page is to document a peculiar problem that I encountered when upgrading my systems to 2.6 kernels. Some time ago I wanted to run a piece of code that allocated a large amount of stack on startup. I noticed that this behaved differently on different machines and that on the machines where it seg. faulted the stack size was limited to a small value. I fiddled around with /etc/security/limits.conf to set soft stack to a large number (these machines have plenty of physical memory):

* soft stack 4294967296

Fast-forward a couple of years. I am now happily running gnome 2.x and want to upgrade to a 2.6 kernel (stock Debian sarge). The upgrade goes fine, but gnome won't start up correctly: various utilities such as Nautilus and gnome-terminal don't come up. On investigation I find that they are failing like this:

centaurus:> gnome-terminal
gnome-terminal: allocatestack.c:370: allocate_stack: Assertion `size != 0' failed.

After much head-scratching I found that this problem goes away if I reduce the limit to something smaller, like 512 Mb.

But in fact the problem was still there, and mozilla still had trouble running (DNS problems, random seg. faults) unless I removed the soft stack entry altogether, when it all works perfectly.

Now of course I'm doing what I should have done in the first place and using calloc for my code...