A new class of attacks has been recently discovered that relies on the way the 2.6 Linux Kernel (2.4 does not do this) executes a setuid() call. If I hit my nproc limit (ulimit -u), then if root tried to setuid(myuid), the setuid() fails, returning a -1. There are a number of programs that aren’t checking the return value of setuid() calls, which causes the program in question to continue execution as root, even though it thinks it’s running as a normal user.

I’ve been spending the past few days to verify that these issues

http://lists.freedesktop.org/archives/xorg/2006-June/016146.html

are not exploitable on RHEL or Fedora Core. My analysis says they’re not. This was important since an X update is non trivial for RHEL and FC4 (it’s one big giant package). On FC5 an update wouldn’t be so bad since it’s just one small self contained package. I’ve learned a great deal about user limits and how they work in Linux.

User limits are not enforced by default. Children processes inherit them from the parent processes, which in most instances is fine. My initial testing was a simple suid root program that tried to setuid(), which never failed. I expected this to fail since I had a test user with a nproc hard limit of 2 processes, and the test user was logged in twice already. I needed to initialize pam in order to properly set the user limits, since a suid root program will inherit the limits of root, not of the user who you are switching to (which does make sense). I should also note that simply calling pam is not enough, you also have to have set pam_limits in the process’ pam configuration file.

The most surprising thing I learned was that the 2.6 kernel sets an arbitrary hard user limit for each user. This bit of code exists in the kernel:

init_task.signal->rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
init_task.signal->rlim[RLIMIT_NPROC].rlim_max = max_threads/2;

That limit gets set in init, then is inherited throughout the system. This means that every user has an nproc limit, which also means that the setuid() attack will work on every 2.6 kernel system, even if user limits are not enabled.