[PATCH] Fix two bugs with process limits (RLIMIT_NPROC)

From: Neil Brown <neilb@cse.unsw.edu.au>

1/ If a setuid process swaps it's real and effective uids and then forks,
 the fork fails if the new realuid has more processes
 than the original process was limited to.
 This is particularly a problem if a user with a process limit
 (e.g. 256) runs a setuid-root program which does setuid() + fork()
 (e.g. lprng) while root already has more than 256 process (which
 is quite possible).

 The root problem here is that a limit which should be a per-user
 limit is being implemented as a per-process limit with
 per-process (e.g. CAP_SYS_RESOURCE) controls.
 Being a per-user limit, it should be that the root-user can over-ride
 it, not just some process with CAP_SYS_RESOURCE.

 This patch adds a test to ignore process limits if the real user is root.

2/ When a root-owned process (e.g. cgiwrap) sets up process limits and then
  calls setuid, the setuid should fail if the user would then be running
  more than rlim_cur[RLIMIT_NPROC] processes, but it doesn't.  This patch
  adds an appropriate test.  With this patch, and per-user process limit
  imposed in cgiwrap really works.

kernel/fork.c

@@ -759,8 +759,10 @@ struct task_struct *copy_process(unsigned long clone_flags,
 		goto fork_out;
 
 	retval = -EAGAIN;
-	if (atomic_read(&p->user->processes) >= p->rlim[RLIMIT_NPROC].rlim_cur) {
-		if (!capable(CAP_SYS_ADMIN) && !capable(CAP_SYS_RESOURCE))
+	if (atomic_read(&p->user->processes) >=
+			p->rlim[RLIMIT_NPROC].rlim_cur) {
+		if (!capable(CAP_SYS_ADMIN) && !capable(CAP_SYS_RESOURCE) &&
+				p->user != &root_user)
 			goto bad_fork_free;
 	}
 

kernel/sys.c

@@ -601,6 +601,14 @@ static int set_user(uid_t new_ruid, int dumpclear)
 	new_user = alloc_uid(new_ruid);
 	if (!new_user)
 		return -EAGAIN;
+
+	if (atomic_read(&new_user->processes) >=
+				current->rlim[RLIMIT_NPROC].rlim_cur &&
+			new_user != &root_user) {
+		free_uid(new_user);
+		return -EAGAIN;
+	}
+
 	switch_uid(new_user);
 
 	if(dumpclear)