[PATCH] sched: honor the "sync" wakeup bit

The scheduler changes had another thing missing: the appreciation of
sync wakeups.  (I had this in one of the earlier sched-domains cleanup
patches before but it got lost in the shuffle.)

When a sync waker is waking, we should subtract its load from the
current load - it will schedule away for sure in the near future. 
That's what the "sync" bit means.

This change is necessary because with the sched-domains balancer we have
a much more sensitive cpu-load estimator, and in this particular context
of try_to_wake_up() the sync waker's effect will always be part of the
load.  Patch against your patch attached. 

In my testing there's an additional increase in bw_pipe numbers on a
dual P2 box, it went from 110-120 MB/sec to 120-130 MB/sec. 
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

kernel/sched.c

@@ -762,6 +762,13 @@ static int try_to_wake_up(task_t * p, unsigned int state, int sync)
 	load = source_load(cpu);
 	this_load = target_load(this_cpu);
 
+	/*
+	 * If sync wakeup then subtract the (maximum possible) effect of
+	 * the currently running task from the load of the current CPU:
+	 */
+	if (sync)
+		this_load -= SCHED_LOAD_SCALE;
+
 	/* Don't pull the task off an idle CPU to a busy one */
 	if (load < SCHED_LOAD_SCALE && load + this_load > SCHED_LOAD_SCALE
 			&& this_load > load)