When fast user switching a lot eventually we get to the point,
where we were checking for the wrong thing in this function.
Signed-off-by: Dave Airlie <airlied@redhat.com>