If the bit is already set, we don't need to reset it, and vice versa.
Because we don't need to make the caches dirty for that.
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>