Avoids potential issues if we read back unexpected values during
a read/modify/write cycle.
Signed-off-by: Mark Brown <broonie@opensource.wolfsonmicro.com>