This removes a possible race condition and allows for further improvement
of the driver.
Signed-off-by: Alexandre Belloni <alexandre.belloni@free-electrons.com>