This reverts commit 108d23e3.
It turns out this causes a regression on the OMAP, Marvell
and Renesas.
Reported-by: Tony Lindgren <tony@atomide.com>
Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reported-by: Jacopo Mondi <jacopo@jmondi.org>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>