This reverts commit 06e26b75.

According to discussions, it causes a regression.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>