I should have let this soak for a while in linux-next, since we have at
least one board that hit a regression from it. Revert from 4.20-rc, and
we'll queue it for next merge window once regression is fixed.

This reverts commit 513eb985.
Signed-off-by: Olof Johansson <olof@lixom.net>