Some Layerscape/QoriQ SoCs have input buffers which needs to be enabled
first. This was done in two different ways in the driver. Unify it.

This was tested on a LS1028A SoC.
Signed-off-by: Michael Walle <michael@walle.cc>
Signed-off-by: Bartosz Golaszewski <bgolaszewski@baylibre.com>