Kbuild should descend into drivers/pinctrl/ only when CONFIG_PINCTRL
is enabled because everything under that directory depends on
CONFIG_PINCTRL.

We can avoid the conditional, ifeq ($(CONFIG_OF),y) ... endif.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>