Also avoid use NULL pointer in error message.

v2-changes:
- use port pinter only after checking
Signed-off-by: Sonic Zhang <sonic.zhang@analog.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>