In the TB10x pin database, a port index of -1 is used to indicate
unmuxed GPIO pin groups. This bug fixes a 'cast to unsigned' bug of
this value.

Thanks to Dan Carpenter for highlighting this.

CC: Dan Carpenter <dan.carpenter@oracle.com>
Signed-off-by: Christian Ruppert <christian.ruppert@abilis.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>