Handle the error properly instead of calling the pci remove function.
Signed-off-by: Sudip Mukherjee <sudip@vectorindia.org>
Signed-off-by: Felipe Balbi <balbi@ti.com>