When removing a SA1111 device, we try to remove all child devices.
However, we must only remove our own RAB bus typed devices from the
tree, there may be other devices present which should not be touched.

This is necessary before we introduce gpiochip to SA1111 to avoid
incorrectly trying to remove the gpiochip device, leading to an oops
in __release_resource().
Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>