irqdomain now supports removal of domains on exit so we can properly clean
up on deletion of a regmap irqchip.
Signed-off-by: Mark Brown <broonie@linaro.org>