We first need to make sure no one else can get at us anymore,
before we can proceed to tear down all the datastructures.

Just a small step towards eventually the perfect unload code ...
Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20170714224656.6431-3-daniel.vetter@ffwll.ch