This could free things twice, just deref the GEM object
and hope its enough.
Signed-off-by: Dave Airlie <airlied@redhat.com>