We only really need it for a process going away, so move it to
those locations.
Signed-off-by: Jens Axboe <jens.axboe@oracle.com>