Ditch the deffered list, lock, and workqueue handling. Just mark the
set as being blocking, so we are invoked from a workqueue already.
Signed-off-by: Jens Axboe <axboe@kernel.dk>