Clean up the unnecessarily complicated blacklist implementation
somewhat.
Signed-off-by: Johan Hovold <johan@kernel.org>