of_find_device_by_node() grabs a dev reference, so make sure we clear it
on error and remove.

Changes in v2:
- Added to the set (Jordan)

Cc: Jordan Crouse <jcrouse@codeaurora.org>
Reviewed-by: Jordan Crouse <jcrouse@codeaurora.org>
Signed-off-by: Sean Paul <seanpaul@chromium.org>
Link: https://patchwork.freedesktop.org/patch/msgid/20190523171653.138678-5-sean@poorly.run