Simplifying the code a bit.

The new implementation unifies the handling between drivers and so
results in waiting for all shared fernces in all cases.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Link: https://patchwork.freedesktop.org/patch/msgid/20211005113742.1101-26-christian.koenig@amd.com