I'm pretty sure this optimisation is actually not a great idea,
and is racy with other things waiting for fences.

Just nuke it, there should be no need to do fence waits in a
busy CPU loop.
Signed-off-by: Dave Airlie <airlied@redhat.com>
Reviewed-by: Ben Skeggs <bskeggs@nvidia.com>
Signed-off-by: Danilo Krummrich <dakr@redhat.com>
Link: https://patchwork.freedesktop.org/patch/msgid/20240417054032.3145721-1-airlied@gmail.com