This reverts commit f2efc6e6.

This was fixed properly for 5.5, but came back via 5.4 merge
into drm-next, so revert it again.
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>