1,should use late_fini to kfree all resource otherwise
the released pointer maybe accessed in IRQ ip fini routine.

2,hwmgr should not be kfree by pem_fini which is invoked
by hw fini path.
Signed-off-by: Monk Liu <Monk.Liu@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>