No need to allocate and deallocate endpoint memory if the physical link is
down.

Link: https://lore.kernel.org/r/20200408064332.19377-3-mrangankar@marvell.comReviewed-by: Lee Duncan <lduncan@suse.com>
Signed-off-by: Manish Rangankar <mrangankar@marvell.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>