Turns out this is not such a great idea. Once the SP clock is disabled,
it's not sufficient to just enable in order to bring the SP core back up.

It seems that the kernel has no business managing this clock. Just let
the firmware keep it enabled.

This reverts commit ed22cee9.

Link: https://lore.kernel.org/lkml/154783267051.169631.3197836544646625747@swboyd.mtv.corp.google.com/Signed-off-by: Lubomir Rintel <lkundrak@v3.sk>
Acked-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>
Signed-off-by: Stephen Boyd <sboyd@kernel.org>