Remove the extra code left over when the serial driver was changed
to require a clock. There is no fallback to 24Mhz as a clock is
now required.

Also remove a second call to of_clk_get which is unnecessary.
Signed-off-by: Tony Prisk <linux@prisktech.co.nz>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>