QSPI doesn't need any pullups of any sort,
let's remove them.
Signed-off-by: Felipe Balbi <balbi@ti.com>
Signed-off-by: Tony Lindgren <tony@atomide.com>