Avoid iterating the child devices. This should be a non-functional
change, but theoretically this might enable LSPCON on some extra ports
with buggy VBTs.

Cc: Ville Syrjälä <ville.syrjala@linux.intel.com>
Reviewed-by: Ville Syrjälä <ville.syrjala@linux.intel.com>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Link: https://patchwork.freedesktop.org/patch/msgid/4bbaff16abb3461ccb67abf1537f68bb50823390.1559308269.git.jani.nikula@intel.com