The problem was caused by the following scenario:

Subquery's table has two indexes, KEY a(a), KEY a_b(a,b)

- LATERAL DERIVED optimization decides to use index a.
  = The subquery uses ref access over key a.
- test_if_skip_sort_order() sees that KEY a_b satisfies the
  subquery's GROUP BY clause, and attempts to switch to it.
  = It fails to do so, because KEYUSE objects for index a_b
    are switched off.

Fixed by disallowing to change the ref access key if it uses KEYUSE
objects injected by LATERAL DERIVED optimization.