The problem was that join_buffer_size conflicted with
join_buffer_space_limit, which caused the query to be run without join
buffer. However this caused wrong results as the optimizer assumed
that hash+join buffer would ensure that the equi-join condition
would be satisfied, and didn't check it itself.

Fixed by not using join_buffer_space_limit when
optimize_join_buffer_size=off. This matches the documentation at
https://mariadb.com/kb/en/block-based-join-algorithms

Other things:
- Removed not used variable JOIN_TAB::join_buffer_size_limit
- Give an error if we cannot allocate a join buffer. This can
  only happen if the join_buffer variables are wrongly configured or
  we are running out of memory.
  In the future, instead of returning an error, we could properly
  convert the query plan that uses BNL-H join into one that doesn't
  use join buffering:
  make sure the equi-join condition is checked where appropriate.

Reviewer: Sergei Petrunia <sergey@mariadb.com>