1. 04 Feb, 2024 16 commits
    • Sergei Golubchik's avatar
      auto-disable --ssl-verify-server-cert in clients, if · 853bdf57
      Sergei Golubchik authored
      * --ssl-verify-server-cert was not enabled explicitly, and
      * CA was not specified, and
      * fingerprint was not specified, and
      * protocol is TCP, and
      * no password was provided
      
      insecure passwordless logins are common in test environment, let's
      not break them. practically, it hardly makes sense to have strong
      MitM protection if an attacker can simply login without a password.
      
      Covers mariadb, mariadb-admin, mariadb-binlog, mariadb-dump
      853bdf57
    • Sergei Golubchik's avatar
      MDEV-31857 enable --ssl-verify-server-cert by default in the internal client · abcd23ad
      Sergei Golubchik authored
      enable ssl + ssl_verify_server_cert in the internal client too
      
      * fix replication tests to disable master_ssl_verify_server_cert
        because accounts are passwordless - except rpl.rpl_ssl1
      * fix federated/federatedx/connect to disable SSL_VERIFY_SERVER_CERT
        because they cannot configure an ssl connection
      * fix spider to disable ssl_verify_server_cert, if configuration
        says so, as spider _can_ configure an ssl connection
      * memory leak in embedded test-connect
      abcd23ad
    • Sergei Golubchik's avatar
      MDEV-31855 validate ssl certificates using client password in the internal client · e0c30390
      Sergei Golubchik authored
      port the client-side implementation from C/C to the internal client.
      add the test.
      e0c30390
    • Sergei Golubchik's avatar
      disable SSL via named pipes in the internal client · 386df879
      Sergei Golubchik authored
      because it doesn't work. CONC-635.
      386df879
    • Sergei Golubchik's avatar
      free mysql->connector_fd correctly in the internal client · 3c36ed18
      Sergei Golubchik authored
      it's not an ssl option, so shouldn't be in mysql_ssl_free(),
      which frees ssl options, and only unless CLIENT_REMEMBER_OPTIONS is set.
      
      mysql->connector_fd must be freed when mysql->net.vio is closed
      and fd becomes no longer valid
      3c36ed18
    • Sergei Golubchik's avatar
      change how self-signed certs are accepted by internal client · 2f13f7d7
      Sergei Golubchik authored
      use SSL_VERIFY_PEER with the "always ok" callback,
      instead of SSL_VERIFY_NONE with no callback.
      
      The latter doesn't work correctly in wolfSSL, it accepts self-signed
      certificates just fine (as in OpenSSL), but after that
      SSL_get_verify_result() returns X509_V_OK, while it returns an error
      (e.g. X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN) in OpenSSL.
      2f13f7d7
    • Sergei Golubchik's avatar
      cleanup: X509_check_host() in the internal client · 05a421eb
      Sergei Golubchik authored
      X509_check_host() and X509_check_ip_asc() exist in all
      supported SSL libraries
      
      in OpenSSL >= 1.0.2 and in the bundled WolfSSL
      
      And X509_free() handles NULL pointers all right.
      05a421eb
    • Sergei Golubchik's avatar
      cleanup: ssl handling in the internal rpl client · f4e174e1
      Sergei Golubchik authored
      * type of mi->ssl_verify_server_cert must be my_bool, because it's
        passed by address to mysql_options(), and the latter expects my_bool
      * explicitly disable ssl in MYSQL if mi->ssl is 0
      * remove dead code (`#ifdef NOT_USED`)
      * remove useless casts and checks replacing empty strings with NULL
        (new_VioSSLFd() does that internally)
      f4e174e1
    • Sergei Golubchik's avatar
      e951edd8
    • Sergei Golubchik's avatar
      enable --ssl in the server by default · ea921fd8
      Sergei Golubchik authored
      except in bootstrap
      ea921fd8
    • Sergei Golubchik's avatar
      MDEV-31856 use ephemeral ssl certificates · 9f93630d
      Sergei Golubchik authored
      if the server is started with --ssl but without neither --ssl-key nor
      --ssl-cert, let it automatically generate a self-signed certificate.
      It's generated in memory only and never saved to disk.
      9f93630d
    • Sergei Golubchik's avatar
      wrong error for bare --ssl on the server side · d33a8ab1
      Sergei Golubchik authored
      when neither --ssl-key nor --ssl-cert were set, the errror
      was "Private key does not match the certificate public key"
      
      changed to "Unable to get certificate"
      d33a8ab1
    • Sergei Golubchik's avatar
      cleanup · d772c4fb
      Sergei Golubchik authored
      d772c4fb
    • Sergei Golubchik's avatar
      test SSL MitM attack · 68f0af2b
      Sergei Golubchik authored
      verify that --ssl-verify-server-cert detects cert mismatch,
      but with --disable-ssl-verify-server-cert the connection succeeds
      68f0af2b
    • Sergei Golubchik's avatar
      client support for --ssl-fp and --ssl--fplist · bac0f899
      Sergei Golubchik authored
      implement --ssl-fp and --ssl-fplist for all clients.
      --ssl-fp takes one certificate fingerprint, for example,
      00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33
      
      --ssl-fplist takes a path to a file with one fingerprint per line.
      
      if the server's certificate fingerprint matches ssl-fp or is found
      in the file - the certificate is considered verified.
      If the fingerprint is specified but doesn't match - the connection
      is aborted independently from the --ssl-verify-server-cert
      bac0f899
    • Sergei Golubchik's avatar
      MDEV-31855 validate ssl certificates using client password · 1ef1bab9
      Sergei Golubchik authored
      if the client enabled --ssl-verify-server-cert, then
      the server certificate is verified as follows:
      
      * if --ssl-ca or --ssl-capath were specified, the cert must have
        a proper signature by the specified CA (or CA in the path)
        and the cert's hostname must match the server's hostname.
        If the cert isn't signed or a hostname is wrong - the
        connection is aborted.
      
      * if MARIADB_OPT_TLS_PEER_FP was used and the fingerprint matches,
        the connection is allowed, if it doesn't match - aborted.
      
      * If the connection uses unix socket or named pipes - it's allowed.
        (consistent with server's --require-secure-transport behavior)
      
      otherwise the cert is still in doubt, we don't know if we can trust
      it or there's an active MitM in progress.
      
      * If the user has provided no password or the server requested an
        authentication plugin that sends the password in cleartext -
        the connection is aborted.
      
      * Perform the authentication. If the server accepts the password,
        it'll send SHA2(scramble || password hash || cert fingerprint)
        with the OK packet.
      
      * Verify the SHA2 digest, if it matches - the connection is allowed,
        otherwise it's aborted.
      1ef1bab9
  2. 03 Feb, 2024 6 commits
  3. 02 Feb, 2024 1 commit
    • Vladislav Vaintroub's avatar
      MDEV-33075 Resolve server shutdown issues on macOS, Solaris, and FreeBSD · 2f5174e5
      Vladislav Vaintroub authored
      This commit addresses multiple server shutdown problems observed on macOS,
      Solaris, and FreeBSD:
      
      1. Corrected a non-portable assumption where socket shutdown was expected
      to wake up poll() with listening sockets in the main thread.
      
      Use more robust self-pipe to wake up poll() by writing to the pipe's write
      end.
      
      2. Fixed a random crash on macOS from pthread_kill(signal_handler)
      when the signal_handler was detached and the thread had already exited.
      
      Use more robust `kill(getpid(), SIGTERM)` to wake up the signal handler
      thread.
      
      3. Made sure, that signal handler thread always exits once `abort_loop` is
      set, and also calls `my_thread_end()` and clears `signal_thread_in_use`
      when exiting.
      
      This fixes warning "1 thread did not exit"  by `my_global_thread_end()`
      seen on FreeBSD/macOS when the process is terminated via signal.
      
      Additionally, the shutdown code underwent light refactoring
      for better readability and maintainability:
      - Modified `break_connect_loop()` to no longer wait for the main thread,
        aligning behavior with Windows (since 10.4).
      - Removed dead code related to the unused `USE_ONE_SIGNAL_HAND`
        preprocessor constant.
      - Eliminated support for `#ifndef HAVE_POLL` in `handle_connection_sockets`
        This code is also dead, since 10.4
      2f5174e5
  4. 29 Jan, 2024 6 commits
  5. 27 Jan, 2024 1 commit
    • Kristian Nielsen's avatar
      MDEV-4991: GTID binlog indexing · d039346a
      Kristian Nielsen authored
      Improve the performance of slave connect using B+-Tree indexes on each binlog
      file. The index allows fast lookup of a GTID position to the corresponding
      offset in the binlog file, as well as lookup of a position to find the
      corresponding GTID position.
      
      This eliminates a costly sequential scan of the starting binlog file
      to find the GTID starting position when a slave connects. This is
      especially costly if the binlog file is not cached in memory (IO
      cost), or if it is encrypted or a lot of slaves connect simultaneously
      (CPU cost).
      
      The size of the index files is generally less than 1% of the binlog data, so
      not expected to be an issue.
      
      Most of the work writing the index is done as a background task, in
      the binlog background thread. This minimises the performance impact on
      transaction commit. A simple global mutex is used to protect index
      reads and (background) index writes; this is fine as slave connect is
      a relatively infrequent operation.
      
      Here are the user-visible options and status variables. The feature is on by
      default and is expected to need no tuning or configuration for most users.
      
      binlog_gtid_index
        On by default. Can be used to disable the indexes for testing purposes.
      
      binlog_gtid_index_page_size (default 4096)
        Page size to use for the binlog GTID index. This is the size of the nodes
        in the B+-tree used internally in the index. A very small page-size (64 is
        the minimum) will be less efficient, but can be used to stress the
        BTree-code during testing.
      
      binlog_gtid_index_span_min (default 65536)
        Control sparseness of the binlog GTID index. If set to N, at most one
        index record will be added for every N bytes of binlog file written.
        This can be used to reduce the number of records in the index, at
        the cost only of having to scan a few more events in the binlog file
        before finding the target position
      
      Two status variables are available to monitor the use of the GTID indexes:
      
        Binlog_gtid_index_hit
        Binlog_gtid_index_miss
      
      The "hit" status increments for each successful lookup in a GTID index.
      The "miss" increments when a lookup is not possible. This indicates that the
      index file is missing (eg. binlog written by old server version
      without GTID index support), or corrupt.
      Signed-off-by: default avatarKristian Nielsen <knielsen@knielsen-hq.org>
      d039346a
  6. 24 Jan, 2024 1 commit
  7. 23 Jan, 2024 1 commit
  8. 22 Jan, 2024 1 commit
    • Brandon Nesterenko's avatar
      MDEV-7850: Extend GTID Binlog Events with Thread Id · c37b2087
      Brandon Nesterenko authored
      This patch augments Gtid_log_event with the user thread-id.
      In particular that compensates for the loss of this info in
      Rows_log_events.
      
      Gtid_log_event::thread_id gets visible in mysqlbinlog output like
      
        #231025 16:21:45 server id 1  end_log_pos 537 CRC32 0x1cf1d963  GTID 0-1-2 ddl thread_id=10
      
      as 64 bit unsigned integer.
      
      While the size of Gtid event has grown by 8-9 bytes
      replication from OLD <-> NEW is not affected by it.
      
      This work was started by the late Sujatha Sivakumar.
      Brandon Nesterenko took it over, reviewed initial patches and extended
      the work.
      
      Reviewed-by: <andrei.elkin@mariadb.com>
      c37b2087
  9. 18 Jan, 2024 1 commit
    • Libing Song's avatar
      MDEV-32894 mysqlbinlog flashback support binlog_row_image FULL_NODUP mode · 8bf9f218
      Libing Song authored
      Summary
      =======
      With FULL_NODUP mode, before image inclues all columns and after
      image inclues only the changed columns. flashback will swap the
      value of changed columns from after image to before image.
      For example:
        BI: c1, c2, c3_old, c4_old
        AI: c3_new, c4_new
      flashback will reconstruct the before and after images to
        BI: c1, c2, c3_new, c4_new
        AI: c3_old, c4_old
      
      Implementation
      ==============
      When parsing the before and after image, position and length of
      the fields are collected into ai_fields and bi_fields, if it is an
      Update_rows_event and the after image doesn't includes all columns.
      
      The changed fields are swapped between bi_fields and ai_fields.
      Then it recreates the before image and after image by using
      bi_fields and ai_fields. nullbit will be set to 1 if the
      field is NULL, otherwise nullbit will be 0.
      
      It also optimized flashback a little bit.
      - calc_row_event_length is used instead of print_verbose_one_row
      - swap_buff1 and swap_buff2 are removed.
      8bf9f218
  10. 17 Jan, 2024 1 commit
  11. 12 Jan, 2024 1 commit
    • Libing Song's avatar
      MDEV-33049 Assertion `marked_for_write_or_computed()' failed in bool · be6d48fd
      Libing Song authored
                 Field_new_decimal::store_value(const my_decimal*, int*)
      
      Analysis
      ========
      When rpl applier is unpacking a before row image, Field::reset() will be
      called before setting a field to null if null bit of the field is set in
      the row image. For Field_new_decimal::reset(), it calls
      Field_new_decimal::store_value() to reset the value. store_value() asserts
      that the field is in the write_set bitmap since it thinks the field is
      updating.
      
      But that is not true for the row image generated in FULL_NODUP
      mode. In the mode, the before image includes all fields and the after
      image includes only updated fields.
      
      Fix
      ===
      In the case unpacking binlog row images, the assertion is meaningless.
      So the unpacking field is marked in write_set temporarily to avoid the
      assertion failure.
      be6d48fd
  12. 10 Jan, 2024 4 commits