MDEV-32511 Assertion !os_aio_pending_writes() failed

In MemorySanitizer builds of 10.10 and 10.11, we would rather often
have the assertion fail in innodb_init() during mariadb-backup --prepare.
The assertion could also fail during InnoDB startup, but less often.

Before commit 685d958e in 10.8 the
log file cleanup after a successfully applied backup is different,
and the os_aio_pending_writes() assertion is in srv0start.cc.

IORequest::write_complete(): Invoke node->complete_write() before
releasing the page latch, so that a log checkpoint that is about to
execute concurrently will not miss a fdatasync() or fsync() on the
file, in case this was the first write since the last such call.

create_log_file(), srv_start(): Replace the debug assertion with
a debug check. For all intents and purposes, all writes could have
been completed but some write_io_callback() may not have invoked
io_slots::release() yet.

storage/innobase/fil/fil0fil.cc

@@ -2850,6 +2850,7 @@ void IORequest::write_complete(int io_error) const
   ut_ad(fil_validate_skip());
   ut_ad(node);
   ut_ad(is_write());
+  node->complete_write();
 
   if (!bpage)
   {
@@ -2862,7 +2863,6 @@ void IORequest::write_complete(int io_error) const
   else
     buf_page_write_complete(*this, io_error);
 
-  node->complete_write();
   node->space->release();
 }
 

storage/innobase/srv/srv0start.cc

@@ -244,10 +244,16 @@ static dberr_t create_log_file(bool create_new_db, lsn_t lsn,
 
 	DBUG_PRINT("ib_log", ("After innodb_log_abort_6"));
 	ut_ad(!os_aio_pending_reads());
-	ut_ad(!os_aio_pending_writes());
 	ut_d(mysql_mutex_lock(&buf_pool.flush_list_mutex));
 	ut_ad(!buf_pool.get_oldest_modification(0));
 	ut_d(mysql_mutex_unlock(&buf_pool.flush_list_mutex));
+	/* os_aio_pending_writes() may hold here if some
+	write_io_callback() did not release the slot yet.  However,
+	the page write itself must have completed, because the
+	buf_pool.flush_list is empty. In debug builds, we wait for
+	this to happen, hoping to get a hung process if this
+	assumption does not hold. */
+	ut_d(os_aio_wait_until_no_pending_writes(false));
 
 	DBUG_EXECUTE_IF("innodb_log_abort_7", return DB_ERROR;);
 	DBUG_PRINT("ib_log", ("After innodb_log_abort_7"));
@@ -1659,10 +1665,17 @@ dberr_t srv_start(bool create_new_db)
 			end of create_log_file(). */
 			ut_d(recv_no_log_write = true);
 			ut_ad(!os_aio_pending_reads());
-			ut_ad(!os_aio_pending_writes());
 			ut_d(mysql_mutex_lock(&buf_pool.flush_list_mutex));
 			ut_ad(!buf_pool.get_oldest_modification(0));
 			ut_d(mysql_mutex_unlock(&buf_pool.flush_list_mutex));
+			/* os_aio_pending_writes() may hold here if
+			some write_io_callback() did not release the
+			slot yet. However, the page write itself must
+			have completed, because the buf_pool.flush_list
+			is empty. In debug builds, we wait for this to
+			happen, hoping to get a hung process if this
+			assumption does not hold. */
+			ut_d(os_aio_wait_until_no_pending_writes(false));
 
 			DBUG_EXECUTE_IF("innodb_log_abort_3",
 					return(srv_init_abort(DB_ERROR)););