MDEV-34307 On startup, [FATAL] InnoDB: Page ... still fixed or dirty

buf_pool_invalidate(): Properly wait for
os_aio_wait_until_no_pending_writes() to ensure so that there
are no pending buf_page_t::write_complete() or buf_page_write_complete()
operations. This will avoid a failure of buf_pool.assert_all_freed().

This bug should affect debug builds only. At this point, the
buf_pool.flush_list should be clear and all changes should have
been written out. The loop around buf_LRU_scan_and_free_block() should
have eventually completed and freed all pages as soon as
buf_page_t::write_complete() had a chance to release the page latches.

It is worth noting that buf_flush_wait() is working as intended.
As soon as buf_flush_page_cleaner() invokes
buf_pool.get_oldest_modification() it will observe that
buf_page_t::write_complete() had assigned oldest_modification_ to 1,
and remove such blocks from buf_pool.flush_list. Upon reaching
buf_pool.flush_list.count=0 the buf_flush_page_cleaner() will mark
itself idle and wake buf_flush_wait() by broadcasting
buf_pool.done_flush_list.

This regression was introduced in
commit a55b951e (MDEV-26827).

Reviewed by: Debarun Banerjee

storage/innobase/buf/buf0buf.cc

@@ -3700,15 +3700,13 @@ void buf_refresh_io_stats()
 All pages must be in a replaceable state (not modified or latched). */
 void buf_pool_invalidate()
 {
-	mysql_mutex_lock(&buf_pool.mutex);
-
 	/* It is possible that a write batch that has been posted
 	earlier is still not complete. For buffer pool invalidation to
 	proceed we must ensure there is NO write activity happening. */
 
-	ut_d(mysql_mutex_unlock(&buf_pool.mutex));
+	os_aio_wait_until_no_pending_writes(false);
 	ut_d(buf_pool.assert_all_freed());
-	ut_d(mysql_mutex_lock(&buf_pool.mutex));
+	mysql_mutex_lock(&buf_pool.mutex);
 
 	while (UT_LIST_GET_LEN(buf_pool.LRU)) {
 		buf_LRU_scan_and_free_block();