Added versioning of row data

Will in future changeset (soon) av versioning of status variables (number of rows) and index
Changed some LEX_STRING to LEX_CUSTRING to avoid casts and warnings
Removed some not needed variables (as noticed by Guilhem)


include/maria.h:
  Added prototypes for maria_chk_init_for_check(), maria_versioning() and maria_ignore_trids()
include/my_base.h:
  Add new error HA_ERR_ROW_NOT_VISIBLE
include/myisamchk.h:
  Added variables for checking visibility of rows during maria_chk
include/thr_lock.h:
  Changed argument type from int to my_bool for get_status
  Added variable allow_multiple_concurrent_insert, to signal if table supports multiple concurrent inserts
mysql-test/r/maria-page-checksum.result:
  Added missing drop table
mysql-test/t/maria-page-checksum.test:
  Added missing drop table
mysys/my_handler.c:
  Added new error messages
mysys/thr_lock.c:
  Added support for multiple concurrent inserts, if table handler supports it
sql/sql_yacc.yy:
  Added LOCK TABLE table_name WRITE CONCURRENT
  This was added (temporarly?) to be able to check versioning with Maria
storage/csv/ha_tina.cc:
  Updated parameter for get_status
storage/maria/ha_maria.cc:
  Added calls to maria_chk_init_status()
  Fixed call to ma_control_file_open()
storage/maria/ma_blockrec.c:
  Changed some LEX_STRING to LEX_CUSTRING to avoid casts and warnings
  Changed back some 'header' parameters to const char*
  Removed some casts
  
  Added support for versioning:
  - If info->row_flag & ROW_FLAG_TRANSID is set, store transaction id together with the row
  - When reading rows, check if rows are visible. Give error if not
  - When scanning table, ignore not visible rows
  - Added function parameters to some functions, to be able to call _ma_compact_block_page() with different parameters depending of if the page is a HEAD or TAIL page
  - _ma_compact_block_page() deletes transaction id's that are visible by all running transactions
  - Added functions for thr_lock() to enable multiple concurrent inserts
  - Added helper function 'mysql_versioning()' to enable/disable versioning
  - Added helper function maria_ignore_trids(), used by maria_chk and maria_pack to see all rows.
storage/maria/ma_blockrec.h:
  Updated parameters for some functions.
  Added new functions to read/store state with thr_lock
storage/maria/ma_check.c:
  Enable handling of transaction id's in rows
  Give a readable error if a table contains a transation id that makes rows not visible
storage/maria/ma_control_file.c:
  Added option to not give warning if control file doesn't exists.
storage/maria/ma_control_file.h:
  Updated parameter lists for ma_control_file_open()
storage/maria/ma_delete.c:
  Removed not used variable (suggestion by Guilhem)
storage/maria/ma_locking.c:
  Changed type of argument from int -> my_bool
storage/maria/ma_open.c:
  Removed not used variables 'key_write_undo_lsn' and 'key_delete_undo_lsn'
  Added new thr_lock interface functions for BLOCK_RECORD to enable multiple concurrent insert
storage/maria/ma_test1.c:
  Added option --versioning (-C) to check versioning
storage/maria/ma_test2.c:
  Added option -C to check versioning
storage/maria/ma_test_recovery:
  Forward argumetns to ma_test_recovery.pl
storage/maria/ma_write.c:
  Removed not used variable key_write_undo_lsn
storage/maria/maria_chk.c:
  Always read control file (if exist) at start
  Initialize checking of tables by calling maria_chk_init_for_check()
  In verbose mode and in case of error, print max found transaction id
storage/maria/maria_def.h:
  Added Trid to MARIA_ROW to be able to check transaction id for found row
  Moved 'base_length' from MARIA_ROW to MARIA_HA to be able to handle different base length (with and without TRANSID) without if's
  Added default row_flag to MARIA_HA for the same reason
  Changed LEX_STRING -> LEX_CUSTRING to avoid casts in ma_blockrec.c
  Removed not needed variables key_write_undo_lsn and key_delete_undo_lsn
  Added prototypes for new functions and fixed those that had changed
storage/maria/maria_pack.c:
  Ensure we can read all rows from the file, independent of the used transaction id
storage/maria/maria_read_log.c:
  Updated arguments to ma_control_file_open()
storage/maria/trnman.c:
  If we have only one transaction, fixed that min_read_from contains current transaction
  Fixed that trnman_can_read_from() returns that row is readable if it was written by current transaction
storage/maria/unittest/ma_control_file-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_all-t:
  Added test of versioning
  Removed printing of one extra space
storage/maria/unittest/ma_test_loghandler-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_loghandler_nologs-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_loghandler_pagecache-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_loghandler_purge-t.c:
  Updated arguments to ma_control_file_open()
storage/maria/unittest/ma_test_recovery.expected:
  Updated file with result from new tests
storage/maria/unittest/ma_test_recovery.pl:
  Added options --abort-on-error and --verbose
  In case of --verbose, print all excuted shell commands
  Added test of versioning
storage/myisam/mi_locking.c:
  Updated type of parameter
storage/myisam/myisamdef.h:
  Updated type of parameter
mysql-test/r/maria-mvcc.result:
  New BitKeeper file ``mysql-test/r/maria-mvcc.result''
mysql-test/t/maria-mvcc.test:
  New BitKeeper file ``mysql-test/t/maria-mvcc.test''

include/maria.h

@@ -391,6 +391,7 @@ typedef struct st_maria_sort_param
 
 /* functions in maria_check */
 void maria_chk_init(HA_CHECK *param);
+void maria_chk_init_for_check(HA_CHECK *param, MARIA_HA *info);
 int maria_chk_status(HA_CHECK *param, MARIA_HA *info);
 int maria_chk_del(HA_CHECK *param, MARIA_HA *info, ulonglong test_flag);
 int maria_chk_size(HA_CHECK *param, MARIA_HA *info);
@@ -432,6 +433,8 @@ int maria_assign_to_pagecache(MARIA_HA *info, ulonglong key_map,
 void maria_change_pagecache(PAGECACHE *old_key_cache,
 			    PAGECACHE *new_key_cache);
 int maria_preload(MARIA_HA *info, ulonglong key_map, my_bool ignore_leaves);
+void maria_versioning(MARIA_HA *info, my_bool versioning);
+void maria_ignore_trids(MARIA_HA *info);
 
 /* fulltext functions */
 FT_INFO *maria_ft_init_search(uint,void *, uint, uchar *, uint,

include/my_base.h

@@ -440,9 +440,12 @@ enum ha_base_keytype {
 #define HA_ERR_INITIALIZATION     173    /* Error during initialization */
 #define HA_ERR_FILE_TOO_SHORT	  174	 /* File too short */
 #define HA_ERR_WRONG_CRC	  175	 /* Wrong CRC on page */
-#define HA_ERR_ROWS_EVENT_APPLY   176    /* The event could not be processed */
-                                         /* no other hanlder error happened  */
-#define HA_ERR_LAST               176    /* Copy of last error nr */
+
+/* The event could not be processed; no other handler error happened  */
+#define HA_ERR_ROWS_EVENT_APPLY   176
+#define HA_ERR_ROW_NOT_VISIBLE    177
+
+#define HA_ERR_LAST               177    /* Copy of last error nr */
 
 /* Number of different errors */
 #define HA_ERR_ERRORS            (HA_ERR_LAST - HA_ERR_FIRST + 1)

include/myisamchk.h

@@ -134,6 +134,11 @@ typedef struct st_handler_check_param
   ha_checksum tmp_record_checksum;
   ulonglong   org_key_map;
   ulonglong   testflag;
+
+  /* Following is used to check if rows are visible */
+  ulonglong max_trid, max_found_trid;
+  ulonglong not_visible_rows_found;
+
   size_t use_buffers, read_buffer_length, write_buffer_length;
   size_t sort_buffer_length, sort_key_blocks;
   ulong rec_per_key_part[HA_MAX_KEY_SEG * HA_MAX_POSSIBLE_KEY];

include/thr_lock.h

@@ -123,11 +123,12 @@ typedef struct st_thr_lock {
   /* write_lock_count is incremented for write locks and reset on read locks */
   ulong write_lock_count;
   uint read_no_write_count;
-  void (*get_status)(void*, int);	/* When one gets a lock */
+  void (*get_status)(void*, my_bool);	/* When one gets a lock */
   void (*copy_status)(void*,void*);
   void (*update_status)(void*);		/* Before release of write */
   void (*restore_status)(void*);         /* Before release of read */
   my_bool (*check_status)(void *);
+  my_bool allow_multiple_concurrent_insert;
 } THR_LOCK;
 
 

mysql-test/r/maria-mvcc.result

+drop table if exists t1;
+create table t1 (i int) engine=maria;
+show create table t1;
+Table	Create Table
+t1	CREATE TABLE `t1` (
+  `i` int(11) DEFAULT NULL
+) ENGINE=MARIA DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1
+lock tables t1 write concurrent;
+insert into t1 values (1);
+insert into t1 values (2);
+/* should see 1 and 2 */
+select i from t1;
+i
+1
+2
+/* should see nothing */
+select i from t1;
+i
+lock tables t1 write concurrent;
+insert into t1 values (3);
+insert into t1 values (4);
+/* should see 3 and 4 */
+select i from t1;
+i
+3
+4
+unlock tables;
+lock tables t1 write concurrent;
+insert into t1 values (5);
+/* should see 3, 4 and 5 */
+select i from t1;
+i
+3
+4
+5
+insert into t1 values (6);
+/* Should see 1, 2, 6 */
+select i from t1;
+i
+1
+2
+6
+unlock tables;
+lock tables t1 write concurrent;
+/* Should see 1, 2, 3, 4 and 6 */
+select i from t1;
+i
+1
+2
+3
+4
+6
+/* should see 3, 4, 5 */
+select i from t1;
+i
+3
+4
+5
+unlock tables;
+/* should see 1, 2, 3, 4, 5, 6 */
+select i from t1;
+i
+1
+2
+3
+4
+5
+6
+unlock tables;
+/* should see 1, 2, 3, 4, 5, 6 */
+select i from t1;
+i
+1
+2
+3
+4
+5
+6
+drop table t1;

mysql-test/r/maria-page-checksum.result

+drop table if exists t1;
 select @@global.maria_page_checksum;
 @@global.maria_page_checksum
 1

mysql-test/t/maria-mvcc.test

+#
+# Testing insert and select on a table with two threads
+# using locking
+#
+
+-- source include/have_maria.inc
+
+--disable_warnings
+drop table if exists t1;
+--enable_warnings
+
+connect (con1,localhost,root,,);
+connection con1;
+
+create table t1 (i int) engine=maria;
+show create table t1;
+lock tables t1 write concurrent;
+insert into t1 values (1);
+insert into t1 values (2);
+/* should see 1 and 2 */
+select i from t1;
+
+connect (con2,localhost,root,,);
+connection con2;
+/* should see nothing */
+select i from t1;
+lock tables t1 write concurrent;
+insert into t1 values (3);
+insert into t1 values (4);
+/* should see 3 and 4 */
+select i from t1;
+unlock tables;
+lock tables t1 write concurrent;
+insert into t1 values (5);
+/* should see 3, 4 and 5 */
+select i from t1;
+
+connection con1;
+insert into t1 values (6);
+/* Should see 1, 2, 6 */
+select i from t1;
+unlock tables;
+lock tables t1 write concurrent;
+/* Should see 1, 2, 3, 4 and 6 */
+select i from t1;
+
+connection con2;
+/* should see 3, 4, 5 */
+select i from t1;
+unlock tables;
+/* should see 1, 2, 3, 4, 5, 6 */
+select i from t1;
+
+connection con1;
+unlock tables;
+/* should see 1, 2, 3, 4, 5, 6 */
+select i from t1;
+
+drop table t1;

mysql-test/t/maria-page-checksum.test

+-- source include/have_maria.inc
+
+--disable_warnings
+drop table if exists t1;
+--enable_warnings
+
 select @@global.maria_page_checksum;
 
 --echo # iteration 1

mysys/my_handler.c

@@ -629,6 +629,8 @@ static const char *handler_error_messages[]=
   "Got a fatal error during initialzaction of handler",
   "File to short; Expected more data in file",
   "Read page with wrong checksum"
+  "Could not apply row event",
+  "Row is not visible by the current transaction",
 };
 
 

mysys/thr_lock.c

@@ -69,9 +69,11 @@ get_status:
 	for concurrent reads.
 
 The lock algorithm allows one to have one TL_WRITE_ALLOW_READ,
-TL_WRITE_CONCURRENT_INSERT or one TL_WRITE_DELAYED lock at the same time as
-multiple read locks.
+TL_WRITE_CONCURRENT_INSERT or one TL_WRITE_DELAYED lock at the same
+time as multiple read locks.
 
+In addition, if lock->allow_multiple_concurrent_insert is set then there can
+be any number of TL_WRITE_CONCURRENT_INSERT locks aktive at the same time.
 */
 
 #if !defined(MAIN) && !defined(DBUG_OFF) && !defined(EXTRA_DEBUG)
@@ -152,7 +154,8 @@ static int check_lock(struct st_lock_list *list, const char* lock_type,
       }
       if (same_owner &&
           !thr_lock_owner_equal(data->owner, first_owner) &&
-	  last_lock_type != TL_WRITE_ALLOW_WRITE)
+	  last_lock_type != TL_WRITE_ALLOW_WRITE &&
+          last_lock_type != TL_WRITE_CONCURRENT_INSERT)
       {
 	fprintf(stderr,
 		"Warning: Found locks from different threads in %s: %s\n",
@@ -205,7 +208,7 @@ static void check_locks(THR_LOCK *lock, const char *where,
       THR_LOCK_DATA *data;
       for (data=lock->read.data ; data ; data=data->next)
       {
-	if ((int) data->type == (int) TL_READ_NO_INSERT)
+	if (data->type == TL_READ_NO_INSERT)
 	  count++;
         /* Protect against infinite loop. */
         DBUG_ASSERT(count <= lock->read_no_write_count);
@@ -254,7 +257,22 @@ static void check_locks(THR_LOCK *lock, const char *where,
 	}	      
       }
       else
-      {						/* Have write lock */
+      {
+        /* We have at least one write lock */
+        if (lock->write.data->type == TL_WRITE_CONCURRENT_INSERT)
+        {
+          THR_LOCK_DATA *data;
+          for (data=lock->write.data->next ; data ; data=data->next)
+          {
+            if (data->type != TL_WRITE_CONCURRENT_INSERT)
+            {
+              fprintf(stderr,
+                      "Warning at '%s': Found TL_WRITE_CONCURRENT_INSERT lock mixed with other write locks\n",
+                      where);
+              break;
+            }
+          }
+        }
 	if (lock->write_wait.data)
 	{
 	  if (!allow_no_locks && 
@@ -514,7 +532,8 @@ thr_lock(THR_LOCK_DATA *data, THR_LOCK_OWNER *owner,
     /* Request for READ lock */
     if (lock->write.data)
     {
-      /* We can allow a read lock even if there is already a write lock
+      /*
+        We can allow a read lock even if there is already a write lock
 	 on the table in one the following cases:
 	 - This thread alread have a write lock on the table
 	 - The write lock is TL_WRITE_ALLOW_READ or TL_WRITE_DELAYED
@@ -558,11 +577,11 @@ thr_lock(THR_LOCK_DATA *data, THR_LOCK_OWNER *owner,
       (*lock->read.last)=data;			/* Add to running FIFO */
       data->prev=lock->read.last;
       lock->read.last= &data->next;
-      if (lock->get_status)
-	(*lock->get_status)(data->status_param, 0);
       if (lock_type == TL_READ_NO_INSERT)
 	lock->read_no_write_count++;
       check_locks(lock,"read lock with no write locks",0);
+      if (lock->get_status)
+	(*lock->get_status)(data->status_param, 0);
       statistic_increment(locks_immediate,&THR_LOCK_lock);
       goto end;
     }
@@ -626,16 +645,18 @@ thr_lock(THR_LOCK_DATA *data, THR_LOCK_OWNER *owner,
 	The following test will not work if the old lock was a
 	TL_WRITE_ALLOW_WRITE, TL_WRITE_ALLOW_READ or TL_WRITE_DELAYED in
 	the same thread, but this will never happen within MySQL.
+
+        The idea is to allow us to get a lock at once if we already have
+        a write lock or if there is no pending write locks and if all
+        write locks are of the same type and are either
+        TL_WRITE_ALLOW_WRITE or TL_WRITE_CONCURRENT_INSERT
       */
       if (thr_lock_owner_equal(data->owner, lock->write.data->owner) ||
-	  (lock_type == TL_WRITE_ALLOW_WRITE &&
-	   !lock->write_wait.data &&
-	   lock->write.data->type == TL_WRITE_ALLOW_WRITE))
+          (!lock->write_wait.data && lock_type == lock->write.data->type &&
+           (lock_type == TL_WRITE_ALLOW_WRITE ||
+            (lock_type == TL_WRITE_CONCURRENT_INSERT &&
+             lock->allow_multiple_concurrent_insert))))
       {
-	/*
-          We have already got a write lock or all locks are
-          TL_WRITE_ALLOW_WRITE
-        */
         DBUG_PRINT("info", ("write_wait.data: 0x%lx  old_type: %d",
                             (ulong) lock->write_wait.data,
                             lock->write.data->type));
@@ -644,8 +665,9 @@ thr_lock(THR_LOCK_DATA *data, THR_LOCK_OWNER *owner,
 	data->prev=lock->write.last;
 	lock->write.last= &data->next;
 	check_locks(lock,"second write lock",0);
-	if (data->lock->get_status)
-	  (*data->lock->get_status)(data->status_param, 0);
+	if (lock->get_status)
+	  (*lock->get_status)(data->status_param,
+                              lock_type == TL_WRITE_CONCURRENT_INSERT);
 	statistic_increment(locks_immediate,&THR_LOCK_lock);
 	goto end;
       }
@@ -678,8 +700,8 @@ thr_lock(THR_LOCK_DATA *data, THR_LOCK_OWNER *owner,
 	  (*lock->write.last)=data;		/* Add as current write lock */
 	  data->prev=lock->write.last;
 	  lock->write.last= &data->next;
-	  if (data->lock->get_status)
-	    (*data->lock->get_status)(data->status_param, concurrent_insert);
+	  if (lock->get_status)
+	    (*lock->get_status)(data->status_param, concurrent_insert);
 	  check_locks(lock,"only write lock",0);
 	  statistic_increment(locks_immediate,&THR_LOCK_lock);
 	  goto end;
@@ -809,7 +831,6 @@ static void wake_up_waiters(THR_LOCK *lock)
 {
   THR_LOCK_DATA *data;
   enum thr_lock_type lock_type;
-
   DBUG_ENTER("wake_up_waiters");
 
   if (!lock->write.data)			/* If no active write locks */
@@ -1372,8 +1393,8 @@ my_bool thr_upgrade_write_delay_lock(THR_LOCK_DATA *data)
   {
     if (!lock->read.data)			/* No read locks */
     {						/* We have the lock */
-      if (data->lock->get_status)
-	(*data->lock->get_status)(data->status_param, 0);
+      if (lock->get_status)
+	(*lock->get_status)(data->status_param, 0);
       pthread_mutex_unlock(&lock->mutex);
       DBUG_RETURN(0);
     }
@@ -1511,7 +1532,7 @@ struct st_test {
   enum thr_lock_type lock_type;
 };
 
-THR_LOCK locks[5];			/* 4 locks */
+THR_LOCK locks[6];			/* Number of locks +1 */
 
 struct st_test test_0[] = {{0,TL_READ}};	/* One lock */
 struct st_test test_1[] = {{0,TL_READ},{0,TL_WRITE}}; /* Read and write lock of lock 0 */
@@ -1531,9 +1552,20 @@ struct st_test test_14[] = {{0,TL_WRITE_CONCURRENT_INSERT},{1,TL_READ}};
 struct st_test test_15[] = {{0,TL_WRITE_ALLOW_WRITE},{1,TL_READ}};
 struct st_test test_16[] = {{0,TL_WRITE_ALLOW_WRITE},{1,TL_WRITE_ALLOW_WRITE}};
 
-struct st_test *tests[] = {test_0,test_1,test_2,test_3,test_4,test_5,test_6,
-			   test_7,test_8,test_9,test_10,test_11,test_12,
-			   test_13,test_14,test_15,test_16};
+struct st_test test_17[] = {{5,TL_WRITE_CONCURRENT_INSERT}};
+struct st_test test_18[] = {{5,TL_WRITE_CONCURRENT_INSERT}};
+struct st_test test_19[] = {{5,TL_READ}};
+struct st_test test_20[] = {{5,TL_READ_NO_INSERT}};
+struct st_test test_21[] = {{5,TL_WRITE}};
+
+
+struct st_test *tests[]=
+{
+  test_0, test_1, test_2, test_3, test_4, test_5, test_6, test_7, test_8,
+  test_9, test_10, test_11, test_12, test_13, test_14, test_15, test_16,
+  test_17, test_18, test_19, test_20, test_21
+};
+
 int lock_counts[]= {sizeof(test_0)/sizeof(struct st_test),
 		    sizeof(test_1)/sizeof(struct st_test),
 		    sizeof(test_2)/sizeof(struct st_test),
@@ -1550,7 +1582,12 @@ int lock_counts[]= {sizeof(test_0)/sizeof(struct st_test),
 		    sizeof(test_13)/sizeof(struct st_test),
 		    sizeof(test_14)/sizeof(struct st_test),
 		    sizeof(test_15)/sizeof(struct st_test),
-		    sizeof(test_16)/sizeof(struct st_test)
+		    sizeof(test_16)/sizeof(struct st_test),
+		    sizeof(test_17)/sizeof(struct st_test),
+		    sizeof(test_18)/sizeof(struct st_test),
+		    sizeof(test_19)/sizeof(struct st_test),
+		    sizeof(test_20)/sizeof(struct st_test),
+		    sizeof(test_21)/sizeof(struct st_test)
 };
 
 
@@ -1594,7 +1631,6 @@ static void *test_thread(void *arg)
 
   printf("Thread %s (%d) started\n",my_thread_name(),param); fflush(stdout);
 
-
   thr_lock_info_init(&lock_info);
   thr_lock_owner_init(&owner, &lock_info);
   for (i=0; i < lock_counts[param] ; i++)
@@ -1640,7 +1676,8 @@ int main(int argc __attribute__((unused)),char **argv __attribute__((unused)))
 {
   pthread_t tid;
   pthread_attr_t thr_attr;
-  int i,*param,error;
+  int *param,error;
+  uint i;
   MY_INIT(argv[0]);
   if (argc > 1 && argv[1][0] == '-' && argv[1][1] == '#')
     DBUG_PUSH(argv[1]+2);
@@ -1660,13 +1697,14 @@ int main(int argc __attribute__((unused)),char **argv __attribute__((unused)))
     exit(1);
   }
 
-  for (i=0 ; i < (int) array_elements(locks) ; i++)
+  for (i=0 ; i < array_elements(locks) ; i++)
   {
     thr_lock_init(locks+i);
     locks[i].check_status= test_check_status;
     locks[i].update_status=test_update_status;
     locks[i].copy_status=  test_copy_status;
     locks[i].get_status=   test_get_status;
+    locks[i].allow_multiple_concurrent_insert= 1;
   }
   if ((error=pthread_attr_init(&thr_attr)))
   {
@@ -1692,7 +1730,7 @@ int main(int argc __attribute__((unused)),char **argv __attribute__((unused)))
 #ifdef HAVE_THR_SETCONCURRENCY
   VOID(thr_setconcurrency(2));
 #endif
-  for (i=0 ; i < (int) array_elements(lock_counts) ; i++)
+  for (i=0 ; i < array_elements(lock_counts) ; i++)
   {
     param=(int*) malloc(sizeof(int));
     *param=i;
@@ -1724,7 +1762,7 @@ int main(int argc __attribute__((unused)),char **argv __attribute__((unused)))
   }
   if ((error=pthread_mutex_unlock(&LOCK_thread_count)))
     fprintf(stderr,"Got error: %d from pthread_mutex_unlock\n",error);
-  for (i=0 ; i < (int) array_elements(locks) ; i++)
+  for (i=0 ; i < array_elements(locks) ; i++)
     thr_lock_delete(locks+i);
 #ifdef EXTRA_DEBUG
   if (found_errors)

sql/sql_yacc.yy

@@ -11179,6 +11179,16 @@ table_lock:
 lock_option:
           READ_SYM               { $$= TL_READ_NO_INSERT; }
         | WRITE_SYM              { $$= TL_WRITE_DEFAULT; }
+        | WRITE_SYM CONCURRENT
+          {
+#ifdef HAVE_QUERY_CACHE
+            if (Lex->sphead != 0)
+             $$= TL_WRITE_DEFAULT;
+           else
+#endif
+             $$= TL_WRITE_CONCURRENT_INSERT;
+          }
+
         | LOW_PRIORITY WRITE_SYM { $$= TL_WRITE_LOW_PRIORITY; }
         | READ_SYM LOCAL_SYM     { $$= TL_READ; }
         ;

storage/csv/ha_tina.cc

@@ -715,7 +715,7 @@ const char **ha_tina::bas_ext() const
   for CSV engine. For more details see mysys/thr_lock.c
 */
 
-void tina_get_status(void* param, int concurrent_insert)
+void tina_get_status(void* param, my_bool concurrent_insert)
 {
   ha_tina *tina= (ha_tina*) param;
   tina->get_status();

storage/maria/ha_maria.cc

@@ -951,7 +951,8 @@ int ha_maria::check(THD * thd, HA_CHECK_OPT * check_opt)
                                               0)))))
     return HA_ADMIN_ALREADY_DONE;
 
-  error= maria_chk_status(&param, file);                // Not fatal
+  maria_chk_init_for_check(&param, file);
+  (void) maria_chk_status(&param, file);                // Not fatal
   error= maria_chk_size(&param, file);
   if (!error)
     error|= maria_chk_del(&param, file, param.testflag);
@@ -2768,7 +2769,7 @@ static int ha_maria_init(void *p)
   maria_hton->flags= HTON_CAN_RECREATE | HTON_SUPPORT_LOG_TABLES;
   bzero(maria_log_pagecache, sizeof(*maria_log_pagecache));
   maria_tmpdir= &mysql_tmpdir_list;             /* For REDO */
-  res= maria_init() || ma_control_file_open(TRUE) ||
+  res= maria_init() || ma_control_file_open(TRUE, TRUE) ||
     !init_pagecache(maria_pagecache,
                     (size_t) pagecache_buffer_size, pagecache_division_limit,
                     pagecache_age_threshold, maria_block_size, 0) ||

storage/maria/ma_blockrec.h

@@ -63,14 +63,13 @@
 
 #define PAGE_TYPE_MASK 7
 enum en_page_type { UNALLOCATED_PAGE, HEAD_PAGE, TAIL_PAGE, BLOB_PAGE, MAX_PAGE_TYPE };
+#define PAGE_CAN_BE_COMPACTED   128             /* Bit in PAGE_TYPE */
 
 #define PAGE_TYPE_OFFSET        LSN_SIZE
 #define DIR_COUNT_OFFSET        (LSN_SIZE+PAGE_TYPE_SIZE)
 #define DIR_FREE_OFFSET         (DIR_COUNT_OFFSET+DIR_COUNT_SIZE)
 #define EMPTY_SPACE_OFFSET      (DIR_FREE_OFFSET+DIR_FREE_SIZE)
 
-#define PAGE_CAN_BE_COMPACTED   128             /* Bit in PAGE_TYPE */
-
 /* Bits used for flag uchar (one byte, first in record) */
 #define ROW_FLAG_TRANSID                1
 #define ROW_FLAG_VER_PTR                2
@@ -174,7 +173,8 @@ my_bool _ma_write_abort_block_record(MARIA_HA *info);
 my_bool _ma_compare_block_record(register MARIA_HA *info,
                                  register const uchar *record);
 void    _ma_compact_block_page(uchar *buff, uint block_size, uint rownr,
-                               my_bool extend_block);
+                               my_bool extend_block, TrID min_read_from,
+                               uint min_row_length);
 TRANSLOG_ADDRESS
 maria_page_get_lsn(uchar *page, pgcache_page_no_t page_no, uchar* data_ptr);
 
@@ -243,9 +243,9 @@ my_bool _ma_apply_redo_bitmap_new_page(MARIA_HA *info, LSN lsn,
 my_bool _ma_apply_undo_row_insert(MARIA_HA *info, LSN undo_lsn,
                                   const uchar *header);
 my_bool _ma_apply_undo_row_delete(MARIA_HA *info, LSN undo_lsn,
-                                  uchar *header, size_t length);
+                                  const uchar *header, size_t length);
 my_bool _ma_apply_undo_row_update(MARIA_HA *info, LSN undo_lsn,
-                                  uchar *header, size_t length);
+                                  const uchar *header, size_t length);
 my_bool _ma_apply_undo_bulk_insert(MARIA_HA *info, LSN undo_lsn);
 
 my_bool write_hook_for_redo(enum translog_record_type type,
@@ -272,3 +272,7 @@ my_bool write_hook_for_undo_bulk_insert(enum translog_record_type type,
 my_bool write_hook_for_file_id(enum translog_record_type type,
                                TRN *trn, MARIA_HA *tbl_info, LSN *lsn,
                                void *hook_arg);
+void _ma_block_get_status(void* param, my_bool concurrent_insert);
+void _ma_block_update_status(void *param);
+void _ma_block_restore_status(void *param);
+my_bool _ma_block_check_status(void *param);

storage/maria/ma_check.c

@@ -99,8 +99,11 @@ static my_bool create_new_data_handle(MARIA_SORT_PARAM *param, File new_file);
 static my_bool _ma_flush_table_files_before_swap(HA_CHECK *param,
                                                  MARIA_HA *info);
 static TrID max_trid_in_system(void);
+static void _ma_check_print_not_visible_error(HA_CHECK *param, TrID used_trid);
 
 
+/* Initialize check param with default values */
+
 void maria_chk_init(HA_CHECK *param)
 {
   bzero((uchar*) param,sizeof(*param));
@@ -121,9 +124,30 @@ void maria_chk_init(HA_CHECK *param)
   param->stats_method= MI_STATS_METHOD_NULLS_NOT_EQUAL;
 }
 
+
+/* Initialize check param and maria handler for check of table */
+
+void maria_chk_init_for_check(HA_CHECK *param, MARIA_HA *info)
+{
+  param->not_visible_rows_found= 0;
+  param->max_found_trid= 0;
+
+  /*
+    Set up transaction handler so that we can see all rows. When rows is read
+    we will check the found id against param->max_tried
+  */
+  if (!ma_control_file_inited())
+    param->max_trid= 0;                 /* Give warning for first trid found */
+  else
+    param->max_trid= max_trid_in_system();
+
+  maria_ignore_trids(info);
+}
+
+
 	/* Check the status flags for the table */
 
-int maria_chk_status(HA_CHECK *param, register MARIA_HA *info)
+int maria_chk_status(HA_CHECK *param, MARIA_HA *info)
 {
   MARIA_SHARE *share= info->s;
 
@@ -1600,19 +1624,22 @@ static my_bool check_head_page(HA_CHECK *param, MARIA_HA *info, uchar *record,
     if (length < share->base.min_block_length)
     {
       _ma_check_print_error(param,
-                            "Page %9s:  Row %3u is too short (%d bytes)",
-                            llstr(page, llbuff), row, length);
+                            "Page %9s:  Row %3u is too short "
+                            "(%d of min %d bytes)",
+                            llstr(page, llbuff), row, length,
+                            (uint) share->base.min_block_length);
       DBUG_RETURN(1);
     }
     flag= (uint) (uchar) page_buff[pos];
     if (flag & ~(ROW_FLAG_ALL))
       _ma_check_print_error(param,
-                            "Page %9s: Row %3u has wrong flag: %d",
+                            "Page %9s: Row %3u has wrong flag: %u",
                             llstr(page, llbuff), row, flag);
 
     DBUG_PRINT("info", ("rowid: %s  page: %lu  row: %u",
                         llstr(ma_recordpos(page, row), llbuff),
                         (ulong) page, row));
+    info->cur_row.trid= 0;
     if (_ma_read_block_record2(info, record, page_buff+pos,
                                page_buff+pos+length))
     {
@@ -1623,6 +1650,10 @@ static my_bool check_head_page(HA_CHECK *param, MARIA_HA *info, uchar *record,
         DBUG_RETURN(1);
       continue;
     }
+    set_if_bigger(param->max_found_trid, info->cur_row.trid);
+    if (info->cur_row.trid > param->max_trid)
+      _ma_check_print_not_visible_error(param, info->cur_row.trid);
+
     if (share->calc_checksum)
     {
       ha_checksum checksum= (*share->calc_checksum)(info, record);
@@ -2070,6 +2101,11 @@ int maria_chk_data_link(HA_CHECK *param, MARIA_HA *info, my_bool extend)
            llstr(param->empty, llbuff),llstr(param->link_used, llbuff2));
     if (param->lost)
       printf("Lost space:   %12s", llstr(param->lost, llbuff));
+    if (param->max_found_trid)
+    {
+      printf("Max trans. id: %11s\n",
+             llstr(param->max_found_trid, llbuff));
+    }
   }
   my_free((uchar*) record,MYF(0));
   DBUG_RETURN (error);
@@ -2196,6 +2232,16 @@ static int initialize_variables_for_repair(HA_CHECK *param,
                     share->base.min_block_length);
     sort_info->max_records= (ha_rows) (sort_info->filelength / rec_length);
   }
+
+  /* Set up transaction handler so that we can see all rows */
+  if (!ma_control_file_inited())
+    param->max_trid= 0;                 /* Give warning for first trid found */
+  else
+    param->max_trid= max_trid_in_system();
+
+  maria_ignore_trids(info);
+  /* Don't write transid's during repair */
+  maria_versioning(info, 0);
   return 0;
 }
 
@@ -3143,7 +3189,7 @@ static my_bool maria_zerofill_data(HA_CHECK *param, MARIA_HA *info,
        pos+= block_size, page++)
   {
     uchar *buff;
-    uint page_type;
+    enum en_page_type page_type;
 
     /* Ignore bitmap pages */
     if ((page % share->bitmap.pages_covered) == 0)
@@ -3159,8 +3205,8 @@ static my_bool maria_zerofill_data(HA_CHECK *param, MARIA_HA *info,
                             llstr(pos, llbuff), my_errno);
       goto err;
     }
-    page_type= buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK;
-    switch ((enum en_page_type) page_type) {
+    page_type= (enum en_page_type) (buff[PAGE_TYPE_OFFSET] & PAGE_TYPE_MASK);
+    switch (page_type) {
     case UNALLOCATED_PAGE:
       if (zero_lsn)
         bzero(buff, block_size);
@@ -3192,7 +3238,10 @@ static my_bool maria_zerofill_data(HA_CHECK *param, MARIA_HA *info,
       if (max_entry != 0)
       {
         dir= dir_entry_pos(buff, block_size, max_entry - 1);
-        _ma_compact_block_page(buff, block_size, max_entry -1, 0);
+        _ma_compact_block_page(buff, block_size, max_entry -1, 0,
+                               page_type == HEAD_PAGE ? ~(TrID) 0 : 0,
+                               page_type == HEAD_PAGE ?
+                               share->base.min_block_length : 0);
 
         /* Zerofille the not used part */
         offset= uint2korr(dir) + uint2korr(dir+2);
@@ -4456,8 +4505,15 @@ static int sort_get_next_record(MARIA_SORT_PARAM *sort_param)
           It requires a reliable data_file_length so we set it.
         */
         info->state->data_file_length= sort_info->filelength;
+        info->cur_row.trid= 0;
         flag= _ma_scan_block_record(info, sort_param->record,
                                     info->cur_row.nextpos, 1);
+        set_if_bigger(param->max_found_trid, info->cur_row.trid);
+        if (info->cur_row.trid > param->max_trid)
+        {
+          _ma_check_print_not_visible_error(param, info->cur_row.trid);
+          flag= HA_ERR_ROW_NOT_VISIBLE;
+        }
       }
       if (!flag)
       {
@@ -6463,3 +6519,25 @@ static TrID max_trid_in_system(void)
   /* 'id' may be far bigger, if last shutdown is old */
   return max(id, max_trid_in_control_file);
 }
+
+
+static void _ma_check_print_not_visible_error(HA_CHECK *param, TrID used_trid)
+{
+  char buff[22], buff2[22];
+  if (!param->not_visible_rows_found++)
+  {
+    if (!ma_control_file_inited())
+    {
+      _ma_check_print_warning(param,
+                              "Found row with transaction id %s but no maria_control_file was specified.  The table may be corrupted",
+                              llstr(used_trid, buff));
+    }
+    else
+    {
+      _ma_check_print_error(param,
+                            "Found row with transaction id %s when max transaction id according to maria_control_file is %s",
+                            llstr(used_trid, buff),
+                            llstr(param->max_trid, buff2));
+    }
+  }
+}

storage/maria/ma_control_file.c

@@ -250,7 +250,8 @@ static int lock_control_file(const char *name)
     @retval 1      Error (in which case the file is left closed)
 */
 
-CONTROL_FILE_ERROR ma_control_file_open(my_bool create_if_missing)
+CONTROL_FILE_ERROR ma_control_file_open(my_bool create_if_missing,
+                                        my_bool print_error)
 {
   uchar buffer[CF_MAX_SIZE];
   char name[FN_REFLEN], errmsg_buff[256];
@@ -423,9 +424,10 @@ ok:
   DBUG_RETURN(0);
 
 err:
-  my_printf_error(HA_ERR_INITIALIZATION,
-                  "Error when trying to use maria control file '%s': %s", 0,
-                  name, errmsg);
+  if (print_error)
+    my_printf_error(HA_ERR_INITIALIZATION,
+                    "Got error '%s' when trying to use maria control file "
+                    "'%s'", 0, errmsg, name);
   ma_control_file_end(); /* will unlock file if needed */
   DBUG_RETURN(error);
 }

storage/maria/ma_control_file.h

@@ -61,7 +61,8 @@ typedef enum enum_control_file_error {
 } CONTROL_FILE_ERROR;
 
 C_MODE_START
-CONTROL_FILE_ERROR ma_control_file_open(my_bool create_if_missing);
+CONTROL_FILE_ERROR ma_control_file_open(my_bool create_if_missing,
+                                        my_bool print_error);
 int ma_control_file_write_and_force(LSN checkpoint_lsn, uint32 logno, TrID trid);
 int ma_control_file_end(void);
 my_bool ma_control_file_inited(void);

storage/maria/ma_delete.c

@@ -184,7 +184,6 @@ int _ma_ck_delete(register MARIA_HA *info, uint keynr, uchar *key,
     struct st_msg_to_write_hook_for_undo_key msg;
     enum translog_record_type log_type= LOGREC_UNDO_KEY_DELETE;
 
-    info->key_delete_undo_lsn[keynr]= info->trn->undo_lsn;
     lsn_store(log_data, info->trn->undo_lsn);
     key_nr_store(log_data + LSN_STORE_SIZE + FILEID_STORE_SIZE, keynr);
     log_pos= log_data + LSN_STORE_SIZE + FILEID_STORE_SIZE + KEY_NR_STORE_SIZE;

storage/maria/ma_locking.c

@@ -255,7 +255,7 @@ int maria_lock_database(MARIA_HA *info, int lock_type)
 			(THR_WRITE_CONCURRENT_INSERT was used)
 */
 
-void _ma_get_status(void* param, int concurrent_insert)
+void _ma_get_status(void* param, my_bool concurrent_insert)
 {
   MARIA_HA *info=(MARIA_HA*) param;
   DBUG_ENTER("_ma_get_status");

storage/maria/ma_open.c

@@ -118,10 +118,6 @@ static MARIA_HA *maria_clone_internal(MARIA_SHARE *share, int mode,
 		       &info.first_mbr_key, share->base.max_key_length,
 		       &info.maria_rtree_recursion_state,
                        share->have_rtree ? 1024 : 0,
-                       &info.key_write_undo_lsn,
-                       (uint) (sizeof(LSN) * share->base.keys),
-                       &info.key_delete_undo_lsn,
-                       (uint) (sizeof(LSN) * share->base.keys),
                        &changed_fields_bitmap,
                        bitmap_buffer_size(share->base.fields),
 		       NullS))
@@ -780,26 +776,29 @@ MARIA_HA *maria_open(const char *name, int mode, uint open_flags)
     {
       share->concurrent_insert=
 	((share->options & (HA_OPTION_READ_ONLY_DATA | HA_OPTION_TMP_TABLE |
-			   HA_OPTION_COMPRESS_RECORD |
-			   HA_OPTION_TEMP_COMPRESS_RECORD)) ||
+                            HA_OPTION_COMPRESS_RECORD |
+                            HA_OPTION_TEMP_COMPRESS_RECORD)) ||
 	 (open_flags & HA_OPEN_TMP_TABLE) ||
-         share->data_file_type == BLOCK_RECORD ||
+         (share->data_file_type == BLOCK_RECORD &&
+          !share->now_transactional) ||
 	 share->have_rtree) ? 0 : 1;
       if (share->concurrent_insert)
       {
-	share->lock.get_status=_ma_get_status;
-	share->lock.copy_status=_ma_copy_status;
-        /**
-           @todo RECOVERY
-           INSERT DELAYED and concurrent inserts are currently disabled for
-           transactional tables; when enabled again, we should re-evaluate
-           what problems the call to _ma_update_status() by
-           thr_reschedule_write_lock() can do (it may hurt Checkpoint as it
-           would be without intern_lock, and it modifies the state).
-        */
-	share->lock.update_status=_ma_update_status;
-	share->lock.restore_status=_ma_restore_status;
-	share->lock.check_status=_ma_check_status;
+        if (share->data_file_type == BLOCK_RECORD)
+        {
+          share->lock.get_status=    _ma_block_get_status;
+          share->lock.update_status= _ma_block_update_status;
+          share->lock.check_status=  _ma_block_check_status;
+          share->lock.allow_multiple_concurrent_insert= 1;
+        }
+        else
+        {
+          share->lock.get_status=    _ma_get_status;
+          share->lock.copy_status=   _ma_copy_status;
+          share->lock.update_status= _ma_update_status;
+          share->lock.restore_status=_ma_restore_status;
+          share->lock.check_status=  _ma_check_status;
+        }
       }
     }
 #endif

storage/maria/ma_test1.c

@@ -43,6 +43,7 @@ static uint unique_key=HA_NOSAME;
 static uint die_in_middle_of_transaction;
 static my_bool pagecacheing, null_fields, silent, skip_update, opt_unique;
 static my_bool verbose, skip_delete, transactional;
+static my_bool opt_versioning= 0;
 static MARIA_COLUMNDEF recinfo[4];
 static MARIA_KEYDEF keyinfo[10];
 static HA_KEYSEG keyseg[10];
@@ -77,7 +78,7 @@ int main(int argc,char *argv[])
   if (maria_init() ||
       (init_pagecache(maria_pagecache, maria_block_size * 16, 0, 0,
                       maria_block_size, MY_WME) == 0) ||
-      ma_control_file_open(TRUE) ||
+      ma_control_file_open(TRUE, TRUE) ||
       (init_pagecache(maria_log_pagecache,
                       TRANSLOG_PAGECACHE_SIZE, 0, 0,
                       TRANSLOG_PAGE_SIZE, MY_WME) == 0) ||
@@ -210,6 +211,8 @@ static int run_test(const char *filename)
 
   if (maria_begin(file))
     goto err;
+  if (opt_versioning)
+    maria_versioning(file, 1);
   my_errno=0;
   row_count=deleted=0;
   for (i=49 ; i>=1 ; i-=2 )
@@ -339,6 +342,8 @@ static int run_test(const char *filename)
     goto err;
   if (maria_begin(file))
     goto err;
+  if (opt_versioning)
+    maria_versioning(file, 1);
   if (!skip_delete)
   {
     if (!silent)
@@ -767,7 +772,7 @@ static struct my_option my_long_options[] =
    "Test in transactional mode. (Only works with block format)",
    (uchar**) &transactional, (uchar**) &transactional, 0, GET_BOOL, NO_ARG,
    0, 0, 0, 0, 0, 0},
-  {"unique", 'C', "Undocumented", (uchar**) &opt_unique,
+  {"unique", 'E', "Check unique handling", (uchar**) &opt_unique,
    (uchar**) &opt_unique, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
   {"update-rows", 'u', "Max number of rows to update", (uchar**) &update_count,
    (uchar**) &update_count, 0, GET_UINT, REQUIRED_ARG, 1000, 0, 0, 0, 0, 0},
@@ -775,6 +780,9 @@ static struct my_option my_long_options[] =
    (uchar**) &verbose, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
   {"version", 'V', "Print version number and exit",
    0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0},
+  {"versioning", 'C', "Use row versioning (only works with block format)",
+   (uchar**) &opt_versioning,  (uchar**) &opt_versioning, 0, GET_BOOL,
+   NO_ARG, 0, 0, 0, 0, 0, 0},
   { 0, 0, 0, 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0}
 };
 
@@ -872,7 +880,8 @@ static void get_options(int argc, char *argv[])
 
   if ((ho_error=handle_options(&argc, &argv, my_long_options, get_one_option)))
     exit(ho_error);
-
+  if (transactional)
+    record_type= BLOCK_RECORD;
   return;
 } /* get options */
 

storage/maria/ma_test2.c

@@ -44,6 +44,7 @@ static int silent= 0, opt_quick_mode= 0, transactional= 0, skip_update= 0;
 static int die_in_middle_of_transaction= 0, pack_fields= 1;
 static int pack_seg= HA_SPACE_PACK, pack_type= HA_PACK_KEY, remove_count= -1;
 static int create_flag= 0, srand_arg= 0, checkpoint= 0;
+static my_bool opt_versioning= 0;
 static uint use_blob= 0, update_count= 0;
 static ulong pagecache_size=8192*32;
 static enum data_file_type record_type= DYNAMIC_RECORD;
@@ -83,7 +84,7 @@ int main(int argc, char *argv[])
   if (maria_init() ||
       (init_pagecache(maria_pagecache, pagecache_size, 0, 0,
 		      maria_block_size, MY_WME) == 0) ||
-      ma_control_file_open(TRUE) ||
+      ma_control_file_open(TRUE, TRUE) ||
       (init_pagecache(maria_log_pagecache,
 		      TRANSLOG_PAGECACHE_SIZE, 0, 0,
 		      TRANSLOG_PAGE_SIZE, MY_WME) == 0) ||
@@ -230,6 +231,8 @@ int main(int argc, char *argv[])
   if (!(file=maria_open(filename,2,HA_OPEN_ABORT_IF_LOCKED)))
     goto err;
   maria_begin(file);
+  if (opt_versioning)
+    maria_versioning(file, 1);
   if (testflag == 1)
     goto end;
   if (checkpoint == 1 && ma_checkpoint_execute(CHECKPOINT_MEDIUM, FALSE))
@@ -1136,12 +1139,15 @@ static void get_options(int argc, char **argv)
     case 'g':
       skip_update= TRUE;
       break;
+    case 'C':
+      opt_versioning= 1;
+      break;
     case '?':
     case 'I':
     case 'V':
-      printf("%s  Ver 1.1 for %s at %s\n",progname,SYSTEM_TYPE,MACHINE_TYPE);
+      printf("%s  Ver 1.2 for %s at %s\n",progname,SYSTEM_TYPE,MACHINE_TYPE);
       puts("By Monty, for testing Maria\n");
-      printf("Usage: %s [-?AbBcDIKLPRqSsTVWltv] [-k#] [-f#] [-m#] [-e#] [-E#] [-t#]\n",
+      printf("Usage: %s [-?AbBcCDIKLPRqSsTVWltv] [-k#] [-f#] [-m#] [-e#] [-E#] [-t#]\n",
 	     progname);
       exit(0);
     case '#':

storage/maria/ma_test_recovery

@@ -5,4 +5,4 @@
 
 # This file is deprecated and has been replaced with ma_test_recovery.pl
 
-unittest/ma_test_recovery.pl
+unittest/ma_test_recovery.pl $@

storage/maria/ma_write.c

@@ -403,7 +403,6 @@ static int _ma_ck_write_btree_with_log(MARIA_HA *info, MARIA_KEYDEF *keyinfo,
     struct st_msg_to_write_hook_for_undo_key msg;
 
     /* Save if we need to write a clr record */
-    info->key_write_undo_lsn[keyinfo->key_nr]= info->trn->undo_lsn;
     lsn_store(log_data, info->trn->undo_lsn);
     key_nr_store(log_data + LSN_STORE_SIZE + FILEID_STORE_SIZE,
                   keyinfo->key_nr);

storage/maria/maria_chk.c

@@ -106,14 +106,21 @@ int main(int argc, char **argv)
   error=0;
   maria_init();
 
+  if (ma_control_file_open(FALSE, opt_require_control_file) &&
+      (opt_require_control_file ||
+       (opt_transaction_logging && (check_param.testflag & T_REP_ANY))))
+  {
+    error= 1;
+    goto end;
+  }
+
   /*
     If we are doing a repair, user may want to store this repair into the log
     so that the log has a complete history and can be used to replay.
   */
   if (opt_transaction_logging && (check_param.testflag & T_REP_ANY))
   {
-    if (ma_control_file_open(FALSE) ||
-        init_pagecache(maria_log_pagecache,
+    if (init_pagecache(maria_log_pagecache,
                        TRANSLOG_PAGECACHE_SIZE, 0, 0,
                        TRANSLOG_PAGE_SIZE, MY_WME) == 0 ||
         translog_init(opt_log_dir, TRANSLOG_FILE_SIZE,
@@ -127,14 +134,6 @@ int main(int argc, char **argv)
       goto end;
     }
   }
-  else
-  {
-    if (ma_control_file_open(FALSE) && opt_require_control_file)
-    {
-      error= 1;
-      goto end;
-    }
-  }
 
   while (--argc >= 0)
   {
@@ -1237,9 +1236,10 @@ static int maria_chk(HA_CHECK *param, char *filename)
       printf("Data records: %7s   Deleted blocks: %7s\n",
              llstr(info->state->records,llbuff),
              llstr(info->state->del,llbuff2));
-    error =maria_chk_status(param,info);
+    maria_chk_init_for_check(param, info);
+    error= maria_chk_status(param,info);
     maria_intersect_keys_active(share->state.key_map, param->keys_in_use);
-    error =maria_chk_size(param,info);
+    error|= maria_chk_size(param,info);
     if (!error || !(param->testflag & (T_FAST | T_FORCE_CREATE)))
       error|=maria_chk_del(param, info,param->testflag);
     if ((!error || (!(param->testflag & (T_FAST | T_FORCE_CREATE)) &&
@@ -1327,7 +1327,15 @@ end2:
                           T_ZEROFILL)))
     error= write_log_record(param);
 
+  if (param->not_visible_rows_found && (param->testflag & T_VERBOSE))
+  {
+    char buff[22];
+    printf("Max transaction id found: %s\n",
+           llstr(param->max_found_trid, buff));
+  }
+
   VOID(fflush(stdout)); VOID(fflush(stderr));
+
   if (param->error_printed)
   {
     if (param->testflag & (T_REP_ANY | T_SORT_RECORDS | T_SORT_INDEX))

storage/maria/maria_def.h

@@ -432,10 +432,11 @@ typedef struct st_maria_row
   MARIA_RECORD_POS *tail_positions;
   ha_checksum checksum;
   LSN orig_undo_lsn;			/* Lsn at start of row insert */
+  TrID trid;                            /* Transaction id for current row */
   uchar *empty_bits, *field_lengths;
   uint *null_field_lengths;             /* All null field lengths */
   ulong *blob_lengths;                  /* Length for each blob */
-  ulong base_length, min_length, normal_length, char_length, varchar_length;
+  ulong min_length, normal_length, char_length, varchar_length;
   ulong blob_length, head_length, total_length;
   size_t extents_buffer_length;         /* Size of 'extents' buffer */
   uint field_lengths_length;            /* Length of data in field_lengths */
@@ -470,12 +471,10 @@ struct st_maria_handler
   DYNAMIC_ARRAY pinned_pages;
   /* accumulate indexfile changes between write's */
   TREE *bulk_insert;
-  LEX_STRING *log_row_parts;		/* For logging */
+  LEX_CUSTRING *log_row_parts;		/* For logging */
   DYNAMIC_ARRAY *ft1_to_ft2;		/* used only in ft1->ft2 conversion */
   MEM_ROOT      ft_memroot;             /* used by the parser               */
   MYSQL_FTPARSER_PARAM *ftparser_param;	/* share info between init/deinit */
-  LSN   *key_write_undo_lsn;            /* Pointer to undo for each key */
-  LSN   *key_delete_undo_lsn;           /* Pointer to undo for each key */
   uchar *buff;				/* page buffer */
   uchar *keyread_buff;                   /* Buffer for last key read */
   uchar *lastkey, *lastkey2;		/* Last used search key */
@@ -510,6 +509,8 @@ struct st_maria_handler
   IO_CACHE rec_cache;			/* When cacheing records */
   LIST open_list;
   MY_BITMAP changed_fields;
+  ulong row_base_length;                /* Length of row header */
+  uint row_flag;                        /* Flag to store in row header */
   uint opt_flag;			/* Optim. for space/speed */
   uint update;				/* If file changed since open */
   int lastinx;				/* Last used index */
@@ -608,9 +609,9 @@ struct st_maria_handler
 #define _ma_store_keynr(share, x, nr) x[(share)->keypage_header - KEYPAGE_KEYID_SIZE - KEYPAGE_FLAG_SIZE - KEYPAGE_USED_SIZE]= (nr)
 #define _ma_get_keynr(share, x) ((uchar) x[(share)->keypage_header - KEYPAGE_KEYID_SIZE - KEYPAGE_FLAG_SIZE - KEYPAGE_USED_SIZE])
 #define _ma_store_transid(buff, transid) \
-  int6store((buff) + LSN_STORE_SIZE, (transid))
+  transid_store((buff) + LSN_STORE_SIZE, (transid))
 #define _ma_korr_transid(buff) \
-  uint6korr((buff) + LSN_STORE_SIZE)
+  transid_korr((buff) + LSN_STORE_SIZE)
 #define _ma_get_keypage_flag(share,x) x[(share)->keypage_header - KEYPAGE_USED_SIZE - KEYPAGE_FLAG_SIZE]
 #define _ma_store_keypage_flag(share,x,flag) x[(share)->keypage_header - KEYPAGE_USED_SIZE - KEYPAGE_FLAG_SIZE]= (flag)
 
@@ -1042,7 +1043,7 @@ my_bool _ma_cmp_dynamic_unique(MARIA_HA *info, MARIA_UNIQUEDEF *def,
                                const uchar *record, MARIA_RECORD_POS pos);
 my_bool _ma_unique_comp(MARIA_UNIQUEDEF *def, const uchar *a, const uchar *b,
                         my_bool null_are_equal);
-void _ma_get_status(void *param, int concurrent_insert);
+void _ma_get_status(void *param, my_bool concurrent_insert);
 void _ma_update_status(void *param);
 void _ma_restore_status(void *param);
 void _ma_copy_status(void *to, void *from);
@@ -1148,4 +1149,5 @@ extern my_bool maria_flush_log_for_page(uchar *page,
 extern my_bool maria_flush_log_for_page_none(uchar *page,
                                              pgcache_page_no_t page_no,
                                              uchar *data_ptr);
+void maria_concurrent_inserts(MARIA_HA *info, my_bool concurrent_insert);
 extern PAGECACHE *maria_log_pagecache;

storage/maria/maria_pack.c

@@ -435,6 +435,7 @@ static MARIA_HA *open_maria_file(char *name,int mode)
     DBUG_RETURN(0);
   }
   VOID(maria_lock_database(isam_file,F_WRLCK));
+  maria_ignore_trids(isam_file);
   DBUG_RETURN(isam_file);
 }
 
@@ -1083,7 +1084,7 @@ static int get_statistic(PACK_MRG_INFO *mrg,HUFF_COUNTS *huff_counts)
     }
     else if (error != HA_ERR_RECORD_DELETED)
     {
-      VOID(fprintf(stderr, "Got error %d while reading rows", error));
+      VOID(fprintf(stderr, "Got error %d while reading rows\n", error));
       break;
     }
 

storage/maria/maria_read_log.c

@@ -56,7 +56,7 @@ int main(int argc, char **argv)
     goto err;
   }
   /* we don't want to create a control file, it MUST exist */
-  if (ma_control_file_open(FALSE))
+  if (ma_control_file_open(FALSE, TRUE))
   {
     fprintf(stderr, "Can't open control file (%d)\n", errno);
     goto err;

storage/maria/trnman.c

@@ -315,7 +315,13 @@ TRN *trnman_new_trn(pthread_mutex_t *mutex, pthread_cond_t *cond,
   pthread_mutex_unlock(&LOCK_trn_list);
 
   if (unlikely(!trn->min_read_from))
-    trn->min_read_from= trn->trid;
+  {
+    /*
+      We are the only transaction. Set min_read_from so that we can read
+      our own rows
+    */
+    trn->min_read_from= trn->trid + 1;
+  }
 
   trn->commit_trid= 0;
   trn->rec_lsn= trn->undo_lsn= trn->first_undo_lsn= 0;
@@ -534,9 +540,19 @@ int trnman_can_read_from(TRN *trn, TrID trid)
   LF_REQUIRE_PINS(3);
 
   if (trid < trn->min_read_from)
-    return 1; /* can read */
-  if (trid > trn->trid)
-    return 0; /* cannot read */
+    return 1; /* Row is visible by all transactions in the system */
+
+  if (trid >= trn->trid)
+  {
+    /*
+      We have now two cases
+      trid > trn->trid, in which case the row is from a new transaction
+      and not visible, in which case we should return 0.
+      trid == trn->trid in which case the row is from the current transaction
+      and we should return 1
+    */
+    return trid == trn->trid;
+  }
 
   found= lf_hash_search(&trid_to_committed_trn, trn->pins, &trid, sizeof(trid));
   if (found == NULL)

storage/maria/unittest/ma_control_file-t.c

@@ -110,7 +110,7 @@ static CONTROL_FILE_ERROR local_ma_control_file_open(void)
 {
   CONTROL_FILE_ERROR error;
   error_handler_hook= my_ignore_message;
-  error= ma_control_file_open(TRUE);
+  error= ma_control_file_open(TRUE, TRUE);
   error_handler_hook= default_error_handler_hook;
   return error;
 }

storage/maria/unittest/ma_test_all-t

@@ -87,9 +87,9 @@ sub run_tests
   # make the unit test fail during 'make test'. $nr_tests must be right.
   #
 
-  $nr_tests+= run_check_tests(0, 0, 0, 0, 1) * 4;  #
-  $nr_tests+= run_repair_tests(0, 0, 0, 0, 1) * 4; # called 4 times
-  $nr_tests+= run_pack_tests(0, 0, 0, 0, 1) * 4;   #
+  $nr_tests+= run_check_tests(0, 0, 0, 0, 1) * 5;  #
+  $nr_tests+= run_repair_tests(0, 0, 0, 0, 1) * 5; # called 4 times
+  $nr_tests+= run_pack_tests(0, 0, 0, 0, 1) * 5;   #
   $nr_tests+= run_tests_on_warnings_and_errors(0, 0, 0, 1);
   $nr_tests+= run_ma_test_recovery(0, 1);
   $nr_tests+= run_tests_on_clrs(0, 0, 1);
@@ -185,6 +185,19 @@ sub run_tests
   run_repair_tests($suffix, $opt_silent, "-M -T", $opt_verbose, 0);
   run_pack_tests($suffix, $opt_silent, "-M -T", $opt_verbose, 0);
 
+  if ($opt_verbose)
+  {
+    print "\nRunning tests with block row format, transactions and versioning\n";
+  }
+  run_check_tests($suffix, $opt_silent, "-M -T -C", $opt_verbose, 0);
+  run_repair_tests($suffix, $opt_silent, "-M -T -C", $opt_verbose, 0);
+  run_pack_tests($suffix, $opt_silent, "-M -T -C", $opt_verbose, 0);
+
+
+  if ($opt_verbose)
+  {
+    print "\nRunning tests with warnings and recovery\n";
+  }
   run_tests_on_warnings_and_errors($suffix, $opt_silent, $opt_verbose, 0);
   run_ma_test_recovery($opt_verbose, 0);
   run_tests_on_clrs($suffix, $opt_verbose, 0);
@@ -523,7 +536,6 @@ sub ok
   if ($verbose)
   {
     print " " x (62 - $len);
-    print " ";
   }
   $err= $?;
   if ((!$err && !$expected_error) ||

storage/maria/unittest/ma_test_loghandler-t.c

@@ -196,7 +196,7 @@ int main(int argc __attribute__((unused)), char *argv[])
   }
 #endif
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);

storage/maria/unittest/ma_test_loghandler_first_lsn-t.c

@@ -67,7 +67,7 @@ int main(int argc __attribute__((unused)), char *argv[])
   }
 #endif
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);

storage/maria/unittest/ma_test_loghandler_max_lsn-t.c

@@ -64,7 +64,7 @@ int main(int argc __attribute__((unused)), char *argv[])
   }
 #endif
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);

storage/maria/unittest/ma_test_loghandler_multigroup-t.c

@@ -184,7 +184,7 @@ int main(int argc __attribute__((unused)), char *argv[])
   }
 #endif
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);
@@ -348,7 +348,7 @@ int main(int argc __attribute__((unused)), char *argv[])
   end_pagecache(&pagecache, 1);
   ma_control_file_end();
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE,TRUE))
   {
     fprintf(stderr, "pass2: Can't init control file (%d)\n", errno);
     exit(1);

storage/maria/unittest/ma_test_loghandler_multithread-t.c

@@ -283,7 +283,7 @@ int main(int argc __attribute__((unused)),
 
   my_thread_global_init();
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);

storage/maria/unittest/ma_test_loghandler_noflush-t.c

@@ -70,7 +70,7 @@ int main(int argc __attribute__((unused)), char *argv[])
   }
 #endif
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);

storage/maria/unittest/ma_test_loghandler_nologs-t.c

@@ -67,7 +67,7 @@ int main(int argc __attribute__((unused)), char *argv[])
   }
 #endif
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);
@@ -140,7 +140,7 @@ int main(int argc __attribute__((unused)), char *argv[])
     }
   }
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);

storage/maria/unittest/ma_test_loghandler_pagecache-t.c

@@ -95,7 +95,7 @@ int main(int argc __attribute__((unused)), char *argv[])
   }
 #endif
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);

storage/maria/unittest/ma_test_loghandler_purge-t.c

@@ -67,7 +67,7 @@ int main(int argc __attribute__((unused)), char *argv[])
   }
 #endif
 
-  if (ma_control_file_open(TRUE))
+  if (ma_control_file_open(TRUE, TRUE))
   {
     fprintf(stderr, "Can't init control file (%d)\n", errno);
     exit(1);

storage/maria/unittest/ma_test_recovery.pl

@@ -11,6 +11,8 @@ $VER= "1.2";
 
 $opt_version= 0;
 $opt_help=    0;
+$opt_verbose= 0;
+$opt_abort_on_error=0;
 
 my $silent= "-s";
 my $maria_path;     # path to "storage/maria"
@@ -33,7 +35,7 @@ sub main
 {
   my ($res, $table);
 
-  if (!GetOptions("help","version"))
+  if (!GetOptions("abort-on-error", "help", "version", "verbose"))
   {
     $flag_exit= 1;
   }
@@ -86,7 +88,10 @@ sub main
   my @t= ("ma_test1$suffix $silent -M -T -c",
           "ma_test2$suffix $silent -L -K -W -P -M -T -c -d500",
           "ma_test2$suffix $silent -M -T -c -b65000",
-          "ma_test2$suffix $silent -M -T -c -b65000 -d800");
+          "ma_test2$suffix $silent -M -T -c -b65000 -d800",
+          "ma_test1$suffix $silent -M -T -c -C",
+          "ma_test2$suffix $silent -L -K -W -P -M -T -c -d500 -C"
+         );
 
   foreach my $prog (@t)
   {
@@ -94,7 +99,7 @@ sub main
     my $prog_no_suffix= $prog;
     $prog_no_suffix=~ s/$suffix// if ($suffix);
     print MY_LOG "TEST WITH $prog_no_suffix\n";
-    $res= `$maria_exe_path/$prog`;
+    $res= my_exec("$maria_exe_path/$prog");
     print MY_LOG $res;
     # derive table's name from program's name
     if ($prog =~ m/ma_(test[0-9]+).*/)
@@ -104,8 +109,8 @@ sub main
     $com=  "$maria_exe_path/maria_chk$suffix -dvv $table ";
     $com.= "| grep -v \"Creation time:\" | grep -v \"file length\" ";
     $com.= "> $tmp/maria_chk_message.good.txt 2>&1";
-    `$com`;
-    my $checksum=`$maria_exe_path/maria_chk$suffix -dss $table`;
+    my_exec($com);
+    my $checksum= my_exec("$maria_exe_path/maria_chk$suffix -dss $table");
     move("$table.MAD", "$tmp/$table-good.MAD") ||
       die "Can't move $table.MAD to $tmp/$table-good.MAD\n";
     move("$table.MAI", "$tmp/$table-good.MAI") ||
@@ -137,6 +142,9 @@ sub main
            "ma_test1$suffix $silent -M -T -c -N blob -H2",
            "--testflag=3",
            "--testflag=4 --test-undo=",
+           "ma_test1$suffix $silent -M -T -c -N blob -H2 --versioning",
+           "--testflag=3",
+           "--testflag=4 --test-undo=",
            "ma_test1$suffix $silent -M -T -c -N blob -H2",
            "--testflag=2",
            "--testflag=3 --test-undo=",
@@ -170,7 +178,7 @@ sub main
           my $prog_no_suffix= $prog;
           $prog_no_suffix=~ s/$suffix// if ($suffix);
           print MY_LOG "TEST WITH $prog_no_suffix $commit_run_args (commit at end)\n";
-          $res= `$maria_exe_path/$prog $commit_run_args`;
+          $res= my_exec("$maria_exe_path/$prog $commit_run_args");
           print MY_LOG $res;
           # derive table's name from program's name
           if ($prog =~ m/ma_(test[0-9]+).*/)
@@ -180,16 +188,16 @@ sub main
           $com=  "$maria_exe_path/maria_chk$suffix -dvv $table ";
           $com.= "| grep -v \"Creation time:\" | grep -v \"file length\" ";
           $com.= "> $tmp/maria_chk_message.good.txt 2>&1";
-          $res= `$com`;
+          $res= my_exec($com);
           print MY_LOG $res;
-          $checksum= `$maria_exe_path/maria_chk$suffix -dss $table`;
+          $checksum= my_exec("$maria_exe_path/maria_chk$suffix -dss $table");
           move("$table.MAD", "$tmp/$table-good.MAD") ||
             die "Can't move $table.MAD to $tmp/$table-good.MAD\n";
           move("$table.MAI", "$tmp/$table-good.MAI") ||
             die "Can't move $table.MAI to $tmp/$table-good.MAI\n";
           unlink <maria_log.* maria_log_control>;
           print MY_LOG "TEST WITH $prog_no_suffix $abort_run_args$test_undo[$j] (additional aborted work)\n";
-          $res= `$maria_exe_path/$prog $abort_run_args$test_undo[$j]`;
+          $res= my_exec("$maria_exe_path/$prog $abort_run_args$test_undo[$j]");
           print MY_LOG $res;
           copy("$table.MAD", "$tmp/$table-before_undo.MAD") ||
             die "Can't copy $table.MAD to $tmp/$table-before_undo.MAD\n";
@@ -271,6 +279,11 @@ sub check_table_is_same
   # Data/key file length is random in ma_test2 (as it uses srand() which
   # may differ between machines).
 
+  if ($opt_verbose)
+  {
+    print "checking if table $table has changed\n";
+  }
+
   $com=  "$maria_exe_path/maria_chk$suffix -dvv $table | grep -v \"Creation time:\" ";
   $com.= "| grep -v \"file length\"> $tmp/maria_chk_message.txt 2>&1";
   $res= `$com`;
@@ -326,7 +339,7 @@ sub apply_log
     $log_md5.= md5_conv($_);
   }
   print MY_LOG "applying log\n";
-  `$maria_exe_path/maria_read_log$suffix -a > $tmp/maria_read_log_$table.txt`;
+  my_exec("$maria_exe_path/maria_read_log$suffix -a > $tmp/maria_read_log_$table.txt");
   foreach (<maria_log.*>)
   {
     $log_md5_2.= md5_conv($_);
@@ -414,6 +427,23 @@ sub physical_cmp
 }
 
 
+sub my_exec
+{
+  my($command)= @_;
+  my $res;
+  if ($opt_verbose)
+  {
+    print "$command\n";
+  }
+  $res= `$command`;
+  if ($? != 0 && $opt_abort_on_error)
+  {
+    exit(1);
+  }
+  return $res;
+}
+
+
 ####
 #### usage
 ####
@@ -429,7 +459,11 @@ Run various maria recovery tests and print the results
 
 Options
 --help             Show this help and exit.
+
+--abort-on-error   Abort at once in case of error.
+--verbose          Show commands while there are executing.
 --version          Show version number and exit.
+
 EOF
   exit(0);
 }

storage/myisam/mi_locking.c

@@ -281,7 +281,7 @@ int mi_lock_database(MI_INFO *info, int lock_type)
 			(THR_WRITE_CONCURRENT_INSERT was used)
 */
 
-void mi_get_status(void* param, int concurrent_insert)
+void mi_get_status(void* param, my_bool concurrent_insert)
 {
   MI_INFO *info=(MI_INFO*) param;
   DBUG_ENTER("mi_get_status");

storage/myisam/myisamdef.h

@@ -700,7 +700,7 @@ int _mi_cmp_dynamic_unique(MI_INFO *info, MI_UNIQUEDEF *def,
                            const uchar *record, my_off_t pos);
 int mi_unique_comp(MI_UNIQUEDEF *def, const uchar *a, const uchar *b,
                    my_bool null_are_equal);
-void mi_get_status(void *param, int concurrent_insert);
+void mi_get_status(void *param, my_bool concurrent_insert);
 void mi_update_status(void *param);
 void mi_restore_status(void *param);
 void mi_copy_status(void *to, void *from);