Bug #16776528 RACE CONDITION CAN CAUSE MYSQLD TO REMOVE SOCKET FILE ERRANTLY

Problem Description:
A mysqld_safe instance is started. An InnoDB crash recovery begins which takes
few seconds to complete. During this crash recovery process happening, another
mysqld_safe instance is started with the same server startup parameters. Since
the mysqld's pid file is absent during the crash recovery process the second
instance assumes there is no other process and tries to acquire a lock on the
ibdata files in the datadir.  But this step fails and the 2nd instance keeps 
retrying 100 times each with a delay of 1 second. Now after the 100 attempts, 
the server goes down, but while going down it hits the mysqld_safe script's 
cleanup section and without any check it blindly deletes the socket and pid 
files. Since no lock is placed on the socket file, it gets deleted.

Solution:
We create a mysqld_safe.pid file in the datadir, which protects the presence 
server instance resources by storing the mysqld_safe's process id in it. We
place a check if the mysqld_safe.pid file is existing in the datadir. If yes
then we check if the pid it contains is an active pid or not. If yes again,
then the scripts logs an error saying "A mysqld_safe instance is already 
running". Otherwise it will log the present mysqld_safe's pid into the 
mysqld_safe.pid file.

scripts/mysqld_safe.sh

@@ -518,6 +518,31 @@ else
 fi
 plugin_dir="${plugin_dir}${PLUGIN_VARIANT}"
 
+# A pid file is created for the mysqld_safe process. This file protects the
+# server instance resources during race conditions.
+safe_pid="$DATADIR/mysqld_safe.pid"
+if test -f $safe_pid
+then
+  PID=`cat "$safe_pid"`
+  if @CHECK_PID@
+  then
+    if @FIND_PROC@
+    then
+      log_error "A mysqld_safe process already exists"
+      exit 1
+    fi
+  fi
+  rm -f "$safe_pid"
+  if test -f "$safe_pid"
+  then
+    log_error "Fatal error: Can't remove the mysqld_safe pid file"
+    exit 1
+  fi
+fi
+
+# Insert pid proerply into the pid file.
+ps -e | grep  [m]ysqld_safe | awk '{print $1}' | sed -n 1p > $safe_pid
+# End of mysqld_safe pid(safe_pid) check.
 
 # Determine what logging facility to use
 
@@ -528,6 +553,7 @@ then
   if [ $? -ne 0 ]
   then
     log_error "--syslog requested, but no 'logger' program found.  Please ensure that 'logger' is in your PATH, or do not specify the --syslog option to mysqld_safe."
+    rm -f "$safe_pid"                 # Clean Up of mysqld_safe.pid file.
     exit 1
   fi
 fi
@@ -542,6 +568,7 @@ then
 
     # mysqld does not add ".err" to "--log-error=foo."; it considers a
     # trailing "." as an extension
+    
     if expr "$err_log" : '.*\.[^/]*$' > /dev/null
     then
         :
@@ -632,6 +659,7 @@ does not exist or is not executable. Please cd to the mysql installation
 directory and restart this script from there as follows:
 ./bin/mysqld_safe&
 See http://dev.mysql.com/doc/mysql/en/mysqld-safe.html for more information"
+  rm -f "$safe_pid"                 # Clean Up of mysqld_safe.pid file.
   exit 1
 fi
 
@@ -725,6 +753,7 @@ then
     if @FIND_PROC@
     then    # The pid contains a mysqld process
       log_error "A mysqld process already exists"
+      rm -f "$safe_pid"                 # Clean Up of mysqld_safe.pid file.
       exit 1
     fi
   fi
@@ -735,6 +764,7 @@ then
 $pid_file
 Please remove it manually and start $0 again;
 mysqld daemon not started"
+    rm -f "$safe_pid"                 # Clean Up of mysqld_safe.pid file.
     exit 1
   fi
 fi
@@ -858,3 +888,5 @@ done
 
 log_notice "mysqld from pid file $pid_file ended"
 
+rm -f "$safe_pid"                       # Some Extra Safety. File is deleted
+                                        # once the mysqld process ends.