ATRT documentation

storage/ndb/test/run-test/ATRT_SETUP_README.txt:
  ATRT Setup documentation
storage/ndb/test/run-test/ATRT_USAGE_README.txt:
  ATRT Usage documentation

storage/ndb/test/run-test/ATRT_SETUP_README.txt

+ATRT_SETUP_README.txt
+Author: Serge Kozlov, MySQL
+Date: 03/23/2006
+
+Contents
+
+1. Overview
+2. Setup
+3. Preparing for testing
+4. CPCD. Command line and configuration options
+5. d.tmp. Examples
+6. Test Case File format. Examples
+
+Note: how to run ATRT tests please look ATRT_USAGE_README.txt
+
+
+  1.Overview
+  ==========
+
+  Auto Test Run Test (ATRT) is a home made test frame work. This frame work can 
+start and stop processes on different Linux hosts through another executable 
+called Cluster Process Control Daemon (CPCD) running on the other hosts. 
+It uses for testing cluster configurations located on different machines. 
+ATRT isn't replacement for mysqltest. In fact, ATRT can invoke mysql-test as 
+samples blow show. This framework has been designed to run most any test 
+(mysqltest testcases, stress tests, any applications) in cluster and detects 
+all errors and issues which happened in any node. In addition, ATRT starts 
+applications that are defined as test with arguments and will analyze log files 
+from all nodes that are produced.
+
+
+  2.Setup
+  =======
+
+  Following steps described how to setup ATRT framework on a cluster:
+* You need to have cloned source tree (e.g mysql-5.1-new) on machine where you 
+  plan to use ATRT.
+* Compile and install build (e.g. for Linux/x86 can use 
+  BUILD/compile-pentium-max --prefix=/path/to/installation).
+* Copy $MYSQL_DIR on all machines and on same path which you plan to use as 
+  nodes (you can use scp utility or ask JonathanMiller about distribution on 
+  ndbXX servers).
+* Compile and install ATRT:
+  * Go to $TREE_ROOT/storage/ndb/test
+  * do make
+  * do make install
+  * Make sure that $MYSQL_DIR/bin now contains files like that: testBasic, 
+    testBlobs, testDict and so on
+  * Make sure that $MYSQL_DIR/mysql-test/ndb now contains files like that: atrt,
+    atrt-analyze-result.sh, atrt-setup.sh, make-config.sh and so on
+  * Add $MYSQL_DIR/mysql-test/ndb to PATH and make sure that you can call atrt 
+    from any place on disk.
+* Configure CPCD processes on all machines. Repeat following steps for each 
+  node:
+  * Create $VAR_DIR/run/ndb_cpcd directory
+  * Create /etc/ndb_cpcd.cnf file. It will configuration file ndb_cpcd daemon. 
+    It strongly recommended instead command line options because more simply. 
+    More option for configuration can fe found in this document in section CPCD
+  * Add to file following text:
+
+	[ndb_cpcd]
+	work-dir= $VAR_DIR/run/ndb_cpcd # e.g. /mysql/builds/5.1/var/run/ndb_cpcd
+	logfile= $VAR_DIR/run/ndb_cpcd/log.txt # e.g. /mysql/builds/5.1/var/run/ndb_cpcd/log.txt
+	debug= 1
+	user= ndbdev
+  
+  * Register ndb_cpcd in etc/initab. Hint: you can use more simply way if use 
+    Linux - add following line to /etc/rc.d/rc.local file
+
+	$MYSQL_DIR/libexec/ndb_cpcd > /dev/null 2>&1 &
+
+  * Start ndb_cpcd
+
+	$MYSQL_DIR/libexec/ndb_cpcd > /dev/null 2>&1 &
+
+  * Open log file and make sure that ndb_cpcd process started properly. Ususal mistakes are: wrong definition of paths and ndb_cpcd process already running. You can test ndb_cpcd from any ndb system by using the command line below. If nothing is returned the process is up and running. Otherwise you will see Failed to connect to node:1234:
+
+	$MYSQL_DIR/libexec/ndb_cpcc node
+
+Now ATRT Setup done. After accomplishment all steps above you should be have 
+following:
+* Each node contains fresh build
+* Each node has configured and started ndb_cpcd process.
+* The node that will used for starting ATRT has installed atrt binaries/scripts
+  and some tests.
+
+
+  3.Preparing for testing
+  =======================
+
+* Create new empty directory where you plan to use for testing. Note: if you 
+  plan to test different cluster configuration you need to create own directory 
+  for each cluster configuration. Do not use same directory.
+* Create d.tmp file for your cluster configuration. See details in this 
+  document in section d.tmp.
+* Run make-config.sh ./d.tmp. It will create necessary directories and files for 
+  ATRT.
+* Create ATRT testcase file(s) (e.g. test1.atrt). See details in this document 
+  in section Test Case File format. Examples.
+* Put your test application into $MYSQL_DIR/bin directory. If your application 
+  requires another directory (e.g. mysql-test-run.pl) create redirect script 
+  such as example below:
+
+	#!/bin/sh
+       
+	set  -x
+	cd $MYSQL_BASE_DIR/mysql-test
+	./mysql-test-run.pl --with-ndbcluster --ndb-connectstring=$*
+        
+Now preparing for testing done.
+
+  
+  4.CPCD.
+  =======
+
+  The CPCD process needs to be running on each host contacts CPCD to tell it 
+what process to execute. In other word for testing each node should have to 
+running ndb_cpcd. Usually ndb_cpcd can be found in storage/ndb/src/cw/cpcd for 
+source tree or in libexec/ndb_cpcd for binary distribution. ndb_cpcd uses 
+configuration file /etc/ndb_cpcd.cnf but all options from one can be replaced 
+by command line. Names of options in command line are same as from 
+configuration file except some some commands have short notation (-X).
+
+ndb_cpcd command line options
+
+-w, --work-dir=name
+    Work directory. Should be exist before starting ndb_cpcd. Usually it's 
+    $VARDIR/run/ndb_cpcd
+-p, --port=#
+    TCP port to listen on. By default 1234
+-S, --syslog
+    Log events to syslog
+-L, --logfile=name
+    File to log events to. Usually it's $VARDIR/run/ndb_cpcd/logfile.txt
+-D, --debug
+    Enable debug mode.
+-u, --user=name
+    Run as user
+
+
+  5.d.tmp
+  =======
+
+  The d.tmp file is used to create the d.txt file (configuration file for ATRT) 
+and the config.ini file (configuration file for MySQL Cluster). This file does 
+not have to be called d.tmp, it can be called by any name. The file will be feed 
+to make-config.sh. All options in the file separated by two parts:
+* Original options. They are located before '-- cluster config'. Description of 
+all these options are below.
+* Options for config.ini. They are located after '-- cluster config'. All 
+options, sections and values completely coincide with config.ini.
+
+d.tmp options
+
+baseport
+    Port used for communicating to the cluster on.
+basedir
+    basedir has to point to the root of the mysql install. Note that ATRT will 
+    create a run directory under the base directory. All test directories and 
+    files created will be copied to all hosts in the test under the basedir/run directory.
+mgm
+    Host to NDB Cluster manager on. Put hostnames separated by blanks.
+ndb
+    Host(s) to run NDB Data Nodes on. Put hostnames separated by blanks.
+api
+    Host(s) that NDB API should be ran on. Put hostnames separated by blanks.
+mysqld
+    Host(s) that mysqld processes should be started on. Put hostnames separated by blanks.
+mysql
+    Host(s) that mysql processes should be started on. Put hostnames separated 
+    by blanks.
+
+Example d.tmp for cluster configuration: 1 ndb node, 1 replica, 1 mgm, 1 api, 
+1 mysql, 1 mysqld. Available hosts for nodes: ndb16, ndb17
+
+baseport: 14000
+basedir: /home/ndbdev/skozlov/builds
+mgm: ndb16
+ndb: ndb17
+api: ndb16
+mysqld ndb16
+mysql ndb16
+-- cluster config
+[DB DEFAULT]
+NoOfReplicas: 1
+
+[MGM DEFAULT]
+PortNumber: 14000
+ArbitrationRank: 1
+
+Example d.tmp for cluster configuration: 2 ndb nodes, 2 replicas, 1 mgm, 1 api, 
+1 mysql, 1 mysqld. Available hosts for nodes: ndb16, ndb17, ndb18
+
+baseport: 14000
+basedir: /home/ndbdev/skozlov/builds
+mgm: ndb16
+ndb: ndb17 ndb18
+api: ndb16
+mysqld ndb16
+mysql ndb16
+-- cluster config
+[DB DEFAULT]
+NoOfReplicas: 2
+
+[MGM DEFAULT]
+PortNumber: 14000
+ArbitrationRank: 1
+               
+Example d.tmp for cluster configuration: 4 ndb nodes, 4 replicas, 1 mgm, 3 api, 
+2 mysql, 1 mysqld. Available hosts for nodes: ndb14, ndb15, ndb16, ndb17, ndb18
+
+baseport: 14000
+basedir: /home/ndbdev/skozlov/builds
+mgm: ndb16
+ndb: ndb17 ndb18 ndb15 ndb14
+api: ndb16 ndb17 ndb18
+mysqld ndb16
+mysql ndb16 ndb17
+-- cluster config
+[DB DEFAULT]
+NoOfReplicas: 4
+
+[MGM DEFAULT]
+PortNumber: 14000
+ArbitrationRank: 1
+               
+
+  6.Test Case File Format. Examples.
+  ==================================
+
+  A test file consists of a list of test cases. Each test case is ended by an 
+empty/blank line. Each test case is described by a set of name value pairs. 
+ATRT looks for tests executables, shell and Perl scripts in the basedir/bin 
+directory.
+
+Test Case File options:
+
+max-time
+    This value is in seconds. Test ATRT how long to allow the test to run before 
+    marking the test a failure and moving on to the next test. If the test 
+   finishes before max-time, it will move on to the next test.
+cmd
+    Tells ATRT which test or script to run from the bin directory
+args
+    These are the command-line arguments to be passed to the test or script 
+    that is being executed.
+type
+    Currently only bench. See ATRT Command-line parameters for details.
+run-all
+    will start the same command for each defined api/mysql (normally it only 
+    started in 1 instance)
+
+Example 1.
+ATRT starts test $MYSQL_DIR/bin/testBlobs without arguments and sets time for 
+execution as 10 min. testBlobs is binary application
+
+max-time: 600
+cmd: testBlobs
+args:
+
+Example 2.
+ATRT starts test $MYSQL_DIR/bin/testRead -n PkRead and sets time for execution 
+as 20 min. testRead is binary application
+
+max-time: 1200
+cmd: testRead
+args: -n PkRead
+
+Example 3.
+ATRT starts test $MYSQL_DIR/bin/atrt-mysql-test-run -force and sets time for 
+execution as one hour. atrt-mysql-test-run is bash script and it points to 
+$MYSQL_DIR/mysql-test/mysql-test-run. In fact this test will start 
+mysql-test-run --force that means the execution all mysqltest testcases in 
+mysql-test/t directory.
+
+max-time: 3600
+cmd: atrt-mysql-test-run 
+args: --force
+
+Example 4.
+ATRT starts test $MYSQL_DIR/bin/MyTest1 -n and sets time for execution as 2 min.
+Then starts $MYSQL_DIR/bin/MyTest2 for each defined mysql/api node and set 
+timeout 3 min.
+
+max-time: 120
+cmd: MyTest1 
+args: -n
+
+max-time: 180
+cmd: MyTest2 
+run-all: yes
+

storage/ndb/test/run-test/ATRT_USAGE_README.txt

+ATRT_USAGE_README.txt
+Author: Serge Kozlov, MySQL
+Date: 03/23/2006
+
+Contents
+
+1. How to run
+2. Results
+3. ATRT Command Line Options
+
+Note: how to setup ATRT please look ATRT_SETUP_README.txt
+  
+  1.How to run.
+  =============
+
+Simple way to start atrt:
+
+	atrt --testcase-file=atrt_test_case_file
+
+Command line above doesn't produce any log files. Better use following :
+
+	atrt --log-file=log.txt --testcase-file=atrt_test_case_file
+
+Now we can look log.txt for investigation any issues. If it is insufficiently
+then add one or more -v arguments:
+
+	atrt -v -v --log-file=log.txt --testcase-file=atrt_test_case_file
+
+If the test case file contains two or more test we can add -r options for 
+preventing stopping testing if one test fails (like --force for mysql-test-run)
+
+	atrt -v -v -r --log-file=log.txt --testcase-file=atrt_test_case_file
+
+The line below is optimal solution for testing:
+
+	atrt -v -v -r -R --log-file=log.txt --testcase-file=atrt_test_case_file
+
+All additional command line arguments and description of used in exampes above 
+available in this document in section atrt command line options
+
+  2.Results.
+  ==========
+
+  Unlike mysql-test-run frame work atrt doesn't inform to console passed/failed 
+status of tests. You need to use --log-file option and look into log file for 
+getting information about status of tests. When atrt finished you can look into 
+log file defined --log-file option. It's main source of information about how 
+were performed atrt tests. Below added the examples of content of log-file for 
+different failures (except example 1 for passed test). Examples include 
+probable cases with reasons and recommended solutions and cover not test issues 
+only but mistakes of atrt configuration or cluster settings. As ATRT testcase 
+file used simple test included in MySQL installation:
+
+	max-time: 600
+	cmd: testBasic
+	args: -n PkRead? T1
+
+
+Of course these examples don't cover all possible failures but at least most 
+probable and often appearing.
+Note: Before start atrt I recommend try to run selected cluster configuration 
+by manual and make sure that it can work: enough to run mgmd, ndbd, mysqld and 
+look via mgm to status of these nodes
+
+Example 1. Test passed
+
+2006-03-02 15:36:51 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup
+2006-03-02 15:36:51 [ndb_atrt] INFO     -- Starting...
+2006-03-02 15:36:51 [ndb_atrt] INFO     -- Connecting to hosts
+2006-03-02 15:36:51 [ndb_atrt] DEBUG    -- Connected to node1:1234
+2006-03-02 15:36:51 [ndb_atrt] DEBUG    -- Connected to node2:1234
+2006-03-02 15:36:51 [ndb_atrt] DEBUG    -- Connected to node3:1234
+2006-03-02 15:36:55 [ndb_atrt] INFO     -- (Re)starting ndb processes
+2006-03-02 15:37:11 [ndb_atrt] INFO     -- Ndb start completed
+2006-03-02 15:37:11 [ndb_atrt] INFO     -- #1 - testBasic -n PkRead? T1
+2006-03-02 16:37:16 [ndb_atrt] INFO     -- #1 OK (0)
+
+Example 2. Test failed.
+Reason: ATRT not started properly. d.txt not found. Probably necessary folders 
+and d.txt file don't exist.
+Solution: run make-config.sh d.tmp and try again.
+
+2006-03-02 18:32:08 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup
+2006-03-02 18:32:08 [ndb_atrt] INFO     -- Starting...
+2006-03-02 18:32:08 [ndb_atrt] CRITICAL -- Failed to open process config file: d.txt
+
+Example 3. Test failed.
+Reason: ATRT not started properly. Necessary folders were removed but d.txt file 
+exists.
+Solution: run make-config.sh d.tmp and try again.
+
+2006-03-02 18:30:54 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup
+2006-03-02 18:30:54 [ndb_atrt] INFO     -- Starting...
+2006-03-02 18:30:54 [ndb_atrt] INFO     -- Connecting to hosts
+2006-03-02 18:30:54 [ndb_atrt] DEBUG    -- Connected to node1:1234
+2006-03-02 18:30:54 [ndb_atrt] DEBUG    -- Connected to node2:1234
+2006-03-02 18:30:54 [ndb_atrt] DEBUG    -- Connected to node3:1234
+2006-03-02 18:30:58 [ndb_atrt] INFO     -- (Re)starting ndb processes
+2006-03-02 18:31:00 [ndb_atrt] CRITICAL -- Failed to setup process
+
+Example 4. Test failed.
+Reason: ATRT not started properly. node2 hasn't running ndb_cpcd process.
+Solution: log into node2 and starts ndb_cpcd process.
+
+2006-03-02 18:15:05 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup
+2006-03-02 18:15:05 [ndb_atrt] INFO     -- Starting...
+2006-03-02 18:15:05 [ndb_atrt] INFO     -- Connecting to hosts
+2006-03-02 18:15:05 [ndb_atrt] DEBUG    -- Connected to node1:1234
+2006-03-02 18:15:05 [ndb_atrt] ERROR    -- Unable to connect to cpc node2:1234
+
+Example 5. Test failed.
+Reason: ATRT not started properly. baseport option isn't defined in d.tmp or 
+baseport and PortNumber are different.
+Solution: correct d.tmp, run make-config.sh d.tmp and try again.
+
+2006-03-02 18:25:31 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup
+2006-03-02 18:25:31 [ndb_atrt] INFO     -- Starting...
+2006-03-02 18:25:31 [ndb_atrt] INFO     -- Connecting to hosts
+2006-03-02 18:25:31 [ndb_atrt] DEBUG    -- Connected to node1:1234
+2006-03-02 18:25:31 [ndb_atrt] DEBUG    -- Connected to node2:1234
+2006-03-02 18:25:31 [ndb_atrt] DEBUG    -- Connected to node3:1234
+2006-03-02 18:25:36 [ndb_atrt] INFO     -- (Re)starting ndb processes
+2006-03-02 18:26:08 [ndb_atrt] CRITICAL -- Unable to connect to ndb mgm node1:0
+
+Example 6. Test failed.
+Reason: ATRT not started properly. basedir option points to wrong path.
+Solution: correct basedir, run make-config.sh d.tmp and try again.
+
+2006-03-02 18:40:10 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup
+2006-03-02 18:40:10 [ndb_atrt] INFO     -- Starting...
+2006-03-02 18:40:10 [ndb_atrt] INFO     -- Connecting to hosts
+2006-03-02 18:40:10 [ndb_atrt] DEBUG    -- Connected to node1:1234
+2006-03-02 18:40:10 [ndb_atrt] DEBUG    -- Connected to node2:1234
+2006-03-02 18:40:10 [ndb_atrt] DEBUG    -- Connected to node3:1234
+2006-03-02 18:40:14 [ndb_atrt] INFO     -- (Re)starting ndb processes
+2006-03-02 18:40:16 [ndb_atrt] ERROR    -- Unable to start process: Failed to start
+
+Example 7. Test failed.
+Reason: ndb nodes have problems on starting.
+Solution: Check configuration of ndb nodes.
+
+2006-03-02 18:46:44 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup
+2006-03-02 18:46:44 [ndb_atrt] INFO     -- Starting...
+2006-03-02 18:46:44 [ndb_atrt] INFO     -- Connecting to hosts
+2006-03-02 18:46:44 [ndb_atrt] DEBUG    -- Connected to node1:1234
+2006-03-02 18:46:44 [ndb_atrt] DEBUG    -- Connected to node2:1234
+2006-03-02 18:46:44 [ndb_atrt] DEBUG    -- Connected to node3:1234
+2006-03-02 18:46:50 [ndb_atrt] INFO     -- (Re)starting ndb processes
+2006-03-02 18:52:59 [ndb_atrt] CRITICAL -- wait ndb timed out 3 3 4
+2006-03-02 18:58:59 [ndb_atrt] CRITICAL -- wait ndb timed out 3 3 4
+
+Example 8. Test failed.
+Reason: test application not found.
+Solution: Correct file name in ATRT test case file and make sure that the file 
+exists in $MYSQL_DIR/bin directory.
+
+2006-03-02 20:21:54 [ndb_atrt] INFO     -- Starting...
+2006-03-02 20:21:54 [ndb_atrt] INFO     -- Connecting to hosts
+2006-03-02 20:21:54 [ndb_atrt] DEBUG    -- Connected to node1:1234
+2006-03-02 20:21:54 [ndb_atrt] DEBUG    -- Connected to node2:1234
+2006-03-02 20:21:54 [ndb_atrt] DEBUG    -- Connected to node3:1234
+2006-03-02 20:21:57 [ndb_atrt] INFO     -- (Re)starting ndb processes
+2006-03-02 20:22:12 [ndb_atrt] INFO     -- Ndb start completed
+2006-03-02 20:22:12 [ndb_atrt] INFO     -- #1 - testBasic123 -n PkRead? T2
+2006-03-02 20:22:13 [ndb_atrt] ERROR    -- Unable to start process: Failed to start
+
+Example 9. Test failed.
+Probable reasons:
+* wrong arguments for test application
+* itself test failed
+* timeout reached
+Solution: Since mgmd/ndbd nodes started properly in such case then try to 
+investigate log files in result/X.api/, result/X.mysqld, result/X.mysql 
+directories.
+
+2006-03-02 19:59:35 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup
+2006-03-02 19:59:35 [ndb_atrt] INFO     -- Starting...
+2006-03-02 19:59:35 [ndb_atrt] INFO     -- Connecting to hosts
+2006-03-02 19:59:35 [ndb_atrt] DEBUG    -- Connected to node1:1234
+2006-03-02 19:59:35 [ndb_atrt] DEBUG    -- Connected to node2:1234
+2006-03-02 19:59:35 [ndb_atrt] DEBUG    -- Connected to node3:1234
+2006-03-02 20:00:49 [ndb_atrt] INFO     -- (Re)starting ndb processes
+2006-03-02 20:01:05 [ndb_atrt] INFO     -- Ndb start completed
+2006-03-02 20:01:05 [ndb_atrt] INFO     -- #1 - testBasic -n PkRead? T1
+2006-03-02 20:01:38 [ndb_atrt] INFO     -- #1 FAILED(256)
+
+
+  3.ATRT Command Line Options.
+  ============================
+
+--process-config=string
+    Specify ATRT configuration file. If not specified, ATRT will look in local 
+    directory for the d.txt file.
+
+--setup-path=string
+    This path points to place where necessary directories (created by 
+    make-config) can be found. Note: d.txt should be in same directory where 
+    you start atrt!
+
+-v (verbose)
+
+    * without the option: atrt prints only [CRITICAL] events
+    * one -v: atrt prints [CRITICAL], [INFO] events
+    * two -v: atrt prints [CRITICAL], [INFO], [DEBUG] events
+
+--log-file=string
+    Used to specify file to log ATRT's results on starting application and 
+    running tests.
+
+--testcase-file=string -f
+    Used to feed ATRT test cases in a text file.
+
+--report-file=string
+    File to record test results
+
+-i, --interactive
+    ATRT terminates on first test failure
+
+-r, --regression
+    Continues even on test failures
+
+-b, --bench
+    Always produce report
+