updates after writing https://kb.askmonty.org/en/mysql-test-auxiliary-files

mysql-test/README.suites:
  removed. it was turned into a manual, eventually
mysql-test/include/default_mysqld.cnf:
  use normal environment variables instead of magic @OPT.vardir
mysql-test/lib/My/ConfigFactory.pm:
  use normal environment variables instead of magic @OPT.vardir
mysql-test/mysql-test-run.pl:
  initialize environment variables before parsing the first my.cnf template - it may refer to them
mysql-test/suite/sphinx/my.cnf:
  use normal environment variables instead of magic @OPT.vardir

mysql-test/README.suites

-These are the assorted notes that will be turned into a manual eventually.
-
-==========================
-Tests are organized in suites.
-A "suite" is a subdirectory inside, one of,
-
-  <basedir>/mysql-test/suite
-  <basedir>/mysql-test
-  <basedir>/share/mysql-test/suite
-  <basedir>/share/mysql-test
-  <basedir>/share/mysql/mysql-test/suite
-  <basedir>/share/mysql/mysql-test
-  <basedir>/storage/*/mysql-test-suites
-
-This is supposed to cover running mtr from a source directory and installed.
-
-==========================
-A suite contains *.test and *.result files. They can be in the t/ and r/
-subdirectories under the suitedir or directly in the suitedir
-(that is suitedir/t/*.test or suitedir/*.test, same for *.result))
-
-==========================
-A suite can contain a suite.opt file - at the same location where .test files
-are or in the suite directory. As usual, the .opt file can use $-substitutions
-for the environment variables.
-
-Usually, using my.cnf template (see below) is preferrable.
-
-But command line options (.opt files and combinations file) get special
-treatment - they can have special options that affect mtr behavior. cnf
-files cannot. Special options are
- --timezone, --plugin-load, --default-time-zone,
-
-In particular, all --plugin-load instances on the command line (on the
-combined command line, assembled from different .opt and combinations
-files) are merged into one. That is, if, say, test-master.opt file contains
---plugin-load=aaa.so and suite.opt has --plugin-load=bbb.so that mysqld
-will get --plugin-load=aaa.so:bbb.so. Also, empty --plugin-load options are
-removed from the command line. Which means that one can safely specify
---plugin-load=$AAA_SO and if aaa.so was not built (perhaps, the plugin was
-statically linked into the server), the .opt file will not result in the
-invalid command line option that can cause the server to refuse to start.
-
-==========================
-A suite can have suite.pm file in the suitedir. It must declare a
-package that inherits from My::Suite.
-
-The suite.pm needs to have @ISA=qw(My::Suite) and it must end
-with bless {}; - that is it must return an object of that class.
-It can also return a string - in this case all tests in the suite
-will be skipped, with this string being printed as a reason.
-
-A suite class can define config_files(), servers(), list_cases(),
-start_test() methods, and skip_combinations() methods.
-
-A config_files() method returns a list of additional config files (besides
-my.cnf), that this suite needs to be created. For every file it specifies
-a function that will create it, when given a My::Config object. For example:
-
-   sub config_files { ( 'config.ini' => \&write_ini, 'new.conf' => \&do_new ) }
-
-A servers() method returns a list of processes that needs to be started for
-this suite. A process is specified as a pair (regex, hash). A regex must
-match a section in the my.cnf template (for example, qr/mysqld\./ corresponds
-to all mysqld processes), a hash contains these options:
-
-   SORT => a number, processes are started in the order of increasing SORT
-           values (and stopped in the reverse order). mysqld has number 300.
-   START => a function to start a process. It takes two arguments,
-            My::Config::Group and My::Test. If START is undefined the process
-            will not be started.
-   WAIT => a function waits for the process to be started. It takes
-           My::Config::Group as an argument. Internallys mtr first invokes
-           START for all processes, then WAIT for all started processes.
-
-example: sub servers { ( qr/^foo$/ => { SORT => 200,
-                                        START => \&start_foo,
-                                        WAIT => \&wait_foo } ) }
-
-See sphinx suite for an example.
-
-A list_cases() method returns a complete list of cases for this suite.
-By default it will be the list of files that have .test extension,
-but without the extension. This list will be filtered by mtr, subject to
-different mtr options (--big-test, --start-from, etc), the suite does not have
-to do it.
-
-A start_test() method starts one test process, by default it will be mysqltest.
-
-See unit suite for an example of list_cases() and start_test() methods.
-
-A skip_combinations() method returns a hash that maps file names
-(where combinations are defined) to a list of combinations that should
-be skipped. For example
-
-  sub skip_combinations { (
-      'combinations' => [ 'mix', 'rpl' ],
-      'inc/many.combinations' => [ 'a', 'bb', 'c' ]
-  ) }
-
-==========================
-A suite can have my.cnf template file in the suitedir.
-A my.cnf template uses a normal my.cnf syntax - groups, options,
-and values - with templating extensions. They are
-
-* There can be groups with non-standard names, not used by mysqld.
-  These groups may be used by the suite.pm file somehow.
-  For example, they can be written to the additional config files.
-  See sphinx suite for an example.
-
-* There can be ENV group. It sets values for the environment variables.
-
-* Values can refer to each other - they will be expanded as needed.
-  A reference to a value of an option looks like @groupname.optionname.
-  For example
-
-  [mysqld.2]
-  master-port= @mysqld.1.port
-
-  it sets the master-port in the mysqld.2 group to the value of
-  port in the mysqld.1 group.
-
-* An option name may start from '#'. In the resulting my.cnf it will look
-  like a comment, but it still can be referred to. For example:
-
-  [example]
-  #foo = localhost:@mysqld.1.port
-  bar = http://@example.#foo/index.html
-
-* There are two special - in this regard - groups.
-
-  Via the ENV group one can refer to any environment variable, not only
-    to values in the [ENV] group of my.cnf file.
-
-  Via the OPT group one can refer to special values:
-    @OPT.vardir - a path to vardir
-    @OPT.port - a new port number is reserved out of the pool. It will not
-                match any other port number used by this test run.
-  See sphinx suite for an example.
-
-Most probably a suite my.cnf will need to start from
-
-  !include include/default_my.cnf
-
-and then modify the configuration as necessary.
-
-==========================
-A suite can have combinations file in the suitedir. It uses my.cnf syntax
-but it cannot use @-substitutions. Instead, it can use $-substitutions for
-the environment variables. Because the combination options will not be
-merged to a my.cnf, but will be added to the command line. Example:
-
-  [conf1]
-  opt1=val1
-
-  [conf2]
-  opt1=val2
-  opt2=$HAVE_SOMETHING
-
-Such a file will cause every test from the suite to be run twice - once
-with mysqld using --opt1=val1 and the other one with mysqld using
---opt1=val2 --opt2=$HAVE_SOMETHING
-==========================
-

mysql-test/include/default_mysqld.cnf

@@ -23,7 +23,7 @@ character-set-server=      latin1
 # disconnects when test servers are put under load see BUG#28359
 connect-timeout=            60
 
-secure-file-priv= @OPT.vardir
+secure-file-priv= @ENV.MYSQLTEST_VARDIR
 
 log-basename=mysqld
 debug-no-sync

mysql-test/lib/My/ConfigFactory.pm

@@ -37,7 +37,6 @@ sub add_opt_values {
 
   # add auto-options
   $config->insert('OPT', 'port'   => sub { fix_port($self, $config) });
-  $config->insert('OPT', 'vardir' => sub { $self->{ARGS}->{vardir} });
   $config->insert('mysqld', "loose-skip-$_" => undef) for (@::optional_plugins);
 }
 

mysql-test/mysql-test-run.pl

@@ -2357,7 +2357,7 @@ sub environment_setup {
   $ENV{'DEFAULT_MASTER_PORT'}= $mysqld_variables{'port'};
   $ENV{'MYSQL_TMP_DIR'}=      $opt_tmpdir;
   $ENV{'MYSQLTEST_VARDIR'}=   $opt_vardir;
-  $ENV{'MYSQL_BINDIR'}=       "$bindir";
+  $ENV{'MYSQL_BINDIR'}=       $bindir;
   $ENV{'MYSQL_SHAREDIR'}=     $path_language;
   $ENV{'MYSQL_CHARSETSDIR'}=  $path_charsetsdir;
   
@@ -3402,6 +3402,7 @@ sub sql_to_bootstrap {
 
 sub default_mysqld {
   # Generate new config file from template
+  environment_setup();
   my $config= My::ConfigFactory->new_config
     ( {
        basedir         => $basedir,

mysql-test/suite/sphinx/my.cnf

@@ -8,7 +8,7 @@ xmlpipe_command = cat @ENV.MTR_SUITE_DIR/testdata.xml
 source = src1
 docinfo = extern
 charset_type = utf-8
-path = @OPT.vardir/searchd/test1
+path = @ENV.MYSQLTEST_VARDIR/searchd/test1
 
 [indexer]
 mem_limit = 32M
@@ -20,10 +20,10 @@ max_matches = 1000
 seamless_rotate = 1
 preopen_indexes = 0
 unlink_old = 1
-log = @OPT.vardir/searchd/sphinx-searchd.log
-query_log = @OPT.vardir/searchd/sphinx-query.log
-#log-error = @OPT.vardir/searchd/sphinx.log
-pid_file = @OPT.vardir/run/searchd.pid
+log = @ENV.MYSQLTEST_VARDIR/searchd/sphinx-searchd.log
+query_log = @ENV.MYSQLTEST_VARDIR/searchd/sphinx-query.log
+#log-error = @ENV.MYSQLTEST_VARDIR/searchd/sphinx.log
+pid_file = @ENV.MYSQLTEST_VARDIR/run/searchd.pid
 port = @ENV.SPHINXSEARCH_PORT
 
 [ENV]