How does Samba do testing? $ make test TESTS=foo to run all the tests matching the regular-ish expression "foo". e.g. "ldap.food", "locking.shoot-foot" $ make test TESTS='foo bar' to match more than one pattern (not "foo|bar"). `make test TESTS=foo` calls `./buildtools/bin/waf --test TESTS=foo` which runs: (/usr/bin/perl /home/douglas/src/samba/selftest/selftest.pl --target=samba --prefix=./st --srcdir=/home/douglas/src/samba --exclude=/home/douglas/src/samba/selftest/skip --testlist="/usr/bin/python /home/douglas/src/samba/selftest/tests.py|" --testlist="/usr/bin/python /home/douglas/src/samba/source3/selftest/tests.py|" --testlist="/usr/bin/python /home/douglas/src/samba/source4/selftest/tests.py|" --exclude=/home/douglas/src/samba/selftest/slow --nss_wrapper_so_path=/home/douglas/src/samba/bin/default/lib/nss_wrapper/libnss-wrapper.so --resolv_wrapper_so_path=/home/douglas/src/samba/bin/default/lib/resolv_wrapper/libresolv-wrapper.so --socket_wrapper_so_path=/home/douglas/src/samba/bin/default/lib/socket_wrapper/libsocket-wrapper.so --uid_wrapper_so_path=/home/douglas/src/samba/bin/default/lib/uid_wrapper/libuid-wrapper.so --use-dns-faking --socket-wrapper foo && touch ./st/st_done) | /usr/bin/python -u /home/douglas/src/samba/selftest/filter-subunit --expected-failures=/home/douglas/src/samba/selftest/knownfail --flapping=/home/douglas/src/samba/selftest/flapping | tee ./st/subunit | /usr/bin/python -u /home/douglas/src/samba/selftest/format-subunit --prefix=./st --immediate s {/home/douglas/src/samba/}{.} s {/usr/bin/}{} (perl ./selftest/selftest.pl --target=samba --prefix=./st --srcdir=. --exclude=./selftest/skip --testlist="python ./selftest/tests.py|" --testlist="python ./source3/selftest/tests.py|" --testlist="python ./source4/selftest/tests.py|" --exclude=./selftest/slow --nss_wrapper_so_path=./bin/default/lib/nss_wrapper/libnss-wrapper.so --resolv_wrapper_so_path=./bin/default/lib/resolv_wrapper/libresolv-wrapper.so --socket_wrapper_so_path=./bin/default/lib/socket_wrapper/libsocket-wrapper.so --uid_wrapper_so_path=./bin/default/lib/uid_wrapper/libuid-wrapper.so --use-dns-faking --socket-wrapper foo && touch ./st/st_done) | python -u ./selftest/filter-subunit --expected-failures=./selftest/knownfail --flapping=./selftest/flapping | tee ./st/subunit | python -u ./selftest/format-subunit --prefix=./st --immediate (perl ./selftest/selftest.pl --target=samba --prefix=./st --srcdir=. \ --exclude=./selftest/skip \ --testlist="python ./selftest/tests.py|" \ --testlist="python ./source3/selftest/tests.py|" \ --testlist="python ./source4/selftest/tests.py|" \ --exclude=./selftest/slow \ --nss_wrapper_so_path=./bin/default/lib/nss_wrapper/libnss-wrapper.so \ --resolv_wrapper_so_path=./bin/default/lib/resolv_wrapper/libresolv-wrapper.so \ --socket_wrapper_so_path=./bin/default/lib/socket_wrapper/libsocket-wrapper.so \ --uid_wrapper_so_path=./bin/default/lib/uid_wrapper/libuid-wrapper.so \ --use-dns-faking \ --socket-wrapper \ foo \ && touch ./st/st_done) | python -u ./selftest/filter-subunit --expected-failures=./selftest/knownfail --flapping=./selftest/flapping | tee ./st/subunit | python -u ./selftest/format-subunit --prefix=./st --immediate --testlist="a-list-of-tests.txt" --testlist="a-script-that-generates-a-list-of-tests.sh|" --exclude=./selftest/slow # takes some out foo # a regular expression (case insensitive) match leads to: if (should_run_test($name) == 1) { push (@ret, [$name, $env, $cmdline, $loadlist]); } foo example: -- TEST -- samba3.blackbox.smbclient_posix_large -e NTLM(nt4_dc:local) nt4_dc:local /home/douglas/src/samba/source3/script/tests/test_smbclient_posix_large.sh none bin/smbclient $SERVER $PREFIX -e -U$USERNAME%$PASS WORD --configfile=$SMB_CONF_PATH 2>&1 | /home/douglas/src/samba/selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox .smbclient_posix_large -e NTLM." --suffix="(nt4_dc:local)" -- TEST -- samba3.blackbox.smbclient_netbios_aliases [foo](ad_member:local) ad_member:local /home/douglas/src/samba/source3/script/tests/test_smbclient_netbios_aliases.sh bin/smbclient foo $DC_USERNAME $DC_PASSWORD $PREFIX -e --configfile=$SMB_CONF_PATH 2>&1 | /home/douglas/src/samba/selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." --suffix="(ad_member:local)" -- TEST -- samba3.blackbox.smbclient_netbios_aliases [bar](ad_member:local) ad_member:local /home/douglas/src/samba/source3/script/tests/test_smbclient_netbios_aliases.sh bin/smbclient bar $DC_USERNAME $DC_PASSWORD $PREFIX -e --configfile=$SMB_CONF_PATH 2>&1 | /home/douglas/src/samba/selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox.smbclient_netbios_aliases [bar]." --suffix="(ad_member:local)" -- TEST-LOADLIST -- samba3.rpc.lsa.lookupsids over ncacn_ip_tcp with [] (nt4_dc) nt4_dc bin/smbtorture --list --configfile=$SMB_CONF_PATH --option='fss:sequence timeout=1' --maximum-runtime=$SELFTEST_MAXTIME --basedir=$SELFTEST_TMPDIR --format=subunit --option=torture:progress=no --option=torture:sharedelay=100000 --option=torture:writetimeupdatedelay=500000 --target=samba3 ncacn_ip_tcp:$SERVER_IP[] -U$USERNAME%$PASSWORD rpc.lsa.lookupsids | /home/douglas/src/samba/selftest/filter-subunit --list --fail-on-empty --prefix="samba3.rpc.lsa.lookupsids over ncacn_ip_tcp with [] ." --suffix="(nt4_dc)" bin/smbtorture $LOADLIST --configfile=$SMB_CONF_PATH --option='fss:sequence timeout=1' --maximum-runtime=$SELFTEST_MAXTIME --basedir=$SELFTEST_TMPDIR --format=subunit --option=torture:progress=no --option=torture:sharedelay=100000 --option=torture:writetimeupdatedelay=500000 --target=samba3 ncacn_ip_tcp:$SERVER_IP[] -U$USERNAME%$PASSWORD rpc.lsa.lookupsids 2>&1 | /home/douglas/src/samba/selftest/filter-subunit --fail-on-empty --prefix="samba3.rpc.lsa.lookupsids over ncacn_ip_tcp with [] ." --suffix="(nt4_dc)" -- TEST-LOADLIST -- containing: -- TEST -- samba3.blackbox.smbclient_netbios_aliases [foo](ad_member:local) ad_member:local /home/douglas/src/samba/source3/script/tests/test_smbclient_netbios_aliases.sh bin/smbclient foo $DC_USERNAME $DC_PASSWORD $PREFIX -e --configfile=$SMB_CONF_PATH 2>&1 | /home/douglas/src/samba/selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." --suffix="(ad_member:local)" -- TEST -- generated by: --testlist="python ./source3/selftest/tests.py|" \ which runs tests with LD_PRELOAD etc LD_PRELOAD=/home/douglas/src/samba/bin/default/lib/nss_wrapper/libnss-wrapper.so:/home/douglas/src/samba/bin/default/lib/resolv_wrapper/libresolv-wrapper.so:/home/douglas/src/samba/bin/default/lib/socket_wrapper/libsocket-wrapper.so:/home/douglas/src/samba/bin/default/lib/uid_wrapper/libuid-wrapper.so SOCKET_WRAPPER_DIR=/home/douglas/src/samba/st/w and runs in a test environment (using wrappers): command: /home/douglas/src/samba/source3/script/tests/test_smbclient_netbios_aliases.sh /home/douglas/src/samba/bin/smbclient foo $DC_USERNAME $DC_PASSWORD $PREFIX -e --configfile=$SMB_CONF_PATH 2>&1 | /home/douglas/src/samba/selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." --suffix="(ad_member:local)" expanded command: /home/douglas/src/samba/source3/script/tests/test_smbclient_netbios_aliases.sh /home/douglas/src/samba/bin/smbclient foo Administrator locDCpass1 ./st -e --configfile=/home/douglas/src/samba/st/ad_member/lib/server.conf 2>&1 | /home/douglas/src/samba/selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." --suffix="(ad_member:local)" command: ./source3/script/tests/test_smbclient_netbios_aliases.sh ./bin/smbclient foo $DC_USERNAME $DC_PASSWORD $PREFIX -e --configfile=$SMB_CONF_PATH 2>&1 | ./selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." --suffix="(ad_member:local)" expanded command: ./source3/script/tests/test_smbclient_netbios_aliases.sh ./bin/smbclient foo Administrator locDCpass1 ./st -e --configfile=./st/ad_member/lib/server.conf 2>&1 | ./selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." --suffix="(ad_member:local)" ./source3/script/tests/test_smbclient_netbios_aliases.sh ./bin/smbclient foo Administrator locDCpass1 ./st -e --configfile=./st/ad_member/lib/server.conf 2>&1 |\ ./selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." --suffix="(ad_member:local)" # the actual test: ./source3/script/tests/test_smbclient_netbios_aliases.sh ./bin/smbclient foo Administrator locDCpass1 ./st -e --configfile=./st/ad_member/lib/server.conf \ #redirected to 2>&1 |\ ./selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." --suffix="(ad_member:local)" # filter-subunit fiddles with the script output, discarding lines it thinks irrelevant #XXX example ######### all of this goes into filter-subunit AGAIN: # this time with expected failures and flapping tests | python -u ./selftest/filter-subunit --expected-failures=./selftest/knownfail --flapping=./selftest/flapping # a copy is saved here | tee ./st/subunit # and the output is reformatted again. | python -u ./selftest/format-subunit --prefix=./st --immediate # this prints a fairly useless summary # and prints some messages to the terminal # takes out time -time: 2017-04-30 08:19:24.070516Z # puts in: +[1(0)/1 at 0s] samba3.blackbox.smbclient_netbios_aliases [foo](ad_member:local) +ALL OK (2 tests in 1 testsuites) + +A summary with detailed information can be found in: + ./st/summary +TOP 10 slowest tests # etc # to measure times, we made a new mode for filter-subunit that converts time: 2017-04-30 08:19:24.070516Z test: samba3.blackbox.smbclient_netbios_aliases [foo].kinit(ad_member:local) time: 2017-04-30 08:19:24.153745Z # into # it would have been sufficient to make a format-subunit-json #to avoid having to work out the correct amount of escaping: make test TESTS='\(foo\|bar\)' #...space separated regexes work: make test TESTS='foo bar' ------------- Actual testenv output for foo test: (SELFTEST_TESTENV=ad_member:local make testenv) $ ./source3/script/tests/test_smbclient_netbios_aliases.sh ./bin/smbclient foo Administrator locDCpass1 ./st -e --configfile=./st/ad_member/lib/server.conf time: 2017-04-30 11:32:36.488603Z test: kinit time: 2017-04-30 11:32:36.616158Z success: kinit time: 2017-04-30 11:32:36.618076Z test: smbclient time: 2017-04-30 11:32:37.399131Z success: smbclient ./source3/script/tests/test_smbclient_netbios_aliases.sh ./bin/smbclient foo Administrator locDCpass1 ./st -e --configfile=./st/ad_member/lib/server.conf | selftest/filter-subunit --perf-test-output elapsed-time: kinit: 0.105706 elapsed-time: smbclient: 0.429435 Add | python selftest/format-subunit-json --prefix=st for: A JSON file summarising these tests performance found in: st/summary.json $ cat st/summary.json { "kinit": 0.084216, "smbclient": 0.43144 }