Measuring Samba performance

Douglas Bagnall

douglas.bagnall@catalyst.net.nz

Hurriedly reducing the scope of the talk

A very large topic

I will talk about a couple of specific things

make perftest
generating realistic traffic

Motivation

A scalability or performance issue is noticed
You come up with a theory and a patch
does it help?
Do we sometimes regress?

Specific thing 1: `make perftest`

Samba has a well-developed, slightly weird, test system

make test runs regression and unit tests

`make test`

To test everything*:

$ make test

To run some tests:

$ make test TESTS=foo

* this presentation may contain lies

`make test`

$ make test TESTS=foo

is converted to

./buildtools/bin/waf --test TESTS=foo

`make test`

$ ./buildtools/bin/waf --test TESTS=foo

calls

(/usr/bin/perl /home/douglas/src/samba/selftest/selftest.pl --target=samba --prefix=./st --srcdir=/home/douglas/src/samba --exclude=/home/douglas/src/samba/selftest/skip --testlist="/usr/bin/python /home/douglas/src/samba/selftest/tests.py|" --testlist="/usr/bin/python /home/douglas/src/samba/source3/selftest/tests.py|" --testlist="/usr/bin/python /home/douglas/src/samba/source4/selftest/tests.py|" --exclude=/home/douglas/src/samba/selftest/slow --nss_wrapper_so_path=/home/douglas/src/samba/bin/default/lib/nss_wrapper/libnss-wrapper.so --resolv_wrapper_so_path=/home/douglas/src/samba/bin/default/lib/resolv_wrapper/libresolv-wrapper.so --socket_wrapper_so_path=/home/douglas/src/samba/bin/default/lib/socket_wrapper/libsocket-wrapper.so --uid_wrapper_so_path=/home/douglas/src/samba/bin/default/lib/uid_wrapper/libuid-wrapper.so --use-dns-faking --socket-wrapper foo && touch ./st/st_done) | /usr/bin/python -u /home/douglas/src/samba/selftest/filter-subunit --expected-failures=/home/douglas/src/samba/selftest/knownfail --flapping=/home/douglas/src/samba/selftest/flapping | tee ./st/subunit | /usr/bin/python -u /home/douglas/src/samba/selftest/format-subunit --prefix=./st --immediate

`make test`

reformatted with abbreviated paths:

(perl ./selftest/selftest.pl --target=samba \
            --prefix=./st \
            --srcdir=. \
            --exclude=./selftest/skip \
            --testlist="python ./selftest/tests.py|" \
            --testlist="python ./source3/selftest/tests.py|" \
            --testlist="python ./source4/selftest/tests.py|"  \
            --exclude=./selftest/slow \
            --nss_wrapper_so_path=$SOMEWHERE/nss_wrapper/libnss-wrapper.so \
            --resolv_wrapper_so_path=$SOMEWHERE/resolv_wrapper/libresolv-wrapper.so \
            --socket_wrapper_so_path=$SOMEWHERE/socket_wrapper/libsocket-wrapper.so \
            --uid_wrapper_so_path=$SOMEWHERE/uid_wrapper/libuid-wrapper.so \
            --use-dns-faking \
            --socket-wrapper \
            foo \
 && touch ./st/st_done \
         ) \
| python -u ./selftest/filter-subunit \
           --expected-failures=./selftest/knownfail \
           --flapping=./selftest/flapping \
| tee ./st/subunit \
| python -u ./selftest/format-subunit \
           --prefix=./st \
           --immediate

make test continued

(perl ./selftest/selftest.pl --target=samba \
            --prefix=./st \
            --srcdir=. \
            --exclude=./selftest/skip \
            --testlist="python ./selftest/tests.py|" \
            --testlist="python ./source3/selftest/tests.py|" \
            --testlist="python ./source4/selftest/tests.py|"  \
            --exclude=./selftest/slow \
    [...]

test are defined in these lists

  --testlist="a-list-of-tests.txt"
  
  --testlist="a-script-that-generates-a-list-of-tests.sh|"

test list format


-- TEST --
samba3.blackbox.smbclient_posix_large -e NTLM(nt4_dc:local)
nt4_dc:local
/home/douglas/src/samba/source3/script/tests/test_smbclient_posix_large.sh none bin/smbclient $SERVER $PREFIX -e -U$USERNAME%$PASS
WORD --configfile=$SMB_CONF_PATH 2>&1  | /home/douglas/src/samba/selftest/filter-subunit --fail-on-empty --prefix="samba3.blackbox
.smbclient_posix_large -e NTLM." --suffix="(nt4_dc:local)"
-- TEST --
samba3.blackbox.smbclient_netbios_aliases [foo](ad_member:local)
ad_member:local
/home/douglas/src/samba/source3/script/tests/test_smbclient_netbios_aliases.sh \
  bin/smbclient foo $DC_USERNAME $DC_PASSWORD $PREFIX -e --configfile=$SMB_CONF_PATH 2>&1 \
  | /home/douglas/src/samba/selftest/filter-subunit --fail-on-empty \
  --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." --suffix="(ad_member:local)"
-- TEST --
samba3.blackbox.smbclient_netbios_aliases [bar](ad_member:local)
ad_member:local
/home/douglas/src/samba/source3/script/tests/test_smbclient_netbios_aliases.sh bin/smbclient bar $DC_US

generated by:

  --testlist="python ./source3/selftest/tests.py|"

`make test TESTS=foo`

 ./source3/script/tests/test_smbclient_netbios_aliases.sh \
     ./bin/smbclient foo Administrator locDCpass1 \
     ./st -e --configfile=./st/ad_member/lib/server.conf 2>&1  \
 | \
 ./selftest/filter-subunit --fail-on-empty \
     --prefix="samba3.blackbox.smbclient_netbios_aliases [foo]." \
     --suffix="(ad_member:local)"

the script output can contain all sorts of stuff

filter-subunit reduces it to lines beginning with certain words

(“success”, “failure”, “error”, etc).

`make test TESTS=foo`

through filter-subunit again and into format-subunit:

    ( [the bit just discussed] \
         && touch ./st/st_done \
         ) \
    | python -u ./selftest/filter-subunit \
             --expected-failures=./selftest/knownfail \
             --flapping=./selftest/flapping \
    |  tee ./st/subunit \
    | python -u ./selftest/format-subunit \
             --prefix=./st \
             --immediate

format-subunit

takes out test details
adds summary

@@ -178,20 +177,12 @@
 Copyright Andrew Tridgell and the Samba Team 1992-2017
 initialize_winbindd_cache: clearing cache and re-creating with version number 2
 STATUS=daemon 'winbindd' finished starting up and ready to serve connections
-testsuite: samba3.blackbox.smbclient_netbios_aliases [foo](ad_member:local)
-progress: push
-time: 2017-04-30 20:19:24.000000Z
[...]
-time: 2017-04-30 08:19:24.974298Z
-successful: samba3.blackbox.smbclient_netbios_aliases [foo].smbclient(ad_member:local)
-time: 2017-04-30 20:19:24.000000Z
-progress: pop
-command: /home/douglas/src/samba/source3/script/tests/test_smbclient_netbios_aliases.sh [...]
-expanded command: /home/douglas/src/samba/source3/script/tests/test_smbclient_netbios_al[...]
-testsuite-success: samba3.blackbox.smbclient_netbios_aliases [foo](ad_member:local)
+[1(0)/1 at 0s] samba3.blackbox.smbclient_netbios_aliases [foo](ad_member:local) 
+ALL OK (2 tests in 1 testsuites)
+
+A summary with detailed information can be found in:
+  ./st/summary
+TOP 10 slowest tests

Running a test as a performance test

make perf-test TESTS=foo TEST_LIST="python ./source3/selftest/tests.py|"

changes the filter and format stages

filter-subunit --perf-test-output looks like

elapsed-time: kinit: 0.084216
elapsed-time: smbclient: 0.43144

which format-subunit-json converts to:

{
    "kinit": 0.084216,
    "smbclient": 0.43144
}

performance specific tests

Most tests make bad performance tests

unrealistic edge cases
too quick (causing jitter)

I aim for 10 or 20 seconds per test

test interdependence is OK

Adding performance specific tests

Supposing you want this to work:

make perftest  TESTS='foo'

add the test to selftest/perf_tests.py:

+for alias in ["foo", "bar"]:
+    plantestsuite("samba3.blackbox.smbclient_netbios_aliases [%s]" % alias, 
+                  "ad_member:local", 
+                  [os.path.join(samba3srcdir, 
+                                "script/tests/test_smbclient_netbios_aliases.sh"),
+                   smbclient3, alias, "$DC_USERNAME", "$DC_PASSWORD", "$PREFIX",
+                   "-e", configuration])

(copy-paste-reformat from ./source3/selftest/tests.py)

aggregating results

We only really care about changes in perfomance

samba-cloud-autobuild

Catalysts repository of miscellaneous Samba testing stuff.

http://git.catalyst.net.nz/gw?p=samba-cloud-autobuild.git;a=tree

a carefully crafted mess

using multi-perf-test

run the foo test multiple times on multiple commits in a temporary directory

./scripts/multi-perf-test -o results.json \
     -r git://git.catalyst.net.nz/samba.git \
     --best-of 5 -t foo \
     --preserve-tmp-dir \
     948a1dab4d3 ca0077ee73^^^..ca0077ee73

... but the foo test are not backported

backporting make perftest TESTS='foo'

multi-perf-test maintains a set of patches

backporting tests and test infrastructure
fixing new compile errors in old versions

viewing results

./scripts/graph-perf-json -i results.json  \\
   --label-commits-from-git-tree ~/src/samba

See Andrew's talk for examples

Generating realistic loads

We want to test real use patterns

but real-world Samba use is always secret

AD DC load modelling (work in progress)

Take a pcap file and throw away sensitive information

timestamp             IP prot. stream   src    dest  protocol  opcode   op name       | operation specific...

1487921562.592911000    11              1       3       dns     1       response
1487921562.593315000    06      3       5       1       ldap    3       searchRequest   2       DC,DC           cn
1487921562.596247000    11              3       1       dns     0       query
1487921562.596362000    11              1       4       dns     0       query
1487921562.596697000    11              4       1       dns     1       response
1487921562.596921000    11              1       3       dns     1       response
1487921562.598308000    11              3       1       dns     0       query
1487921562.598414000    11              1       4       dns     0       query
1487921562.598729000    11              4       1       dns     1       response
1487921562.598963000    11              1       3       dns     1       response
1487921562.607624000    11              6       1       dns     0       query
1487921562.607956000    11              6       1       dns     0       query
1487921562.608009000    11              1       6       dns     1       response
1487921562.608232000    11              1       6       dns     1       response
1487921562.612424000    11              6       1       dns     0       query
1487921562.612648000    11              1       6       dns     1       response
1487921562.720442000    11              6       1       cldap   3       searchRequest                           Netlogon                
1487921562.720706000    11              6       1       cldap   3       searchRequest                           Netlogon                
1487921562.721004000    11              6       1       cldap   3       searchRequest                           Netlogon                
1487921562.724801000    11              1       6       cldap   5       searchResDone                                           
1487921562.728632000    11              1       6       cldap   5       searchResDone                                           
1487921562.732508000    11              1       6       cldap   5       searchResDone                                           
1487921562.748004000    06      3       1       5       ldap    5       searchResDone                                           
1487921562.820387000    06      3       5       1       ldap    2       unbindRequest                                           
1487921562.831445000    06      14      6       1       dcerpc  11      Bind

separate DNS

DNS is modelled as a DNS hammer


timestamp             IP prot. stream   src    dest  protocol  opcode   op name       | operation specific...
1487921562.593315000    06      3       5       1       ldap    3       searchRequest   2       DC,DC           cn
1487921562.720442000    11              6       1       cldap   3       searchRequest                           Netlogon                
1487921562.720706000    11              6       1       cldap   3       searchRequest                           Netlogon                
1487921562.721004000    11              6       1       cldap   3       searchRequest                           Netlogon                
1487921562.724801000    11              1       6       cldap   5       searchResDone                                           
1487921562.728632000    11              1       6       cldap   5       searchResDone                                           
1487921562.732508000    11              1       6       cldap   5       searchResDone                                           
1487921562.748004000    06      3       1       5       ldap    5       searchResDone                                           
1487921562.820387000    06      3       5       1       ldap    2       unbindRequest                                           
1487921562.831445000    06      14      6       1       dcerpc  11      Bind

model conversations


timestamp             IP prot. stream   src    dest  protocol  opcode   op name       | operation specific...
1487921562.593315000    06      3       5       1       ldap    3       searchRequest   2       DC,DC           cn
1487921562.720442000    11              6       1       cldap   3       searchRequest                           Netlogon                
1487921562.720706000    11              6       1       cldap   3       searchRequest                           Netlogon                
1487921562.721004000    11              6       1       cldap   3       searchRequest                           Netlogon                
1487921562.724801000    11              1       6       cldap   5       searchResDone                                           
1487921562.728632000    11              1       6       cldap   5       searchResDone                                           
1487921562.732508000    11              1       6       cldap   5       searchResDone                                           
1487921562.748004000    06      3       1       5       ldap    5       searchResDone                                           
1487921562.820387000    06      3       5       1       ldap    2       unbindRequest                                           
1487921562.831445000    06      14      6       1       dcerpc  11      Bind

model conversations

conversation 1, 5

NULL → NULL → ▿ldap:3 → ▵ldap:5 → ▿ldap:2 → NULL

conversation 1, 6

NULL → NULL → ▿cldap:3 → ▿cldap:3 → ▿ldap:3 → ▵cldap:5 → \
     ▵cldap:5 → ▵ldap:5 → ▿dcerpc:11 → NULL

model conversations

simple trigram Markov model

with a wait pseudo-operation

the next operation is predicted by the previous two

Conversation 1, 5 contributes these possibilities:

NULL     →  NULL     →  ▿ldap:3
NULL     →  ▿ldap:3  →  ▵ldap:5
▿ldap:3  →  ▵ldap:5  →  ▿ldap:2
▵ldap:5  →  ▿ldap:2  →  NULL

generate new conversations from the models

start with NULL NULL and randomly walk until hitting NULL.

restore ldap request parameters based on

replay the client side

Hopefully the server answers

fork for each client

reconstruction problems

Samba client primitives are

higher level than packets
not always available in python
or unused or difficult to find
traffic overloads the test client

reconstruction problems continued

we didn't capture quite enough information in the first round

Load modelling branch

Work in progress in a place like:

http://git.catalyst.net.nz/gw?p=samba.git;h=refs/heads/performance-testing-tool

Measuring Samba performance

Hurriedly reducing the scope of the talk

Motivation

Specific thing 1: make perftest

make test

make test

make test

make test

make test continued

test list format

make test TESTS=foo

make test TESTS=foo

format-subunit

Running a test as a performance test

performance specific tests

Adding performance specific tests

aggregating results

samba-cloud-autobuild

using multi-perf-test

backporting make perftest TESTS='foo'

viewing results

Generating realistic loads

AD DC load modelling (work in progress)

separate DNS

model conversations

model conversations

conversation 1, 5

conversation 1, 6

model conversations

generate new conversations from the models

replay the client side

reconstruction problems

reconstruction problems continued

Load modelling branch

the end

Specific thing 1: `make perftest`

`make test`

`make test`

`make test`

`make test`

`make test TESTS=foo`

`make test TESTS=foo`