More Cscope benchmarks

A patch to cscope was just posted: https://sourceforge.net/p/cscope/patches/86/. This claims to speed up the building of the inverted index by using a more efficient search algorithm in one place, and a better sorting implementation in another. Since I did some cscope benchmarks earler (here and here), I can easily evaluate this patch, so I did this.

Test description

The results aren't directly comparable to the timings in the previous posts, since the project being indexed (Linux kernel) is at a very different version; much more recent and with many more sources. The test machine is the same as before. All the tests were done with a real ext3 hard disk, not a ramdisk. The cscope is the stock cscope 15.8a-2 from Debian.

Results

All timings are in seconds.

Cold disk cache

	Stock	Patched
Initial database build	123.572	95.5225
Database re-build after touching a file	57.2912	30.91
Initial search	9.11125	8.415
Re-search after touching a file	59.6287	31.92
Initial no-db-update search	0.80625	1.2075
No-db-update re-search after touching a file	0.805	0.95

Warm disk cache

	Stock	Patched
Initial database build	55.3537	29.5287
Database re-build after touching a file	45.4975	18.805
Initial search	0.12125	0.12
Re-search after touching a file	45.985	19.0437
Initial no-db-update search	0	0
No-db-update re-search after touching a file	0	0.00125

Note that this tests only the timings. I did not actually look at the results being produced. Presumably they match, but I did not check.

Conclusions

Yeah. Much faster. Hopefully this produces the correct results, and gets merged in some form.

Benchmark script

Here's the script that was used to get the timings. It's pretty much the same as before, with small modifications to set what is being tested. As before, this is a zsh script. It uses some zsh-isms, but could be converted to bash if somebody cares to do it.

#!/bin/zsh

# needed in cleandb()
setopt nonomatch

function dropcaches() {
    if [[ $warmcold == "cold" ]]; then
        sync ;
        sudo sysctl -w vm.drop_caches=3;
    fi
    sleep 2;
}

function cleandb() {
    # requires nonomatch option to ignore missing globs
    rm -f cscope.out* G*;
}

function touchfile() {
    sleep 2; # very important. cscope needs this to see the file update
    touch include/drm/drm_edid.h;
}

TIMEFMT='%E'

awktally='
BEGIN {
  skip = ENVIRON["skip"]
}

/^[0-9\.]+s$/ {
  gsub("s","");
  str = str " " $1
  if( n >= skip )
  {
    sum += $1;
  }
  n++;
}

END {
  print ENVIRON["name"] ": skipping: " skip " all: " str " mean: " sum/(n-skip)
}'

typeset -A skipcounts
skipcounts=(cold 2 warm 2)

modeoptions="-k -q"

cscope-indexer -l -r

Nrepeat=8

for mode (kernel patched)
{
    if [[ $mode == "patched" ]]; then
        cmd="/tmp/cscope-15.8a-patched/src/cscope $modeoptions";
    else
        cmd="/tmp/cscope-15.8a/src/cscope $modeoptions";
    fi

    for dotouch (0 1)
    {
        for warmcold (cold warm)
        {
            export name="$warmcold initial build; $mode mode; touching: $dotouch";
            export skip=$skipcounts[$warmcold];
            repeat $(($Nrepeat + $skip)) {
                if (($dotouch)); then
                    touchfile;
                else
                    cleandb;
                fi
                dropcaches;
                time ${(z)cmd} -b;
            } |& awk $awktally
        }
    }

    for dotouch (0 1)
    {
        for warmcold (cold warm)
        {
            export name="$warmcold initial search; $mode mode; touching: $dotouch";
            export skip=$skipcounts[$warmcold];
            repeat $(($Nrepeat + $skip)) {
                if (($dotouch)); then
                    touchfile;
                fi
                dropcaches;
                time ${(z)cmd} -L0 main > /dev/null;
            } |& awk $awktally
        }
    }

    for dotouch (0 1)
    {
        for warmcold (cold warm)
        {
            export name="$warmcold initial no-db search; $mode mode; touching: $dotouch";
            export skip=$skipcounts[$warmcold];
            repeat $(($Nrepeat + $skip)) {
                if (($dotouch)); then
                    touchfile;
                fi
                dropcaches;
                time ${(z)cmd} -d -L0 main > /dev/null;
            } |& awk $awktally
        }
    }
}