I have played a bit more with the tree, see below.

> In that case, perhaps you can tell me how many files are detected by whatsnew -ls?
6927 files

> I presume you have a huge number of files that aren't added.  Are any/many of
them large?
The largest ignored file is ~ 1.6 GB. There's 2 ignored files above 1GB, ~30
above 300kB, ~ 90 above 100kB.

> If there are lots of added files, than this issue could be the same as
http://bugs.darcs.net/issue79

Memory consumption of darcs 2 stagnates around 80kB (VIRT)  56m (RES) 4,2kB
(SHR), which is 3.0% of my RAM (2GB). darcs1 uses a maximum of 1.4%.
I don't think this easily explains a slow-down by ~60.

> A quick check involving 600 binary files and 1000 text files adding up to 3.6G
show a small performance regression from 1.1s to 2.5s.
That's fewer files than I had, but larger total size. So the problem is likely
the number of files, not file size.


The whole issue is highly nonlinear in the size of the tree. I have stripped
down the tree to a few subdirectories that have the largest effect, and now the
slowdown is around 3 to 4 only.
However, if I replicate the main subdirectory (i.e. double the problem), I find
a slowdown around 10.

In fact, you can easily reproduce this, using the darcs unstable repo:
prompt>  darcs get http://darcs.net/repos/unstable; cd unstable
prompt>  (for darcs in /usr/bin/darcs /usr/bin/darcs /usr/bin/darcs darcs2
darcs2 darcs2 ; do rm -rf _darcs ; $darcs init ; time $darcs w -ls > /dev/null;
done)
  0.524u 0.040s  0.56  99.98%
  0.512u 0.044s  0.55  99.82%
  0.576u 0.068s  0.64  99.52%
  0.544u 0.032s  0.57  99.91%
  0.460u 0.036s  0.49  100.00%
  0.456u 0.040s  0.49  100.00%

Not much of a difference, right? Now we increase the repo size:

prompt>  mkdir large; cp -r src large/src1; cp -r src large/src2

and get

  0.680u 0.052s  0.73  100.00%
  0.472u 0.028s  0.50  99.02%
  0.472u 0.012s  0.48  99.79%
  1.364u 0.052s  1.42  99.22%
  1.236u 0.060s  1.32  98.14%
  1.140u 0.036s  1.17  99.71%

Increasing again gives
prompt>  mkdir larger; cp -r large larger/large1; cp -r large larger/large2
  1.268u 0.064s  1.33  99.80%
  0.920u 0.052s  0.97  99.30%
  0.944u 0.036s  0.98  99.21%
  4.000u 0.104s  4.11  99.79%
  4.052u 0.076s  4.14  99.71%
  3.964u 0.072s  4.04  99.74%

And after the next iteration:
wdobler@frenesi:~/tmp/unstable> mkdir yetlarger; cp -r larger yetlarger/larger1;
cp -r larger yetlarger/larger2;
  2.484u 0.100s  2.59  99.76%
  2.360u 0.104s  2.47  99.67%
  2.408u 0.060s  2.47  99.79%
  17.701u 0.260s  18.11  99.17%
  17.833u 0.200s  18.16  99.30%
  18.033u 0.224s  18.28  99.87%