1. Summarise the issue (what were doing, what went wrong?)

> darcs whatsnew -l
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.

3. What darcs version are you using? (Try: darcs --exact-version)

darcs-2.8.4

4. What operating system are you running?

> uname -a             
Linux aragon 2.6.18-6-686 #1 SMP Fri Feb 19 23:40:03 UTC 2010 i686 GNU/Linux

For reproduction, a tar file of the complete repo (~400MB) is available
from http://www-csr.bessy.de/control/Soft/BII-Controls-2013-03-08.tar.gz

On a newer machine under Ubuntu I get the same. BTW, it takes some time
before the message appears:

tiber: .../BII-Controls/head > time darcs whatsnew -l
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.
darcs whatsnew -l  5.99s user 0.32s system 89% cpu 7.057 total
tiber: .../BII-Controls/head > uname -a
Linux tiber 3.2.0-38-generic #61-Ubuntu SMP Tue Feb 19 12:20:02 UTC 2013
i686 i686 i386 GNU/Linux

The same happens with the latest darcs (freshly compiled from an
up-to-date copy of screened):

> darcs --version
2.9.7 (+ 90 patches)
tiber: .../BII-Controls/head > darcs whatsnew -l
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.

I downloaded your project and started tinkering with it. When doing a

    for i in *
    do
    echo $i
    darcs whatnews --boring -l 
    done

is shown that darcs fails on the "ioc3s1g.sumlog" file. Adding this file and
recording just it also fails because the file is 63MB.

So the problem isn't "whatsnew", is that darcs cannot import this huge
blob of
63MB in one go patch, thus failing.

I don't know how darcs should handle this (nor will pretend to know) but
you can
sidetrack this by doing this

    gzip ioc3s1g.sumlog   #passes from 64MB to 4.4MB
    darcs add ioc3s1g.sumlog.gz
    darcs record ./ioc3s1g.sumlog.gz

this will put the file in your repository ass binary, is not that bad
because I
don't think you will be modifying your log (just appending to it), and
you can
use zcat & friends

rubenAstudillo, thanks for looking into this. I did not notice I had
this large file in there, probably because it matched some boring
pattern. It was there by accident, and I never planned to add it.

Indeed, if I put this file into a clean empty repository, the error is
the same. Darcs add works, but whatsnew (with or without -l), darcs
record, etc all fail with the stack overflow.

So there are two problems: (1) 'darcs whatsnew -l' does not need to, and
therefore should not, look at the contents of a file that is not in the
repo, nor has been added. (2) Darcs should be able to handle a 65MB
file; a "Stack space overflow" is unacceptable IMO. From what I read in
Haskell-Cafe it typically appears when you left fold over a large list
in a lazy manner. Unfortunately I have no idea how to debug this, I
don't even know where to start. I have added --debug but that just gives me:

ben@sarun[1]: .../tmp/darcs-bug > darcs whatsnew -l --debug-verbose
Beginning identifying repository .
Done identifying repository .
Identified darcs-2 repo: /home/ben/tmp/darcs-bug
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.

I found out it is not the file size alone (of the accidentally present
ioc3s1g.sumlog) that causes the stack overflow, but rather that the file
does not end in a newline! Adding a final '\n' makes the problem
disappear. Also shrinking the file size does:

> cp ~/tmp/head/ioc3s1g.sumlog .
> truncate ioc3s1g.sumlog --size=50M
> time darcs whatsnew -l            
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.
darcs whatsnew -l  6,70s user 0,35s system 99% cpu 7,061 total
> truncate ioc3s1g.sumlog --size=40M
> time darcs whatsnew -l            
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.
darcs whatsnew -l  5,08s user 0,32s system 99% cpu 5,406 total
> truncate ioc3s1g.sumlog --size=30M
> time darcs whatsnew -l            
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize -RTS' to increase it.
darcs whatsnew -l  3,89s user 0,24s system 99% cpu 4,141 total
> truncate ioc3s1g.sumlog --size=20M
> time darcs whatsnew -l            
a ./ioc3s1g.sumlog
darcs whatsnew -l  2,55s user 0,13s system 99% cpu 2,686 total
> truncate ioc3s1g.sumlog --size=15M
> time darcs whatsnew -l            
a ./ioc3s1g.sumlog
darcs whatsnew -l  1,88s user 0,16s system 99% cpu 2,048 total
> truncate ioc3s1g.sumlog --size=10M
> time darcs whatsnew -l            
a ./ioc3s1g.sumlog
darcs whatsnew -l  1,30s user 0,07s system 99% cpu 1,375 total
> truncate ioc3s1g.sumlog --size=1M 
> time darcs whatsnew -l           
a ./ioc3s1g.sumlog
darcs whatsnew -l  0,12s user 0,04s system 96% cpu 0,165 total

I have verified that each truncation leaves the file ending without a
newline. Note that even at sizes below 20M when Darcs properly
terminates, the performance is still bad (but gets better the smaller
the file is). Now, contrast this with leaving the file at its original
65M, only add a final newline:

> time darcs whatsnew -l
a ./ioc3s1g.sumlog
darcs whatsnew -l  0,07s user 0,07s system 97% cpu 0,139 total

This is much faster!

I hope these findings make it easier to pinpoint the bug in the code.

Guillaume, would it make sense for Jose to check this case as part of his 
GSoC project? I know the overflow is from whatsnew, but I think it is 
probably using the same underlying code as record.

I made a patch that solves this special case when the file doesn't end in a
newline character[patch1076]. But then i look into it a little more and i
found that in any file that you changes a big number of lines you will end
with a stack overflow. The mkAmap have to make a map too big to handle i
think. Anyway, I was implementing/patching darcs to use patience diff
algorithm and with this implementation there are no stack overflows. I hope
it helps.


2013/5/28 Ganesh Sittampalam <bugs@darcs.net>

>
> Ganesh Sittampalam <ganesh@earth.li> added the comment:
>
> Guillaume, would it make sense for Jose to check this case as part of his
> GSoC project? I know the overflow is from whatsnew, but I think it is
> probably using the same underlying code as record.
>
> ----------
> nosy: +ganesh, gh
>
> __________________________________
> Darcs bug tracker <bugs@darcs.net>
> <http://bugs.darcs.net/issue2313>
> __________________________________
> _______________________________________________
> darcs-devel mailing list
> darcs-devel@darcs.net
> http://lists.osuosl.org/mailman/listinfo/darcs-devel
>

Attachments

• unnamed

The following patch sent by Jose Luis Neder <jlneder@gmail.com> updated issue issue2313 with
status=resolved;resolvedin=2.10.0 HEAD

* resolve issue2313: whatsnew -l: Stack space overflow  
Ignore-this: 7c8e2b0426dc5139c1a5de4b27fffd3a