I have a large repository (the complete history of libcurl as converted from CVS
to darcs by tailor), and the following command failed:
HACN Wonwin-McBrootles-Computer:~/playground/libcurl/dw$ time darcs put -v -v -v
zooko@nooxie.zooko.com:libcurl
Creating repository
Applying patches in new repository...
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize' to increase it.
Apply failed!

real    9m47.742s
user    0m24.380s
sys     0m6.924s

This is with darcs-2.0.2.

It seems that darcs put is ridden with problems. I would suggest cutting that
functionality. Just today I got from darcs put:

darcs failed:  Refusing to apply patches leading to conflicts.
If you would rather apply the patch and mark the conflicts,
There are conflicts in the following files:
./test/Makefile.in
use the --mark-conflicts or --allow-conflicts options to apply
These can set as defaults by adding
 apply mark-conflicts
to _darcs/prefs/defaults in the target repo. 
Apply failed!

I'd argue it's terminally broken and should be preemptively removed. When we have
a better version we could put it back... Opinions?

If it is true that darcs put is currently buggy, then I would approve of
removing it for now.  People can work-around the lack of darcs put either by
doing a darcs init on the far side followed by a darcs push, or else by using
the handy-dandy rsync.

This may be related to zooko's issue.

On my system, which has 512MB of RAM, attempting to "darcs put" the
darcs darcs repo resulted in it being killed by the kernel due to
out-of-memory (OOM).  I used darcs 2.0.2 to do the put.

twb@Clio[Desktop]$ while x=`pgrep darcs`; do ps u $x | tail -1; sleep 1; done
twb      14985 18.0  0.6  29300  3500 pts/3    Sl+  18:19   0:00 darcs put peach:/tmp/huzzar
twb      14985 25.0  1.3  30324  6656 pts/3    Rl+  18:19   0:00 darcs put peach:/tmp/huzzar
twb      14985 35.3  2.7  37492 13912 pts/3    Dl+  18:19   0:01 darcs put peach:/tmp/huzzar
twb      14985 42.0  3.5  42612 18252 pts/3    Dl+  18:19   0:01 darcs put peach:/tmp/huzzar
twb      14985 50.0  4.8  48756 24624 pts/3    Rl+  18:19   0:02 darcs put peach:/tmp/huzzar
twb      14985 54.8  6.6  57972 33740 pts/3    Rl+  18:19   0:03 darcs put peach:/tmp/huzzar
twb      14985 57.7  6.7  57972 34272 pts/3    Rl+  18:19   0:04 darcs put peach:/tmp/huzzar
twb      14985 60.3  9.1  70260 46544 pts/3    Rl+  18:19   0:04 darcs put peach:/tmp/huzzar
twb      14985 62.3  9.1  70260 46676 pts/3    Rl+  18:19   0:05 darcs put peach:/tmp/huzzar
twb      14985 58.0  9.1  70260 46688 pts/3    Rl+  18:19   0:06 darcs put peach:/tmp/huzzar
twb      14985 60.3 12.7  88692 64908 pts/3    Rl+  18:19   0:07 darcs put peach:/tmp/huzzar
twb      14985 62.2 12.7  88692 64912 pts/3    Rl+  18:19   0:08 darcs put peach:/tmp/huzzar
twb      14985 63.7 12.8  88692 65120 pts/3    Rl+  18:19   0:08 darcs put peach:/tmp/huzzar
twb      14985 65.6 18.2 116340 92536 pts/3    Dl+  18:19   0:09 darcs put peach:/tmp/huzzar
twb      14985 67.0 18.2 116340 92552 pts/3    Rl+  18:19   0:10 darcs put peach:/tmp/huzzar
twb      14985 67.7 18.2 116340 92552 pts/3    Dl+  18:19   0:11 darcs put peach:/tmp/huzzar
twb      14985 68.7 18.2 116340 92768 pts/3    Dl+  18:19   0:12 darcs put peach:/tmp/huzzar
twb      14985 69.4 18.8 119412 95732 pts/3    Dl+  18:19   0:13 darcs put peach:/tmp/huzzar
twb      14985 70.9 25.8 155252 131164 pts/3   Dl+  18:19   0:14 darcs put peach:/tmp/huzzar
twb      14985 71.9 25.8 155252 131164 pts/3   Rl+  18:19   0:15 darcs put peach:/tmp/huzzar
twb      14985 72.5 25.8 155252 131164 pts/3   Rl+  18:19   0:15 darcs put peach:/tmp/huzzar
twb      14985 73.3 25.9 155252 131684 pts/3   Rl+  18:19   0:16 darcs put peach:/tmp/huzzar
twb      14985 74.0 25.9 156276 131912 pts/3   Rl+  18:19   0:17 darcs put peach:/tmp/huzzar
twb      14985 72.1 35.6 205428 180920 pts/3   Rl+  18:19   0:18 darcs put peach:/tmp/huzzar
twb      14985 72.7 35.7 205428 181444 pts/3   Rl+  18:19   0:19 darcs put peach:/tmp/huzzar
twb      14985 73.4 35.7 205428 181448 pts/3   Rl+  18:19   0:20 darcs put peach:/tmp/huzzar
twb      14985 74.4 35.7 205428 181452 pts/3   Rl+  18:19   0:21 darcs put peach:/tmp/huzzar
twb      14985 75.3 35.7 205428 181484 pts/3   Rl+  18:19   0:22 darcs put peach:/tmp/huzzar
twb      14985 76.2 35.8 205428 181864 pts/3   Rl+  18:19   0:23 darcs put peach:/tmp/huzzar
twb      14985 77.1 35.8 205428 181864 pts/3   Rl+  18:19   0:24 darcs put peach:/tmp/huzzar
twb      14985 78.0 35.8 205428 181864 pts/3   Rl+  18:19   0:25 darcs put peach:/tmp/huzzar
twb      14985 78.8 47.8 266868 243156 pts/3   Rl+  18:19   0:26 darcs put peach:/tmp/huzzar
twb      14985 79.2 49.4 275060 251160 pts/3   Rl+  18:19   0:27 darcs put peach:/tmp/huzzar
twb      14985 78.3 49.4 275060 251164 pts/3   Rl+  18:19   0:28 darcs put peach:/tmp/huzzar
twb      14985 77.7 49.4 275060 251164 pts/3   Rl+  18:19   0:28 darcs put peach:/tmp/huzzar
twb      14985 76.0 49.4 275060 251164 pts/3   Rl+  18:19   0:29 darcs put peach:/tmp/huzzar
twb      14985 76.4 49.4 275060 251164 pts/3   Rl+  18:19   0:30 darcs put peach:/tmp/huzzar
twb      14985 77.0 49.4 275060 251164 pts/3   Rl+  18:19   0:31 darcs put peach:/tmp/huzzar
twb      14985 77.6 49.5 275060 251496 pts/3   Rl+  18:19   0:32 darcs put peach:/tmp/huzzar
twb      14985 78.3 49.5 275060 251496 pts/3   Rl+  18:19   0:33 darcs put peach:/tmp/huzzar
twb      14985 78.5 49.5 275060 251496 pts/3   Rl+  18:19   0:34 darcs put peach:/tmp/huzzar
twb      14985 79.0 49.5 275060 251496 pts/3   Rl+  18:19   0:35 darcs put peach:/tmp/huzzar
twb      14985 79.0 52.0 289396 264488 pts/3   Rl+  18:19   0:36 darcs put peach:/tmp/huzzar
twb      14985 76.5 53.9 298612 273900 pts/3   Dl+  18:19   0:36 darcs put peach:/tmp/huzzar
twb      14985 76.3 55.8 308852 283692 pts/3   Rl+  18:19   0:37 darcs put peach:/tmp/huzzar
twb      14985 76.0 60.5 332404 307480 pts/3   Dl+  18:19   0:38 darcs put peach:/tmp/huzzar
twb@Clio[Desktop]$ dmesg | tail
[28821.407594] 1931 reserved pages
[28821.407598] 30716 pages shared
[28821.407602] 0 pages swap cached
[28821.407606] 0 pages dirty
[28821.407610] 0 pages writeback
[28821.407614] 5 pages mapped
[28821.407619] 1480 pages slab
[28821.407623] 318 pages pagetables
[28821.407631] Out of memory: kill process 14988 (darcs) score 87048 or a child
[28821.407662] Killed process 14998 (ssh)
twb@Clio[Desktop]$

I've opened a new ticket for Petr's observation about a potential darcs put bug
(see issue1518), which I don't think is particularly relevant here.

As far as I understand

1. ssh is not relevant to the stack space overflow (but thanks for the details)
2. darcs put creates a massive patch bundle and then calls darcs apply on the
remote end (so presumably, darcs init and darcs push -a would have the same result)

It sounds from your error message like the darcs apply is where the stack space
overflow is happening.

Zooko: I'd like to narrow this down a little bit. Could you please
1. repeat this procedure, this time logging in to the remote machine and
watching to see how darcs behaves there?  Alternatively, you could create a fake
darcs which just saves the bundle to a file
2. see what happens when you do a darcs send against an empty repo?

If we manage to narrow this down, then we know that we just need to fix darcs
apply on large patch bundles.

I suppose if you're looking for a workaround in the meantime, darcs init and
then pushing 100 patches at a time with some shell trick would work.

(I hate roundup web UI *types the comment again*)

In darcs darcs repo:
darcs send -O ../empty -- 210M RAM
darcs apply ../...dpatch -- 290M RAM

bundle size: 30M

-> the parsed bundle is likely kept in memory all the time -> bug.

I get the impression that this is something people could look into with Haskell
experience but not much darcs experience.

If possible, I'd like us to somehow turn this into something we could ask the
wider Haskell community to help us with.

Does this make any sense? Is that feasible? How can we present this to them?

On Sat, Aug 15, 2009 at 14:10:24 +0000, Petr Ročkai wrote:
> In darcs darcs repo:
> darcs send -O ../empty -- 210M RAM
> darcs apply ../...dpatch -- 290M RAM
> 
> bundle size: 30M
> 
> -> the parsed bundle is likely kept in memory all the time -> bug.

One other thought: does the stack/heap distinction matter here?

OK, I'm treating the memory use of send/apply as being unrelated to the stack
space overflow for now (see issue1540 and issue1539 for those).

As with the above bugs, it looks like we need to profile apply.