darcs

Issue 471 push/pull => Too much memory used!

Title push/pull => Too much memory used!
Priority bug Status resolved
Milestone Resolved in
Superseder darcs pull memory and cpu usage
View: 123
Nosy List abez, darcs-devel, dmitry.kurochkin, kowey, markstos, thorkilnaur, tommy
Assigned To
Topics Performance

Created on 2007-06-14.17:16:28 by abez, last changed 2009-08-27.13:55:30 by admin.

Messages
msg1717 (view) Author: abez Date: 2007-06-14.17:16:14
Repository: http://churchturing.org/w/test.case/
Problem: It uses too much memory to record changes and to push changes.

In a similar repo I run out of memory pushing changes around.
realloc: resource exhausted (out of memory)
I think it hits near a 2GB limit and then dies. Regardless it shouldn't
use that memory for such a minor change.

The change is to a 70kloc file with additions every 20 lines.

My machines:
linux 2.4.25  i686 , Glasgow Haskell Compiler, Version 6.4, for
              Haskell 98, compiled by GHC version 5.04.3
linux 2.6.15-27-686, Glasgow Haskell Compiler, Version 6.4.1, for
              Haskell 98, compiled by GHC version 6.4.1

This is with darcs 1.0.9 stable, compiled from the repo.

Thanks,

abram

p.s. attached is the chat log and a

--- CHATLOGS! ----

21:54 < abez> Does anyone have problems with running out of memory on a
push or a pull?
21:55 < kc5tja> That depends.  I've had darcs attempt to solve the
halting problem while trying to merge several parallel branches of code
once, but it never ran out of memory.
21:55 < abez> I had a 70kloc file which recieved changes every 20 lines
for one patch
21:55 < abez> It must've hit the 2gb limit
21:56 < abez> realloc: resource exhausted (out of memory)
21:56 < kc5tja> What version of darcs?
21:56 < abez> 1.9
21:56  * kc5tja attempts to replicate.
21:56 < abez> 1.0.9
21:57 -!- MarcWebe1 [n=marc@217.224.137.40] has joined #darcs
21:59 < kc5tja> OK, commiting the source file now.  It's definitely
taking some time to commit the initial import.
21:59 < kc5tja> Just finished.
21:59 < kc5tja> Now to make a script that alters one character randomly
every 20 lines.
22:05 < abez> I got it up to 1200mb
22:06 < abez> (VIRT and 700 RES)
22:06 -!- Jerri [n=gerhard@dslb-084-056-036-081.pools.arcor-ip.net] has
joined #darcs
22:12 < kc5tja> So, let me see if my math is right.
22:12 < kc5tja> 70,000 LOC, one change every 20 lines, that's about 3500
changes total for the file.
22:12 < kc5tja> According to my diff, I have about 5000 changes total
(not sure why, but either way, I've confirmed it's one change per line,
for more lines than my estimate)
22:12 < kc5tja> Here goes nothing....
22:15 < kc5tja> M ./70kloc.src -3498 +3462
22:15 < kc5tja> That's what darcs w -l reports, so I'm close.  diff was
giving me some extra lines in its output.
22:15 < kc5tja> Took 2 minutes to generate the w -l report.
22:15 < kc5tja> now for the commit
22:18 -!- clanehin [n=clanehin@cpe-069-134-236-134.nc.res.rr.com] has
quit [Remote closed the connection]
22:18 -!- MarcWeber [n=marc@pD9E09F07.dip.t-dialin.net] has quit [Read
error: 110 (Connection timed out)]
22:20 -!- Jerri_ [n=gerhard@dslb-084-056-053-202.pools.arcor-ip.net] has
quit [Read error: 110 (Connection timed out)]
22:20 < abez> kc5tja: I just tried to commit a change to a test repo and
I accidently modified the test file. Instead of selecting d to record
the current selections, I pressed s
22:20 < abez> kc5tja: now I'm waiting for linux to tell darcs it is out
of memory
22:20 < abez> 1219m V with 685m RES
22:20 < kc5tja> Ditto, my darcs record is still in progress.
22:21 < kc5tja> However, it's at 78MB total, and has been stable.
22:22 < abez> I had the same behaviour on a 2.4 kernel and on an ubuntu
Dapper system
22:22 < abez> 13810 abez      19   0 1240m 649m  61m R  3.6 85.8
1:50.81 darcs
22:22 < kc5tja> What kernel version?  I don't even know what I'm at.  ;D
22:23 < kc5tja> Ooh, did you compile darcs yourself?  If so, what
version of GHC did you use?
22:23 < kc5tja> 2.6.17-11 here (GoboLinux)
22:23 < kc5tja> GHC 6.6 (but no patch level indicated.)
22:28 < abez> kc5tja: linux 2.4.25  i686 , Glasgow Haskell Compiler,
Version 6.4, for Haskell 98, compiled by GHC version 5.04.3
22:28 < abez> kc5tja:  2.6.15-27-686, Glasgow Haskell Compiler, Version
6.4.1, for Haskell 98, compiled by GHC version 6.4.1
22:28 < abez> and this happened with 1.0.3 and 1.0.7
22:30 < kc5tja> Yeah, the fact that I have a later version of GHC might
have an impact (perhaps better garbage collection).
22:30 < kc5tja> It finished!
22:30 < kc5tja> darcs record -a -m "BUG: BOMBS AWAY"  334.39s user
89.29s system 90% cpu 7:46.38 total
22:32 < abez> kc5tja: http://churchturing.org/w/test.case/ here's my
test case
22:32 < lambdabot> Title: Index of /w/test.case
22:33 < kc5tja> Mine isn't much different from yours.  Written in Python
though.
22:34 < kc5tja> kc5tja@aldeberan ~/tmp/bug]cat 70kloc.py
22:34 < kc5tja> #!/usr/bin/python
22:34 < kc5tja> l = "akhdflakdflakhdf  KLSHFDKLSHFDKL KLjhdflaskjhdflksJ
fLSD FSKLJHFDSKLjhfd skljhfsd fKJH"
22:34 < kc5tja> open("70kloc.src", "w").write( "\n".join([l for x in
xrange(70000)]))
22:34 < kc5tja> That generates the file.
22:35 < kc5tja> http://pastebin.ca/565651
22:35 < kc5tja> That is the modifier script.

-------------------
Top Memory Log:
13512 abez      17   0 48256  47m 3064 R 92.0  6.2   0:08.67 darcs

13512 abez      16   0 50764  49m 3404 R 35.2  6.5   0:13.75 darcs

13512 abez      15   0 81300  79m 3404 R 99.3 10.5   0:19.74 darcs

13512 abez      19   0  179m 179m 3404 R 99.0 23.7   0:22.72 darcs

13512 abez      20   0  311m 311m 3404 R 96.0 41.2   0:25.61 darcs

13512 abez      17   0  461m 461m 3404 R 98.7 61.0   0:28.58 darcs

13512 abez      16   0  643m 626m 3524 D 36.1 82.8   0:32.53 darcs

13599 abez      18   0  255m 255m 3368 R 99.0 33.7   0:23.73 darcs

13599 abez      16   0  407m 407m 3368 R 99.0 53.8   0:26.71 darcs

13599 abez      16   0  558m 558m 3368 R 97.7 73.8   0:29.65 darcs

13599 abez      19   0  645m 645m 3368 R 62.8 85.3   0:31.54 darcs

13599 abez      16   0  751m 697m 3792 D 28.0 92.2   0:34.03 darcs

push
13649 abez      20   0 84568  82m 3396 R 99.0 10.9   0:08.27 darcs

13649 abez      19   0  208m 208m 3396 R 97.3 27.6   0:11.20 darcs

13649 abez      17   0  365m 365m 3396 R 99.0 48.3   0:14.18 darcs

13649 abez      16   0  520m 520m 3396 R 99.3 68.7   0:17.17 darcs

13649 abez      17   0  703m 702m 3288 S 35.5 92.8   0:21.14 darcs

13649 abez      20   0  703m 684m 3796 R 58.8 90.5   0:22.91 darcs

13649 abez      16   0  721m 699m 4420 D 85.1 92.4   0:28.45 darcs

13649 abez      17   0  740m 684m 4496 R 32.2 90.4   0:29.42 darcs

13649 abez      16   0  928m 688m 4752 R 22.6 91.0   0:35.47 darcs

13649 abez      16   0  928m 688m 4752 R 22.6 91.0   0:35.47 darcs

13649 abez      18   0  993m 663m 4752 D 13.3 87.7   0:37.11 darcs

13649 abez      14   0 1119m 697m 4752 D 37.5 92.1   0:40.19 darcs

13649 abez      14   0 1119m 697m 4752 D 37.5 92.1   0:40.19 darcs

13649 abez      14   0 1158m 704m 4752 R 33.9 93.1   0:41.21 darcs

13649 abez      17   0 1201m 703m 4752 R 36.7 93.0   0:42.38 darcs

13649 abez      15   0 1245m 704m 4752 R 41.2 93.1   0:43.62 darcs

13649 abez      14   0  899m 339m 4752 R 27.6 44.8   0:44.45 darcs

13649 abez      13   0  644m 339m 4752 R  0.7 44.8   0:44.48 darcs

... then I got bored and later ran out of memory :( ...
msg1836 (view) Author: kowey Date: 2007-07-16.09:27:01
Marking as a duplicate of issue123.  Deferring so that we know to get back to
you when we fix this (someday...)
msg3489 (view) Author: markstos Date: 2008-02-16.03:54:02
I'm marking this one as resolved because it's a dupe of an issue which is now
considered resolved.
History
Date User Action Args
2007-06-14 17:16:28abezcreate
2007-07-16 09:27:04koweysetstatus: unread -> deferred
title: Too much memory used! -> push/pull => Too much memory used!
nosy: droundy, tommy, beschmi, kowey, abez
messages: + msg1836
topic: + Performance
superseder: + darcs pull memory and cpu usage
2007-08-03 17:26:42droundysetstatus: deferred -> duplicate
2008-02-16 03:54:03markstossetstatus: duplicate -> resolved
nosy: + markstos
messages: + msg3489
2009-08-06 17:32:04adminsetnosy: + jast, Serware, dmitry.kurochkin, darcs-devel, zooko, dagit, mornfall, simon, thorkilnaur, - droundy, abez
2009-08-06 20:28:54adminsetnosy: - beschmi
2009-08-10 22:04:53adminsetnosy: + abez, - darcs-devel, zooko, jast, dagit, Serware, mornfall
2009-08-25 17:46:50adminsetnosy: + darcs-devel, - simon
2009-08-27 13:55:30adminsetnosy: tommy, kowey, markstos, darcs-devel, abez, thorkilnaur, dmitry.kurochkin