Darcs output:

$ time darcs put [...]
darcs(28475) malloc: *** mmap(size=1152278528) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
darcs: malloc: resource exhausted (out of memory)


real	2m25.229s
user	1m35.057s
sys	0m6.252s

Darcs 2 compiled with GHC 6.8.2:

$ darcs -v
2.0.0rc1 (unknown)

Could you give us a link to the repository on which darcs fails? If that's not
possible, can you give the size of the repository in megabytes and in number of
patches? And how big is the largest patch?

On 4/2/08, Reinier Lamers <bugs@darcs.net> wrote:
>  Could you give us a link to the repository on which darcs fails? If that's not
>  possible, can you give the size of the repository in megabytes and in number of
>  patches? And how big is the largest patch?

This is a commercial project, so no repo for you. But I can give you the rest:

* Size of repository including working directory: 181M
* Size of _darcs: 139M
* Number of patches: 6475
* Largest patch is 8MB compressed, 15MB decompressed.

The repo was created by cloning a Darcs 1 repo. So the exact sequence was:

$ darcs get ...
$ darcs put ...

Let me know if there is anything else I can do or provide.

Alexander.

I made a test repo with 6476 patches of which one is a binary patch of 15 MB.

When I darcs put on that, I get the following output:

777test reinier$ darcs put --debug-verbose ../777test-put
Creating repository
Beginning identifying repository .
Done identifying repository .
Beginning identifying repository .
Done identifying repository .
Identified darcs-1 repo: /Users/reinier/Documents/Programs/777test
Beginning reading inventory of repository /Users/reinier/Documents/Programs/777test
Done reading inventory of repository /Users/reinier/Documents/Programs/777test
darcs apply --all --repodir ../777test-put --debug
^C^CwithSignalsHandled: Interrupted!

Darcs tells us about the repository:

777test reinier$ darcs query repo
          Type: darcs
        Format: darcs-1.0
          Root: /Users/reinier/Documents/Programs/777test
      Pristine: PlainPristine "_darcs/pristine"
         Cache: thisrepo:/Users/reinier/Documents/Programs/777test
Default Remote: ../777test-put
   Num Patches: 6476

So apparently we need to look at "darcs apply" here.

This goes wrong because the "darcs apply" command first reads the whole patch
bundle from stdin, and only when all the input has been read, starts applying
it. This is because hGetContentsPS is strict. Am I right that hGetContentsPS
does not have, and is not supposed to have, the lazy I/O behavior of the
standard getContents?

Adding David and Gwern to this bug.  Can either of you two answer Reiner's
question?  Jason?

> Am I right that hGetContentsPS does not have, and is not supposed to have,
> the lazy I/O behavior of the standard getContents?

'getContents' is usually lazy, yes (the Prelude getContents is lazy, System.IO's
hGetContents is lazy, etc.).

But FastPackedString.hs *always* uses strict ByteStrings (and
OldFastPackedStrings.hs is strict as well, I think, from the looks of it). 

http://haskell.org/ghc/docs/latest/html/libraries/bytestring/Data-ByteString-Char8.html#v%3AhGetContents

"Read entire handle contents into a ByteString. This function reads chunks at a
time, doubling the chunksize on each read. The final buffer is then realloced to
the appropriate size. For files > half of available memory, this may lead to
memory exhaustion. Consider using readFile in this case."

Not lazy. If we're getting errors, perhaps one could try taking its advice about
readFile.

So if we use strict bytestrings, that limits the size of the patch bundle to
available working memory. To do something about that, you'd have to read the
patches one by one, I suppose. Are there some patch theoretical objections to
doing that?

On Fri, May 16, 2008 at 2:16 PM, Reinier Lamers <bugs@darcs.net> wrote:

>
> Reinier Lamers <tux_rocker@reinier.de> added the comment:
>
> So if we use strict bytestrings, that limits the size of the patch bundle
> to
> available working memory. To do something about that, you'd have to read
> the
> patches one by one, I suppose. Are there some patch theoretical objections
> to
> doing that?

No, I don't think there is any patch theoretic problem here.  In the case of
patch application, I think it's just as straightforward as reading patches
and modifying the working copy (with perhaps some in memory optimizations, I
don't know because I haven't looked at this code).

But, as I understand it, there can be problems with lazily reading files
that get modified while the read is happening.  Also, the code internally is
easier to verify and maintain if darcs assumes it can hold all the needed
patches in memory at once.

I recently asked Gwern if he could make --use-mmap (or --lazyio) or similar
a command line option so that the user can choose at run-time which way they
want darcs to do reading.  I suspect there are corner cases where using
either lazy or strict IO is not safe but prefered due to resource usage.  It
could be that using mmap is dangerous in some cases (darcs could get a
segfault if a file is modified during reading, or lazy IO could result in a
file descriptor shortage) but that it may be the only way to make a specific
command work in some cases.  For these situations it would be nice if the
user could pick between the IO strategies.  Of course the draw back is the
added complexity.

Jason

Attachments

• unnamed

It turns out the size of the patch bundle was a red herring. My system becomes
unusably swappy already when only 11 MB of it has been generated. So the problem
is with the code that generates the bundle. But I'm going to investigate that
tomorrow.

I'm attaching the heap profile of running darcs put on a repository with a
single 15 MiB binary patch (the first 15 MiB of my hard disk, actually). It
looks as if it's allocating the patch content on the heap three times, once in
reading the patch file and then twice in creating the bundle.

Attachments

• darcs.ps

It seems that hash_bundle is allocating some voluptuous heap space to take a
hash of. I'm going to try to modify that function so that it doesn't need to
hold all the data in memory at once.

darcs put requires that the entire parsed repository be held in memory.  For any
reasonably-large repository, this is impossible.  Changing this requires a
complete rewrite of put.

tux_rocker, you have assigned this issue to yourself: Is there anything that you 
wish to add to this issue?

Thanks and best regards
Thorkil

So if I understand correctly, this can be broken down into generating a giant
patch bundle (issue1540) and consuming it with darcs apply (issue1539).

I recommend we move the discussion over to those bugs.