'darcs convert' fails on my very large repository > 2000 patches with an 'out of
memory' error message (on a machine with 4GB).  So I decided to experiment a bit
and unpulled all patches except for the very first one in the repo.  The first
patch is the head of my old subversion repo and is large (150mb).

'darcs convert' still fails with the same error; always an out of memory error.
 The amount of memory it consumes is > 20X the size of the compressed patch. 
The  contents of the patch are around 600mb (the size of the working tree when
the repo only contains the first patch).

Quite why darcs requires this amount of memory is a mystery to me; I expect
there is a severe memory leak somewhere.

This happens on 2.0.0pre2, 2.0.0pre3 and current head of unstable as of 27th Jan
2008.

James,

Thanks for testing Darcs 2. This is an important test case to know about, as it
sounds like what a lot of people will do as soon as they try out Darcs 2. I'm
upgrading this bug status to "urgent", since it would be blocker for people to
use Darcs2 effectively.

No problem, Mark.  I wish I could be of more help, but the repository
I am using is not open source.

Hi James,

Is this the same repository as fails with darcs get? (Just so we can avoid
forcing you to resend the same sort of information.)

Do I understand you properly that darcs convert on a repository with just this
single huge patch fails?

What is the uncompressed size of the patch itself? Does it have many (or large)
binary files?

I hesitate to ask, but... since this is a single patch, do you think you could
recreate a repository with the same size files (and preferably the same
directory structure) but with the binary files replaced with all '\0' and the
test files replaced with all ' ' (but the same number of lines)? And you could
change the names of the files as well.  I know this would be a fair amount of
work, but it wouldn't be too hard to automate, and then a single amend-record
could convert your existing non-public repository into one that you could share
with us (which would also compress much better... hopefully that wouldn't hide
the bug).

Thanks for any help you can provide! (and for the initial bug report)

The trouble here is that we've run convert on a very large repository (the ghc
repository) and while it runs slowly, it doesn't crash.  So I'd rather not have
to guess in creating a test repository with which to reproduce your bug!

David

I'm downgrading the "urgent" status of this bug since we are stalled waiting for
feedback from the reporter on it. 

###

James, if you are reading this: we still need answers to the following to
possibly pursue further addressing your bug report:

###

Hi James,

Is this the same repository as fails with darcs get? (Just so we can avoid
forcing you to resend the same sort of information.)

Do I understand you properly that darcs convert on a repository with just this
single huge patch fails?

What is the uncompressed size of the patch itself? Does it have many (or large)
binary files?

I hesitate to ask, but... since this is a single patch, do you think you could
recreate a repository with the same size files (and preferably the same
directory structure) but with the binary files replaced with all '\0' and the
test files replaced with all ' ' (but the same number of lines)? And you could
change the names of the files as well.  I know this would be a fair amount of
work, but it wouldn't be too hard to automate, and then a single amend-record
could convert your existing non-public repository into one that you could share
with us (which would also compress much better... hopefully that wouldn't hide
the bug).

Thanks for any help you can provide! (and for the initial bug report)

The trouble here is that we've run convert on a very large repository (the ghc
repository) and while it runs slowly, it doesn't crash.  So I'd rather not have
to guess in creating a test repository with which to reproduce your bug!

David

Sorry for the late response, guys.  I'd been on holiday and got back to work
last week and all of this had fallen off my radar - my bad!

So, here's what I know:

First of all this IS the same repository that fails using darcs get
http://bugs.darcs.net/issue623.

> Do I understand you properly that darcs convert on a repository with just this
single huge patch fails?

Yes.

> What is the uncompressed size of the patch itself? Does it have many (or
large) binary files?

The uncompressed size is 753M.  It's the head of our old subversion repo, and
most of the size is because we have a bunch of binary dependencies in their for
all of our modules.  See http://bugs.darcs.net/issue623 for details.

> I hesitate to ask, but... since this is a single patch, do you think you could
> recreate a repository with the same size files (and preferably the same
> directory structure) but with the binary files replaced with all '\0' and the
> test files replaced with all ' ' (but the same number of lines)? And you could
> change the names of the files as well.  I know this would be a fair amount of
> work, but it wouldn't be too hard to automate, and then a single amend-record
> could convert your existing non-public repository into one that you could share
> with us (which would also compress much better... hopefully that wouldn't hide
the bug).

I am more than happy to provide you with what you need, but could you please
elaborate a little more on the technical details? - I didn't completely
understand the above.  

Cheers,

James.

Hi James, if you could just take the head state, and replace every binary file
with zeros of the same size, and for each text file replace all the characters
other than '\n' with ' ', then record a patch using darcs record -la (or
whatever you did to record the first patch in your actual repository), that
would be perfect (provided I could access the resulting one-patch repository.

You could also rename the files and directories, if the filenames might be
private.  Basically, I'd like to see the patch that's causing you trouble, so I
can figure out how to handle it using less memory (if possible).

If that wasn't clear (and I'm not at all sure it's clear), then let me know, and
I'll try to do better.  Oh yeah, and when you've generated this repository, I'd
like you to confirm that darcs still uses up all your memory when
converting/getting it.

David

This sounds like it could be the same as issue80
"record: memory usage is 2X patch size"

    Mark

On Mon, Feb 18, 2008 at 03:56:42PM -0000, Mark Stosberg wrote:
> This sounds like it could be the same as issue80
> "record: memory usage is 2X patch size"

I'm sure it's related, but from what James says, it seems that in this case
the memory use is way over 2X the uncompressed patch size, and that's
suspicious.
-- 
David Roundy
Department of Physics
Oregon State University