I created a 1 billion byte file (the machine has a mere 500 million  
bytes of physical RAM), darcs added that file, and then darcs  
recorded and I got

darcs failed: out of memory

This was with darcs-2.

Regards,

Zooko

I'm sorry Zooko, but I'm going to have to declare that using darcs with files
that don't fit into memory+swap is just a mistake, not our bug.  Perhaps we
should document this, but the changes needed to allow us to deal with files of
this magnitude are just silly.

David

On Jan 31, 2008, at 9:42 AM, David Roundy wrote:

> I'm sorry Zooko, but I'm going to have to declare that using darcs  
> with files
> that don't fit into memory+swap is just a mistake, not our bug.   
> Perhaps we
> should document this, but the changes needed to allow us to deal  
> with files of
> this magnitude are just silly.

I think that you are saying two things here:  first, that people  
shouldn't want that, and second that it would be hard to implement.

On the first point, I'm sure that rarely someone will want it, but  
not often.

On the second point, I'll have to take your word for it, but I  
continue to be surprised that when working in a programming language  
that was designed from the start for laziness and abstraction, that  
it isn't easy to make an abstraction over data which lazily uses the  
disk only when necessary.  (What about an abstraction which can be  
implemented just by using mmap?)

I don't think it is necessary to document this, as long as the error  
messages are clear when it happens, which was it was for the "darcs  
add bigfile && darcs record" test that I did.

Regards,

Zooko

...
> On the second point, I'll have to take your word for it, but I
> continue to be surprised that when working in a programming language
> that was designed from the start for laziness and abstraction, that
> it isn't easy to make an abstraction over data which lazily uses the
> disk only when necessary.  (What about an abstraction which can be
> implemented just by using mmap?)

In order to use files of this magnitude,  we wouldn't need just lazy
reading or lazy processing of data, we'd need active swapping to disk.
 Laziness would allow a few operations (e.g. get and record and maybe
pull, if there's no merging needed), but we'd need real swapping in
order to avoid failure in nontrivial darcs commands.  Which is why it
doesn't seem worthwhile to implement this.  It's not that you can't
work with files of this size, you just need to use more swap, since we
aren't going to reimplement that functionality of the OS ourselves
(even though in principle we could do a better job with darcs-specific
swap functionality).

David