Folks:

Somehow one of my repositories got corrupted so that when I run

darcs changes --from-tag=^allmydata-tahoe

(with either darcs-1.0.9 or darcs-2pre)

it says:

/home/zooko/darcsbugged/source-darcs-bugged/_darcs/patches/ 
20070817221910-92b7f-e637e92d8ddddea613300ed07f1ce943732c471c.gz:  
openBinaryFile: does not exist (No such file or directory)

I got a new copy from http://allmydata.org/source/tahoe/trunk, and  
that one doesn't have this problem.  The difference between the  
working official repository and the corrupted one is this -- it seems  
like the corrupted one has an "inventory" file with a lot *more*  
entries in it.  The full output from "diff -u -r trunk source-darcs- 
bugged" is here:

https://zooko.com/sdb.diff.u.r.txt

So this raises a few different questions:

1.  Shouldn't "darcs check" detect this corruption?

2.  Are there any known bugs in darcs, or were there recently in pre- 
releases of darcs-2, which could cause this corruption?

I can tell you rather precisely when it happened, since this is part  
of a buildbot-managed automatic build system.  On this date, there  
was no corruption and "darcs changes" worked:

http://allmydata.org/buildbot/builders/solaris/builds/416

and the next day, it was corrupt:

http://allmydata.org/buildbot/builders/solaris/builds/417

Regards,

Zooko

Zooko,

Was this a partial repo? I've seen partial repos report this error when a patch
beyond the partial horizon is requested. 

  Mark

On Feb 11, 2008, at 5:17 PM, Mark Stosberg wrote:
>
> Was this a partial repo? I've seen partial repos report this error  
> when a patch
> beyond the partial horizon is requested.

It was a partial repo, but why did this behavior break between Thu  
and Fri?  The only change between Thu and Fri was that more patches  
were pulled into the repo from the source repo.

Also, I tried to reproduce the problem by pulling with --partial, but  
the resulting partial repo did not exhibit this problem.

Regards,

Zooko

It's possible (I haven't looked at the code to see what we do) that this is an
effect of darcs automatically optimizing the repository.  I don't remember if we
do this... I didn't think we did it on darcs-1 format repositories.

Did you by any chance pull a tag between Thu and Fri? (not that this should
cause this bug, but it'd affect the control flow we'd have to consider to figure
out what's gone wrong.)

David

Another question:  you didn't upgrade darcs between Thu and Fri, by any chance?
I know you said that all you did was some pulls, but I'd like to be sure.  Simon
M recently reported a performance regression in changes --from-tag that could be
related to this if you upgraded to the latest unstable during this period. 
(Just brainstorming.)

David

On Feb 12, 2008, at 7:56 AM, David Roundy wrote:

> It's possible (I haven't looked at the code to see what we do) that  
> this is an
> effect of darcs automatically optimizing the repository.  I don't  
> remember if we
> do this... I didn't think we did it on darcs-1 format repositories.
>
> Did you by any chance pull a tag between Thu and Fri? (not that  
> this should
> cause this bug, but it'd affect the control flow we'd have to  
> consider to figure
> out what's gone wrong.)

No we certainly didn't.  We create tags only to mark releases of  
allmydata.org, and there hasn't been a release for a while, and we  
pull several times a day.

I just tried something else to reproduce the corruption, and again  
failed:  I did a darcs-1.0.9 get --partial, obliterated the patches  
created since Thur, then did a darcs-2pre pull.  This simulates, as  
well as I can figure, what the corrupted repository went through.   
However the test repository showed no ill effects.

Did you look at the diff output that I included in my original bug  
report?  Maybe the difference between the normal and corrupted  
inventory files will tell you something.  There were no other  
differences between the normal and corrupted repositories except in  
the inventory file.

Also, even without knowing what caused this corruption, would it be  
reasonable to make "darcs check" detect this kind of corruption?

Regards,

Zooko

On Feb 12, 2008, at 7:58 AM, David Roundy wrote:

> Another question:  you didn't upgrade darcs between Thu and Fri, by  
> any chance?
> I know you said that all you did was some pulls, but I'd like to be  
> sure.  Simon
> M recently reported a performance regression in changes --from-tag  
> that could be
> related to this if you upgraded to the latest unstable during this  
> period.
> (Just brainstorming.)

Hm...  It is possible that I did upgrade my darcs-2pre to the current  
darcs-2pre on Thu or Fri.

I'm sorry that I don't have records so that I can answer that for  
certain.

Regards,

Zooko

On Tue, Feb 12, 2008 at 08:55:19AM -0700, zooko wrote:
> On Feb 12, 2008, at 7:56 AM, David Roundy wrote:
> Did you look at the diff output that I included in my original bug  
> report?  Maybe the difference between the normal and corrupted  
> inventory files will tell you something.  There were no other  
> differences between the normal and corrupted repositories except in  
> the inventory file.

Yes, I did look at the diff, and from a quick review, it looks like the
repository isn't actually corrupt in one sense... in the sense that I
suspect it's a perfectly valid repository that's just missing some patches.
You can test if this is the case by (in a copy, of course) copying over the
contents of _darcs/patches/* from your full repository (the one you were
pulling from) into the failing one, with an option to not overwrite any
existing files (I'm not sure how to make cp do this, but I'm sure you can).
My guess is that the result will be a repository that passes darcs check.

If this is correct, then the bug was (perhaps) that some darcs operation
caused a valid inventory to be written that wasn't as "optimized" as a
previous one, which exposed the repository's "partial-ness" to darcs
changes.

Another possibility is that this happened long ago, and the problem was
caused by a darcs upgrade caused darcs changes --from-tag to be less
efficient and to try to read older history.  Actually, this second
possibility seems more likely to me, at the moment.  The larger inventory
(particularly if you're not comparing in the diff with the same repository
at an earlier date) could easily be explained by the ordinary behavior of
darcs pull, which doesn't break the inventory up on tags.  And I believe
there's an undiagnosed performance regression in changes --from-tag that
might also trigger this behavior.
-- 
David Roundy
Department of Physics
Oregon State University

This same corruption happened again.  I fixed it by regetting the original
repository with darcs-2.

Hello Zooko,

I am not sure what to do about this: I have changed the status to "need-info", but 
perhaps it is more appropriate to set it "deferred" or "presumed-dead"?

Thanks and best regards
Thorkil

I think the most realistic approach to this and other bugs is to retire support
for partial repositories.

By this I mean pre-existing partial repositories will still work exactly the
same way, but that future darcs will ignore the --partial flag.  We already have
lost the ability to create checkpoint patches, so are creeping towards this.

Sorry!  The 'solution' is to upgrade to hashed repositories and call us in the
morning.