Sorry for the lack of details.  Perhaps Zooko and Zandr can fill us in

Chat log:
http://irclog.perlgeek.de/darcs/2008-12-04#i_737739

Repository tarball:
http://testgrid.allmydata.org:3567/uri/URI%3ACHK%3Ahusxcycnp5bqcocrxw3kqsnhyi%3Ape5h3til4ylosy6brwjpq23m3qw3untmh7e7xap375dk3cmd3wzq%3A3%3A10%3A99473758

They had an 'unapplicable patch'... there is also a claim floating around that
all this compression stuff is a red herring

Zooko says that he was able to fetch it with his darcs 1.0.9 (Ubuntu) whereas
Zandr was not.

Possible differences he suggests are libcurl.so and x86 vs amd64

Another thing to note is that mixed up in these, there is the issue that some of
the repositories they were working with were partial, and missing their
checkpoints, which explains some of the errors.

With the referred repository:

> $ gunzip -t _darcs/patches/20080128195622-e01fd-
a805b1db46503b277fe9dfdbb7978699240f65ec.gz
>
> gunzip: _darcs/patches/20080128195622-e01fd-
a805b1db46503b277fe9dfdbb7978699240f65ec.gz: invalid compressed data--crc error
> $

So, even at the outset, the repository appears to be corrupted.

Best regards
Thorkil

On Sun, Dec 07, 2008 at 22:18:09 -0000, Thorkil Naur wrote:
> > $ gunzip -t _darcs/patches/20080128195622-e01fd-
> a805b1db46503b277fe9dfdbb7978699240f65ec.gz

And if we do a gzip -t _darcs/patchs/*.gz, we can see some patches
from 2007 which are affected.

This leads me to think that the CRC problems have been around for
a long time.  Unless of course, we have any reason to believe that
a later darcs would have somehow opened these files and rezipped
them (maybe patch commutation?)

> This leads me to think that the CRC problems have been around for
> a long time.  Unless of course, we have any reason to believe that
> a later darcs would have somehow opened these files and rezipped
> them (maybe patch commutation?)

Zooko: is this repository push-only?  Or is there any reason to
believe that any patch commutation (which entails changing patch
contents and thereby gzipped patch file contents) has taken
place?

If not, I think it is safe to claim that these CRC problems have
been around for longer than darcs 2.

> Zooko: is this repository push-only?  Or is there any reason to
> believe that any patch commutation (which entails changing patch
> contents and thereby gzipped patch file contents) has taken
> place?
>
> If not, I think it is safe to claim that these CRC problems have
> been around for longer than darcs 2.

Hm...  In terms of the actual set of patches it is push-only, but I  
have run "darcs optimize --reorder".

Do the unix timestamps on the files help?

Regards,

Zooko

On Mon, Dec 08, 2008 at 05:22:26 -0700, zooko wrote:
> Hm...  In terms of the actual set of patches it is push-only, but I have 
> run "darcs optimize --reorder".

Ah, well that shoots down that hypothesis.  Hmm :-/

I'm still trying to figure out which version of darcs produces bad CRCs
but I would need an example of this happening in darcs 1.0.9, or an
explanation about what it was that introduced the broken CRCs, which is
not necessarily related to your bug though.

> Do the unix timestamps on the files help?

I've deleted the repository unfortunately (or rather, I rebooted my
computer, which flushed /tmp), but I think the files go back no further
than 2008-06, which would be darcs 2.0.0...

> I've deleted the repository unfortunately (or rather, I rebooted my  
> computer, which flushed /tmp), but I think the files go back no  
> further than 2008-06, which would be darcs 2.0.0...

http://testgrid.allmydata.org:3567/file/URI%3ACHK% 
3Ahusxcycnp5bqcocrxw3kqsnhyi% 
3Ape5h3til4ylosy6brwjpq23m3qw3untmh7e7xap375dk3cmd3wzq%3A3%3A10% 
3A99473758/@@named=/serverrepo.tar.gz

(Sorry for the fact that my MUA -- Apple's Mail.app -- inserts line  
breaks into that URL.  :-( )

I untarred it and use "ls -altr" in _darcs/patches/.  The most  
recently touched files are:

-rw-rw-r--    1 wonwinmc wonwinmc      199 Nov 17 23:51  
20080930202051-e01fd-2c1992db2d7cd23f056230158aa6f4d4d6f57aa3.gz
-rw-rw-r--    1 wonwinmc wonwinmc      538 Nov 18 00:26  
20081118072525-e01fd-7659fc42d17da89735b42c40e9e9d733293947c6.gz
-rw-rw-r--    1 wonwinmc wonwinmc      248 Nov 18 14:37  
20081118213238-e01fd-a3b22aeb60cc8063d4bfa20405551c44b7312265.gz
-rw-rw-r--    1 wonwinmc wonwinmc      987 Nov 18 14:37  
20081118213015-e01fd-80c1e2b6fd91ede0754912c9c784c8d98ade02f5.gz
-rw-rw-r--    1 wonwinmc wonwinmc      461 Nov 18 14:45  
20081118214431-e01fd-94ab21e962ebc7784f31be277843546cb54b5259.gz
-rw-rw-r--    1 wonwinmc wonwinmc        4 Nov 18 20:06  
pending.tentative
-rw-rw-r--    1 wonwinmc wonwinmc        4 Nov 18 20:06 pending
-rw-rw-r--    1 wonwinmc wonwinmc      619 Nov 18 20:06  
20081028203646-4233b-f8bc92a46d30b0b24ee426f40b4c8b6cd72437d4.gz
drwxrwxr-x  5588 wonwinmc wonwinmc   189992 Nov 18 20:06 .

The least recently touched files are:

-rw-rw-r--    1 wonwinmc wonwinmc    10396 Apr 17  2008 0000026320- 
aa80bc69d3b71101139c91fad94b0b6a1036de998388ec788517ba3d59f45d73
-rw-rw-r--    1 wonwinmc wonwinmc    20765 Apr 17  2008  
0000254386-1fbf97fb279e58b91fbbb277d8a2615685eda8e240f1beea89a371bbe35b0 
2e6
-rw-rw-r--    1 wonwinmc wonwinmc      242 Apr 17  2008  
0000000477-77cdf38215deac3db587d293112abf75fafe0891d7effbc9165beb2035314 
c3a
-rw-rw-r--    1 wonwinmc wonwinmc     2471 Apr 17  2008  
0000008012-86a3d8c1cc26d8d5be8092d152718cd4f922ff839ca8af10ef796a10f9d0e 
3d3
-rw-rw-r--    1 wonwinmc wonwinmc     3126 Apr 14  2008  
0000007998-214b31257aef60932b6ca289956ca3e6a6a041c798a75ed09bf4e570d7f77 
e88

--Z

On Mon, Dec 08, 2008 at 10:42:01 -0700, zooko wrote:
> -rw-rw-r--    1 wonwinmc wonwinmc    10396 Apr 17  2008 0000026320- 
> aa80bc69d3b71101139c91fad94b0b6a1036de998388ec788517ba3d59f45d73

Thanks! I've posted this on issue844 as likely evidence that the broken
CRCs came from darcs > 1.0.9 (sigh!)

On Dec 8, 2008, at 10:57 AM, Eric Kow wrote:

> On Mon, Dec 08, 2008 at 10:42:01 -0700, zooko wrote:
>> -rw-rw-r--    1 wonwinmc wonwinmc    10396 Apr 17  2008 0000026320-
>> aa80bc69d3b71101139c91fad94b0b6a1036de998388ec788517ba3d59f45d73
>
> Thanks! I've posted this on issue844 as likely evidence that the  
> broken
> CRCs came from darcs > 1.0.9 (sigh!)

Note that I've used pre-release versions of darcs quite often.  There  
could have been a bug in darcs that was not in any stable release but  
was in the darcs darcs repo at some point, and I might have used that  
version of darcs.

Regards,

Zooko

In http://bugs.darcs.net/issue844 we have converged on the idea that the bad
CRCs were introduced in darcs 2.0.2.

We need to determine if there is anything we need to do for this bug.

1. What about the 'unapplicable patch' errors?  Why do we have them?
2. Is there any difference between the gzip errors and the stuff on issue844?
3. Is there any relationship between [1] and [2]?

Zooko: as I understand it, this is no longer blocking anything for you guys, right?

Yeah, I'm not aware of any current operational problems at allmydata from this
issue.

If I understand correctly, there is a repository with patches made by lots of
different versions of darcs, and that darcs 1.0.9 has trouble fetching it
(because of a gzread error on a fetched patch) and that darcs 2 does not (?).
One key point to pay attention to is that if they darcs optimize --uncompress
the repository, then darcs 1.0.9 is perfectly happy.

New development: darcs 2.3.0 offers a mechanism (darcs gzcrcs) to repair broken
CRC files.  It fixed 1576 files in this tarball.

I still don't know if the CRC errors are related or if there is something else
going on.  But maybe we can rule them out.

Zooko: could you check to see if (i) you and Zandr can still reproduce this
unapplicable patch w/ darcs 1.0.9 problem and (ii) if repairing the CRCs makes a
difference? (Perhaps you are still sufficiently in touch with the Tahoe team to
do this?)

Thanks!

I'm sorry, but I'm not going to have time to do anything like this for at least
a couple of weeks.  Perhaps someone else could try darcs-1.0.9 against the darcs
repo in msg6784.

Good idea, I recommend somebody download the repo, confirm the failure with
darcs 1.0.9, fix the CRCs then re-confirm the failure.

I tried to download the tarball from msg6784 but I got a 

2010-04-03 17:19:07 ERROR 503: Service Unavailable.

Anyway, I guess we'll have to mark this presumed-dead for lack of time
and focus on other things :-(