I ran this command on Windows, using the darcs.exe from 2008-02-06:

time darcs get --hashed --verbose --debug-verbose $USER@$HOST:$REPO

The remote host repo was in hashed format.  This was the result:

ssh wonwinmcbrootles@192.168.1.104 darcs transfer-mode --repodir  
playground/pycryptopp/pycryptopp-hf/
with ssh wonwinmcbrootles@192.168.1.104: get prefs/motd
with ssh wonwinmcbrootles@192.168.1.104: get format
Beginning identifying repository  
wonwinmcbrootles@192.168.1.104:playground/pycryptopp/pycryptopp-hf
P: Identifying repository wonwinmcbrootles@192.168.1.104:playground/ 
pycryptopp/pycryptopp-hf : inventory
with ssh wonwinmcbrootles@192.168.1.104: get inventory
P: Identifying repository wonwinmcbrootles@192.168.1.104:playground/ 
pycryptopp/pycryptopp-hf : hashed_inventory
with ssh wonwinmcbrootles@192.168.1.104: get hashed_inventory
P: Identifying repository wonwinmcbrootles@192.168.1.104:playground/ 
pycryptopp/pycryptopp-hf : format
with ssh wonwinmcbrootles@192.168.1.104: get format
Done identifying repository wonwinmcbrootles@192.168.1.104:playground/ 
pycryptopp/pycryptopp-hf
with ssh wonwinmcbrootles@192.168.1.104: get prefs/sources
with ssh wonwinmcbrootles@192.168.1.104: get hashed_inventory
Copying hashed pristine tree: eda419cf0416724dd8ba82f5aaa0220fa0d858a6
Beginning copying pristine
Reading hash file eda419cf0416724dd8ba82f5aaa0220fa0d858a6 from  
pristine.hashed/
I'm doing copyFileUsingCache on eda419cf0416724dd8ba82f5aaa0220fa0d858a6
with ssh wonwinmcbrootles@192.168.1.104: get pristine.hashed/ 
eda419cf0416724dd8ba82f5aaa0220fa0d858a6
In fetchFileUsingCachePrivate I'm going manually
     getting eda419cf0416724dd8ba82f5aaa0220fa0d858a6
     from wonwinmcbrootles@192.168.1.104:playground/pycryptopp/ 
pycryptopp-hf/_darcs/pristine.hashed/ 
eda419cf0416724dd8ba82f5aaa0220fa0d858a6
with ssh wonwinmcbrootles@192.168.1.104: get pristine.hashed/ 
eda419cf0416724dd8ba82f5aaa0220fa0d858a6

darcs failed:  Couldn't fetch `eda419cf0416724dd8ba82f5aaa0220fa0d858a6'
in subdir pristine.hashed from sources:

thisrepo:c:/playground/pycryptopp-hf
repo:wonwinmcbrootles@192.168.1.104:playground/pycryptopp/pycryptopp-hf

Writing working directory contents...

Zooko, 

Was the repo you were trying to "get" from a partial one? 

   Mark

> Was the repo you were trying to "get" from a partial one?

I'm pretty sure that it is not, but how can I tell?

Regards,

Zooko

Zooko,

I'm not certain how to identify when a repo is partial. However, I have a
different debugging question:

Is there actually a patch file for "eda419cf0416724dd8ba82f5aaa0220fa0d858a6"
on the server or not? 

  Mark

Zooko, 

And if you move the repos to be on the same machine, does the issue persist? 

And if you substitute a "darcs init; darcs pull -a " for a "darcs get", how
about then? 

Those tests could help narrow down the issue.

Okay now I tried the following with the same client computer  
(Windows) version of darcs (2008-02-06), and remote repo (hashed):

mkdir pycryptopp-hf
cd pycryptopp-hf
darcs init --hashed
time darcs get --all --verbose --debug-verbose  
wonwinmcbrootles@192.168.1.104:playground/pycryptopp/pycryptopp-hf

It ended with:

-------
darcs failed:  binary patch to ./misc/dependencies/setuptools-0.6c7- 
py2.5.egg couldn't apply.
Error applying patch to the working directory.

This may have left your working directory an inconsistent
but recoverable state. If you had no un-recorded changes
by using 'darcs revert' you should be able to make your
working directory consistent again.
-------

Then when I run "darcs repair", it ends with:

-------
darcs failed:  binary patch to ./misc/dependencies/setuptools-0.6c7- 
py2.5.egg couldn't apply.
-------

The full output of those two commands is attached (remember: "sudo  
apt-get install p7zip-full").

You can get a copy of this repository from http://allmydata.org/ 
source/pycryptopp , but note that it is a copy of the repository --  
it is not the actual same repository (hosted on my private  
192.168.1.104 in hashed format) involved in this bug.

Regards,

Zooko

Attachments

• logs.7z

> Is there actually a patch file for  
> "eda419cf0416724dd8ba82f5aaa0220fa0d858a6"
> on the server or not?

Yes.  Here is its location, metadata, and complete contents:

HACK wonwin-mcbrootles-computer:~/playground/pycryptopp/pycryptopp-hf/ 
_darcs$ find . -iname '*eda419cf0416724dd8ba82f5aaa0220fa0d858a6*'
./pristine.hashed/eda419cf0416724dd8ba82f5aaa0220fa0d858a6
HACK wonwin-mcbrootles-computer:~/playground/pycryptopp/pycryptopp-hf/ 
_darcs$ ls -al ./pristine.hashed/ 
eda419cf0416724dd8ba82f5aaa0220fa0d858a6
-rw-rw-r--    1 wonwinmc wonwinmc      301 Feb 13 19:24 ./ 
pristine.hashed/eda419cf0416724dd8ba82f5aaa0220fa0d858a6
HACK wonwin-mcbrootles-computer:~/playground/pycryptopp/pycryptopp-hf/ 
_darcs$ zcat ./pristine.hashed/eda419cf0416724dd8ba82f5aaa0220fa0d858a6
file:
COPYING.TGPPL.html
8ace7d77d0a33c4d32108bde67c7674b1319ac5a
directory:
misc
9cdae64ba66c73e763ae534f75d2084da403cd1e
file:
README.txt
3728871b6efb09179c15b27fa35bf1a227d2f95c
file:
COPYING.GPL
f96b9c958bb8b1645479ddbe30b790d54559a8ab
file:
setup.py
408dc763bb59bc8ea6a654446c3028c79c52eefe
file:
ez_setup.py
6945e0db17459ffb762e6f729defd308e9bcd761
directory:
pycryptopp
3d568e08c47d30eb43ce12ddbe9ff4df0a263539

On Feb 13, 2008, at 9:33 PM, Mark Stosberg wrote:

> And if you substitute a "darcs init; darcs pull -a " for a "darcs  
> get", how
> about then?

My report on this experiment crossed in the mail with your request  
for it.  :-)

On Feb 13, 2008, at 9:33 PM, Mark Stosberg wrote:

> And if you move the repos to be on the same machine, does the issue  
> persist?

I tarred up and scp'ed the source repo onto the same (Windows)  
machine.  Then "darcs get" succeeds, but subsequently "darcs check"  
or "darcs repair" in the gotten repo fail in a similar way: "darcs  
failed:  binary patch to ./misc/dependencies/setuptools-0.6c7- 
py2.5.egg couldn't apply.".

Hm.  That isn't the *exact* same failure though...

Regards,

Zooko

I just tried it with darcs-2-format source repository, and got the  
same failure as earlier:

On Feb 13, 2008, at 8:53 PM, Zooko wrote:

> In fetchFileUsingCachePrivate I'm going manually
>      getting eda419cf0416724dd8ba82f5aaa0220fa0d858a6
>      from wonwinmcbrootles@192.168.1.104:playground/pycryptopp/
> pycryptopp-hf/_darcs/pristine.hashed/
> eda419cf0416724dd8ba82f5aaa0220fa0d858a6
> with ssh wonwinmcbrootles@192.168.1.104: get pristine.hashed/
> eda419cf0416724dd8ba82f5aaa0220fa0d858a6
>
> darcs failed:  Couldn't fetch  
> `eda419cf0416724dd8ba82f5aaa0220fa0d858a6'
> in subdir pristine.hashed from sources:
>
> thisrepo:c:/playground/pycryptopp-hf
> repo:wonwinmcbrootles@192.168.1.104:playground/pycryptopp/ 
> pycryptopp-hf

Zooko,

Thanks for the follow-up. One more triage question for you: were any of the
files in the repo already gzip'ped? I think there may be a currently open bug
when that's that the case. 

   Mark

> Thanks for the follow-up. One more triage question for you: were  
> any of the
> files in the repo already gzip'ped? I think there may be a  
> currently open bug
> when that's that the case.

The ones mentioned in the error messages -- setuptools-0.6c7.egg for  
example -- are zipped:

setuptools-0.6c7.egg:       Zip archive data, at least v1.0 to extract

(Note: not gzipped.)

Regards,

Zooko

Note that this bug is blocking me from using darcs-2 for now.

I'll upgrade this to "urgent", since it's a potential blocker. 

Perhaps I can play around with generating a triggering test case by using zipped
or gzipped files.

Tonight I tried to create a reduced test case by adding just a "zip" file and
then just a "gz" file, and then trying different operations such as "put",
"darcs changes" and "darcs check". I was unable to trigger this bug.

Zooko,

I tried "get'ing" the provided repo and re-getting it locally, as well running
"darcs check" on it. I don't trigger any error with Darcs 2 built 2008-02-15. 

I'm giving this bug the "Windows" keyword since that's the only platform the bug
has been triggered on at this point. Perhaps if you tried again on a unix
platform you'd get a different result.

Here is a related error:

darcs-2 get http://darcs.net/repos/unstable

on Windows ends with:

-------
P: Applying patch : Sun Oct 31 06:04:47 Mountain Daylight Time 2004  Mark
Stosberg <mark@summersault.com>
  * homepage overhaul: new text and design, and temp. logo
Unapplicable patch:
Sun Oct 31 06:04:47 Mountain Daylight Time 2004  Mark Stosberg
<mark@summersault.com>
  * homepage overhaul: new text and design, and temp. logo

darcs failed:  binary patch to ./logo.png couldn't apply.
Repo read
Repo local: "False"
-------

full log:

https://zooko.com/zooko.darcs_get.log.txt.7z

and attached

Attachments

• zooko.darcs_get.log.txt.7z

On Sat, Feb 16, 2008 at 01:10:17AM -0000, Mark Stosberg wrote:
> Tonight I tried to create a reduced test case by adding just a "zip" file and
> then just a "gz" file, and then trying different operations such as "put",
> "darcs changes" and "darcs check". I was unable to trigger this bug.

It may be that you need to create a larger test file, as I believe gzip
refuses to compress small files (where that compression would actually
expand them).
-- 
David Roundy
Department of Physics
Oregon State University

I wonder if we might somewhere be opening a file in text mode? That would
certainly give us a windows-specific bug like this.  I had thought that we had
expunged all of those, but if we can't find this bug anywhere else, a review of
that possibility might be in order.

Actually, Zooko, could you post a link to the logo.png file that exists in the
working directory after this failure? Also, a quick look to see whether it's a
valid png file would be instructive.  If it's been corrupted by the addition of
extra '\r', we ought to be able to recognize this just by looking at the file
that's left over.

David

Oh, this is interesting!  This succeeds:

darcs-2 get http://darcs.net/repos/unstable-hashed

even though this still fails:

darcs-2 get http://darcs.net/repos/unstable

Here are the resulting logo file:

https://zooko.com/logo.png
(The one that failed.)

It doesn't look corrupted.  Note that there is no logo.png file in the base
directory after getting the unstable-hashed repo.  There is one in the
doc/logos/ subdirectory, but that is a different logo.

On Mon, Feb 18, 2008 at 05:27:40PM -0000, Zooko wrote:
> Oh, this is interesting!  This succeeds:
> 
> darcs-2 get http://darcs.net/repos/unstable-hashed
> 
> even though this still fails:
> 
> darcs-2 get http://darcs.net/repos/unstable

Good, good, we're making progress! Can you try running darcs check on the
repository that you got successfully with that first call?

> Here are the resulting logo file:
> 
> https://zooko.com/logo.png
> (The one that failed.)

Oh my, that's the cheesy logo I made myself, which Mark replaced with the
current logo! I haven't see that logo in ages.  :)

You're right, it certainly doesn't look corrupted.
-- 
David Roundy
Department of Physics
Oregon State University

I'm marking this as FixForDarcs20, as we really want windows systems to work
with Darcs 2.0.

Zooko,

You said this bug was triggered on Windows. Was it with Cygwin?

On Mar 7, 2008, at 8:29 AM, Mark Stosberg wrote:
>
> You said this bug was triggered on Windows. Was it with Cygwin?

I don't think it can possibly matter whether cygwin is installed on  
the system or not, since no cygwin code is executed when darcs runs.

But if it matters, cygwin is installed on that system.

Regards,

Zooko

This has something to do with content that is already gzipped, right?

Today I added a .tar.gz into my darcs repository, and now I can't look at
changesets other changesets in trac:

http://allmydata.org/trac/tahoe/changeset/2269

Running (cd /home/source/darcs/tahoe/trunk; TZ=UTC darcs diff --diff-command
"cat %2" --to-match "hash
20080310224405-e01fd-3c50e9d8aee44e82be50e4669ffc6897b0733b1d.gz"
"src/allmydata/mutable.py") failed: 2, darcs failed: binary patch to
./misc/dependencies/darcsver-1.1.1.tar.gz couldn't apply. :

Running this command on the command-line gets the same result.

To reproduce:

$ darcs get http://allmydata.org/source/tahoe/trunk tahoe
$ cd tahoe
$ darcs diff --to-match "hash
20080310224405-e01fd-3c50e9d8aee44e82be50e4669ffc6897b0733b1d.gz"
"src/allmydata/mutable.py"

The same thing happens with darcs-1.1.0pre (with darcs-1-format repository), and
with darcs-2pre with darcs-1-format, darcs-hashed-format, and darcs-2-format.

P.S.  So previously this bug was only triggered on windows, but this issue
described in msg3859 is triggered on Linux and Mac OS X.

I just double-checked, and the three-line "to reproduce" script in msg3859 does
indeed result in this error on Mac OS X:

darcs failed:  binary patch to ./misc/dependencies/darcsver-1.1.1.tar.gz
couldn't apply.

Hm.  I *was* using darcs-2pre when this corruption happened.  So this may be a
bug in darcs-2pre, which results in a repository such that both darcs-2pre and
darcs-1.1.0pre exhibit the same behavior when faced with that repository.

By the way, since this is potentially interfering with the http://allmydata.org
project (for example, I wanted to look at a patch today and instead got this: 
http://allmydata.org/trac/tahoe/changeset/2230/src/allmydata/webish.py ), I
might try to perform surgery on my repository to erase the patches that touched
darcsver-1.1.1.tar.gz file from history.  Fortunately, those patches were a
brief mistake and are not used or required by any other patch...

Zooko wrote:
> 
> Hm.  I *was* using darcs-2pre when this corruption happened.  So this may be a
> bug in darcs-2pre, which results in a repository such that both darcs-2pre and
> darcs-1.1.0pre exhibit the same behavior when faced with that repository.
> 
> By the way, since this is potentially interfering with the http://allmydata.org
> project (for example, I wanted to look at a patch today and instead got this: 
> http://allmydata.org/trac/tahoe/changeset/2230/src/allmydata/webish.py ), I
> might try to perform surgery on my repository to erase the patches that touched
> darcsver-1.1.1.tar.gz file from history.  Fortunately, those patches were a
> brief mistake and are not used or required by any other patch...

Zooko,

Thanks for the continued follow-up on this bug. It sounds important. I 
recall both that there was a ".gz" fix committed and I believe an 
automated test for it, too.  Pursuing the angle that this is the result 
of a bug which formerly existed but is not currently in Darcs 2 seems 
interesting to pursue.

Otherwise, reviewing or creating a reduced test to try to trigger it 
would be interesting.

   Mark

Sorry -- I obliterated the offending patch from the
http://allmydata.org/source/tahoe/trunk repository, thus making it harder for
people to reproduce the problem.

The offending patch simply added darcsver-1.1.1.tar.gz to the repo.  Here is
darcsver-1.1.1.tar.gz.

Attachments

• darcsver-1.1.1.tar.gz

Hm.  I just tried to reproduce this bug by "darcs add"'ing
darcsver-1.1.1.tar.gz, and it didn't trigger the bug.

Here is a repository which still has the offending patches in:

http://allmydata.org/source/tahoe/trunk-reget

Zooko,

Thanks for your continued investigation into this. 

Since we seem to have isolated the bug, and it can't be reproduced with the
current darcs, and we know there was a fix made for this in a recent past, but
possibly after your repo got into this state, it seems reasonable that the bug
can be "presumed-dead".

So, I would be comfortable considering this bug resolved now. If there would be
a final thing to do, it would be to examine the repo and determine that it is
indeed corrupt somehow in relation to this, and not the the repo is OK, but
darcs is still buggy. 

Thoughts, David?

   Mark

On Wed, Mar 12, 2008 at 02:21:56PM -0000, Mark Stosberg wrote:
> Zooko,
> 
> Thanks for your continued investigation into this. 

My thanks as well.

> Since we seem to have isolated the bug, and it can't be reproduced with the
> current darcs, and we know there was a fix made for this in a recent past, but
> possibly after your repo got into this state, it seems reasonable that the bug
> can be "presumed-dead".
>
> So, I would be comfortable considering this bug resolved now. If there would be
> a final thing to do, it would be to examine the repo and determine that it is
> indeed corrupt somehow in relation to this, and not the the repo is OK, but
> darcs is still buggy. 

That sounds good, particularly if there were two (or more) patches
involving the tar.gz file, which would explain the continued persistence of
the repository corruption.  Otherwise I'd like to have a chance to look at
the corrupt repository (which won't happen this week while I'm at a
conference).

If is just the one patch adding the tarball, then I'd not close the bug
just yet.  If there are more than one, then I think I know why there's a
problem, and it should be fixed, and you could close the bug before I have
a chance to look at the repo.

Gotta go...
-- 
David Roundy
http://www.darcs.net

On Mar 12, 2008, at 8:21 AM, Mark Stosberg wrote:

> Since we seem to have isolated the bug, and it can't be reproduced  
> with the
> current darcs, and we know there was a fix made for this in a  
> recent past, but
> possibly after your repo got into this state, it seems reasonable  
> that the bug
> can be "presumed-dead".

Sorry, when I said that I was unable to reproduce it, I didn't mean  
that the same process with the current darcs shows no-bug.  What I  
meant was that the process that I tried to use to reproduce it didn't  
reproduce it.  However, I believe the bad patch was created with  
current darcs-2pre or close to it.

Do you know of a recent change to darcs-2 that might have fixed a bug  
in darcs-2 that could have caused a corrupted repository such as:

http://allmydata.org/source/darcs/trunk-reget

?

If so, I could see whether the version of darcs-2 that I have  
installed on allmydata.org is later than the creation of the bug and  
earlier than the bug fix.

Regards,

Zooko

On Mar 12, 2008, at 8:29 AM, David Roundy wrote:

> On Wed, Mar 12, 2008 at 02:21:56PM -0000, Mark Stosberg wrote:
>>
>> Thanks for your continued investigation into this.
>
> My thanks as well.

You're welcome!  And thank you for all the hard work you two have put  
into darcs.  Making software "production grade" is a lot of work.  I  
think darcs is getting there, but I personally won't be happy with it  
until there is a string of all-green buildslaves showing a set of  
unit tests which collectively exercise all of the important branches.

> That sounds good, particularly if there were two (or more) patches
> involving the tar.gz file, which would explain the continued  
> persistence of
> the repository corruption.

There were two -- one that did nothing but add the .gz file, and the  
next did nothing but remove it.

How can we write a unit test that will show that the current version  
of darcs doesn't suffer from this bug?  (Such as test would also give  
us confidence from now on that we haven't regressed and reintroduced  
this bug.)  We could either add darcsver-1.1.1.tar.gz to the  
repository in the tests/ subdirectory, or encode the entire contents  
of darcsver-1.1.1.tar.gz into a shell script or perl script test.   
Or, perhaps you guys, knowing the actual cause of the bug, could  
suggest a smaller file that would stimulate the same logic.  Is it  
just any gzipped file which is touched by two patches?   
darcsver-1.1.1.tar.gz is 6013 bytes.  The minimal gzip file (the gzip  
of the empty file) is 28 bytes.

Regards,

Zooko

The change

Mon Feb 18 11:43:11 EST 2008  David Roundy <droundy@darcs.net>
  * resolve issue673: if hash check fails, try reading file without ungzipping it.

should fix a bug relating to adding of gzipped files to the repository.  This
only affects hashed and darcs-2-format repositories.  Old-style repositories
shouldn't be affected at all (because their pristine cache isn't compressed).

I am unable to access http://allmydata.org/source/darcs/trunk-reget, but from
what you describe (adding and removing gzipped tarball), I believe the
corruption is consistent with what could have been created prior to this bug
fix... or possibly it would require that the second patch is recorded after the
bugfix (I'm not sure).

I'm marking this bug as resolved, since I think we've resolved it.  Thanks for
your patience in tracking this down!

David

On Mar 15, 2008, at 10:55 AM, David Roundy wrote:
>
> I am unable to access http://allmydata.org/source/darcs/trunk- 
> reget, but from

Sorry, this is the correct URL:

http://allmydata.org/source/tahoe/trunk-reget/

Argh, now I attempted to reproduce it by switching back to the  
earlier version of darcs -- at least I *think* it was the same darcs  
executable that I used earlier, and it emits the same version string,  
but now the problem didn't happen:

-------
DARC wonwin-mcbrootles-computer:~/playground/allmydata/tahoe$ time  
darcs get trunk prod_exp2
Copying patches, to get lazy repository hit ctrl-C...
Finished getting.

real    0m12.906s
user    0m3.835s
sys     0m1.097s
DARC wonwin-mcbrootles-computer:~/playground/allmydata/tahoe$ darcs -- 
version
2.0.0 (2.0.0 (+ -1 patch))
-------

Does anyone have an idea what could be going on here?  Does darcs  
"2.0.0 (2.0.0 (+ -1 patch))" have a known bug which is sporadic in  
effect?  (I doubt it.)

The behavior is identical to a bug that I earlier reproduced and that  
was earlier fixed:

http://bugs.darcs.net/issue687

The fix went into darcs before the darcs-2.0.0 release, so it ought  
to be in both of these versions of darcs that I have here.

Speaking of which, what does the "(2.0.0 (+ -1 patch))" version  
string mean?

Appended is the exact-version of this darcs executable.

Oh, I have a theory.  Maybe the "darcs 2.0.0 +37", when I ran it,  
cleaned up some state in my system-wide cache, so that now even the  
older "darcs 2.0.0 + -1" works.  Hm.

Does this suggest that we ought to put out a 2.0.1 stable release of  
darcs?  Not unless we know what is the actual problem.  I don't see  
any patches in my "darcs 2.0.0 +37" which aren't already in "darcs  
2.0.0 + -1" whose descriptions suggest that they fix a bug.

Regards,

Zooko

-------
darcs compiled on Apr  8 2008, at 18:37:34
# configured Tue Apr  8 18:35:12 PDT 2008
./configure --prefix=/usr/local/stow/darcs-2.0.0

Context:

[update README url links
gwern0@gmail.com**20080407201333]

[add a bit more debugging info to repository identification.
David Roundy <droundy@darcs.net>**20080408151912]

[resolve issue385: don't worry if we can't get local changes.
David Roundy <droundy@darcs.net>**20080408151823]

[add test for issue385.
David Roundy <droundy@darcs.net>**20080408151808]

[resolve issue786:  implement oops command.
David Roundy <droundy@darcs.net>**20080408145856]

[fix URL in network test.
David Roundy <droundy@darcs.net>**20080408145407]

[fix manual bug.
David Roundy <droundy@darcs.net>**20080407191736]

[add new show bug command (hidden) to see what darcs will report if  
we encounter a bug.
David Roundy <droundy@darcs.net>**20080407175410]

[automatically work out the version of the stable release.
David Roundy <droundy@darcs.net>**20080407171850]

[set prefs again (they got lost on convert).
David Roundy <droundy@darcs.net>**20080407171559]

[update darcs repository URL.
David Roundy <droundy@darcs.net>**20080407164601]

[fix up website for new release.
David Roundy <droundy@darcs.net>**20080407164010]

[simplify determine_release_state.pl.
David Roundy <droundy@darcs.net>**20080407153000]

[make determine_release_state.pl use changes --count.
David Roundy <droundy@darcs.net>**20080407152347]

[add --count output option to changes.
David Roundy <droundy@darcs.net>**20080407151825]

[TAG 2.0.0
David Roundy <droundy@darcs.net>**20080407150638]

(By the way, I opened a new ticket -- http://bugs.darcs.net/issue801  
-- for this before I realized that it looks the same as the resolved  
issue687.)

On Apr 18, 2008, at 2:38 PM, Zooko wrote:
>
>
> Argh, now I attempted to reproduce it by switching back to the
> earlier version of darcs -- at least I *think* it was the same darcs
> executable that I used earlier, and it emits the same version string,
> but now the problem didn't happen:

I just confirmed that there are only two darcs executables on this  
system, so I was definitely reusing the same "darcs 2.0.0 + -1" when  
I ran the test the third time as when I did it the first time and  
experienced the bug.

Regards,

Zooko

Okay, I suspect that what happened was that I had some corruption in my global
cache from using pre-releases of darcs-2 with a global cache.

I can't reproduce this problem with either of the post-darcs-2 executables that
I have here, so let's consider it fixed.

Okay, I got this kind of error message again.  This time I know exactly what I
was doing when it happened.  By the way, it happened on Mac OS X, this time.

What I was doing was running tailor to do "darcs record" in a repo, named
"tailored" at the same time that I ran a "darcs get" in another repo, named "dw"
to get a copy of tailored.  The result was that when I used "dw", I got this
message:

{{{
HACO wonwin-mcbrootles-computer:~/playground/libcurl/dw$ darcs check
                                                           
darcs failed:  Couldn't fetch
`0000081564-08e2f575191165b8d044792021e00a786951b7f770add74aa2a5f01c59b7f581'
in subdir pristine.hashed from sources:

thisrepo:/Users/wonwinmcbrootles/playground/libcurl/dw
repo:/Users/wonwinmcbrootles/playground/libcurl/tailored
cache:/Users/wonwinmcbrootles/.darcs/cache
}}}

Note that I *did* have a global cache in use.  Perhaps the darcs locking doesn't
correctly control simultaneous readers and writers into the global cache?

This is with a current darcs-2:

HACO wonwin-mcbrootles-computer:~/playground/libcurl/dw$ darcs --version
2.0.0 (2.0.0 (+ 220 patches))

I tried to reproduce this and couldn't, but this is consistent with the
hypothesis that it is a problem of unsynchronized simultaneous access to the
global cache.  (Such a problem would be hard to reproduce on demand.)

I did, however, capture the entire state of my global cache and the two
repositories after it happened...  Here, I will attach that state to this ticket.

Oh!  It happened again.  Yes, I am able to reproduce this *some* of the time by
doing a darcs get from the "tailored" repo while tailor is busy updating the
working directory of that repo and invoking 'darcs record' in that repo.

To summarise: trying to get from a repository that is simultaneously being
recorded to sometimes blows up.  Is that correct?

Hmm.  I guess you could cook up a little Perl script to try and reproduce this
scenario with a toy repository.  The trick of doing many iterations of the same
thing until it blows up could be helpful.

On May 20, 2008, at 3:13 AM, Eric Kow wrote:

>
> Eric Kow <eric.kow@gmail.com> added the comment:
>
> To summarise: trying to get from a repository that is  
> simultaneously being
> recorded to sometimes blows up.  Is that correct?

Yes.  It happened twice out of about six attempts last night here.

Regards,

Zooko

Ok, zooko, do you think this recordget_race.sh script more or less reflects what
is going on? 

I get errors like 'Duplicate patch' and other darcs check failures.

It may be that the resolution for this is something that amounts to "tell the
user not to do that", but it depends on how realistic it is for this kind of
scenario to occur; if it happens with something like push/get, it would be big
trouble, I'm guessing.

Attachments

• recordget_race.sh

Come to think of it, "tell the user not to do that" is clearly out of the question.

David?

On May 20, 2008, at 6:35 AM, Eric Kow wrote:
>
> Ok, zooko, do you think this recordget_race.sh script more or less  
> reflects what
> is going on?

Yes, this looks like a reasonable simulation of my behavior.

Interesting that you don't get the same error message that I got two  
or three times now.

Maybe the platform (Mac OS X at the moment, Windows previously) matters.

Regards,

Zooko

http://progetti.arstecnica.it/tailor/ticket/152

This might be a similar issue -- I was running tailor to produce a darcs
repository from a CVS repository.  I was pulling from that resulting darcs
repository.  Tailor stopped, saying that darcs had exited with exit code 1 when
asked to do "darcs record".  I tried darcs record by hand and it worked.

Then I retried the tailor process and it worked, so it does kind of seem like
there was a transient failure which went away on the second attempt.

The following patch updated the status of issue687 to be resolved:

* resolve issue687: don't clean out pristine cache except when optimizing. 
The trouble is that we can never tell when someone might be waiting to grab
one of these older files.  Atomicity doesn't gain us much if we drop the
old files immediately.  The downside is that now repositories will
monotonically grow until an optimize is performed.  :(

For what it is worth, as a user I prefer the simpler and more predictable
behavior of storing and using everything until I clear it out with /bin/rm -rf
~/.darcs/cache.

One question mark is what happens when there are too many files in the cache for
the underlying filesystem to handle smoothly.  As long as the resulting failure
is clean and apparent-to-the-user, then it is okay.

Zooko, clearing out ~/.darcs/cache won't clean out the crud which accumulates in
your repository, so your suggestion doesn't address the issue.

David

Oh, I see.  Well, I am quite satisfied with my repositories monotonically
growing until optimize is performed.