darcs

Issue 1860 noslurps darcs can create pristine that's missing hashed-files

Title noslurps darcs can create pristine that's missing hashed-files
Priority bug Status resolved
Milestone 2.5.0 Resolved in 2.5.0
Superseder Nosy List dmitry.kurochkin, kowey, mornfall
Assigned To
Topics

Created on 2010-06-03.22:17:37 by mornfall, last changed 2010-06-15.22:40:13 by mornfall.

Files
File name Uploaded Type Edit Remove
seven-cxt kowey, 2010-06-05.18:36:42 text/plain
try-narrow.sh kowey, 2010-06-05.17:10:23 application/x-sh
Messages
msg11228 (view) Author: mornfall Date: 2010-06-03.22:17:35
In a copy of current repo.darcs as used by darcs-benchmark:

$ darcs unpull --last 1000 -a
WARNING: Doing a one-time conversion of pristine format.
This may take a while. The new format is backwards-compatible.
Pristine conversion done...
Finished unpulling.
00:15:17 | morn@twi:~/dev/darcs/_test/darcs-unpull -> darcs show pristine 
| head 

darcs failed:  error opening 
_darcs/pristine.hashed/00f9f3c801c250e5c1e9528f80877725b7b7f11a06c893a657
730a0bafc2b628

Bug! Trimming this down into a smaller testcase would be really helpful.
If we can fix this in something like a week or maybe a little less, we 
should wait with alpha 1 for the fix. Otherwise, I'd say it's not a 
blocker, since it's pretty rare as far as I can tell.
msg11231 (view) Author: mornfall Date: 2010-06-03.22:30:24
I have narrowed this down a little in the meantime...
The patch that causes the breakdown is this:

Fri May  8 04:08:00 CEST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Move tools/ to contrib/.
  
  In other projects where I've seen a "tools" directory, it contains
  utilities used during build, rather than by end users.  I think
  convention is to put handy little end-user utilities in a directory
  called "contrib".  This also indicates that they aren't maintained as
  actively as the rest of the code.
  
  Renaming this directory makes me, at least, less confused.
    move ./tools ./contrib

HOWEVER, this does *not* happen when unpulling this patch alone. It only 
happens if some patches on top of this one are unpulled together with 
it. It is not enough to just unpull a set of patches touching contrib 
together with this one either. I could only reproduce this by unpulling 
521 patches from (a get of) repo.darcs. Unpulling 520 patches, then 1, 
then pulling everything back and unpulling 521 works and shows no signs 
of the bug. So this is really obscure. I am thoroughly mystified.
msg11246 (view) Author: kowey Date: 2010-06-05.17:10:23
On Thu, Jun 03, 2010 at 22:30:24 +0000, Petr Ročkai wrote:
> HOWEVER, this does *not* happen when unpulling this patch alone. It only 
> happens if some patches on top of this one are unpulled together with 
> it. It is not enough to just unpull a set of patches touching contrib 
> together with this one either. I could only reproduce this by unpulling 
> 521 patches from (a get of) repo.darcs.

If it helps narrow it down any, I did a little binary chop.  First:

 darcs get repo.darcs t0
 cd t0
 darcs optimize --pristine

and then the attached shell script.

So we know that

 darcs unpull --last=175 followed by darcs unpull --last=346

triggers this bug, but not 176 and 345

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
Attachments
msg11247 (view) Author: kowey Date: 2010-06-05.18:36:43
On Sat, Jun 05, 2010 at 17:10:24 +0000, Eric Kow wrote:
> > HOWEVER, this does *not* happen when unpulling this patch alone. It only 
> > happens if some patches on top of this one are unpulled together with 
> > it. It is not enough to just unpull a set of patches touching contrib 
> > together with this one either. I could only reproduce this by unpulling 
> > 521 patches from (a get of) repo.darcs.

Attached is a context file, for which darcs unpull --last=7 -a
triggers the error.

It's not entirely minimal, but I think it should be something to
go on.

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
Attachments
msg11251 (view) Author: mornfall Date: 2010-06-06.09:11:57
I have narrowed this down to unpulling these two patches in this order, with TreeIO 
buffering enabled. The bug does not happen when buffering is disabled. I can 
hopefully fix the problem at this point...

Wed Sep  9 21:24:13 CEST 2009  Eric Kow <kowey@darcs.net>        
  * Remove darcs.cgi contrib script.
  Will Glozer can no longer maintain it.
Shall I unpull this patch? (1/324)  [ynWvplxdaqjk], or ? for help: y
Fri May  8 04:08:00 CEST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Move tools/ to contrib/.
  
  In other projects where I've seen a "tools" directory, it contains
  utilities used during build, rather than by end users.  I think
  convention is to put handy little end-user utilities in a directory
  called "contrib".  This also indicates that they aren't maintained as
  actively as the rest of the code.
  
  Renaming this directory makes me, at least, less confused.
Shall I unpull this patch? (2/324)  [ynWvplxdaqjk], or ? for help: y
msg11252 (view) Author: mornfall Date: 2010-06-06.09:21:40
Ok, the problem is in how we track changes in TreeMonad -- this tracking 
is path-based, and only paths that have changed are written out. The 
problem happens when a path is changed and its parent is then renamed, 
which would happen exactly in this scenario. I will try to work out a 
fix. One option would be to fix up the paths in the "changed" set, 
another would be to track changes differently (but I am not sure how).
msg11287 (view) Author: mornfall Date: 2010-06-06.19:28:48
The following patch updated the status of issue1860 to be resolved:

* Resolve issue1860: (Un)applying move patches may lead to incomplete pristine. 
Ignore-this: f81b28cc30fbd28b88a87bc33c0d1f06

Fixed in hashed-storage 0.5.2 -- directory renames in TreeIO have caused
subsequent (disk) flush-es to be incomplete in some cases.
msg11450 (view) Author: mornfall Date: 2010-06-15.22:40:12
The following patch updated issue issue1860 with status=resolved;resolvedin=2.5.0 (current)

* Resolve issue1860: (Un)applying move patches may lead to incomplete pristine. 
Ignore-this: f81b28cc30fbd28b88a87bc33c0d1f06

Fixed in hashed-storage 0.5.2 -- directory renames in TreeIO have caused
subsequent (disk) flush-es to be incomplete in some cases.
History
Date User Action Args
2010-06-03 22:17:37mornfallcreate
2010-06-03 22:30:24mornfallsetmessages: + msg11231
2010-06-05 17:10:24koweysetfiles: + try-narrow.sh
nosy: + kowey
messages: + msg11246
title: noslurps darcs can create pristine that's missing hashed-files -> noslurps darcs can create pristine that's missing hashed-files
2010-06-05 18:36:44koweysetfiles: + seven-cxt
messages: + msg11247
title: noslurps darcs can create pristine that's missing hashed-files -> noslurps darcs can create pristine that's missing hashed-files
2010-06-06 09:11:57mornfallsetmessages: + msg11251
2010-06-06 09:21:41mornfallsetmessages: + msg11252
2010-06-06 14:05:00mornfalllinkissue1862 superseder
2010-06-06 14:51:53mornfalllinkissue1853 superseder
2010-06-06 19:28:49mornfallsetstatus: needs-reproduction -> resolved
messages: + msg11287
2010-06-15 20:52:18adminsetmilestone: 2.5.0
2010-06-15 21:00:10adminsettopic: - Target-2.5
2010-06-15 22:40:13mornfallsetmessages: + msg11450
resolvedin: 2.5.0