please see the transcript below. it happens when i convert a rather big repo
using tailor.

ati@ed101:~/local-repos/sbcl/sync$ darcs --version
2.2.0 (+ 2 patches)
ati@ed101:~/local-repos/sbcl/sync$ darcs record --all --pipe --ignore-times
What is the date? 2002-02-02
Who is the author? asd
What is the patch name? asd
What is the log?

darcs failed:  Cannot remove non-empty file ./src/pcl/describe.lisp
Your repository is now in an inconsistent state.
This must be fixed by running darcs repair.


Failed to record patch 'asd'

ati@ed101:~/local-repos/sbcl/sync$ wnew
M ./NEWS +2
M ./src/code/condition.lisp -14
M ./src/code/describe.lisp -319 +546
M ./src/code/typedefs.lisp -7 +9
M ./src/cold/warm.lisp -7 +1
M ./src/compiler/compiler-deftype.lisp -2 +2
M ./src/compiler/main.lisp -2 +4
R ./src/pcl/describe.lisp
M ./src/pcl/documentation.lisp -1 +1
M ./tests/interface.impure.lisp -9 +11
M ./version.lisp-expr -1 +1
ati@ed101:~/local-repos/sbcl/sync$ ll src/pcl/describe.lisp
ls: cannot access src/pcl/describe.lisp: No such file or directory
ati@ed101:~/local-repos/sbcl/sync$ 

hth,

- attila

Hi Attila,

This looks interesting.

Can you somehow boil this down into a minimal test case, notably without Tailor?
Here are some tips on how to do that http://wiki.darcs.net/RegressionTests

If not, can you at least reproduce this reliably, and perhaps give us a recipe
to make it blow up?

Thanks!

sorry, but it is beyond my resources to make a simple test-case out of this.
it's tailor + a huge repo automatically converted from CVS.

all i can add is that i've met this several times since then. i think every time
a file is deleted from the CVS source it comes out.

i've resolved it by hand about 5 times since then, but i still don't really know
what i'm doing... :)

i think i delete and/or darcs revert the offending file and amend the failed
half-recorded patch created by tailor. or something like that, sorry for the
confusion, that's all i can help for now.

i *think* this is what happens: file deleted in CVS, tailor invokes a "darcs
remove offending-file", tailor does not remove the physical file from the disk,
then tailor invokes a "darcs record --all --pipe --ignore-times" which fails as
attached.

ok, here it is again, i'll try to record what i do. the relevant tailor log:

19:00:10 [I] Changeset #28
19:00:10 [I] Upstream revision "2009-08-05 12:11:42 by demoss" by demoss,
2009-08-05 12:11:43+00:00
19:00:10 [I] Log message: 1.0.30.35: turn SB-INTROSPECT into an ASDF system

 * I at least am starting to find myself having to insert

      (EVAL-WHEN (...) (REQUIRE :SB-INTROSPECT))

   in too many places. This may be a sign that things aught to be in
   SB-EXT instead, but ASDFication seems like a logical next step.

   ...arguably all contribs but ASDF itself should be ASDF systems.
19:00:10 [I] /home/ati/local-repos/sbcl/sync $ cvs -f -d
:pserver:anonymous@sbcl.cvs.sourceforge.net:/cvsroot/sbcl -q update -d -P -r 1.1
contrib/sb-introspect/introspect.lisp contrib/sb-introspect/sb-introspect.asd
19:00:13 [I] [Ok]
19:00:13 [I] /home/ati/local-repos/sbcl/sync $ cvs -f -d
:pserver:anonymous@sbcl.cvs.sourceforge.net:/cvsroot/sbcl -q update -d -P -r 1.3
contrib/sb-introspect/Makefile contrib/sb-introspect/xref-test-data.lisp
contrib/sb-introspect/xref-test.lisp
19:00:15 [I] [Ok]
19:00:15 [I] /home/ati/local-repos/sbcl/sync $ cvs -f -d
:pserver:anonymous@sbcl.cvs.sourceforge.net:/cvsroot/sbcl -q update -d -P -r
1.18 contrib/sb-introspect/test-driver.lisp
19:00:17 [I] [Ok]
19:00:17 [I] /home/ati/local-repos/sbcl/sync $ cvs -f -d
:pserver:anonymous@sbcl.cvs.sourceforge.net:/cvsroot/sbcl -q update -d -P -r
1.37 contrib/sb-introspect/sb-introspect.lisp
19:00:18 [I] [Ok]
19:00:18 [I] /home/ati/local-repos/sbcl/sync $ cvs -f -d
:pserver:anonymous@sbcl.cvs.sourceforge.net:/cvsroot/sbcl -q update -d -P -r
1.1596 NEWS
19:00:20 [I] [Ok]
19:00:20 [I] /home/ati/local-repos/sbcl/sync $ cvs -f -d
:pserver:anonymous@sbcl.cvs.sourceforge.net:/cvsroot/sbcl -q update -d -P -r
1.4588 version.lisp-expr
19:00:21 [I] [Ok]
19:00:21 [I] /home/ati/local-repos/sbcl/sync $ darcs record --all --pipe
--ignore-times 2>&1
19:00:22 [W] [Status 2]
19:00:22 [I] /home/ati/local-repos/sbcl/sync $ darcs repair 2>&1
19:00:55 [I] [Ok]
19:00:55 [I] /home/ati/local-repos/sbcl/sync $ darcs whatsnew --summary
--look-for-add 2>&1
19:00:57 [W] [Status 0]
19:00:57 [C] Couldn't replay changeset:
Revision: 2009-08-05 12:11:42 by demoss
Date: 2009-08-05 12:11:43+00:00
Author: demoss
Entries: NEWS(UPD to 1.1596)
	 version.lisp-expr(UPD to 1.4588)
	 contrib/sb-introspect/Makefile(UPD to 1.3)
	 contrib/sb-introspect/introspect.lisp(ADD at 1.1)
	 contrib/sb-introspect/sb-introspect.asd(ADD at 1.1)
	 contrib/sb-introspect/sb-introspect.lisp(DEL at 1.37)
	 contrib/sb-introspect/test-driver.lisp(UPD to 1.18)
	 contrib/sb-introspect/xref-test-data.lisp(UPD to 1.3)
	 contrib/sb-introspect/xref-test.lisp(UPD to 1.3)
Log: 1.0.30.35: turn SB-INTROSPECT into an ASDF system

 * I at least am starting to find myself having to insert

      (EVAL-WHEN (...) (REQUIRE :SB-INTROSPECT))

   in too many places. This may be a sign that things aught to be in
   SB-EXT instead, but ASDFication seems like a logical next step.

   ...arguably all contribs but ASDF itself should be ASDF systems.
19:00:57 [I] 12 pending changesets in state file
19:00:57 [C] Upstream change application failed: Changes left in working dir
after commit:
M ./NEWS +1
M ./contrib/sb-introspect/Makefile -5 +2
A ./contrib/sb-introspect/introspect.lisp
A ./contrib/sb-introspect/sb-introspect.asd
R ./contrib/sb-introspect/sb-introspect.lisp
M ./contrib/sb-introspect/test-driver.lisp -134 +256
M ./contrib/sb-introspect/xref-test-data.lisp +14
M ./contrib/sb-introspect/xref-test.lisp -43 +48
M ./version.lisp-expr -1 +1

Most probably a tailor bug, not everything has been committed.: Changes left in
working dir after commit:
M ./NEWS +1
M ./contrib/sb-introspect/Makefile -5 +2
A ./contrib/sb-introspect/introspect.lisp
A ./contrib/sb-introspect/sb-introspect.asd
R ./contrib/sb-introspect/sb-introspect.lisp
M ./contrib/sb-introspect/test-driver.lisp -134 +256
M ./contrib/sb-introspect/xref-test-data.lisp +14
M ./contrib/sb-introspect/xref-test.lisp -43 +48
M ./version.lisp-expr -1 +1

ati@ed101:~/local-repos/sbcl$ 



the offending file is:
./contrib/sb-introspect/sb-introspect.lisp
(it was renamed to introspect.lisp as i can see)



ati@ed101:~/local-repos/sbcl$ cd sync/contrib/sb-introspect/
ati@ed101:~/local-repos/sbcl/sync/contrib/sb-introspect$ ll
total 68
drwxr-sr-x 2 ati ati  4096 2009-08-19 19:00 CVS
-rw-r--r-- 1 ati ati 30838 2009-08-05 14:11 introspect.lisp
-rw-r--r-- 1 ati ati    47 2009-08-19 19:00 Makefile
-rw-r--r-- 1 ati ati  1761 2009-08-05 14:11 sb-introspect.asd
-rw-r--r-- 1 ati ati  9713 2009-08-19 19:00 test-driver.lisp
-rw-r--r-- 1 ati ati  1133 2007-11-10 17:17 test.lisp
-rw-r--r-- 1 ati ati  3572 2009-08-19 19:00 xref-test-data.lisp
-rw-r--r-- 1 ati ati  2723 2009-08-19 19:00 xref-test.lisp
ati@ed101:~/local-repos/sbcl/sync/contrib/sb-introspect$ 




ooops, the file is not there on the filesystem?! let's revert it and simply
delete it (as opposed to a darcs remove that tailor issues (it's just a guess!))




ati@ed101:~/local-repos/sbcl/sync$ darcs rev
contrib/sb-introspect/sb-introspect.lisp
Reverting changes in "contrib/sb-introspect/sb-introspect.lisp"..

rmfile ./contrib/sb-introspect/sb-introspect.lisp
Shall I revert this change? (1/1)  [ynWsfvplxdaqjk], or ? for help: y
Do you really want to revert this change? y
Finished reverting.



after running tailor again, it's getting worse (notice the duplicate R and a
entries!):


M ./NEWS +1
M ./contrib/sb-introspect/Makefile -5 +2
A ./contrib/sb-introspect/introspect.lisp
A ./contrib/sb-introspect/sb-introspect.asd
R ./contrib/sb-introspect/sb-introspect.lisp
M ./contrib/sb-introspect/test-driver.lisp -134 +256
M ./contrib/sb-introspect/xref-test-data.lisp +14
M ./contrib/sb-introspect/xref-test.lisp -43 +48
M ./version.lisp-expr -1 +1
a ./contrib/sb-introspect/sb-introspect.lisp



ati@ed101:~/local-repos/sbcl/sync$ pico contrib/sb-introspect/sb-introspect.lisp
ati@ed101:~/local-repos/sbcl/sync$ ll contrib/sb-introspect/sb-introspect.lisp
-rw-r--r-- 1 ati ati 0 2009-08-19 19:14 contrib/sb-introspect/sb-introspect.lisp


that didn't help.

but after restarting tailor a few times (end deleting the offending file from
the file system), i've ended up in this state:

19:16:17 [C] Upstream change application failed: Changes left in working dir
after commit:
a ./contrib/sb-introspect/introspect.lisp
a ./contrib/sb-introspect/sb-introspect.asd


ati@ed101:~/local-repos/sbcl/sync/contrib/sb-introspect$ darcs add
introspect.lisp sb-introspect.asd
ati@ed101:~/local-repos/sbcl/sync/contrib/sb-introspect$ darcs amen
Wed Aug  5 14:11:43 CEST 2009  demoss
  * [sbcl @ 2009-08-05 12:11:42 by demoss]
  1.0.30.35: turn SB-INTROSPECT into an ASDF system
  
   * I at least am starting to find myself having to insert
  
        (EVAL-WHEN (...) (REQUIRE :SB-INTROSPECT))
  
     in too many places. This may be a sign that things aught to be in
     SB-EXT instead, but ASDFication seems like a logical next step.
  
     ...arguably all contribs but ASDF itself should be ASDF systems.
Shall I amend this patch? [yNvpxq], or ? for help: y
addfile ./contrib/sb-introspect/introspect.lisp                          
Shall I add this change? (1/4)  [ynWsfvplxdaqjk], or ? for help: y
hunk ./contrib/sb-introspect/introspect.lisp 1

[whole file diff deleted]

Shall I add this change? (2/4)  [ynWsfvplxdaqjk], or ? for help: y
addfile ./contrib/sb-introspect/sb-introspect.asd
Shall I add this change? (3/4)  [ynWsfvplxdaqjk], or ? for help: y
hunk ./contrib/sb-introspect/sb-introspect.asd 1

[whole file diff deleted]

Shall I add this change? (4/4)  [ynWsfvplxdaqjk], or ? for help: y
You're not demoss! Amend anyway? y
Finished amending patch:
Wed Aug 19 19:19:04 CEST 2009  demoss
  * [sbcl @ 2009-08-05 12:11:42 by demoss]
  1.0.30.35: turn SB-INTROSPECT into an ASDF system
  
   * I at least am starting to find myself having to insert
  
        (EVAL-WHEN (...) (REQUIRE :SB-INTROSPECT))
  
     in too many places. This may be a sign that things aught to be in
     SB-EXT instead, but ASDFication seems like a logical next step.
  
     ...arguably all contribs but ASDF itself should be ASDF systems.
ati@ed101:~/local-repos/sbcl/sync/contrib/sb-introspect$                 



and tailor goes on happily...


sorry for any confusion in this transcript, it's not without a reason: i'm still
confused how i got through that state after several tries, especially that
tailor is not logging all darcs invocations.

hrm, some more info:

the repo in question is a darcs-1 repo. i just converted it to a darcs-2 repo
and run a darcs check:

alendvai@dwim:/opt/repos/sbcl$ darcs check
WARNING: Fixing removal of non-empty file ./contrib/sb-introspect/sb-introspect.lisp
WARNING: Fixing removal of non-empty file ./src/pcl/describe.lisp
WARNING: Fixing removal of non-empty file ./src/code/early-pcounter.lisp
WARNING: Fixing removal of non-empty file ./src/code/pcounter.lisp

Looks like we have a difference...                               
Nothing

Inconsistent repository!
Found broken patches.



as you can see, the sb-introspect.lisp file from my previous transcript shows up
there.

one more addition:

i had a backup of the tailor conversion from a few months back lying around.
when i created the first conversion, none of these errors came up (probably
because the darcs version i've used at that time didn't have this bug? or the
version of tailor i used was using different darcs commands?).

but anyway, i grabbed that backup darcs-1 repo, converted it to darcs-2, started
tailor HEAD with darcs 2.3.0 (so, the same environment as in my previous
attempts, only on a darcs-2 sync repo this time) and it ran without any issues!

Attila, thanks for going through all this trouble!

Could you let us know what your tailor version was?  See issue693 in which some
versions of tailor would create patches that removed files without removing
their contents through direct manipulation of the pending patch.

Also, could you darcs check the darcs 1 repo? Perhaps it was created by a
version of tailor that was too old to have the pending patch bug?

Thanks!

On Sat, 22 Aug 2009 11:54:20 +0000
Eric Kow <bugs@darcs.net> wrote:

> Could you let us know what your tailor version was?  See issue693 in
> which some versions of tailor would create patches that removed files
> without removing their contents through direct manipulation of the
> pending patch.

Actually, current tailor still uses that clever-but-maybe-wrong way of
doing:

  http://progetti.arstecnica.it/tailor/browser/vcpx/repository/darcs/target.py#L177

Unfortunately, darcs 2.x changed its behaviour of that, I assume for
very good reasons :-) With darcs 1.x it was perfectly fine doing the
following:

    darcs --version
    mkdir p
    cd p
    darcs init
    echo "foo" > bar
    darcs add bar
    darcs rec -a -m "1st"
    echo "{" > _darcs/patches/pending
    echo "rmfile ./bar" >> _darcs/patches/pending
    echo "}" >> _darcs/patches/pending
    darcs rec -a -m "2nd"
    darcs check

which outputs the following with 1.0.9

    $ darcs --version
    1.0.9rc1 (release candidate 1)
    $ mkdir p
    $ cd p
    $ darcs init
    $ echo "foo" > bar
    $ darcs add bar
    $ darcs rec -a -m "1st"
    Finished recording patch '1st'
    $ echo "{" > _darcs/patches/pending
    $ echo "rmfile ./bar" >> _darcs/patches/pending
    $ echo "}" >> _darcs/patches/pending
    $ darcs rec -a -m "2nd"
    Finished recording patch '2nd'
    $ darcs check
    Applying patch 2 of 2... done.
    The repository is consistent!

while current 2.3 gives

    $ darcs --version
    2.3.0 (release)
    $ mkdir p
    $ cd p
    $ darcs init
    $ echo "foo" > bar
    $ darcs add bar
    $ darcs rec -a -m "1st"
    Finished recording patch '1st'
    $ echo "{" > _darcs/patches/pending
    $ echo "rmfile ./bar" >> _darcs/patches/pending
    $ echo "}" >> _darcs/patches/pending
    $ darcs rec -a -m "2nd"
    Finished recording patch '2nd'
    $ darcs check
    WARNING: Fixing removal of non-empty file ./bar
    Looks like we have a difference...
    Nothing

    Inconsistent repository!
    Found broken patches.

Further inspection reveals that the last step, "darcs check",
effectively found and fixed the bad patch (and maybe a simple
reordering of the command output would be clearer, postponing the
WARNING to come *after* the "Found broken patches."), changing it from

    [2nd
    lele@nautilus.homeip.net**20090822122930
     Ignore-this: 483a9cc8245c0424079acd38fa180448
    ] rmfile ./bar

to

    [2nd
    lele@nautilus.homeip.net**20090822122930
     Ignore-this: 483a9cc8245c0424079acd38fa180448
    ] binary ./bar
    oldhex
    *666f6f0a
    newhex
    *
    rmfile ./bar

I don't understand why it switched to binary kind, but maybe tailor
could dump the hexified version of "darcs query content bar" into the
pending file... Will try that once I get back from vacation.

Happy hacking,
ciao, lele.
-- 
nickname: Lele Gaifax    | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas    | comincerò ad aver paura di chi mi copia.
lele@nautilus.homeip.net |                 -- Fortunato Depero, 1929.

On Sat, Aug 22, 2009 at 12:41:17 +0000, Lele Gaifax wrote:
> Actually, current tailor still uses that clever-but-maybe-wrong way of
> doing:
> 
>   http://progetti.arstecnica.it/tailor/browser/vcpx/repository/darcs/target.py#L177

Crucial detail there!  Thanks :-)

>     $ darcs --version
>     1.0.9rc1 (release candidate 1)
>     $ mkdir p
>     $ cd p
>     $ darcs init
>     $ echo "foo" > bar
>     $ darcs add bar
>     $ darcs rec -a -m "1st"
>     Finished recording patch '1st'
>     $ echo "{" > _darcs/patches/pending
>     $ echo "rmfile ./bar" >> _darcs/patches/pending
>     $ echo "}" >> _darcs/patches/pending
>     $ darcs rec -a -m "2nd"
>     Finished recording patch '2nd'
>     $ darcs check
>     Applying patch 2 of 2... done.
>     The repository is consistent!

What happens when you check this darcs-1-created repo with darcs 2?

>     [2nd
>     lele@nautilus.homeip.net**20090822122930
>      Ignore-this: 483a9cc8245c0424079acd38fa180448
>     ] binary ./bar
>     oldhex
>     *666f6f0a
>     newhex
>     *
>     rmfile ./bar
> 
> I don't understand why it switched to binary kind, but maybe tailor
> could dump the hexified version of "darcs query content bar" into the
> pending file... Will try that once I get back from vacation.

I think it switched to binary because that was the easiest thing to
implement (you don't know what that file might contain).  I don't see
why we don't do binary-detection in this repair operation, though.
Probably for simplicity.

On Sat, 22 Aug 2009 15:01:44 +0000
Eric Kow <bugs@darcs.net> wrote:

> What happens when you check this darcs-1-created repo with darcs 2?

The same thing happen, either on (a copy of) original darcs-1 or on a
clone done with "darcs2.3 get".

ciao, lele.
-- 
nickname: Lele Gaifax    | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas    | comincerò ad aver paura di chi mi copia.
lele@nautilus.homeip.net |                 -- Fortunato Depero, 1929.

On Sun, Aug 23, 2009 at 07:13:49 +0000, Lele Gaifax wrote:
> > What happens when you check this darcs-1-created repo with darcs 2?
> 
> The same thing happen, either on (a copy of) original darcs-1 or on a
> clone done with "darcs2.3 get".

Hang on, I just copy and pasted your minimal example and ran it as a
script (*) and found that no matter what version of darcs creates the
repository, darcs 2 check complains about the inconsistency.

So this does not appear to be a regression, just darcs 2 being more
alert about errors than darcs 1.

Could you confirm this assessment of the situation?

(*) By the way, I love it when people make things like that easy,
    so thanks!

    One trick I used for switching between versions was just to stick 
    alias darcs=darcs-1.0.9 on top of my script

On Sun, 23 Aug 2009 10:16:47 +0000
Eric Kow <bugs@darcs.net> wrote:

> Hang on, I just copy and pasted your minimal example and ran it as a
> script (*) and found that no matter what version of darcs creates the
> repository, darcs 2 check complains about the inconsistency.

Yes, sorry if I gave the wrong feeling. The on-disk patches contain
exactly the same information, recording it with whatever darcs version
I tried.

But only darcs 2 perceives the inconsistency.

> So this does not appear to be a regression, just darcs 2 being more
> alert about errors than darcs 1.

Yes: the bug is definitely on tailor's shoulders, it shouldn't create
such an "incomplete" representation of a remove.

Thank you,
ciao, lele.
-- 
nickname: Lele Gaifax    | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas    | comincerò ad aver paura di chi mi copia.
lele@nautilus.homeip.net |                 -- Fortunato Depero, 1929.

Attila: I'm going to operate on the assumption that this is only the tailor bug
from issue693.  Adding you there.