Recently upgraded from darcs 2.2.0 to darcs 2.3.1... and can no longer work 
with one of my repos.

$ /Programs/Darcs/2.3.1/bin/darcs w --debug
/Mount/Work/per_dtest/_darcs/prefs/sources
Beginning identifying repository .
Done identifying repository .
/Mount/Work/per_dtest/_darcs/prefs/sources
Identified darcs-1 repo: /Mount/Work/per_dtest
darcs: Codec.Compression.Zlib: premature end of compressed stream
$

strace makes it look like the following file is at fault:
$ file _darcs/pristine.hashed/* | grep empty
_darcs/pristine.hashed/
da39a3ee5e6b4b0d3255bfef95601890afd80709:                                    
empty

$ time /Programs/Darcs/2.2.0/bin/darcs check
The repository is consistent!                                        
                                                                     
real    0m32.775s
user    0m27.049s
sys     0m1.278s
$ time /Programs/Darcs/2.3.1/bin/darcs check
The repository is consistent!                                        
                                                                     
real    0m38.155s
user    0m33.945s
sys     0m1.910s
$

And as check implies, a repair does nothing because it believes the repository 
is OK.  If I remove the above file from the pristine.hashed, both 2.2.0 and 
2.3.1 checks still report consistency and 2.3.1 whatsnew works.  I'm very 
hesitant to continue in this mode though...

Unfortunately, I cannot supply this repository for testing.

This looks pretty urgent.  Kevin: is there any way you could use this
information to boil this down to a minimal test case?

If you do succeed, have a look at http://wiki.darcs.net/Development/RegressionTests

Meanwhile, Petr: this may be one for you to look at as well

Doesn't seem to be very disturbing. This is just another wart on the current
pristine format, in that all files are compressed, *but* the empty ones are
not. It should be easy to work around, and it should also be safe to remove the
file from pristine and then run repair, which would probably remove that bad
file (new darcs versions create empty files as 0000000000-
da39a3ee5e6b4b0d3255bfef95601890afd80709 -- and the former contains valid gzip
header and an empty compressed file -- the latter is truly empty and disturbs
pristine reading that hashed-storage does).

It won't be very easy to write a testcase for this though, since current darcs
shouldn't produce these files in pristine anymore (or so I'd hope). Anyway, I
have manually produced a repository that exhibits this issue, let me attach it.

Attachments

• t.tar.gz

Oh, a workaround on the user side would be to say:
gzip -c /dev/null >
_darcs/pristine.hashed/da39a3ee5e6b4b0d3255bfef95601890afd80709

in the repository. That's perfectly safe thing to do and fixes the problem with
the repo.

Great! [silly darcs]
Glad to hear it only *looks* scary but is really quite benign.

Could we turn this into a regression test, then?  One way might just be to ship
the tarball with the tests/ directory (there's a subdir for that).  Another,
probably better way would be to reproduce the manual steps.

I'm not really able to create an good test for this because I'm not sure how 
the blank patchfile got introduced... as Petr said, it was probably an earlier 
wart that no longer exists.

I did some more testing though, this time with the darcs dev repo:

* The initial "darcs w" still has the same complaint:
darcs: Codec.Compression.Zlib: premature end of compressed stream

* Running a darcs repair generates the following:
Fixing pristine tree...
Hash mismatch(es)!
d1/d2/d3/d5
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d3/d6
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d3/d7/d8
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d3/d7
    index: 8cf6f36c44d48bdc81f3db9baf8841c1f90bb76dbad9491f261ad695b9755a29
  working: 359f732f818c662b2100a3f750d081614c7c09fa156d8b3d8f105b77eebc547f
d1/d2/d3/d9/d10/d11
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d3/d9/d10
    index: b44d23331f93827466a9a729c04c4720c19b0253ac30a496601ae53559fde62f
  working: 331a518d77a3ea90735f2fd373ba1f2f1360cf68cd5d0ee054d65359c3af3acf
d1/d2/d3/d9
    index: 1e2cc2e0b7c3a17f68f940a6302d5b82811a86653eb648c341501c2d77c3ff24
  working: 46afb29074741bdb07a57c94643b8d3136ab6ef7a8e49291e4f8e0a5dc9c63b5
d1/d2/d3
    index: 72b12a2afc0d640844ada9ace736bebc606e0f03a52254cad42694188d1d81c0
  working: cc911d26941d0263ea57cf250736f372baf66fcf5648311204544a21f002b210
d1/d2/d4/pts
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d4
    index: d38e79bda677b5c30531f7bd10c4b76b4741065b7f847528894bdcf5eed75e90
  working: a0f77e053b9c1fe6519af6ab095d24010d20b8bb2ef8c774cfd58ea116d6c9cb
d1/d2
    index: f2a59061486f35a2f6b4914f951c0c1cce337fa64552819bfc43a83e58b06102
  working: 7987eb8eef0533e1863025255a257227b7e8ff140579d43d1f9e8429124b065b
d1
    index: 04114f9904852dadbd32b9f85629ff96618b6a198012dc403f3972d788dc6309
  working: 83c85ecccb6cebbd152419623acf7a8333360109570e069984580a817dd3e9c8

Bad index discarded.
$

* The good news is that this seemed to address the empty file problem: a "darcs 
w" now returns "No changes" as expected.

* However, the hash errors remain:
$ darcs check
The repository is consistent!
Hash mismatch(es)!
d1/d2/d3/d5
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d3/d6
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d3/d7/d8
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d3/d7
    index: 8cf6f36c44d48bdc81f3db9baf8841c1f90bb76dbad9491f261ad695b9755a29
  working: 359f732f818c662b2100a3f750d081614c7c09fa156d8b3d8f105b77eebc547f
d1/d2/d3/d9/d10/d11
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d3/d9/d10
    index: b44d23331f93827466a9a729c04c4720c19b0253ac30a496601ae53559fde62f
  working: 331a518d77a3ea90735f2fd373ba1f2f1360cf68cd5d0ee054d65359c3af3acf
d1/d2/d3/d9
    index: 1e2cc2e0b7c3a17f68f940a6302d5b82811a86653eb648c341501c2d77c3ff24
  working: 46afb29074741bdb07a57c94643b8d3136ab6ef7a8e49291e4f8e0a5dc9c63b5
d1/d2/d3
    index: 72b12a2afc0d640844ada9ace736bebc606e0f03a52254cad42694188d1d81c0
  working: cc911d26941d0263ea57cf250736f372baf66fcf5648311204544a21f002b210
d1/d2/d4/pts
    index: 0000000000000000000000000000000000000000000000000000000000000000
  working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
d1/d2/d4
    index: d38e79bda677b5c30531f7bd10c4b76b4741065b7f847528894bdcf5eed75e90
  working: a0f77e053b9c1fe6519af6ab095d24010d20b8bb2ef8c774cfd58ea116d6c9cb
d1/d2
    index: f2a59061486f35a2f6b4914f951c0c1cce337fa64552819bfc43a83e58b06102
  working: 7987eb8eef0533e1863025255a257227b7e8ff140579d43d1f9e8429124b065b
d1
    index: 04114f9904852dadbd32b9f85629ff96618b6a198012dc403f3972d788dc6309
  working: 83c85ecccb6cebbd152419623acf7a8333360109570e069984580a817dd3e9c8

Bad index.
$

And the above is persistent.  So... is it safe to use this latest darcs with 
this repo?

That's quite interesting. It *should* be safe, although it probably causes a
small dent in the performance. It's however still something to worry about. Can
you share a tarball of the failing repository?

Sorry, can't share this one: it's from work (the annoying day job thing).  I'd 
be glad to run any analysis tools or anything on it and report back the 
results, but I can't pass along the repository itself.  Unfortunately, this 
hasn't seemed to occur in any of my personal repos.

Kevin: thanks for your continuing work in ferreting this out.  Would it be
possible to send the repository to just Petr  and not the whole team?  Perhaps
that would be less day-job sensitive?

Just a quick note that I have reproduced it and it is mostly harmless -- the 
problem is that empty directories do not get a hash in the index, but they do get 
a hash when doing the check, and that constitutes a mismatch. I'll look into 
fixing the problem...

Fixed. Also confirmed by Trent.

[Eric: I hope this is the right ticket!]

Kevin Quick wrote:
> Hash mismatch(es)!
> d1/d2/d3/d5
>     index: 0000000000000000000000000000000000000000000000000000000000000000
>   working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Here's the minimal test:

    $ PATH=~/.cabal/bin-x86_64:$PATH with-temp-dir bash -c 'darcs --version; darcs --exact-version | sed /]/q; darcs init; mkdir d; darcs rec -qlamd; darcs check'
    with-temp-dir: entering directory `/tmp/with-temp-dir.9Sfr21'
    2.3.1 (+ 426 patches)
    darcs compiled on Jan 10 2010, at 14:41:15

    Context:

    [Resolve issue1720:  Fix cabal haddock problem.
    Dave Love <fx@gnu.org>**20100106162322
     Ignore-this: 1521b3a742711ac3ee979efc765f6b61
    ]
    Finished recording patch 'd'
    The repository is consistent!
    Hash mismatch(es)!
    d
        index: 0000000000000000000000000000000000000000000000000000000000000000
      working: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

    Bad index.
    with-temp-dir: leaving directory `/tmp/with-temp-dir.9Sfr21'
    $

It is no longer present with hashed-storage 0.4.5:

    $ PATH=~/.cabal/bin-linux:$PATH with-temp-dir bash -c 'darcs --version; darcs --exact-version | sed /]/q; darcs init; mkdir d; darcs rec -qlamd; darcs check'
    with-temp-dir: entering directory `/tmp/with-temp-dir.1B8tE9'
    2.3.1 (+ 437 patches)
    darcs compiled on Jan 19 2010, at 11:29:30

    Context:

    [resolve issue1731: bump hashed-storage dependency to 0.4.5
    Reinier Lamers <tux_rocker@reinier.de>**20100117205223
     Ignore-this: c6e300b3c56d85b425306530b4d6c7be
    ]
    Finished recording patch 'd'
    The repository is consistent!
    with-temp-dir: leaving directory `/tmp/with-temp-dir.1B8tE9'
    $

Remind me to submit a regression test for this, then we can close this ticket.

Alright, re-opening for Trent's regression test (which is a good, disciplined,
idea).  Alternatively, some sort of hashed-storage test may be good enough.

The following patch updated the status of issue1677 to be resolved:

* Resolve issue1677: require hashed-storage 0.4.4. 
Ignore-this: 92a9def06cccee05dc1cbaa63b6ef6fa

This version of hashed-storage produces index with correct hashes for empty
directories (the previous versions would use zeroes as the hash instead of the
hash of empty string there, leading to hash mismatches in the index).