darcs

Issue 673 if hash check fails, try reading file without ungzipping it.

Title if hash check fails, try reading file without ungzipping it.
Priority bug Status resolved
Milestone 2.0.x Resolved in
Superseder Nosy List Serware, darcs-devel, dmitry.kurochkin, kowey, markstos, thorkilnaur, tommy
Assigned To markstos
Topics Darcs2

Created on 2008-02-11.14:24:45 by tommy, last changed 2010-06-15.21:20:06 by admin.

Messages
msg3314 (view) Author: tommy Date: 2008-02-11.14:24:44
I get a Hash failure with darcs2 (with both hashed and darcs-2
format, but not with the old format).

I run the following command in a newly initialized repo:

  find -name _darcs -prune -or -print | while (read f) { darcs add $f && darcs
rec -am 'init '$f }

and after a while I get:

  Finished recording patch 'init ./sample.call'
  Finished recording patch 'init ./sounds'
  Finished recording patch 'init ./sounds/Makefile'
  Hash failure in /home/tommy/tmp/asterisk-1.4.0 of hash
7e8f6423d14d055b145c1e482e7de9cf56ceea78

After that all darcs commands fail with:

  darcs failed:  Couldn't fetch `7e8f6423d14d055b145c1e482e7de9cf56ceea78'
  in subdir pristine.hashed from sources:

I've put a 33 MiB .tgz of the repo in this state at:

  http://www.lysator.liu.se/~ptp/pub/hashfailure.tgz
msg3330 (view) Author: droundy Date: 2008-02-11.17:48:07
Thanks for the report, Tommy!

It looks like your repository responds well to darcs repair, which
re-creates this file.  The question then is why wasn't it created in the
first place? The relevant file in the pristine cache is:

asterisk-core-sounds-en-gsm-1.4.4.tar.gz

and I can't help but wonder if what went wrong was that it was already
gzipped.  We store all files in gzipped form using zlib, and the hash that
we use is the hash of the uncompressed contents.  So I'm speculating that
maybe the trouble is that zlib recognized this as a gzipped file and
refused to compress it.  Then when we opened and read it to check the sha1
hash, zlib uncompressed it, resulting in a different checksum.

I haven't tested this, but I think this is enough to write up a little test
script that adds a gzipped file to the repository to try to reproduce this.
It may need to be a large gzipped file, since I think zlib may default to
storing small files in plain text format.

Anyhow, if my analysis is correct, then the solution would most likely be
to check the hash of the uncompressed and non-uncompressed form of files.
We only need do this when the hash check fails, so it shouldn't be common
and shouldn't slow us down much.
-- 
David Roundy
Department of Physics
Oregon State University
msg3514 (view) Author: markstos Date: 2008-02-16.22:53:48
I can see about creating a reduced test case for this.
msg3607 (view) Author: markstos Date: 2008-02-21.00:54:43
droundy published a fix for this in the unstable repo, I'm not sure why the
posthook didn't check in to mark in resolved in Roundup, but I'm doing it now.
msg3645 (view) Author: droundy Date: 2008-02-22.21:20:46
The following patch updated the status of issue673 to be resolved in the unstable branch:

* resolve issue673: if hash check fails, try reading file without ungzipping it. 

You can view the patch details online here: 
http://darcs.net/cgi-bin/darcs.cgi/unstable/?c=annotate&p=20080218164311-72aca-a43f1c3de23a67b84bf76d789993d212b245fba7.gz
History
Date User Action Args
2008-02-11 14:24:45tommycreate
2008-02-11 17:48:10droundysetstatus: unread -> unknown
nosy: droundy, tommy, beschmi, kowey
messages: + msg3330
2008-02-11 18:08:58markstossettopic: + Darcs2
nosy: droundy, tommy, beschmi, kowey
2008-02-16 22:53:06markstoslinkissue659 superseder
2008-02-16 22:53:49markstossetstatus: unknown -> testing
assignedto: markstos
title: hash failure with hashed and darcs-2 format -> record => "hash failure" with pre-gzipped files
messages: + msg3514
nosy: + markstos
2008-02-16 22:54:14markstossetstatus: testing -> has-patch
nosy: droundy, tommy, beschmi, kowey, markstos
2008-02-21 00:54:45markstossetstatus: has-patch -> resolved-in-unstable
nosy: droundy, tommy, beschmi, kowey, markstos
messages: + msg3607
2008-02-21 03:39:50markstossettopic: + Target-2.0
nosy: droundy, tommy, beschmi, kowey, markstos
2008-02-21 03:42:42markstosunlinkissue659 superseder
2008-02-22 15:03:29droundysetstatus: resolved-in-unstable -> resolved
nosy: droundy, tommy, beschmi, kowey, markstos
2008-02-22 21:20:48droundysetstatus: resolved -> resolved-in-unstable
nosy: droundy, tommy, beschmi, kowey, markstos
messages: + msg3645
title: record => "hash failure" with pre-gzipped files -> if hash check fails, try reading file without ungzipping it.
2008-09-04 21:32:29adminsetstatus: resolved-in-unstable -> resolved
nosy: + dagit
2009-08-06 17:54:06adminsetnosy: + jast, Serware, dmitry.kurochkin, darcs-devel, zooko, mornfall, simon, thorkilnaur, - droundy
2009-08-06 20:57:36adminsetnosy: - beschmi
2009-08-10 22:14:41adminsetnosy: - darcs-devel, zooko, jast, Serware, mornfall
2009-08-11 00:06:50adminsetnosy: - dagit
2009-08-25 18:05:27adminsetnosy: + darcs-devel, - simon
2009-08-27 14:08:53adminsetnosy: tommy, kowey, markstos, darcs-devel, thorkilnaur, dmitry.kurochkin
2010-06-15 21:20:05adminsetmilestone: 2.0.x
2010-06-15 21:20:06adminsettopic: - Target-2.0
nosy: + Serware