darcs

Issue 971 darcs check fails (case sensitivity on filenames)

Title darcs check fails (case sensitivity on filenames)
Priority bug Status resolved
Milestone 2.2.x Resolved in
Superseder Nosy List darcs-devel, dmitry.kurochkin, ertai, kowey, mornfall, simonmar, simonpj, thorkilnaur, tommy
Assigned To mornfall
Topics

Created on 2008-07-24.21:37:21 by simonpj, last changed 2010-06-15.21:50:58 by admin.

Files
File name Uploaded Type Edit Remove
unnamed simonpj, 2008-07-24.21:37:16 text/html
Messages
msg5253 (view) Author: simonpj Date: 2008-07-24.21:37:16
Incidentally I expect you already know this, but 'darcs check' doesn't work (on Windows at least) on the GHC repo, even with darcs 2.0.  I recall something to do with a case error but I thought darcs 2 fixed that.  Maybe not.  Or maybe not with the existing GHC repo anyway.

Of course, for anyone who doesn't already know that 'darcs check' fails, it's terribly unsettling.  Is my repo so corrupt that it crashes darcs?  A more polite error message such as "You can't run check on this repository because..." would be much better.

Dump below.

PS: in my previous message about an infinite loop I should have said that I'm on Windows.  Apologies.

Simon

darcs2 check
Checking patch 5/19039 : Mon Jan  8 20:28:12 GMT Standard Time 1996  partain
Checking patch 6/19039 : Tue Jan  9 18:03:28 GMT Standard Time 1996  partain
Checking patch 12/19039 : Thu Jan 11 14:26:13 GMT Standard Time 1996  partain
Unapplicable patch:
Thu Jan 11 14:26:13 GMT Standard Time 1996  partain
  * [project @ 1996-01-11 14:06:51 by partain]

darcs failed:  File './ghc/includes/RtsTypes.lh' already exists!
sh-2.04$
Attachments
msg5256 (view) Author: kowey Date: 2008-07-25.08:41:21
Thanks, Simon.

If this provides any relief, I don't think corruption is the case here. 
Furthermore, you might have more luck if you did :

 mkdir ghc-hashed
 cd ghc-hashed
 darcs init --hashed
 darcs pull -a /the/original/ghc/repo

If this succeeds, you should be able to use ghc-hashed for your regular
darcs'ing, interacting with the old repository on darcs.haskell.org as before. 
The hashed format is a good deal more robust.

-----

Ok, so what's going on here?  There are three darcs repository format:
 1. original
 2. hashed
 3. darcs-2

Hashed provides a sort of halfway-house between original and darcs-2.  It is
still compatible with the original darcs (i.e. you can push to and pull from
original repositories), and provides more robustness wrt case sensitivity, among
other things, but it does not offer the nicer semantics of the darcs-2 format
(the one which better avoids exponential blow ups).

The darcs-2 format provides both the extra robustness and the nicer semantics
(the disadvantage being that everybody has to upgrade)

-----

I'm not very sure how to improve our handling of errors we catch from doing
darcs check on a case insensitive file system (we would have to distinguish
between actual errors and case issues).  But if this already fixed with the
hashed format, I think the only thing is to encourage people to at least upgrade
to the hashed format.
msg5257 (view) Author: kowey Date: 2008-07-25.08:45:57
Actually, you can do this even more simply:
 darcs get --hashed /the/original/ghc/repo
msg5260 (view) Author: simonmar Date: 2008-07-25.09:20:03
> I'm not very sure how to improve our handling of errors we catch from
> doing
> darcs check on a case insensitive file system (we would have to
> distinguish
> between actual errors and case issues).  But if this already fixed with
> the
> hashed format, I think the only thing is to encourage people to at least
> upgrade
> to the hashed format.

I tried darcs check on a hashed GHC repo and it still fails.

$ darcs2 query repo
          Type: darcs
        Format: hashed
          Root: c:/builds/darcs-bugs/ghc-hashed
      Pristine: HashedPristine
         Cache: thisrepo:c:/builds/darcs-bugs/ghc-hashed
boringfile Pref: .darcs-boring
Default Remote: c:/builds/darcs-bugs/ghc
   Num Patches: 18652
$ darcs2 check
Unapplicable patch:
Thu Jan 11 14:26:13 GMT Standard Time 1996  partain
  * [project @ 1996-01-11 14:06:51 by partain]

darcs failed:  File './ghc/includes/RtsTypes.lh' already exists!

Also, darcs whatsnew on a hashed repository is significantly slower than a darcs1 repository, slow enough that I've actually gone back to using darcs1 repos.

Sorry guys :-(

Cheers,
        Simon
msg5261 (view) Author: kowey Date: 2008-07-25.09:28:11
> I tried darcs check on a hashed GHC repo and it still fails.

> darcs failed:  File './ghc/includes/RtsTypes.lh' already exists!

Argh! Well, I guess I can only put David on this bug then.  I thought the hashed
format should do the check fine since we've got hashed pristine.  But maybe
darcs check doesn't work the way I think it does.

:-/

(If only we could get somebody, like an intern, to sit down and just work on
darcs performance for a few months.)
msg5271 (view) Author: simonpj Date: 2008-07-25.23:21:27
No, I can't do

|  darcs pull -a /the/original/ghc/repo

because a different bug in Darcs makes the pull fail.  The only way to get a valid repo on a windows machine (as far as I know) is to start from a tar-ball.  It's very frustrating.

In any case, I'm just reporting that darcs check fails -- I don't actually need it.  I think.

Simon

| -----Original Message-----
| From: Eric Kow [mailto:bugs@darcs.net]
| Sent: 25 July 2008 09:41
| To: beschmi@cloaked.de; dagit@codersbase.com; eric.kow@gmail.com;
| igloo@earth.li; ptp@lysator.liu.se; Simon Marlow; Simon Peyton-Jones
| Subject: [issue971] darcs check fails (case sensitivity on filenames)
|
|
| Eric Kow <eric.kow@gmail.com> added the comment:
|
| Thanks, Simon.
|
| If this provides any relief, I don't think corruption is the case here.
| Furthermore, you might have more luck if you did :
|
|  mkdir ghc-hashed
|  cd ghc-hashed
|  darcs init --hashed
|  darcs pull -a /the/original/ghc/repo
|
| If this succeeds, you should be able to use ghc-hashed for your regular
| darcs'ing, interacting with the old repository on darcs.haskell.org as
| before.
| The hashed format is a good deal more robust.
|
| -----
|
| Ok, so what's going on here?  There are three darcs repository format:
|  1. original
|  2. hashed
|  3. darcs-2
|
| Hashed provides a sort of halfway-house between original and darcs-2.  It is
| still compatible with the original darcs (i.e. you can push to and pull from
| original repositories), and provides more robustness wrt case sensitivity,
| among
| other things, but it does not offer the nicer semantics of the darcs-2 format
| (the one which better avoids exponential blow ups).
|
| The darcs-2 format provides both the extra robustness and the nicer semantics
| (the disadvantage being that everybody has to upgrade)
|
| -----
|
| I'm not very sure how to improve our handling of errors we catch from doing
| darcs check on a case insensitive file system (we would have to distinguish
| between actual errors and case issues).  But if this already fixed with the
| hashed format, I think the only thing is to encourage people to at least
| upgrade
| to the hashed format.
|
| ----------
| nosy: +kowey
| priority:  -> bug
| status: unread -> resolved
| title: darcs check fails -> darcs check fails (case sensitivity on filenames)
|
| __________________________________
| Darcs bug tracker <bugs@darcs.net>
| <http://bugs.darcs.net/issue971>
| __________________________________
msg5332 (view) Author: mornfall Date: 2008-08-07.20:38:50
22:09:44 | morn@eri:/mnt/foobar -> cd ghc-hashedrepo 
22:09:47 | morn@eri:/mnt/foobar/ghc-hashedrepo -> darcs check
The repository is consistent!                                                 
22:29:50 | morn@eri:/mnt/foobar/ghc-hashedrepo -> echo bla > CaseSensitive
22:37:51 | morn@eri:/mnt/foobar/ghc-hashedrepo -> cat CASESENSITIVE
bla

Seems to work here, this being 2.0.2, although it is running on Linux. I'll try
to see if it could have been changed in between 2.0.0 and 2.0.2.
msg5333 (view) Author: kowey Date: 2008-08-07.21:38:44
Fails on my Mac with 2.0.2+

Huh.
msg5335 (view) Author: mornfall Date: 2008-08-07.22:24:08
That's... curious. Could this be interaction with some system-specific GHC or
library behaviour? I'll try to review that code, but I apparently don't have
access to anything where this happens.
msg5341 (view) Author: simon Date: 2008-08-09.02:13:00
Here's a demonstration of this issue:

linux> mkdir casesensitiverepo
linux> cd casesensitiverepo/
linux> darcs init
linux> touch a A
linux> darcs add a A
The following file is already in the repository;
note that to ensure portability we don't allow
files that differ only in case. Use --case-ok to override this:  ## good
a
linux> darcs add a --case-ok
linux> darcs whatsnew -s
A ./A
A ./a
linux> darcs record -am 'two case-overlapping files'
What is your email address? simon
Finished recording patch 'two case-overlapping files'

mac> darcs get linux:casesensitiverepo        ## mac has a case-insensitive
filesystem
Unapplicable patch:
Fri Aug  8 18:27:11 PDT 2008  Simon Michael <simon@joyful.com>
  * two case-overlapping files

darcs failed:  File './a' already exists!     ## fair enough. a higher-level
message would be nice

mac> scp -rq linux:casesensitiverepo casesensitiverepo   ## so we copy the
wretched thing
mac> cd casesensitiverepo
mac> ls
_darcs	a                                                       ## => a corrupt repo.
mac> darcs whatsnew -s
No changes!
mac> darcs check
Unapplicable patch:
Fri Aug  8 18:27:11 PDT 2008  Simon Michael <simon@joyful.com>
  * two case-overlapping files

darcs failed:  File './a' already exists!
mac> darcs repair                                       ## and irreparable on
this filesystem

darcs failed:  File './a' already exists!

mac> scp -rq casesensitiverepo linux:casesensitiverepo-damaged  ## move it back
to linux

linux> cd casesensitiverepo-damaged
linux> ls
A  _darcs
linux> darcs whatsnew -s
No changes!
linux> darcs check                                             ## here, we can
repair
Looks like we have a difference...
Difference:  rmfile ./a

Inconsistent repository!
linux> darcs repair                                                
Fixing pristine tree...
linux> darcs check
The repository is consistent!
linux> ls                                                      ## oops - not
quite done
A  _darcs
linux> darcs whatsnew -s
R ./a
linux> darcs revert -a 
linux> Finished reverting.
linux> ls
a  A  _darcs

mac> darcs get joyful.com:/tmp/casesensitiverepo-hashed   ## what about hashed
repos ?
Copying patches, to get lazy repository hit ctrl-C...
Finished getting.
mac> cd casesensitiverepo-hashed/
mac> ls
A	_darcs
mac> darcs check
Unapplicable patch:
Fri Aug  8 18:35:08 PDT 2008  simon
  * case-overlapping files

darcs failed:  File './a' already exists!
mac> darcs repair

darcs failed:  File './a' already exists!    
                           ## the same, but get didn't warn you. not good. same
for darcs2 repos.
msg5342 (view) Author: simon Date: 2008-08-09.02:14:16
PS:

mac> darcs --version
2.0.0 (2.0.0 (+ 271 patches))
linux> darcs --version
2.0.2 (+ 29 patches)
msg5343 (view) Author: simon Date: 2008-08-09.02:38:42
In summary, 

- a case-sensitive repo (containing case-overlapping file names) works
  only on a case-sensitive filesystem.

- darcs warns you about this and requires a special flag when you add a
  case-overlapping file.

- darcs getting a case-sensitive darcs1 repo to a case-insensitive
  filesystem fails with a not-so-clear message. When the repo is hashed or
  darcs2 format, it succeeds and doesn't indicate any problem.

- once a case-sensitive repo is on a case-insensitive fs (via darcs get or
  manual copying) it is unusable, inconsistent and irreparable. It can be
  fully repaired if moved back to a case-sensitive fs.

- the ghc repo is case-sensitive, as noted at
  http://hackage.haskell.org/trac/ghc/wiki/DarcsRepositories . It says
  there ghc can't make their repo case-insensitive without messing up
  history.

What to do ?
msg5344 (view) Author: simon Date: 2008-08-09.02:54:07
Correction: "unusable" was wrong. Such repos seem to mostly work, if you avoid
the case-overlapping files and don't try darcs check or darcs repair. There are
probably other breakages.
msg5345 (view) Author: simon Date: 2008-08-09.03:45:03
After some #darcs chat, I think the current situation could be described like so:

"patch theory means that you can't fix case overlaps in your repo without
undoing all dependents of the patch that introduced the overlap. case-sensitive
repos don't work cross-platform. So introducing a case overlap is an expensive
mistake."

or

"A repository becomes case-sensitive if you add two files with names differing
only in capitalisation. Case-sensitive repos don't work on windows or mac. You
might discover this problem after some time has passed, and unfotunately fixing
a case-sensitive darcs repo is expensive: patch theory means you must remove the
offending patch plus all later dependent patches. So don't add case-overlapping
files to your repo (current darcs will give a warning and prevent it.)"
msg5347 (view) Author: mornfall Date: 2008-08-09.10:50:10
Not really. You don't really care about case sensitive or not. What you do is fix
how newpristine/ is generated -- iow, using HashedIO, instead of the normal
application. That avoids any case sensitivity issues on repair. Voila. I don't
have a patch for that yet, but I'll conjure one at some point. It would then only
means you can't have consistent *working copies* where the file overlap is
*active*. You can always use rename to fix the overlap, too.
msg5356 (view) Author: simon Date: 2008-08-10.15:31:53
Note there are patches for this on darcs-user needing review, subject "darcs
patch: Parametrize "pristine.subdir" in a bunch".
msg5359 (view) Author: droundy Date: 2008-08-10.18:52:48
On Sat, Aug 09, 2008 at 02:54:10AM -0000, Simon Michael wrote:
> Correction: "unusable" was wrong. Such repos seem to mostly work, if you avoid
> the case-overlapping files and don't try darcs check or darcs repair. There are
> probably other breakages.

Such repos should almost completely work.  It's arguably a bug in repair
and check that makes them fail in this case.  Everything else should work
fine (except certain commands that require construction of impossible
trees, e.g. diff on ancient versions that include case-sensitive namings).
-- 
David Roundy
http://www.darcs.net
msg5647 (view) Author: kowey Date: 2008-08-22.13:12:45
My understanding is that Petr's patches have made it into the repository and
consequently that this bug is resolved-in-unstable.

Is that correct?
msg5702 (view) Author: kowey Date: 2008-08-26.08:24:16
The following three patches by Petr have made it into darcs.net

Parametrize "pristine.hashed" in a bunch of functions.
Add writeSlurpy to roll out a copy of slurpy into a filesystem.
Generalize HashRepo.clean_pristine to HashIO.clean_hashdir.

The issue does not seem to be resolved.  Are there more patches to (re)send?
Thanks!
msg6072 (view) Author: simon Date: 2008-09-20.19:04:05
Status: Petr says "Basically,  there are two fronts where improvement would be good:
1) Make the slurpy update behaviour more smart (especially wrt. memory usage) --
has been discussed in the various patch review threads, IIRC.
2) Migrate the repair code from Repair.lhs to Repository/Internal.lhs and use it
from Check.lhs as well.
The 2 is what would ultimately let 971 to be resolved.
I'll eventually fix that myself...
msg6504 (view) Author: dagit Date: 2008-10-28.17:34:28
I haven't seen any updates on this bug in over a month.  Is anyone still making
progress? Or should we set it back to needs-volunteer?

Thanks!
msg6820 (view) Author: mornfall Date: 2008-12-12.07:20:52
It would be great if someone could check this with darcs 2.2, where check uses
the same codepath as repair now, ie. hashed pristine. This should make it work
fine on case-problematic repos even with case-insensitive filesystems. Or so I
hope, anyway.
msg6866 (view) Author: mornfall Date: 2008-12-22.20:06:43
Hi. I have verified the fix in branch-2.2 and it will therefore become part of 
the 2.2 release. The trick to reproduce on linux is to have /tmp case-sensitive.
History
Date User Action Args
2008-07-24 21:37:21simonpjcreate
2008-07-25 08:41:25koweysetpriority: bug
status: unread -> resolved
title: darcs check fails -> darcs check fails (case sensitivity on filenames)
messages: + msg5256
nosy: + kowey
2008-07-25 08:43:42koweylinkissue970 superseder
2008-07-25 08:45:59koweysetstatus: resolved -> unknown
nosy: tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj
messages: + msg5257
2008-07-25 08:47:03koweysetstatus: unknown -> resolved
nosy: tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj
2008-07-25 09:20:08simonmarsetstatus: resolved -> unknown
nosy: tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj
messages: + msg5260
2008-07-25 09:28:15koweysetnosy: + droundy
messages: + msg5261
2008-07-25 23:21:32simonpjsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj
messages: + msg5271
2008-08-07 20:38:53mornfallsetnosy: + mornfall
messages: + msg5332
2008-08-07 21:38:47koweysetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, mornfall
messages: + msg5333
2008-08-07 22:24:11mornfallsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, mornfall
messages: + msg5335
2008-08-09 02:13:03simonsetnosy: + simon
messages: + msg5341
2008-08-09 02:14:18simonsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, mornfall
messages: + msg5342
2008-08-09 02:38:45simonsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, mornfall
messages: + msg5343
2008-08-09 02:54:10simonsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, mornfall
messages: + msg5344
2008-08-09 03:45:06simonsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, mornfall
messages: + msg5345
2008-08-09 10:50:13mornfallsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, mornfall
messages: + msg5347
2008-08-09 19:02:49ertaisetnosy: + ertai
2008-08-10 15:31:56simonsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, ertai, mornfall
messages: + msg5356
2008-08-10 18:52:52droundysetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, ertai, mornfall
messages: + msg5359
2008-08-11 14:47:35mornfallsetstatus: unknown -> has-patch
nosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, ertai, mornfall
assignedto: mornfall
2008-08-22 13:12:49koweysettopic: + Target-2.1
nosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, ertai, mornfall
messages: + msg5647
2008-08-26 08:24:18koweysetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, ertai, mornfall
messages: + msg5702
2008-09-20 11:25:21simonlinkissue1067 superseder
2008-09-20 19:04:07simonsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, ertai, mornfall
messages: + msg6072
2008-10-22 18:06:36thorkilnaurlinkissue1134 superseder
2008-10-28 17:34:30dagitsetnosy: + dmitry.kurochkin, thorkilnaur
messages: + msg6504
2008-12-12 07:20:55mornfallsetnosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, thorkilnaur, ertai, dmitry.kurochkin, mornfall
messages: + msg6820
2008-12-22 20:06:46mornfallsetstatus: has-patch -> resolved
nosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, thorkilnaur, ertai, dmitry.kurochkin, mornfall
messages: + msg6866
2008-12-29 17:41:21mornfallsettopic: + Target-2.2
nosy: droundy, tommy, beschmi, kowey, dagit, igloo, simonmar, simonpj, simon, thorkilnaur, ertai, dmitry.kurochkin, mornfall
2009-08-06 17:59:20adminsetnosy: + markstos, jast, Serware, darcs-devel, zooko, - droundy, igloo, simonmar, simonpj, ertai
2009-08-06 21:09:57adminsetnosy: - beschmi
2009-08-10 22:21:52adminsetnosy: + igloo, simonmar, simonpj, ertai, - markstos, darcs-devel, zooko, jast, Serware
2009-08-11 00:19:27adminsetnosy: - dagit
2009-08-25 17:19:36adminsetnosy: + darcs-devel, - igloo
2009-08-25 17:23:53adminsetnosy: - simon
2009-08-27 14:14:12adminsetnosy: tommy, kowey, darcs-devel, simonmar, simonpj, thorkilnaur, ertai, dmitry.kurochkin, mornfall
2009-10-23 22:38:26adminsetnosy: + marlowsd, - simonmar
2009-10-23 22:40:52adminsetnosy: + nicolas.pouillard, - ertai
2009-10-23 23:36:56adminsetnosy: + simonmar, - marlowsd
2009-10-24 00:05:55adminsetnosy: + ertai, - nicolas.pouillard
2010-06-15 21:47:59adminsetmilestone: 2.1.x
2010-06-15 21:48:00adminsettopic: - Target-2.1
2010-06-15 21:50:58adminsetmilestone: 2.1.x -> 2.2.x
2010-06-15 21:50:58adminsettopic: - Target-2.2