Incidentally I expect you already know this, but 'darcs check' doesn't work (on Windows at least) on the GHC repo, even with darcs 2.0.  I recall something to do with a case error but I thought darcs 2 fixed that.  Maybe not.  Or maybe not with the existing GHC repo anyway.

Of course, for anyone who doesn't already know that 'darcs check' fails, it's terribly unsettling.  Is my repo so corrupt that it crashes darcs?  A more polite error message such as "You can't run check on this repository because..." would be much better.

Dump below.

PS: in my previous message about an infinite loop I should have said that I'm on Windows.  Apologies.

Simon

darcs2 check
Checking patch 5/19039 : Mon Jan  8 20:28:12 GMT Standard Time 1996  partain
Checking patch 6/19039 : Tue Jan  9 18:03:28 GMT Standard Time 1996  partain
Checking patch 12/19039 : Thu Jan 11 14:26:13 GMT Standard Time 1996  partain
Unapplicable patch:
Thu Jan 11 14:26:13 GMT Standard Time 1996  partain
  * [project @ 1996-01-11 14:06:51 by partain]

darcs failed:  File './ghc/includes/RtsTypes.lh' already exists!
sh-2.04$

Attachments

• unnamed

Thanks, Simon.

If this provides any relief, I don't think corruption is the case here. 
Furthermore, you might have more luck if you did :

 mkdir ghc-hashed
 cd ghc-hashed
 darcs init --hashed
 darcs pull -a /the/original/ghc/repo

If this succeeds, you should be able to use ghc-hashed for your regular
darcs'ing, interacting with the old repository on darcs.haskell.org as before. 
The hashed format is a good deal more robust.

-----

Ok, so what's going on here?  There are three darcs repository format:
 1. original
 2. hashed
 3. darcs-2

Hashed provides a sort of halfway-house between original and darcs-2.  It is
still compatible with the original darcs (i.e. you can push to and pull from
original repositories), and provides more robustness wrt case sensitivity, among
other things, but it does not offer the nicer semantics of the darcs-2 format
(the one which better avoids exponential blow ups).

The darcs-2 format provides both the extra robustness and the nicer semantics
(the disadvantage being that everybody has to upgrade)

-----

I'm not very sure how to improve our handling of errors we catch from doing
darcs check on a case insensitive file system (we would have to distinguish
between actual errors and case issues).  But if this already fixed with the
hashed format, I think the only thing is to encourage people to at least upgrade
to the hashed format.

Actually, you can do this even more simply:
 darcs get --hashed /the/original/ghc/repo

> I'm not very sure how to improve our handling of errors we catch from
> doing
> darcs check on a case insensitive file system (we would have to
> distinguish
> between actual errors and case issues).  But if this already fixed with
> the
> hashed format, I think the only thing is to encourage people to at least
> upgrade
> to the hashed format.

I tried darcs check on a hashed GHC repo and it still fails.

$ darcs2 query repo
          Type: darcs
        Format: hashed
          Root: c:/builds/darcs-bugs/ghc-hashed
      Pristine: HashedPristine
         Cache: thisrepo:c:/builds/darcs-bugs/ghc-hashed
boringfile Pref: .darcs-boring
Default Remote: c:/builds/darcs-bugs/ghc
   Num Patches: 18652
$ darcs2 check
Unapplicable patch:
Thu Jan 11 14:26:13 GMT Standard Time 1996  partain
  * [project @ 1996-01-11 14:06:51 by partain]

darcs failed:  File './ghc/includes/RtsTypes.lh' already exists!

Also, darcs whatsnew on a hashed repository is significantly slower than a darcs1 repository, slow enough that I've actually gone back to using darcs1 repos.

Sorry guys :-(

Cheers,
        Simon

> I tried darcs check on a hashed GHC repo and it still fails.

> darcs failed:  File './ghc/includes/RtsTypes.lh' already exists!

Argh! Well, I guess I can only put David on this bug then.  I thought the hashed
format should do the check fine since we've got hashed pristine.  But maybe
darcs check doesn't work the way I think it does.

:-/

(If only we could get somebody, like an intern, to sit down and just work on
darcs performance for a few months.)

No, I can't do

|  darcs pull -a /the/original/ghc/repo

because a different bug in Darcs makes the pull fail.  The only way to get a valid repo on a windows machine (as far as I know) is to start from a tar-ball.  It's very frustrating.

In any case, I'm just reporting that darcs check fails -- I don't actually need it.  I think.

Simon

| -----Original Message-----
| From: Eric Kow [mailto:bugs@darcs.net]
| Sent: 25 July 2008 09:41
| To: beschmi@cloaked.de; dagit@codersbase.com; eric.kow@gmail.com;
| igloo@earth.li; ptp@lysator.liu.se; Simon Marlow; Simon Peyton-Jones
| Subject: [issue971] darcs check fails (case sensitivity on filenames)
|
|
| Eric Kow <eric.kow@gmail.com> added the comment:
|
| Thanks, Simon.
|
| If this provides any relief, I don't think corruption is the case here.
| Furthermore, you might have more luck if you did :
|
|  mkdir ghc-hashed
|  cd ghc-hashed
|  darcs init --hashed
|  darcs pull -a /the/original/ghc/repo
|
| If this succeeds, you should be able to use ghc-hashed for your regular
| darcs'ing, interacting with the old repository on darcs.haskell.org as
| before.
| The hashed format is a good deal more robust.
|
| -----
|
| Ok, so what's going on here?  There are three darcs repository format:
|  1. original
|  2. hashed
|  3. darcs-2
|
| Hashed provides a sort of halfway-house between original and darcs-2.  It is
| still compatible with the original darcs (i.e. you can push to and pull from
| original repositories), and provides more robustness wrt case sensitivity,
| among
| other things, but it does not offer the nicer semantics of the darcs-2 format
| (the one which better avoids exponential blow ups).
|
| The darcs-2 format provides both the extra robustness and the nicer semantics
| (the disadvantage being that everybody has to upgrade)
|
| -----
|
| I'm not very sure how to improve our handling of errors we catch from doing
| darcs check on a case insensitive file system (we would have to distinguish
| between actual errors and case issues).  But if this already fixed with the
| hashed format, I think the only thing is to encourage people to at least
| upgrade
| to the hashed format.
|
| ----------
| nosy: +kowey
| priority:  -> bug
| status: unread -> resolved
| title: darcs check fails -> darcs check fails (case sensitivity on filenames)
|
| __________________________________
| Darcs bug tracker <bugs@darcs.net>
| <http://bugs.darcs.net/issue971>
| __________________________________

22:09:44 | morn@eri:/mnt/foobar -> cd ghc-hashedrepo 
22:09:47 | morn@eri:/mnt/foobar/ghc-hashedrepo -> darcs check
The repository is consistent!                                                 
22:29:50 | morn@eri:/mnt/foobar/ghc-hashedrepo -> echo bla > CaseSensitive
22:37:51 | morn@eri:/mnt/foobar/ghc-hashedrepo -> cat CASESENSITIVE
bla

Seems to work here, this being 2.0.2, although it is running on Linux. I'll try
to see if it could have been changed in between 2.0.0 and 2.0.2.

Fails on my Mac with 2.0.2+

Huh.

That's... curious. Could this be interaction with some system-specific GHC or
library behaviour? I'll try to review that code, but I apparently don't have
access to anything where this happens.

Here's a demonstration of this issue:

linux> mkdir casesensitiverepo
linux> cd casesensitiverepo/
linux> darcs init
linux> touch a A
linux> darcs add a A
The following file is already in the repository;
note that to ensure portability we don't allow
files that differ only in case. Use --case-ok to override this:  ## good
a
linux> darcs add a --case-ok
linux> darcs whatsnew -s
A ./A
A ./a
linux> darcs record -am 'two case-overlapping files'
What is your email address? simon
Finished recording patch 'two case-overlapping files'

mac> darcs get linux:casesensitiverepo        ## mac has a case-insensitive
filesystem
Unapplicable patch:
Fri Aug  8 18:27:11 PDT 2008  Simon Michael <simon@joyful.com>
  * two case-overlapping files

darcs failed:  File './a' already exists!     ## fair enough. a higher-level
message would be nice

mac> scp -rq linux:casesensitiverepo casesensitiverepo   ## so we copy the
wretched thing
mac> cd casesensitiverepo
mac> ls
_darcs	a                                                       ## => a corrupt repo.
mac> darcs whatsnew -s
No changes!
mac> darcs check
Unapplicable patch:
Fri Aug  8 18:27:11 PDT 2008  Simon Michael <simon@joyful.com>
  * two case-overlapping files

darcs failed:  File './a' already exists!
mac> darcs repair                                       ## and irreparable on
this filesystem

darcs failed:  File './a' already exists!

mac> scp -rq casesensitiverepo linux:casesensitiverepo-damaged  ## move it back
to linux

linux> cd casesensitiverepo-damaged
linux> ls
A  _darcs
linux> darcs whatsnew -s
No changes!
linux> darcs check                                             ## here, we can
repair
Looks like we have a difference...
Difference:  rmfile ./a

Inconsistent repository!
linux> darcs repair                                                
Fixing pristine tree...
linux> darcs check
The repository is consistent!
linux> ls                                                      ## oops - not
quite done
A  _darcs
linux> darcs whatsnew -s
R ./a
linux> darcs revert -a 
linux> Finished reverting.
linux> ls
a  A  _darcs

mac> darcs get joyful.com:/tmp/casesensitiverepo-hashed   ## what about hashed
repos ?
Copying patches, to get lazy repository hit ctrl-C...
Finished getting.
mac> cd casesensitiverepo-hashed/
mac> ls
A	_darcs
mac> darcs check
Unapplicable patch:
Fri Aug  8 18:35:08 PDT 2008  simon
  * case-overlapping files

darcs failed:  File './a' already exists!
mac> darcs repair

darcs failed:  File './a' already exists!    
                           ## the same, but get didn't warn you. not good. same
for darcs2 repos.

PS:

mac> darcs --version
2.0.0 (2.0.0 (+ 271 patches))
linux> darcs --version
2.0.2 (+ 29 patches)

In summary, 

- a case-sensitive repo (containing case-overlapping file names) works
  only on a case-sensitive filesystem.

- darcs warns you about this and requires a special flag when you add a
  case-overlapping file.

- darcs getting a case-sensitive darcs1 repo to a case-insensitive
  filesystem fails with a not-so-clear message. When the repo is hashed or
  darcs2 format, it succeeds and doesn't indicate any problem.

- once a case-sensitive repo is on a case-insensitive fs (via darcs get or
  manual copying) it is unusable, inconsistent and irreparable. It can be
  fully repaired if moved back to a case-sensitive fs.

- the ghc repo is case-sensitive, as noted at
  http://hackage.haskell.org/trac/ghc/wiki/DarcsRepositories . It says
  there ghc can't make their repo case-insensitive without messing up
  history.

What to do ?

Correction: "unusable" was wrong. Such repos seem to mostly work, if you avoid
the case-overlapping files and don't try darcs check or darcs repair. There are
probably other breakages.

After some #darcs chat, I think the current situation could be described like so:

"patch theory means that you can't fix case overlaps in your repo without
undoing all dependents of the patch that introduced the overlap. case-sensitive
repos don't work cross-platform. So introducing a case overlap is an expensive
mistake."

or

"A repository becomes case-sensitive if you add two files with names differing
only in capitalisation. Case-sensitive repos don't work on windows or mac. You
might discover this problem after some time has passed, and unfotunately fixing
a case-sensitive darcs repo is expensive: patch theory means you must remove the
offending patch plus all later dependent patches. So don't add case-overlapping
files to your repo (current darcs will give a warning and prevent it.)"

Not really. You don't really care about case sensitive or not. What you do is fix
how newpristine/ is generated -- iow, using HashedIO, instead of the normal
application. That avoids any case sensitivity issues on repair. Voila. I don't
have a patch for that yet, but I'll conjure one at some point. It would then only
means you can't have consistent *working copies* where the file overlap is
*active*. You can always use rename to fix the overlap, too.

Note there are patches for this on darcs-user needing review, subject "darcs
patch: Parametrize "pristine.subdir" in a bunch".

On Sat, Aug 09, 2008 at 02:54:10AM -0000, Simon Michael wrote:
> Correction: "unusable" was wrong. Such repos seem to mostly work, if you avoid
> the case-overlapping files and don't try darcs check or darcs repair. There are
> probably other breakages.

Such repos should almost completely work.  It's arguably a bug in repair
and check that makes them fail in this case.  Everything else should work
fine (except certain commands that require construction of impossible
trees, e.g. diff on ancient versions that include case-sensitive namings).
-- 
David Roundy
http://www.darcs.net

My understanding is that Petr's patches have made it into the repository and
consequently that this bug is resolved-in-unstable.

Is that correct?

The following three patches by Petr have made it into darcs.net

Parametrize "pristine.hashed" in a bunch of functions.
Add writeSlurpy to roll out a copy of slurpy into a filesystem.
Generalize HashRepo.clean_pristine to HashIO.clean_hashdir.

The issue does not seem to be resolved.  Are there more patches to (re)send?
Thanks!

Status: Petr says "Basically,  there are two fronts where improvement would be good:
1) Make the slurpy update behaviour more smart (especially wrt. memory usage) --
has been discussed in the various patch review threads, IIRC.
2) Migrate the repair code from Repair.lhs to Repository/Internal.lhs and use it
from Check.lhs as well.
The 2 is what would ultimately let 971 to be resolved.
I'll eventually fix that myself...

I haven't seen any updates on this bug in over a month.  Is anyone still making
progress? Or should we set it back to needs-volunteer?

Thanks!

It would be great if someone could check this with darcs 2.2, where check uses
the same codepath as repair now, ie. hashed pristine. This should make it work
fine on case-problematic repos even with case-insensitive filesystems. Or so I
hope, anyway.

Hi. I have verified the fix in branch-2.2 and it will therefore become part of 
the 2.2 release. The trick to reproduce on linux is to have /tmp case-sensitive.