New users trying to fetch the repository always results in an error:
$ darcs -v
2.4.4 (release)
$ darcs get --lazy http://code.haskell.org/gentoo/gentoo-haskell --no-cache
This is the gentoo-haskell darcs overlay.
**********************
darcs: bug at src/URL.hs:246 compiled May 30 2010 13:33:24                   
Another possible bug in URL.waitNextUrl:  curl_multi_perform() - no running handles
See http://wiki.darcs.net/BugTracker/Reporting for help on bug reporting.

I kill it with Ctrl+C. Sometimes you get this error after 10 pathes, sometimes after 1500. It seems random.

So, let's run 'darcs check' on the remote site to make sure the repo is intact:
$ darcs -v
2.4.4 (release)
$ darcs show repo                   
          Type: darcs
        Format: hashed
          Root: /srv/code/gentoo/gentoo-haskell
      Pristine: HashedPristine
         Cache: thisrepo:/srv/code/gentoo/gentoo-haskell, cache:/home/kolmodin/.darcs/cache
   Num Patches: 3551

$ darcs check
The repository is consistent!
Missing items in index!
[snip, long list of very changes]
Hash mismatch(es)!
[snip, long list of recent changes]
Bad index.

The patches missing in the index, and with hash mismatches are all very recent. It might be since we upgraded darcs on the remote site to 2.4.1.

So, I'd like to 'darcs repair'. I make a copy and do the changes there:

$ cd ~/public_html
$ cp -r /srv/code/gentoo/gentoo-haskell g-h
$ cd g-h/
$ darcs repair
The repository is already consistent, no changes made.                          
Missing items in index!
[snip]
Hash mismatch(es)!
[snip]
Bad index discarded.

Trying to fetch again from repaired copy:

$ darcs get --lazy http://code.haskell.org/~kolmodin/g-h --no-cache
This is the gentoo-haskell darcs overlay.
**********************
Another possible bug in URL.waitNextUrl:  curl_multi_perform() - no running handles
See http://wiki.darcs.net/BugTracker/Reporting for help on bug reporting.
Copying pristine 890 done, 147 queued. mmap-0.5.4.ebuild

And it hangs again.

So, I make a copy on the remote site using darcs itself:

$ darcs get g-h/ g-h2 --no-cache
This is the gentoo-haskell darcs overlay.
**********************
Copying patches, to get lazy repository hit ctrl-C...              
Finished getting.

That worked out well, let's check it:

$ cd g-h2
$ darcs check
The repository is consistent!

Now, again on my local machine, lets try fetching a copy from the g-h2 repo:
$ darcs get --lazy http://code.haskell.org/~kolmodin/g-h2 --no-cache
Finished getting.

Do you know what's up?
'darcs repair' didn't fix the issue, but 'darcs get' did?

Hi Petr,

Could you have a look at this bug, please?  Is there any information
that the Gentoo guys could provide to help us reproduce the cause of the
error?

Note that I'm primarily concerned with the 'Missing items in index' (I
had split this bug off from issue1876 which is a duplicate of the curl
multi_perform bug in issue1770).

Note also that they mention a darcs 2.4.1 in there.  Is that potentially
useful as a hint?

The only thing I can offer is that we had a similar issue with the darcs
darcs repo in issue1710 but we were never able to reproduce it.

Is it safe for us to optimistically mark this resolved?

Also, I can confirm the missing index (if I scp -r
code.haskell.org:/srv/code/gentoo/gentoo-haskell) which doesn't tell us
much.  But I seem to remember this was fixed in one of the stable darcsen?

And yes, to be clear, the issue isn't that check complains about missing
items in index, but that the items are missing in the first place.

Petr: could you confirm there's a plausible explanation for their repo
being broken by an older darcs?  Many thanks!

Right, and now the remote site is also using darcs 2.4.4.
Would it have worked better if we would use the darcs2 format?

You can scp -r code.haskell.org:/srv/code/gentoo/gentoo-haskell to get a copy of the repo.
Then 'darcs get gentoo-haskell filesystem-copy' to make a local copy.

Then, I'd assume, the gentoo-haskell/_darcs and filesystem-copy/_darcs should look fairly the same?
The local copy is called 'g-h2' in this run:

Only in srv/code/gentoo/gentoo-haskell/_darcs: email
Only in srv/code/gentoo/gentoo-haskell/_darcs: fixed_files
Files srv/code/gentoo/gentoo-haskell/_darcs/index and g-h2/_darcs/index differ
Only in srv/code/gentoo/gentoo-haskell/_darcs/inventories: 0000049760-4f815e3247649e344e6d283d82ebc4b0ee8a4a7ba69f93274c05f5f6728021e4
[snip, lots of more files]
Only in srv/code/gentoo/gentoo-haskell/_darcs/inventories: 0000096215-4d3315ad3ba2d3cc5095db5397be01b867b3d8c9cb661f45e45094dd80cbd4b8
Only in srv/code/gentoo/gentoo-haskell/_darcs/patches: pending
Only in srv/code/gentoo/gentoo-haskell/_darcs/patches: unrevert
Files srv/code/gentoo/gentoo-haskell/_darcs/prefs/binaries and g-h2/_darcs/prefs/binaries differ
Only in srv/code/gentoo/gentoo-haskell/_darcs/prefs: defaults
Only in srv/code/gentoo/gentoo-haskell/_darcs/prefs: email
Files srv/code/gentoo/gentoo-haskell/_darcs/prefs/motd and g-h2/_darcs/prefs/motd differ
Files srv/code/gentoo/gentoo-haskell/_darcs/prefs/repos and g-h2/_darcs/prefs/repos differ
Only in g-h2/_darcs/prefs: sources
Only in g-h2/_darcs/: tentative_hashed_inventory
Files srv/code/gentoo/gentoo-haskell/_darcs/tentative_pristine and g-h2/_darcs/tentative_pristine differ

So, by large the directories are identical, but there are several files missing from the 'filesystem-copy'.

I'm afraid I don't know exactly what the different directories represent, maybe it's documented in the manual?

On Wed, Jun 23, 2010 at 06:28:40 +0000, Lennart Kolmodin wrote:
> Right, and now the remote site is also using darcs 2.4.4.
> Would it have worked better if we would use the darcs2 format?

I doubt that would have made a difference since the problem is
presumably something to do with the hashed pristine
> 
> Then, I'd assume, the gentoo-haskell/_darcs and filesystem-copy/_darcs should look fairly the same?
> The local copy is called 'g-h2' in this run:
> 
> Files srv/code/gentoo/gentoo-haskell/_darcs/index and g-h2/_darcs/index differ
> Only in srv/code/gentoo/gentoo-haskell/_darcs/inventories: 0000049760-4f815e3247649e344e6d283d82ebc4b0ee8a4a7ba69f93274c05f5f6728021e4
> [snip, lots of more files]
> Only in srv/code/gentoo/gentoo-haskell/_darcs/inventories: 0000096215-4d3315ad3ba2d3cc5095db5397be01b867b3d8c9cb661f45e45094dd80cbd4b8
> Files srv/code/gentoo/gentoo-haskell/_darcs/prefs/binaries and g-h2/_darcs/prefs/binaries differ
> Files srv/code/gentoo/gentoo-haskell/_darcs/prefs/motd and g-h2/_darcs/prefs/motd differ
> Files srv/code/gentoo/gentoo-haskell/_darcs/prefs/repos and g-h2/_darcs/prefs/repos differ

Snipped files that can be easily accounted for

> I'm afraid I don't know exactly what the different directories
> represent, maybe it's documented in the manual?

The manual does not have very much to say about darcs repository
internals, but we're making an effort to document them.
  http://wiki.darcs.net/DarcsInternals/HashedPristine
may help a little bit.

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9

This might be interesting too:

kolmodin@nun:/srv/code/gentoo/gentoo-haskell$ darcs wh . -l
R ./dev-vcs/dpm/Manifest
[snip, lots of removes]
R ./www-apps/gitit/gitit-0.7.3.6.ebuild
a ./www-apps/gitit/gitit-0.7.3.6.ebuild
[snip, lots of adds]
a ./app-admin/haskell-updater/haskell-updater-1.1.0.0.ebuild
a ./.current-hash

Seems that the same files has both 'R' and 'a'.
Is that due to the broken index?
Please let me know if you need me to provide you with anything.
As kowey already noted, you can already get the repo at code.haskell.org:/srv/code/gentoo/gentoo-haskell

As a lot of users ask how to get a copy of our repo, I'm keen on finding 
a workaround.

Is it safe to use the 'darcs get' (via the filesystem, not http or ssh) 
copy? It seems to work, maybe it ignored the broken index and the hash 
mismatches?

We can still keep a copy of the broken repo if you need it for 
debugging.

On Thu, Jun 24, 2010 at 13:39:15 +0000, Lennart Kolmodin wrote:
> As a lot of users ask how to get a copy of our repo, I'm keen on finding 
> a workaround.

Yikes, bug-tracking slip-up, sorry!
(one of my rules is "make sure the user has a workaround first")
http://wiki.darcs.net/BugTracker

If you have a repository that people can fetch safely (ie, your g-h-2)
and with the same patches, then by all means, swap them.  Just be
careful to copy your prefs (eg. the email file, motd, etc).

Meanwhile, it could be nice if you could confirm that the broken
repository fetches fine with Darcs HEAD.  It does for me.

> Is it safe to use the 'darcs get' (via the filesystem, not http or ssh) 
> copy? It seems to work, maybe it ignored the broken index and the hash 
> mismatches?

The index file is a purely local affair, so I don't think it should
affect anything.  Petr should correct me if I'm wrong.

> We can still keep a copy of the broken repo if you need it for 
> debugging.

Please do.  We should notify you when you can delete it when we close
this ticket.

Thanks,

PS: your repo sounds like a good candidate for the optimize --http
    optimisation which may appear in Darcs 2.5 (no guarantees, eh?)

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9

Sorry, but I am quite confused. What is the bug again?
- doing a darcs get over http works from code.haskell.org (I tried that
  couple days ago) and the resulting repo is consistent and has no index
  problems (using darcs HEAD here)
- grabbing a copy of the repository with scp -r yield a repo which is
  ostensibly consistent, but has broken index

Anyway, I don't think this has anything to do with "darcs get" or 
getting a copy of the repository: index is never accessed remotely.

The index problem is only relevant for direct access to the repository, 
using darcs on code.haskell.org. The curl problem (waitNextUrl) has been 
fixed on HEAD and has nothing to do with index.

If you ever use an old version of darcs to locally access the repository 
on code.haskell.org, the index corruption is quite expectable: the pre-
2.3 darcs won't know about or touch the index in any way. If the 
repository is updated and the index not, this is what you will get.

FYI, the default darcs on code.haskell.org is 2.0.2 -- maybe you are 
using this one to apply patches in your repository? (Or at least some of  
your contributors are?) If you use old darcs on an indexed repo, you 
need to rm _darcs/index before using new darcs again.

Hi Lennart: could you confirm that it is likely/plausible that the
repository has been written to on code.haskell.org (eg. via darcs apply)
with darcs-2.0.2 (or some other pre darcs-2.3.1 version)?

Sorry for being confusing, my lack of knowledge about darcs internals makes my report a bit fuzzy.

The broken repository does fetch OK with darcs-HEAD, thanks.
I still don't understand why 'darcs get' gives me a more usable repo than 'darcs repair' does, though.

About the missing files in the index, I've found an embarrassing reason...
I noticed many of the missing files in the index were created by myself, but I knew that I had set my PATH to 
include darcs-2.4.4 on code.haskell.org.

However, I got surprised when I executed
$ ssh code.haskell.org "darcs -v"
2.0.2 (release)

I executed the PATH prepending in .bashrc, so it only gets executed during interactive logins, not when darcs 
logs in.
I have not yet figured out how to set the path so that 'ssh code.haskell.org "darcs -v"' shows 2.4.4.
I'd prefer to set the PATH, than to add --remote-darcs to all c.h.o repos I'm using.

Hi,

Lennart Kolmodin <bugs@darcs.net> writes:
> $ ssh code.haskell.org "darcs -v"
> 2.0.2 (release)

> I executed the PATH prepending in .bashrc, so it only gets executed
> during interactive logins, not when darcs logs in.  I have not yet
> figured out how to set the path so that 'ssh code.haskell.org "darcs
> -v"' shows 2.4.4.  I'd prefer to set the PATH, than to add
> --remote-darcs to all c.h.o repos I'm using.

The only way to do this, I expect, is to set up .ssh/environment -- this
is something that sshd needs to allow though, so you need to find out
whether c.h.o does and maybe consult the admins if it doesn't. When you
run noninteractive ssh, it is not running your shell at all, AFAIR.

Yours,
   Petr.

Using darcs-2.0.2 explains why the index appeared broken, so I'm closing the bug.
mornfall and kowey, thanks for the clarification! And sorry to waste your time :(

c.h.o does not allow users to set their environment, and indeed it does not run my shell, not even in 
non-interactive mode.
As a workaround we'll just add --remote-darcs.