Dear Darcs crew

I have a complete *not partial* repo for GHC (roughly, the HEAD).  Call it "NEW".

I unpacked a complete (not partial) old repo for GHC, ghc-HEAD-2007-08-29-ghc-corelibs-testsuite.tar.  (It's available at darcs.haskell.org).  Call it "OLD".

So NEW has a lot of patches that OLD does not have, but all OLD patches are in NEW.  NEW has no conflicts.

These two repos are peer directories on my machine.  Then, in OLD I say
        darcs pull ../NEW -av

I get lots of output, and then a pause. Then disaster:

darcs failed:  problem gzopening file for write: _darcs/patches/20080126191939-e23c7-efc06647390321527cc18de0e08c017b568b36ae.gz-0

The same thing happens (but much more slowly) if I darcs pull from the true HEAD (at darcs.haskell.org).

 I thought of doing 'darcs check' on NEW, but that doesn't work because of a different (and well known) bug in darcs, to do with line endings:

sh-2.04$  darcs check
Applying patch 12 of 18450...
darcs failed:  Error applying hunk to file ./ghc/includes/rtsTypes.lh
Unapplicable patch:
Thu Jan 11 14:26:13 GMT Standard Time 1996  partain
  * [project @ 1996-01-11 14:06:51 by partain]
sh-2.04$

At this point I seem to be stuck.  All this is darcs 1.0.9 on Windows

Simon

Do you have direct links to these repositories? You mention that
they're at darcs.haskell.org, but don't give a specific URL...

David

darn.   The particular baseline I started from isn't on darcs.haskell.org any more, it seems, so I'll put it up at

        http://research.microsoft.com/~simonpj/tmp/ghc-HEAD-2007-08-29-ghc-corelibs-testsuite.tar.bz2

Untar that and try
        darcs pull http://darcs.haskell.org/ghc -av

I get the "problem gzopening.." error.

Simon

| -----Original Message-----
| From: David Roundy [mailto:bugs@darcs.net]
| Sent: 09 May 2008 12:25
| To: beschmi@cloaked.de; dagit@codersbase.com; daveroundy@gmail.com; igloo@earth.li; ptp@lysator.liu.se;
| Simon Marlow; Simon Peyton-Jones
| Subject: [issue842] Darcs gzip error
|
|
| David Roundy <daveroundy@gmail.com> added the comment:
|
| Do you have direct links to these repositories? You mention that
| they're at darcs.haskell.org, but don't give a specific URL...
|
| David
|
| ----------
| nosy: +daveroundy
| status: unread -> chatting
|
| __________________________________
| Darcs bug tracker <bugs@darcs.net>
| <http://bugs.darcs.net/issue842>
| __________________________________

On Fri, May 09, 2008 at 05:27:26PM +0100, Simon Peyton-Jones wrote:
> darn.   The particular baseline I started from isn't on darcs.haskell.org
> any more, it seems, so I'll put it up at
> 
>         http://research.microsoft.com/~simonpj/tmp/ghc-HEAD-2007-08-29-ghc-corelibs-testsuite.tar.bz2

Hmmm.  Doesn't seem to be available.  Is there maybe a problem with this
URL?

David

Probably just wait an hour or two for the mirroring to work

S

| -----Original Message-----
| From: David Roundy [mailto:droundy@darcs.net]
| Sent: 09 May 2008 17:29
| To: Simon Peyton-Jones
| Cc: Darcs bug tracker; beschmi@cloaked.de; dagit@codersbase.com; igloo@earth.li; ptp@lysator.liu.se;
| Simon Marlow
| Subject: Re: [issue842] Darcs gzip error
|
| On Fri, May 09, 2008 at 05:27:26PM +0100, Simon Peyton-Jones wrote:
| > darn.   The particular baseline I started from isn't on darcs.haskell.org
| > any more, it seems, so I'll put it up at
| >
| >         http://research.microsoft.com/~simonpj/tmp/ghc-HEAD-2007-08-29-ghc-corelibs-testsuite.tar.bz2
|
| Hmmm.  Doesn't seem to be available.  Is there maybe a problem with this
| URL?
|
| David

On Fri, May 09, 2008 at 05:29:42PM +0100, Simon Peyton-Jones wrote:
> 
> | >         http://research.microsoft.com/~simonpj/tmp/ghc-HEAD-2007-08-29-ghc-corelibs-testsuite.tar.bz2
> |
> | Hmmm.  Doesn't seem to be available.  Is there maybe a problem with this
> | URL?
>
> Probably just wait an hour or two for the mirroring to work

Still doesn't work for me.

Thanks
Ian

| > | >         http://research.microsoft.com/~simonpj/tmp/ghc-HEAD-2007-08-29-ghc-corelibs-
| testsuite.tar.bz2
| > |
| > | Hmmm.  Doesn't seem to be available.  Is there maybe a problem with this
| > | URL?
| >
| > Probably just wait an hour or two for the mirroring to work
|
| Still doesn't work for me.

Indeed. Rats.  I'll have to ask tech suppport

S

OK try this instead
http://darcs.haskell.org/~simonpj/ghc-HEAD-2007-08-29-ghc-corelibs-testsuite.tar.bz2

Simon

| -----Original Message-----
| From: Ian Lynagh [mailto:igloo@earth.li]
| Sent: 10 May 2008 02:32
| To: Simon Peyton-Jones
| Cc: David Roundy; Darcs bug tracker; beschmi@cloaked.de;
| dagit@codersbase.com; ptp@lysator.liu.se; Simon Marlow
| Subject: Re: [issue842] Darcs gzip error
|
| On Fri, May 09, 2008 at 05:29:42PM +0100, Simon Peyton-Jones wrote:
| >
| > | >         http://research.microsoft.com/~simonpj/tmp/ghc-HEAD-2007-08-29-
| ghc-corelibs-testsuite.tar.bz2
| > |
| > | Hmmm.  Doesn't seem to be available.  Is there maybe a problem with this
| > | URL?
| >
| > Probably just wait an hour or two for the mirroring to work
|
| Still doesn't work for me.
|
|
| Thanks
| Ian

Simon and David,

Just reporting that I have retrieved the tarball and tried the darcs pull on
Linux using darcs 2.  It took about 20 minutes, but it succeeded.  So I could
not reproduce it here :-/ 

Off-topic: performance
======================
I realise this is not terribly relevant, but re: timing, it spent a minute after
reading the inventory, spent about 6 minutes fetching patches, stopped a while,
fetched more patches for another 22 minutes, did a little bit of work and
successfully applied.  

(One puzzling thing is that I tried this twice, and the first time, I could have
sworn it only took about 10 minutes or so, whereas the second time took twice as
long)

That's 82% of the time spent recuperating patches (or more; I re-did the test
using a local pull, and it only took about 4 minutes).  For the record, I
counted something like 1700 patches in the remote inventory and 6 in the local
one.  Hopefully the pipelining work will improve this.  I haven't had a chance
to try it.

For other people doing triage on this kind of thing, the new --debug-verbose
--timings switches in darcs 2 are quite handy.

Correction, it took about 30 minutes, but no matter; the point is still that I
couldn't reproduce it :-(

On another note, I tried doing a zcat
_darcs/patches/20080126191939-e23c7-efc06647390321527cc18de0e08c017b568b36ae.gz
and I got this: 

gzip:
_darcs/patches/20080126191939-e23c7-efc06647390321527cc18de0e08c017b568b36ae.gz:
invalid compressed data--crc error

Out of curiosity, I googled the error message and stumbled upon this FAQ from
gzip.org.  I tried their fixgz program, but the resulting file still gives a CRC
error.  

Simon, maybe it would be worthwhile to see what happens if you

 zcat
_darcs/patches/20080126191939-e23c7-efc06647390321527cc18de0e08c017b568b36ae.gz
> tmp-patch
 gzip tmp-patch
 mv tmp-patch.gz
_darcs/patches/20080126191939-e23c7-efc06647390321527cc18de0e08c017b568b36ae.gz

... and then try the darcs pull.

And here is the FAQ:

========8<==================
gunzip complains about corrupted data or a CRC error

99.9% of the problems with gzip are due to file transfers done in ASCII mode
instead of BINARY mode. In particular, gopher is known to corrupt binary files
by considering them as ASCII. Make sure that your local copy of the file has
exactly the same byte size as the original.

If you have transferred a file in ASCII mode and you no longer have access to
the original, you can try the program fixgz to remove the extra CR (carriage
return) bytes inserted by the transfer. A Windows 9x/NT/2000/ME/XP binary is
here. But there is absolutely no guarantee that this will actually fix your
file. Conclusion: never transfer binary files in ASCII mode. To compile fixgz
and run it, do:

   cc -o fixgz fixgz.c
   fixgz  bad.gz  fixed.gz
   gzip -tv fixed.gz
======8<=======================

Ok, I stop here.  Over to David (or somebody else!)

I also haven't been able to reproduce your bug, Simon.  From the page that Eric
quoted, I'd guess that what we're seeing is corruption-in-transit of a patch
that we're pulling.

You've mentioned that this is darcs 1.0.9 on Windows.  Do you know whether this
darcs was compiled with libcurl? All I can think is that maybe the bug is
library-related... so if you're using a transport method other than libcurl
(e.g. straight wget or curl) that would be interesting.

David

| I also haven't been able to reproduce your bug, Simon.  From the page that Eric
| quoted, I'd guess that what we're seeing is corruption-in-transit of a patch
| that we're pulling.

That'd be surprising, because it *also* happens if I pull from a local repo of the HEAD, which itself seems to be working fine. (Maybe it's illusory that it's working fine.)

| You've mentioned that this is darcs 1.0.9 on Windows.  Do you know whether this
| darcs was compiled with libcurl? All I can think is that maybe the bug is
| library-related... so if you're using a transport method other than libcurl
| (e.g. straight wget or curl) that would be interesting.

I'm afraid I don't know -- I just took the binary download from the darcs site, so it depends how you built it.

Meanwhile, I can probably work around this, so don't worry about fixing.  I though you should know about it, though, because it's an obscure message that might leave someone else totally stuck.

Simon

Zooko: we're trying to debug an issue that Simon PJ is facing.  I'm guessing the
1.0.9 binary on Windows (linked from the darcs wiki) is yours.  Do you remember
if it uses libcurl or some other method?  Issue834 makes me guess that it does
not use libcurl...

| Simon, maybe it would be worthwhile to see what happens if you
|
|  zcat
| _darcs/patches/20080126191939-e23c7-efc06647390321527cc18de0e08c017b568b36ae.gz
| > tmp-patch
|  gzip tmp-patch
|  mv tmp-patch.gz
| _darcs/patches/20080126191939-e23c7-efc06647390321527cc18de0e08c017b568b36ae.gz

I don't have zcat on my machine, but gunzip -c worked fine on that file.  (That isn't the file mentioned in the error message, incidentally.)

Simon

PS.  Ah I see that the patch Eric mentions was indeed the one mentioned in my original bug report.

But when I do this
        start from the tar ball I've put up
        darcs pull http://darcs.haskell.org/ghc -av
then I get the error

darcs failed:  problem gzopening file for write: _darcs/patches/20080125161308-e23c7-641f90a729379274c5079732c681bcc47fc3e314.gz-0

Same error, different file.

Simon

| -----Original Message-----
| From: Eric Kow [mailto:bugs@darcs.net]
| Sent: 13 May 2008 19:57
| To: Simon@osuosl.org; beschmi@cloaked.de; dagit@codersbase.com;
| daveroundy@gmail.com; droundy@darcs.net; eric.kow@gmail.com; igloo@earth.li;
| ptp@lysator.liu.se; Simon Marlow; Simon Peyton-Jones
| Subject: [issue842] pull => problem gzopening file for write (1.0.9)
|
|
| Eric Kow <eric.kow@gmail.com> added the comment:
|
| Out of curiosity, I googled the error message and stumbled upon this FAQ from
| gzip.org.  I tried their fixgz program, but the resulting file still gives a
| CRC
| error.
|
| Simon, maybe it would be worthwhile to see what happens if you
|
|  zcat
| _darcs/patches/20080126191939-e23c7-
| efc06647390321527cc18de0e08c017b568b36ae.gz
| > tmp-patch
|  gzip tmp-patch
|  mv tmp-patch.gz
| _darcs/patches/20080126191939-e23c7-
| efc06647390321527cc18de0e08c017b568b36ae.gz
|
| ... and then try the darcs pull.
|
| And here is the FAQ:
|
| ========8<==================
| gunzip complains about corrupted data or a CRC error
|
| 99.9% of the problems with gzip are due to file transfers done in ASCII mode
| instead of BINARY mode. In particular, gopher is known to corrupt binary
| files
| by considering them as ASCII. Make sure that your local copy of the file has
| exactly the same byte size as the original.
|
| If you have transferred a file in ASCII mode and you no longer have access to
| the original, you can try the program fixgz to remove the extra CR (carriage
| return) bytes inserted by the transfer. A Windows 9x/NT/2000/ME/XP binary is
| here. But there is absolutely no guarantee that this will actually fix your
| file. Conclusion: never transfer binary files in ASCII mode. To compile fixgz
| and run it, do:
|
|    cc -o fixgz fixgz.c
|    fixgz  bad.gz  fixed.gz
|    gzip -tv fixed.gz
| ======8<=======================
|
| Ok, I stop here.  Over to David (or somebody else!)
|
| __________________________________
| Darcs bug tracker <bugs@darcs.net>
| <http://bugs.darcs.net/issue842>
| __________________________________

On Wed, May 14, 2008 at 08:53:41AM +0100, Simon Peyton-Jones wrote:
> | I also haven't been able to reproduce your bug, Simon.  From the page
> | that Eric quoted, I'd guess that what we're seeing is
> | corruption-in-transit of a patch that we're pulling.
> 
> That'd be surprising, because it *also* happens if I pull from a local
> repo of the HEAD, which itself seems to be working fine. (Maybe it's
> illusory that it's working fine.)

Hmmm.  Yes, that does seem to eliminate the possibility of this being a
transport issue, which only makes it more puzzling.  You aren't running out
of disk space, are you? (To ask the obvious...)

Just to be clear: does darcs check pass on the local HEAD repo that you get
the error when pulling from?

> | You've mentioned that this is darcs 1.0.9 on Windows.  Do you know whether this
> | darcs was compiled with libcurl? All I can think is that maybe the bug is
> | library-related... so if you're using a transport method other than libcurl
> | (e.g. straight wget or curl) that would be interesting.
> 
> I'm afraid I don't know -- I just took the binary download from the darcs
> site, so it depends how you built it.
> 
> Meanwhile, I can probably work around this, so don't worry about fixing.
> I though you should know about it, though, because it's an obscure
> message that might leave someone else totally stuck.

It'd be nice to be able to reproduce this, though, so we could know that
it's fixed.  I can't reproduce the problem with either the stock debian
darcs 1.0.9rc1 or with darcs 2.0.0 (+ 212 patches).  :(

Could you perhaps try the same test with a darcs 2 (either compiled by
Simon M. or a precompiled binary from the wiki)? If you can't reproduce it
with darcs 2, I'll be less concerned.  As it is, I'm worried that this may
be a windows-specific bug.

Does windows have something like the open file-descriptor limit that posix
systems have? If so, do you have any idea how to track that kind of trouble
down? At this point I'm just brainstorming what could be going wrong...

David

>  Hmmm.  Yes, that does seem to eliminate the possibility of this being a
>  transport issue, which only makes it more puzzling.  You aren't running out
>  of disk space, are you? (To ask the obvious...)

It may be worthwhile to do the crc check on the repos you're pulling
from, both local (NEW) and remote ( http://darcs.haskell.org/ghc )

I think it's:
 for i in _darcs/patches/*.gz; do zcat $i > /dev/null; done 2> crc-errors
 grep crc crc-errors | sed -e 's/gzip: //' -e 's/gz.*/gz/' > bad-patches

> Could you perhaps try the same test with a darcs 2 (either compiled by
> Simon M. or a precompiled binary from the wiki)? If you can't reproduce it
> with darcs 2, I'll be less concerned.  As it is, I'm worried that this may
> be a windows-specific bug.

I'm building a darcs2 now, we'll see how that goes.

> Does windows have something like the open file-descriptor limit that posix
> systems have? If so, do you have any idea how to track that kind of
> trouble
> down? At this point I'm just brainstorming what could be going wrong...

Yes, the file descriptor thing had occurred to me too.  I'm sure there are similar limits on Windows.

Is there any way to get more info from gzopen about the error?

Cheers,
        Simon

| Hmmm.  Yes, that does seem to eliminate the possibility of this being a
| transport issue, which only makes it more puzzling.  You aren't running out
| of disk space, are you? (To ask the obvious...)

No, 37Gbyte free.

| Just to be clear: does darcs check pass on the local HEAD repo that you get
| the error when pulling from?

Alas, I can't do that!  Remember, a different Darcs bug means that one cannot pull, or check, a full GHC repo from scratch.  Something to do with line endings I believe. That's why we always have to start from a tar ball.  Alas.

Simon

I guess David and Simon M have noticed this already, but I only just now saw
that this was a problem opening the file for /write/ not read...

indeed.  does that crccheck still make sense? I'm installing cygwin so I can do that for you if you want

| -----Original Message-----
| From: Eric Kow [mailto:bugs@darcs.net]
| Sent: 14 May 2008 11:47
| To: Simon@osuosl.org; beschmi@cloaked.de; dagit@codersbase.com; daveroundy@gmail.com; droundy@darcs.net;
| eric.kow@gmail.com; igloo@earth.li; ptp@lysator.liu.se; Simon Marlow; Simon Peyton-Jones;
| zooko@zooko.com
| Subject: [issue842] pull => problem gzopening file for write (1.0.9)
|
|
| Eric Kow <eric.kow@gmail.com> added the comment:
|
| I guess David and Simon M have noticed this already, but I only just now saw
| that this was a problem opening the file for /write/ not read...
|
| __________________________________
| Darcs bug tracker <bugs@darcs.net>
| <http://bugs.darcs.net/issue842>
| __________________________________

>  indeed.  does that crccheck still make sense? I'm installing cygwin so I can do that for you if you want

I guess so.  It may not tell us very much about the bug at hand, but
it may be useful for a new ticket that we may open...

Well, I guess you could do it on darcs.haskell.org at least.
Attaching my copy of it.

Attachments

• crc-errors-kowey

| Well, I guess you could do it on darcs.haskell.org at least.
| Attaching my copy of it.

Sounds as if you've done that, so I won't.  If you want any more experiments on my setup, I'm at your service.

S

On Wed, May 14, 2008 at 11:55:09AM +0100, Simon Peyton-Jones wrote:
> | Hmmm.  Yes, that does seem to eliminate the possibility of this being a
> | transport issue, which only makes it more puzzling.  You aren't running out
> | of disk space, are you? (To ask the obvious...)
> 
> No, 37Gbyte free.

Ah well.  One can always hope for an easy answer!  :)

> | Just to be clear: does darcs check pass on the local HEAD repo that you get
> | the error when pulling from?
> 
> Alas, I can't do that!  Remember, a different Darcs bug means that one
> cannot pull, or check, a full GHC repo from scratch.  Something to do
> with line endings I believe. That's why we always have to start from a
> tar ball.  Alas.

Argh.  This is the case-insensitive filesystem issue.  :(

Actually, you could look for the problem I was curious about by running
darcs changes -s on the local HEAD repo.  That'll force darcs to open and
parse every patch, and that's where the error seems to be coming.
-- 
David Roundy
Department of Physics
Oregon State University

| Actually, you could look for the problem I was curious about by running
| darcs changes -s on the local HEAD repo.  That'll force darcs to open and
| parse every patch, and that's where the error seems to be coming.

I did darcs cha -s > /dev/null, which seemed to complete normally, both on the unpacked tarball, and on the local copy of the HEAD.

Remember the gzopen message seems to be concerned with writing not reading

S

The darcs-1.0.9 executables were actually built by Will Glozer, and I  
downloaded them from here:

http://glozer.net/darcs/

I don't know off the top of my head if libcurl is statically linked in.

strings ./realdarcs.exe | grep -i curl

suggests that it is.

It would be cool if darcs --exact-version would emit the  
configuration options that were used to configure it.

Regards,

Zooko

On Wed, May 14, 2008 at 02:47:46PM +0100, Simon Peyton-Jones wrote:
> | Actually, you could look for the problem I was curious about by running
> | darcs changes -s on the local HEAD repo.  That'll force darcs to open and
> | parse every patch, and that's where the error seems to be coming.
> 
> I did darcs cha -s > /dev/null, which seemed to complete normally, both
> on the unpacked tarball, and on the local copy of the HEAD.
> 
> Remember the gzopen message seems to be concerned with writing not reading

Ah yes, I forgot about that.  That is puzzling indeed.  Perhaps we're
opening the same file for write twice? Is there anything like systrace you
could use to see what's going on?
-- 
David Roundy
Department of Physics
Oregon State University

In case this helps, I did a darcs check on my copy of HEAD (from
darcs.haskell.org) and it said the repository was consistent.  [I'm on Linux,
but I'll bet that the new hashed pristine makes it possible to do this check
successfully on case-insensitive file systems with the new darcs]

On May 14, 2008, at 9:25 AM, David Roundy wrote:

> Ah yes, I forgot about that.  That is puzzling indeed.  Perhaps we're
> opening the same file for write twice? Is there anything like  
> systrace you
> could use to see what's going on?

cygwin has "strace.exe".  I haven't used it much.

Regards,

Zooko

From Simon on issue892.  Assigning to myself to reproduce on Windows with both
darcses 1.0.9 and 2.0.0

---------8<-------------------

Dear Darcs crew

I'm just writing to advertise my continued failure to get 'darcs pull' to work.

To reproduce:

1.        Unpack
http://darcs.haskell.org/~simonpj/ghc-HEAD-2007-08-29-ghc-corelibs-testsuite.tar.bz2

2.       darcs pull http://darcs.haskell.org/ghc -av

Result is

darcs failed:  problem gzopening file for write:
_darcs/patches/20080126191939-e23c7-efc06647390321527cc18de0e08c017b568b36ae.gz-0

(In fact I did step 2 from a local rep, because doing it over the network takes
a very long time, but I saw the same behaviour earlier when pulling from
http://darcs.haskell.org/ghc.)

I'm doing this under MSYS on Windows, with
sh-2.04$ darcs --version
1.0.9 (release)

I'm giving up on this, and instead I'll start from a more recent snapshot of the
repo.  Nevertheless I have to say that it is very unsettling that Darcs fails in
mysterious ways.

I'll keep the bits around in case you want me to try any other experiments.

Simon

So, I *can* reproduce this under Windows with darcs 1.0.9.

Unfortunately, the cygwin strace doesn't report anything but
--- Process 716, exception C0000008 at 7C90EB74 

I think this is because darcs is not built against cygwin

Anyone know how to run strace under msys?

Here's what I get if I try to reproduce this with darcs2, an error applying a
patch to the working dir.  Is this related?  Attached

Unassigning myself.   I don't know what else to do...

Attachments

• darcs2-debug

> Eric Kow <eric.kow@gmail.com> added the comment:
>
> Here's what I get if I try to reproduce this with darcs2, an error
> applying a
> patch to the working dir.  Is this related?  Attached
>
> Unassigning myself.   I don't know what else to do...

I've also encoutnered a similar error (see my recent post on darcs-users), although in my case the file was different (RnExpr.lhs-boot).

Cheers,
        Simon

Alright, it looks like it would be a good idea to track down which patch(es)
is/are trying to modify
  ./compiler/rename/RnPat.lhs
and
  ./compiler/rename/RnExpr.lhs-boot
and breaking (and of course, it may not be the patch itself that is at fault,
but a prior one)

Also, if we get it to a state where without trying to pull that patch darcs2
works fine, but darcs1 still has the gzopen error, then we can assume we've got
a separate issue to deal with.

With http://bugs.darcs.net/issue913 taken into account, we do indeed have the
situation where darcs2 can pull these patches, but darcs1 cannot.

Is is worth trying to figure out what went wrong with darcs 1.0.9, or should we
just consider this resolved?

I declare this bug resolved (since only darcs 1.0.9 seems to be affected)