/usr/local/bin/darcs get --verbose --partial --repo-name build
http://207.7.153.190/source/tahoe/trunk-hashedformat
darcs: bug in darcs!
Another possible bug in URL.waitNextUrl:  select() failed at src/URL.hs:193
compiled Aug 27 2008 11:49:16
You can check to see if this bug is already known at http://bugs.darcs.net/
If it is not, please report this to bugs@darcs.net
If possible include the output of 'darcs --exact-version'.

Are you running on Solaris? Can you please attach --debug output?

Regards,
  Dmitry

Yes, GNU OpenSolaris a.k.a. Nexenta.

/usr/local/bin/darcs get --debug --verbose --partial
http://207.7.153.190/source/tahoe/trunk-hashedformat
Wed Aug 27 12:24:28 PDT 2008
Starting work on get...
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/motd
                      -> darcs12212)
URL.waitUrUlR Lh.tutrpl:T/h/r2e0a7d.
7(.h1t5t3p.:1/9/02/0s7o.u7r.c1e5/3t.a1h9oe/trunk-hashedformat/_darcs/prefs/motd
0/source/tahoe/trunk-hashedformat/_darcs/prefs/motd
            -> darcs12212)
URL.request_url
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/motd
              -> darcs12212)
URL.request_url succeeded
URL.waitNexRteUardli nsgu cccheckpoint...
Beginning identifying repository
http://207.7.153.190/source/tahoe/trunk-hashedformat
eeded: http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/motd
darcs12212
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
                      -> darcs12212)
URL.waitUrl http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
URL.urlThread (http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
            -> darcs12212)
URL.request_url (http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
              -> darcs12212)
URL.request_url succeeded
URL.waitNextUrl succeeded:
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format darcs12212
Done identifying repository http://207.7.153.190/source/tahoe/trunk-hashedformat
Found the format of http://207.7.153.190/source/tahoe/trunk-hashedformat...
Finished initializing new directory.
Beginning identifying repository .
Done identifying repository .
Beginning identifying repository .
Done identifying repository .
Identified darcs-1 repo: /home/zooko/dx/trunk-hashedformat_6
Identifying and copying repository...
Beginning identifying repository
http://207.7.153.190/source/tahoe/trunk-hashedformat
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
                      -> darcs12212)
URL.waitUrl http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
URL.urlThread (http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
            -> darcs12212)
URL.request_url (http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
              -> darcs12212)
URL.request_url succeeded
Done identifying repository http://207.7.153.190/source/tahoe/trunk-hashedformat
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources
                      -> darcs12212)
URL.waitUrl
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources
URL.waitNextUrl succeeded:
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format darcs12212
URL.urlThread
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources
            -> darcs12212)
URL.request_url
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources
              -> darcs12212)
URL.request_url succeeded
Beginning identifying repository .
Done identifying repository .
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/hashed_inventory
                      ->
/home/zooko/dx/trunk-hashedformat_6/_darcs/hashed_inventory)
URL.waitUrl
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/hashed_inventory
URL.waitNextUrl succeeded:
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources darcs12212
URL.urlThread
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/hashed_inventory
            -> /home/zooko/dx/trunk-hashedformat_6/_darcs/hashed_inventory)
URL.request_url
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/hashed_inventory
              -> /home/zooko/dx/trunk-hashedformat_6/_darcs/hashed_inventory)
URL.request_url succeeded
Directory '/home/zooko/dx/trunk-hashedformat' already exists, creating
repository as '/home/zooko/dx/trunk-hashedformat_6'
Fetching over HTTP:  http://darcs.net/maintenance
darcs: bug in darcs!
Another possible bug in URL.waitNextUrl:  select() failed at src/URL.hs:193
compiled Aug 27 2008 11:49:16
You can check to see if this bug is already known at http://bugs.darcs.net/
If it is not, please report this to bugs@darcs.net
If possible include the output of 'darcs --exact-version'.

Looks like I need to add some debug messages to see what's going on. Expect a
patch soon...

FYI, this is how I ./configured libcurl:

./configure --prefix=/usr/local/stow/libcurl-7.18.2 --disable-ftp --disable-file
--disable-ldap --disable-ldaps --disable-dict --disable-telnet --disable-tftp
--disable-manual --disable-ipv6 --disable-ares --disable-verbose --disable-sspi
--disable-debug --disable-crypto-auth --disable-cookies --enable-hidden-symbols
--without-libssh2 --without-nss --without-ca-bundle --without-libidn

So, I'm afraid I don't really understand what's going on with these HTTP bugs,
so please excuse me if this question is completely random:

Could pipelining be an issue here and with the other HTTP bugs?  If so, should
we recommend that users try the --disable-pipelining runtime (!) switch as an
easy debugging step?

Is the --disable-pipelining runtime switch already in there?  I just tried it
and got the same failure:

/usr/local/bin/darcs get --disable-pipelining --debug --verbose --partial
http://207.7.153.190/source/tahoe/trunk-hashedformat
Wed Aug 27 12:53:06 PDT 2008
Starting work on get...
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/motd
                      -> darcs19878)
URL.waitUrl UhRtLt.pu:r/l/T2h0r7e.a7d.
1(5h3t.t1p9:0///s2o0u7r.c7e./1t5a3h.o1e9/0/source/tahoe/trunk-hashedformat/_darcs/prefs/motd
            -> darcs19878)
URL.request_url
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/motd
              -> darcs19878)
URL.request_url succeeded
URL.waitNextUrl succeeded:
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/motd darcs19878
trunk-hashedformat/_darcs/prefs/motd
Reading checkpoint...
Beginning identifying repository
http://207.7.153.190/source/tahoe/trunk-hashedformat
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
                      -> darcs19878)
URL.waitUrl http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
URL.urlThread (http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
            -> darcs19878)
URL.request_url (http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
              -> darcs19878)
URL.request_url succeeded
Done identifying repository http://207.7.153.190/source/tahoe/trunk-hashedformat
Found the format of http://207.7.153.190/source/tahoe/trunk-hashedformat...
Finished initializing new directory.
Beginning identifying repository .
Done identifying repository .
Beginning identifying repository .
Done identifying repository .
Identified darcs-1 repo: /home/zooko/dx/trunk-hashedformat_8
Identifying and copying repository...
Beginning identifying repository
http://207.7.153.190/source/tahoe/trunk-hashedformat
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
                      -> darcs19878)
URL.waitUrl http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
URL.waitNextUrl succeeded:
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format darcs19878
URL.urlThread (http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
            -> darcs19878)
URL.request_url (http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format
              -> darcs19878)
URL.request_url succeeded
Done identifying repository http://207.7.153.190/source/tahoe/trunk-hashedformat
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources
                      -> darcs19878)
URL.waitUrl
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources
URL.waitNextUrl succeeded:
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/format darcs19878
URL.urlThread
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources
            -> darcs19878)
URL.request_url
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources
              -> darcs19878)
URL.request_url succeeded
Beginning identifying repository .
Done identifying repository .
URL.copyUrlWithPriority
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/hashed_inventory
                      ->
/home/zooko/dx/trunk-hashedformat_8/_darcs/hashed_inventory)
URL.waitUrl
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/hashed_inventory
URL.waitNextUrl succeeded:
http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/prefs/sources darcs19878
URL.urlThread
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/hashed_inventory
            -> /home/zooko/dx/trunk-hashedformat_8/_darcs/hashed_inventory)
URL.request_url
(http://207.7.153.190/source/tahoe/trunk-hashedformat/_darcs/hashed_inventory
              -> /home/zooko/dx/trunk-hashedformat_8/_darcs/hashed_inventory)
URL.request_url succeeded
Directory '/home/zooko/dx/trunk-hashedformat' already exists, creating
repository as '/home/zooko/dx/trunk-hashedformat_8'
Fetching over HTTP:  http://darcs.net/maintenance
darcs: bug in darcs!
Another possible bug in URL.waitNextUrl:  select() failed at src/URL.hs:193
compiled Aug 27 2008 12:23:46
You can check to see if this bug is already known at http://bugs.darcs.net/
If it is not, please report this to bugs@darcs.net
If possible include the output of 'darcs --exact-version'.

darcs-2.0.2 release works for this command on this system.

I took the liberty of marking this ReleaseCritical_2.0.3, but if it turns out
that I'm the only one who has this problem then I agree that we could release
2.0.3 without fixing this.

Dmitri (or anyone else who wants to debug this): I will happily give you an
account on my solaris server.  Just send me your SSH public key.

How frustrating -- I can no longer reproduce this bug, although I was doing it
every single time earlier today.

Ah -- I can reproduce this failure more reliably by telling darcs not to use the
local cache or by rm -rf'ing the contents of the local cache before trying the
"darcs get".

I have found the root cause - select is interrupted by signal. Such things are
supposed to be handled by (errno != EINTR) but errno is 2 and EINTR is 4 (still
perror prints interrupted by signal, strange).

Now the question is how to fix this. Can we actually use errno in an unbound
thread? (it is bad if we can not, because curl is likely using errno).

I have tried changing forkIO to forkOS, but errno is still strange. Perhaps it
is a nexenta issue, but unlikely.

Now I try calling pthread_sigmask before curl is initialized. Does anybody know
if it works? There is System.Posix.Signals.blockSignals function, but
documentation says it uses sigprocmask. And sigprocmask man page says result is
undefined in multi threaded program.

An advice from haskell thread expert would be nice :)

BTW looks like pthread_sigprocmask did the trick, but I get "hash failure" errors.

Regards,
  Dmitry

Hi Simon,

Could you have a look at this bug report, please?  Dmitry said
> An advice from haskell thread expert would be nice :)

so I thought of you

What is errno 2?

Yes, you can use errno in any thread.  It is saved across context switches.

I can not tell what errno 2 is right now (no ssh access). But IIRC it is an
error code that is not documented for select, which is odd.

Besides, perror() and strerror(errno) give different strings...

What do you think about blocking signals with pthread_sigmask? Is it a valid
operation in unbound thread? If not what about bound thread?

I still believe that the real cause of failed select is signal interrupt. And
instead of checking errno and restarting select we can just block signals.

Regards,
  Dmitry

Dmitry: do you have ssh access to nooxie now?  Note that I spelled your name
wrong in your username...

Yes, I have ssh access to nooxie. Thanks for it! I was able to reproduce the
select bug. And looks like pthread_sigmask fixes the issue, but I want to hear
comments from Simon before submitting the patch.

BTW do you see the bug only on
http://207.7.153.190/source/tahoe/trunk-hashedformat repo?

Regards,
  Dmitry

No, as you can see from the buildbot -- http://buildbot.darcs.net -- the
following commands always seem to hang indefinitely on some machines:

When the darcs tests are running, we execute "make test_network", which executes
"get.sh".  This always hangs on certain machines.

pthread_sigmask sounds like a bit of a hack.  You'd have to remember to restore
the mask after calling into curl.  I'd like to understand a bit more about
what's causing the problem here - we think it's that select() is getting EINTR,
but select() apparently already checks for EINTR, so that's not the whole story.
 (strace might help).

Is libcurl supposed to be thread-safe?  If not, we need to be calling it from
just one thread.

We had a problem with libedit recently where it was failing to check for EINTR.
 The workaround we use is to disable the timer interrupt during calls into
libedit, by calling stopTimer()/startTimer() in GHC's RTS.  This is a horrible
hack though.

On Mon, Sep 1, 2008 at 2:27 PM, Simon Marlow <bugs@darcs.net> wrote:
>
> Simon Marlow <simonmar@microsoft.com> added the comment:
>
> pthread_sigmask sounds like a bit of a hack.  You'd have to remember to restore
> the mask after calling into curl.

Perhaps we should use a bound thread to prevent OS thread switching?

>  I'd like to understand a bit more about
> what's causing the problem here - we think it's that select() is getting EINTR,
> but select() apparently already checks for EINTR, so that's not the whole story.
>  (strace might help).

Yeah, something odd is going on there.

>
> Is libcurl supposed to be thread-safe?  If not, we need to be calling it from
> just one thread.

Curl is not thread safe. But I believe it is fine to call it from
different OS threads as long as we do not do it in parallel.

Besides, IIRC I have tried changing forkIO to forkOS - and it does not help.

>
> We had a problem with libedit recently where it was failing to check for EINTR.
>  The workaround we use is to disable the timer interrupt during calls into
> libedit, by calling stopTimer()/startTimer() in GHC's RTS.  This is a horrible
> hack though.

I think it's worth trying. At least that would show us that problem is
with timer interrupts.

I do not have ssh access at work. And I can only work on nooxie at
evening. If you give me a set of questions/things to try I will test
it and post results. Or zooko can give you ssh access to the server
and you can check it on your own.

One thing I want to test is a pure C program that is interrupted
during select(). To check that we should (not) get EINTR.

Regards,
  Dmitry

Ah, I just had a brainwave.  I bet you need to compile all your C code with

#define _REENTRANT 1

otherwise you don't get the thread-safe version of errno on Solaris.

Cheers,
        Simon

On Mon, Sep 1, 2008 at 3:04 PM, Simon Marlow <simonmar@microsoft.com> wrote:
> Ah, I just had a brainwave.  I bet you need to compile all your C code with
>
> #define _REENTRANT 1
>
> otherwise you don't get the thread-safe version of errno on Solaris.

Wow! Looks like this explains everything! I have googled on this and
there are different macros for different platforms (AIX, SOLARIS). And
looks like -pthread gcc option is a "portable" variant of turning on
multi threading. Is my understanding correct?

Regards,
  Dmitry

>
> Cheers,
>        Simon
>

> From: Dmitry Kurochkin [mailto:dmitry.kurochkin@gmail.com]
> Sent: 01 September 2008 12:18
> To: Simon Marlow
> Cc: Darcs bug tracker; dagit@codersbase.com; eric.kow@gmail.com;
> zooko@zooko.com
> Subject: Re: [issue1037] select() failed
>
> On Mon, Sep 1, 2008 at 3:04 PM, Simon Marlow <simonmar@microsoft.com>
> wrote:
> > Ah, I just had a brainwave.  I bet you need to compile all your C code
> with
> >
> > #define _REENTRANT 1
> >
> > otherwise you don't get the thread-safe version of errno on Solaris.
>
> Wow! Looks like this explains everything! I have googled on this and
> there are different macros for different platforms (AIX, SOLARIS). And
> looks like -pthread gcc option is a "portable" variant of turning on
> multi threading. Is my understanding correct?

I'm not sure of all the details, but I know we needed to do this in GHC's libraries.  And it seems to work to use _REENTRANT==1 all the time, even when not doing multithreading, because in GHC we have the libraries compiled only once for both -threaded and non-threaded.

Cheers,
        Simon

On Mon, Sep 1, 2008 at 3:22 PM, Simon Marlow <simonmar@microsoft.com> wrote:
> I'm not sure of all the details, but I know we needed to do this in GHC's libraries.  And it seems to work to use _REENTRANT==1 all the time, even when not doing multithreading, because in GHC we have the libraries compiled only once for both -threaded and non-threaded.

Do you set this flag for solaris only? Are there specific flags for
other systems, or solaris is the only one that needs this?

Regards,
  Dmitry

Yeah, Simon, just send me your ssh public key and I'll create an account for you
on nooxie.

> From: Dmitry Kurochkin [mailto:dmitry.kurochkin@gmail.com]
> On Mon, Sep 1, 2008 at 3:22 PM, Simon Marlow <simonmar@microsoft.com>
> wrote:
> > I'm not sure of all the details, but I know we needed to do this in
> GHC's libraries.  And it seems to work to use _REENTRANT==1 all the time,
> even when not doing multithreading, because in GHC we have the libraries
> compiled only once for both -threaded and non-threaded.
>
> Do you set this flag for solaris only? Are there specific flags for
> other systems, or solaris is the only one that needs this?

We set it unconditionally in a couple of places in GHC (once in the RTS and once in the base package).  I wouldn't be surprised if there is C code in other packages that needs it to be set on Solaris, and currently doesn't.

Cheers,
        Simon

> From: Zooko [mailto:bugs@darcs.net]
>
> Zooko <zooko@zooko.com> added the comment:
>
> Yeah, Simon, just send me your ssh public key and I'll create an account
> for you on nooxie.

I'm too busy with GHC releases and other things right now, besides I think we've found the problem now, but thanks anyway!

Cheers,
        Simon

I noticed this in the libcurl 7.19.0 changes file:

#  library will now be built with _REENTRANT symbol defined only if needed 

Is that relevant at all?

That probably means that on Solaris (at least) curl >= 7.19.0 should be
configured with --enable-multithread (or smth like that) to work with darcs.

Regards,
  Dmitry

Fixed by:

Mon Sep  1 21:57:38 MSD 2008  Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
  * Resolve issue1037: compile with -D_REENTRANT.