While trying to get http://darcs.haskell.org/ghc-stable/packages/pretty
by invoking `darcs get --partial
http://darcs.haskell.org/ghc-stable/packages/pretty' command as part of
GHC buildprocess on GHC buildbot I've notices that this darcs never
finishes its `Identifying repository
http://darcs.haskell.org/ghc-stable/packages/pretty
' step. It seems there is some live-lock involved while calling licurl
functions. This is on Solaris 11, ghc 6.8.2.20080430, darcs from May 22,
linked against curl 7.18.0 with curl pipelining enabled.

As this is on Solaris I've saved few process's stacktraces by using
pstack(1) command. It seems darcs is again and again invoking following
trace:

-----------------  lwp# 1 / thread# 1  --------------------
 fea545b7 gettimeofday (80459c0, 8561e60, 80459d8, feed829d, 80459c0,
8561e64) + 7
 feed82da curl_multi_perform (8561e60, 8045a2c, 0, 0, 0, 0) + 8e
 0841e5db curl_wait_next_url (0, 0, 0, 0, 0, 0) + 37
 08189e3d ???????? (fe3ec8c1, fe3ec8ce, fe3ec8e7, fe3ec8f0, 818b29c,
fe3ec8fa)
 00000001 ???????? (c5830818, a79ae9fc, f689002e, 384ab0, 4, ffff0022)
 b11c0045 ???????? ()


Anyway whole log saved by few seconds running `while (true) do pstack
<darcs pid>; done is attached.

Could anybody be so kind and try to duplicate this issue on Linux? Also
if anybody is able to tell me how to get rid of question marks in the
trace and replace them with appropriate GHC generated functions name,
then I would be glad to provide "fixed" stacktrace to show more
information about the issue. Thanks!

PS: Please note this is in Solaris running on real-metal, not in broken
VirtualBox!

Attachments

• darcs_live_lock_stacks.txt.gz

For what it's worth, I was able to get that repository under Linux (darcs 2.0.0
+ 264)

Actually I think Zooko on darcs-user might right claiming that this
might be issue in libcurl badly handling of time changes. i.e. I'm
running ntpd and perhaps at the right time it moved clock backward which
might confuse curl enough to break whole darcs. But this would require
some testing for a proof.

I don't think this is caused by ntpd. So many boxes are running that, that if it
ever caused trouble, it would have been fixed already. Can you reproduce it
under Solaris?

Also, removing the question marks from ptrace output is impossible AFAIK. A tool
like ptrace assumes that the processor registers are used according to some C
ABI, but GHC's stack and register layout is totally different from C's.

Creating a tracing tool for GHC-compiled executables seems like a fun project
though :)

It seems that this issue also might cause strange dead-locks in darcs testsuite
as shown on the darcs buildbots. Sometimes either make test or make test_network
hangs forever. Following little scripts duplicates this issue on Solaris:

[code]
while (true); do gmake test; gmake test_network;  done
[/code]

At the risk of throwing in some noise, I notice that under Windows (darcs
2.0.1rc1 + 15) with curl pipelining enabled, darcs will sit there forever trying
to identify the repository format.

The debugging info says that it has successfully retrieved the format file, but
if you ls it, the tmpfile it's using is 0 byte.

Attached

Attachments

• hang-windows

Argh.  Nevemind.  That was noise.  I was (a) using a silly URL and (b) had not
set my proxy server

Just to check, is it just darcs get that's causing problems?

If we echo changes.sh > tests/network/tests_to_run (disabling the get test), do
we still have a hang?

I suspect this might be related to curl versus threaded runtime. See also
issue977 -- Dmitry is looking into that, I believe.

Dmitry says that issue977 has been resolved.  Any improvements with the latest
darcs, Karel?

The only improvement is that instead of live-lock I get dead-lock. So darcs does
not consume 100% CPU, but just waits silent forever. I've reported this to
darcs-users@ and Dmitry has kindly asked me to duplicate with --debug
--debug-http, but I've not had any free time to do this yet. Sorry, for that.
For more information see Solaris' buildbot and its failing test_network test. It
locks all the time now.
Well, but if you care I may open different bug for this if you like to have this
one closed. Thanks!

Karel, can you test with the latest darcs? There were few HTTP related fixes,
one Solaris specific.

If you are behind HTTP proxy - take a look at issue1035, there is a bug in curl.

Regards,
  Dmitry

Hello, as per Dmitry request I've tested and I'm not able to duplicate
the dead-lock while running make test_network several times (11x). So
perhaps this is really fixed, since previously it happened every time.
Thanks! Karel

Thanks Karel! Closing this one.

Regards,
  Dmitry