Folks:

I just ran into a problem where I did a "darcs push user@host" and it  
"hung".  After a few minutes (instead of waiting I went to do other  
stuff), I realized that the underlying ssh connection must be blocked.

I tried it -- "ssh user@host" -- and sure enough the Windows putty  
ssh posted a warning message to stderr and blocked after asking  
"Store key in cache? (y/n)".

I've noticed that this is a frequent problem for new users of  
darcs.  If anyone is having trouble using darcs over ssh, the first  
step is always to tell them to invoke the equivalent ssh command on  
the command-line so that they can see the stderr.

Brian Warner pointed out to me that if darcs simply passed any stderr  
from the call to ssh back to darcs's stderr, then this would happen  
automatically and new users (as well as experienced users like me)  
would be able to diagnose their problems more quickly.

Regards,

Zooko

I'll submit a patch for this (as I have seen the problem myself)... but I have
the feeling it's a bit more subtle than that, or that my patch will break
something else...

So, as I mention in my patch, the potential cost of doing this is that we're
going to be dumping a lot of ssh output (scp) to the user's screen.  One thought
is that it doesn't matter as much now with darcs transfer-mode.  Another is that
we could make this something parameterisable (--no-ssh-stderr)

Yes, that's a good point.  Another thought is that we can ask ssh to be as quiet
as possible by passing "-q" or equivalent flags.

Heh... I had a feeling we've discussed this before...

Fri Jul  7 03:52:45 BST 2006  Eric Kow <eric.kow@gmail.com>
  * Run ssh/scp/sftp quietly.
  
  This is useful for silencing Putty, and could also be for OpenSSH should
  we decide to stop redirecting to /dev/null.

Resolved in unstable by my patch... (seems our auto-resolver is broken)

Wed May 14 14:55:04 BST 2008  Eric Kow <E.Y.Kow@brighton.ac.uk>
  * Resolve issue845: Pass ssh's stderr to our stderr.
  
  This may have the side effect that users get *a lot* of scp output dumped
  on their screen, but false 'hanging' that Zooko reported seems like the
  greater evil.  Note that darcs transfer-mode is not affected by this, so
  the only users that would see the extra output are those who are interacting
  with servers that don't have darcs 2 installed.

This change seems have been reverted by

http://allmydata.org/trac/darcs-2/changeset/6017

Maybe a good compromise would be to filter out certain normal ssh stderr
messages that we don't want to see, but pass through all unrecognized ones?

So, now if users experience weird hanging over ssh (because putty is asking the
user something) we can at least tell them to try darcs --debug

But I guess Zooko's solution would be ideal (sigh).  

Or we pass along PuTTY's stderr, but not that of any other ssh.

Or PuTTY stops putting its prompts on stderr

Or we stop using PuTTY

What are the messages that ssh puts on stderr that we don't want to show to the
user?

I just experimented with ssh a bit, both with and without a "known_hosts" entry
for the host, and I didn't get any messages on stderr.

On Mon, Sep 01, 2008 at 11:47:36 -0000, Zooko wrote:
> What are the messages that ssh puts on stderr that we don't want to show to the
> user?

scp errors fetching a non-existent _darcs/prefs/format (non-essential
file)

I don't think it is a good strategy to let users experience weird hangs (because
the ssh client is blocked asking the user something), and then hope that the
user will come find the darcs developers and ask them for advice, or will try to
use the darcs "--debug" flag.

I'm hoping that droundy made this change because he wasn't aware that it causes
darcs to mysteriously hang in some situations -- making darcs mysteriously hang
doesn't seem like the kind of thing that he likes to do, even in order to make
the stderr friendlier.

By the way, I don't think that this is limited to putty.  Although there are no
notes on this ticket showing that it has happened with openssh, I seem to
remember that new users frequently have the problem of darcs hanging because
their underlying ssh isn't configured properly.

See related tickets:, issue334, issue219, issue218, issue325, issue1028,
issue29, issue28

See also: http://wiki.darcs.net/DarcsWiki/SSHNotes

On Mon, Sep 01, 2008 at 11:56:30 -0000, Zooko wrote:
> I don't think it is a good strategy to let users experience weird hangs (because
> the ssh client is blocked asking the user something), and then hope that the
> user will come find the darcs developers and ask them for advice, or will try to
> use the darcs "--debug" flag.

Fair enough.  I think we need to either go back to passing stderr
through or trying to capture the err output and filtering out what we
are familiar with.

> By the way, I don't think that this is limited to putty.  Although
> there are no notes on this ticket showing that it has happened with
> openssh, I seem to remember that new users frequently have the problem
> of darcs hanging because their underlying ssh isn't configured
> properly.

So far I have only seen this particular behaviour with PutTTY.

The OpenSSH hanging was genuine, and was due to a bug in their control
master feature.

Otherwise, OpenSSH has the virtue of writing directly to terminal (I
only say this because I tried ssh example.com 2> foo and still got
output)

> See related tickets:, issue334, issue219, issue218, issue325, issue1028,
> issue29, issue28
> 
> See also: http://wiki.darcs.net/DarcsWiki/SSHNotes

Those notes were for diagnosing the ControlMaster problem.

On Mon, Sep 01, 2008 at 11:56:30 -0000, Zooko wrote:
> I don't think it is a good strategy to let users experience weird hangs (because
> the ssh client is blocked asking the user something), and then hope that the
> user will come find the darcs developers and ask them for advice, or will try to
> use the darcs "--debug" flag.

Ok, Zooko, I think I know what may be the right way to fix this, and I
think I want you to do it (*)!

In src/Exec.lhs, there is type called Redirect

  data Redirect = AsIs | Null | File FilePath | Stdout
                  deriving Show

I think we could extend it to

  data Redirect = AsIs | Null | File FilePath | Stdout
                | Filter (String -> String) Redirect
                  deriving Show

If you don't know what I'm getting at, ask #darcs for help.

(*) You've been one of our most productive issue submitters so far,
for which thanks.  But I think what would make you even more useful
is to upgrade you into the kind of issue submitter that sends patches
for some N% of his issues! I realise you may not be a Haskeller, but
I think this falls into ProbablyEasy territory.

(Adding Nosy: droundy)

Folks, I just hit this issue again -- upgraded to darcs-2.1.0 on a Windows
computer, tried to "darcs get" from another computer, and it silently hung. 
Eventually I realized that it was waiting for something, killed it, constructed
an equivalent ssh commandline (which, note, is normally never done from this
windows computer to that server -- it has to be done only in order to debug
darcs and it has to be done using the ssh executable that darcs invokes, which
is not necessarily the one that "ssh" invokes), learned that the host key or IP
address has changed so I have to approve yes/no to continue, and then I could go
back to using darcs.

Until I implement the nice filtering described in this ticket, which will
exclude the specific noisy lines that we don't want but lets through prompts and
unexpected error messages and so forth, could we err on the side of noisiness
instead of on the side of silently hanging?

As a compromise with Zooko's interim measure, we could usually output ssh
warnings, and suppress them when people do darcs --quiet

> > What are the messages that ssh puts on stderr that we don't want to show to the
> > user?

> scp errors fetching a non-existent _darcs/prefs/format (non-essential

Does this happen only if the server is running darcs earlier than 2.0?  (I
thought if both server and client were darcs-2, then it would use the
darcs-protocol-over-ssh instead of using scp.)

Maybe we should make this windows-specific behavior, since ssh prompts
just fine on posix machines?

David

On Oct 21, 2008, at 8:05 AM, David Roundy wrote:

> Maybe we should make this windows-specific behavior, since ssh prompts
> just fine on posix machines?

I thought that something similar could happen on unix, but I don't  
recall an instance of it actually happening.  I'll ask Brian Warner  
(who originally reported this issue to me, and who almost never uses  
Windows) about it.

It is probably a lot more common Windows, for various reasons,  
including the fact that Windows users are less likely to use ssh  
directly on a frequent basis, and the fact that some common  
installations of darcs on Windows have a separate (putty) ssh for  
darcs than the one that the user actually normal uses (e.g. cygwin ssh).

So, in short, +1 on passing stdout/stderr from ssh through on Windows.

By the way, I'm still not clear on what noisy I-didn't-need-to-see- 
that messages we will actually see, with modern darcs >= 2.0.0 on the  
server...

Regards,

Zooko

On Tue, Oct 21, 2008 at 09:34:02AM -0600, zooko wrote:
> By the way, I'm still not clear on what noisy
> I-didn't-need-to-see-that messages we will actually see, with modern
> darcs >= 2.0.0 on the server...

I think it's more common to see ugly messages without darcs-2 on the
server, but it was enough to scare users like SPJ...

David

Here's a user report of how darcs's behavior of hiding all ssh output from the 
user caused a problem for the user:

http://lists.osuosl.org/pipermail/darcs-users/2009-July/020374.html

We need to fix this.

I proposed a rough strategy in msg5901 based on some kind of fine-grained
filtering where we only pass through stuff we don't already recognise as noise.

If that's not going to be feasible, we should at least the last resort option of
being stderr noisy on Windows so people don't get mysterious hangs.

It'd be a good idea to check the Exec, External and SSH modules to figure out
what's going on right now.

I think this bit some folks on Haskell Cafe recently.  
http://www.haskell.org/pipermail/haskell-cafe/2010-March/074055.html

Now that we've got this program of picking out nice ProbablyEasy bugs
(for new Darcs hackers) to work on during hacking sprints, I nominate
this one for ZuriHac.

Removing the probably easy tag.

The problem with filtering the messages is that the only access to the
output of the external program is through a Handle type.  This is
opaque, so it's difficult to filter.

We experimented by using createProcess in conjunction with CreatePipe. 
The basic idea is simple:

  applySomeFilter :: (String -> String) -> IO ()
  applySomeFilter f = do
    (_,Just hout,_,_) <- createProcess (proc "cat" ["/dev/zero"]){
std_out = CreatePipe }
    readWrite hout stdout f
  
  readWrite :: Handle -> Handle -> (String -> String) -> IO ()
  readWrite from to f = do
    x <- hGetContents from
    hPutStr to (f x)

However, it gets very complicated in the general case.  For example,
what if putty/scp goes mad and outputs exceedingly long lines or binary
content?

Perhaps the easiest way forward is to pass through binary content, and
assume that text input is less than N characters per line.  This should
make it possible to filter out particular lines.

The following patch sent by Eric Kow <kowey@darcs.net> updated issue issue845 with
status=has-patch

* Resolve issue845: pass ssh stderr through. 
Ignore-this: 95fb98b084b3c09a360fa99e61099c25

There does not seem to be any reason to hide it, as we are using
the -q flag in pscp.

The following patch sent by Eric Kow <kowey@darcs.net> updated issue issue845 with
status=resolved;resolvedin=2.8.0 HEAD

* Resolve issue845: pass ssh stderr through. 
Ignore-this: 95fb98b084b3c09a360fa99e61099c25

There does not seem to be any reason to hide it, as we are using
the -q flag in pscp.