Darcs is now unusable for my co-worker Rob.  He has started using rsync to
transfer between systems.  There's always the possibility that he will go back
to using SVN.  Much of the analysis below is suspect because this happens only
on Windows as far as I know.

--Z

<zooko> [11:32:35] rob_kinninmont@mac.com: push's been running for 10 min.
<zooko> [11:32:46] rob_kinninmont@mac.com: I wish it were over.
<zooko> [11:32:51] zooko2: The #darcs folks want to know what's the message
	that it prints out.
<zooko> [11:32:54] zooko2: Something about copying a file.
<zooko> [11:33:14] zooko2: They are investigating.
<zooko> [11:33:28] rob_kinninmont@mac.com: this message?
<zooko> [11:33:29] rob_kinninmont@mac.com: darcs4b3406 | 0 kB | 1.0 kB/s |
	ETA: 00:00:00 | 100%
<zooko>  darcsf8004a | 0 kB | 0.2 kB/s | ETA: 00:00:00 | 100%
<zooko>  darcs726601 | 0 kB | 0.5 kB/s | ETA: 00:00:00 | 100%
<zooko>  darcs4ee832 | 6 kB | 6.9 kB/s | ETA: 00:00:00 | 100%
<zooko>  darcs2dd35b | 1 kB | 1.8 kB/s | ETA: 00:00:00 | 100%
<zooko>  darcs1a8291 | 1 kB | 1.5 kB/s | ETA: 00:00:00 | 100%
<zooko>  darcs2eeb7e | 0 kB | 0.0 kB/s | ETA: 00:00:00 | 100%
<zooko>  darcs2bdb50 | 0 kB | 0.2 kB/s | ETA: 00:00:00 | 100%
<zooko>  darcs1ad51c | 0 kB | 0.3 kB/s | ETA: 00:00:00 | 100%
<zooko>  darcs1cf758 | 1 kB | 1.1 kB/s | ETA: 00:00:00 | 100%
<zooko>
<zooko> [11:33:35] rob_kinninmont@mac.com: pages and pages of it.
<zooko>
<zooko> Note that the number of lines grows apparently linearly with how long
	we've been adding patches to our project.
<zooko> So it runs slower every day for Rob.
<kowey> huh... that's ssh that's doing that, i think
<kowey> normally, when darcs calls ssh, we send the output to /dev/null
<kowey> s/ssh/scp/ in this case
<zooko> Yes.
<zooko> So apparently darcs calls ssh O(n) times where n is the number of
	patches.
<zooko> scp I mean.
<kowey> ok, so it's not the output that's the problem, but the fact that it's
	doing all these scp - right?
<zooko> Right.
<kowey> ok... so i think i've got an analysis for the problem -- this doesn't
	mean i want to fix it, but maybe we can write this in a bug report
<zooko> Ok.
<kowey> the thing is that darcs push has to get (minimally) a list of patches
	from the other side, some of the patches (i don't know which, and why)
	in order
<kowey> to make a comparison as to what patches the other guy has and what
	patches you've got
<kowey> to do this, it calls read_repo, which does 2 things
<kowey> 1) it reads the inventory
<kowey> 2) in a NON-LAZY fashion, it fetches all the patches in _darcs/patches
<arjanb> it should need only the inventory for the comparison part..
<kowey> maybe... instead of calling read_repo, we can call lazily_read_repo
	(or just fetch the inventory)
<kowey> does the local end have to do any commutation type stuff? i doubt
	it... i thought push was just sending the stuff over and asking the
	other side to do apply?
<zooko> As far as I know.
<zooko> It should just (a) figure out which patches to prompt the user about,
<zooko> (b) interact with the user,
<zooko> (c) send those patches (including required patches that the other side
	lacks) over to the other side.
<kowey> the problem is that (a) is harder than it looks, i'll bet
<kowey> maybe it is impossible to tell what patches you and the other side
	have in common without actually getting the patches themselves
<zooko> Hm.
<kowey> (i hope it isn't, of course)
<zooko> Ah yes.
<zooko> This is the part of darcs basic architecture that I don't like.
<zooko> It computes this kind of metadata -- which patches depend on which --
	at runtime, from the patches.
<zooko> Rather than at patch-creation time and then storing the metadata as
	metadata.
* kowey suspects that the metadata is often in the interaction between patches
<kowey> and not the individual patches themselves
<kowey> that being said, we might be able to store such metadata as well... a
	bit like chart parsing
<zooko> No, the question of "On which other patches does this patch depend?"
	can be answered using only information present at the time the patch
	is created.
<zooko> What patches I have in my repo, later, when pulling, cannot change the
	answer to that question.
<zooko> At least in my idealized world.

I had similar complaints from users, here, and decided to expose the filesystem 
itself as a samba share and asked people to push and pull to that, instead, 
which resulted in a massive speed increase; operations that used to take 
minutes take a few seconds. Of course, Samba will be less secure than a well-
configured SSH-for-Darcs setup.

Some of Eric Kow's recent patches look related - they use recent features of 
OpenSSH to use fewer SSH connections. As I understand it, you can use recent 
OpenSSH versions via Cygwin, though I haven't tried.

Finally, on the matter of finding the dependencies of patches, there are 
algorithms for such that have been given on the darcs-conflicts mailing list. 
However, one of the advantages of not doing so is that you can save time by not 
calculating dependencies until you need to know about them.

(I'm forced to set a priority, in order to submit a message, so I'm setting it 
to 'bug'; I hope that's not a problem.)

On Tue, Mar 21, 2006 at 04:09:37PM +0000, Zooko wrote:
> Darcs is now unusable for my co-worker Rob.  He has started using rsync
> to transfer between systems.  There's always the possibility that he will
> go back to using SVN.  Much of the analysis below is suspect because this
> happens only on Windows as far as I know.

I've only skimmed this report, but am wondering if your coworker could try
introducing a tag, pulling that tag into the other repositories (so it's in
both) and then running darcs optimize --reorder in both repositories.  I
have a suspicion that this will fix the slowness.
-- 
David Roundy
http://www.darcs.net

Zooko,

How slow is it?  If we're speaking about more than a second per scp,
then you've probably got broken reverse DNS.  If that's the case, I
suggest you check UseDNS in the sshd_config file.

I also second David's suggestion, although I suspect it shouldn't matter
much on a local network.

Marking this bug as a feature request, and will mark it as
resolved-in-unstable when I push Eric's patches.

                                        Juliusz

I'm assuming this is just Issue 32 combined with improper reverse DNS.
I'm therefore marking it as resolved-in-unstable.

If you disagree, please revert that to chatting.