darcs

Issue 1503 prefer local caches to remote ones

Title prefer local caches to remote ones
Priority feature Status resolved
Milestone 2.5.0 Resolved in 2.5.0
Superseder Nosy List WorldMaker, abuiles, darcs-devel, dmitry.kurochkin, kowey, mornfall, thorkilnaur, zooko
Assigned To
Topics Hashed, Performance

Created on 2009-07-21.16:18:05 by zooko, last changed 2010-06-15.22:40:18 by abuiles.

Files
File name Uploaded Type Edit Remove
log builes.adolfo, 2010-05-25.21:59:31 application/octet-stream
log10 builes.adolfo, 2010-05-25.21:59:31 application/octet-stream
unnamed builes.adolfo, 2010-05-25.21:59:31 text/html
Messages
msg7976 (view) Author: zooko Date: 2009-07-21.16:18:02
from <name withheld>:

darcs scares me sometimes
10:13
administrator@SCBLR01SR001:~/trees/www$ darcs pull ../98.home.adm/www/
Pulling from "/home/administrator/trees/98.home.adm/www"...
The authenticity of host 'dev.allmydata.com (207.7.153.140)' can't be  
established.
RSA key fingerprint is 92:d2:54:be:66:cd:20:77:54:9a:d8:dd:ea:40:0a:22.
Are you sure you want to continue connecting (yes/no)? yes
Tue Jul 21 08:10:43 PDT 2009  withheld@allmydata.com
   * Record assorted changes to www code for scancafe deployment

10:15
why would pulling from localhost (relative path) cause it to ssh to  
another host?
10:15
(!)
msg7977 (view) Author: kowey Date: 2009-07-21.16:24:09
This reminds me of http://bugs.darcs.net/issue1153

From the second hackathon report:
Petr has developed a plan to solve these issues by ignoring bad cache
locations, warning about them, and deleting them when it is appropriate
to do so. He has begun studying the darcs caching source code to prepare
for future implementation work.

On Tue, Jul 21, 2009 at 16:18:05 +0000, Zooko wrote:
> administrator@SCBLR01SR001:~/trees/www$ darcs pull ../98.home.adm/www/
> Pulling from "/home/administrator/trees/98.home.adm/www"...
> The authenticity of host 'dev.allmydata.com (207.7.153.140)' can't be  
> established.
> RSA key fingerprint is 92:d2:54:be:66:cd:20:77:54:9a:d8:dd:ea:40:0a:22.
> Are you sure you want to continue connecting (yes/no)? yes
> Tue Jul 21 08:10:43 PDT 2009  withheld@allmydata.com
>    * Record assorted changes to www code for scancafe deployment
> 
> 10:15
> why would pulling from localhost (relative path) cause it to ssh to  
> another host?
> 10:15
> (!)

OK for some reason, darcs must have thought it need to fetch patches
that it had not previously obtained.

Could I just make sure this is a --lazy repo?
Also, what are the contents of _darcs/prefs/sources?
msg7981 (view) Author: WorldMaker Date: 2009-07-21.23:43:27
Zooko wrote:
> New submission from Zooko <zooko@zooko.com>:
> 
> from <name withheld>:
> 
> darcs scares me sometimes
> 10:13
> administrator@SCBLR01SR001:~/trees/www$ darcs pull ../98.home.adm/www/
> Pulling from "/home/administrator/trees/98.home.adm/www"...
> The authenticity of host 'dev.allmydata.com (207.7.153.140)' can't be  
> established.
> RSA key fingerprint is 92:d2:54:be:66:cd:20:77:54:9a:d8:dd:ea:40:0a:22.
> Are you sure you want to continue connecting (yes/no)? yes
> Tue Jul 21 08:10:43 PDT 2009  withheld@allmydata.com
>    * Record assorted changes to www code for scancafe deployment
> 
> 10:15
> why would pulling from localhost (relative path) cause it to ssh to  
> another host?

I would assume its an issue with a set of lazy repositories: 
~/trees/98.home.adm/www/ may have patches listed in inventory that it 
lacks in _darcs/patches/ (and which aren't cached locally for other 
repos either). For whatever reason darcs thinks it needs those patch 
files (or old tag inventory files?) to complete the pull so ~/trees/www 
probably has a _darcs/prefs/sources line that includes the 
dev.allmydata.com server.

It looks like darcs could use better status messages in such a case, but 
other than that I don't think there is a bug/issue here.
msg8032 (view) Author: kowey Date: 2009-08-06.14:43:50
I need a volunteer to produce a minimal scenario that could trigger this sort of
thing.  It's likely enough to use local repositories.

I imagine the scenario would involve something like

init --repo A
get A B # maybe --lazy
get --lazy B C
pull --repo C B
# all of a sudden it wants patches from A: why?
msg8747 (view) Author: kowey Date: 2009-09-07.21:05:38
Here's a minimal example showing how this can happen:

. ../tests/lib                  # Load some portability helpers.
rm -rf R S T log                # Another script may have left a mess.
darcs init --hashed --repo R    # Create our test repos.
darcs tag       --repo R -m 1
darcs get --lazy R S
darcs tag       --repo S -m 2
darcs get --lazy R T
darcs pull      --repo T S -a --debug --verbose 2> log
not grep "R/_darcs" log

[Note that this uses the darcs regression testing infrastructure, so to make
this work, save this as tests/issue1503.sh and run cabal test tests/issue1503.sh]

In this example the repo you are pulling to (T) was originally gotten from the
'remote' repo R.  Even thought you are pulling from S, the fact that R came
first in the cache meant that you tried it first.

Zooko: by any chance could you verify that this matches your use case?

I think a simple solution to this would be to sort the caches so that local ones
come first.
msg8750 (view) Author: zooko Date: 2009-09-07.23:14:01
Yes, that's probably the situation in which <name withheld> was scared by darcs.
msg8752 (view) Author: kowey Date: 2009-09-08.06:12:00
Thanks Zooko.  I'm going to downgrade this to a feature and submit the test case.
msg8753 (view) Author: kowey Date: 2009-09-08.06:14:50
I retract my offer to submit the test case :-) It's going to be tricky to
automate the business of checking that it's not looking at remote repo in the
suite.  You can probably dump something into test/network, or maybe prefer the
the repo we are actually pulling from to any other repo instead.  In any case, I
leave the test to the implementor.
msg10278 (view) Author: kowey Date: 2010-03-18.15:05:57
Perhaps we can fix this for Darcs 2.5?
msg11113 (view) Author: abuiles Date: 2010-05-25.21:59:31
After scratching my head a bit with this issue, I finally found what's going
on here, seems that what is causing all the  mess is the
_dacrs/prefs/sources file and the fact that they are tried before the
repository we are pulling (even if local),  so a work around which could
work is checking first if the repo we are pulling is local, and then adding
it to the sources file, also we have to notice that It tries from top to
bottom, so, why we don't keep the sources files sorted ?  keeping locals
first and so on. (I'm still asking if   this would break something ?  )

I used the test cases Eric put before and put the R repo in a server,
(trying to make it look to the original report).

I'm attaching two files, in the file "log", you can see how it goes to the
server looking for the missing patches/inventories :(.. then I manually
wrote down in the sources file the repo S, and you can see that the result
is different when trying to pull (file "log10") ,  it  looks like this :

abuiles@abuiles-laptop:~/linkProg/gsoc2010/testRepo$ cat
T/_darcs/prefs/sources
cache:/home/abuiles/.darcs/cache
thisrepo:/media/my-disk/stuff/adolfo/programacion/gsoc2010/testRepo/T
repo:/media/my-disk/stuff/adolfo/programacion/gsoc2010/testRepo/S
repo:acadavid@lizarus.com:R
Attachments
msg11156 (view) Author: kowey Date: 2010-05-29.14:25:03
I doubt there's any harm in sorting the sources file (on disk).  It
seems that at the very least, we should sort our internal listing of the
sources.

It's also worth checking/documenting what the interaction is between the
sources list mechanism, the default repo and any repos passed in on the
command line.

Note: I noticed an unrelated error calling darcs transfer-mode and
opened issue1854
msg11217 (view) Author: abuiles Date: 2010-06-03.16:54:51
The following patch updated the status of issue1503 to be resolved:

* Resolve issue1503: prefer local caches to remote ones 
Ignore-this: 8e7af0e4d04a8e58b819bbe93817d681
msg11456 (view) Author: abuiles Date: 2010-06-15.22:40:17
The following patch updated issue issue1503 with status=resolved;resolvedin=2.5.0 (current)

* Resolve issue1503: prefer local caches to remote ones 
Ignore-this: 8e7af0e4d04a8e58b819bbe93817d681
History
Date User Action Args
2009-07-21 16:18:05zookocreate
2009-07-21 16:24:12koweysetstatus: unread -> unknown
nosy: + mornfall
messages: + msg7977
2009-07-21 23:43:30maxbattchersetnosy: + maxbattcher, darcs-devel
messages: + msg7981
2009-08-06 14:43:52koweysetpriority: bug
status: unknown -> needs-reproduction
messages: + msg8032
nosy: kowey, darcs-devel, zooko, maxbattcher, simon, thorkilnaur, dmitry.kurochkin, mornfall
2009-08-25 17:46:20adminsetnosy: - simon
2009-08-27 14:21:11adminsetnosy: kowey, darcs-devel, zooko, maxbattcher, thorkilnaur, dmitry.kurochkin, mornfall
2009-09-07 21:05:43koweysetstatus: needs-reproduction -> waiting-for
nosy: kowey, darcs-devel, zooko, maxbattcher, thorkilnaur, dmitry.kurochkin, mornfall
topic: + Hashed
messages: + msg8747
assignedto: zooko
2009-09-07 23:14:03zookosetnosy: kowey, darcs-devel, zooko, maxbattcher, thorkilnaur, dmitry.kurochkin, mornfall
messages: + msg8750
2009-09-08 03:31:37koweysetstatus: waiting-for -> unknown
nosy: kowey, darcs-devel, zooko, maxbattcher, thorkilnaur, dmitry.kurochkin, mornfall
assignedto: zooko ->
2009-09-08 06:12:02koweysetpriority: bug -> feature
status: unknown -> needs-implementation
messages: + msg8752
nosy: kowey, darcs-devel, zooko, maxbattcher, thorkilnaur, dmitry.kurochkin, mornfall
title: why would pulling from localhost (relative path) cause it to ssh to another host? -> prefer local caches to remote ones
2009-09-08 06:14:52koweysetnosy: kowey, darcs-devel, zooko, maxbattcher, thorkilnaur, dmitry.kurochkin, mornfall
messages: + msg8753
2009-09-08 06:15:07koweysettopic: + Performance
nosy: kowey, darcs-devel, zooko, maxbattcher, thorkilnaur, dmitry.kurochkin, mornfall
2009-10-24 00:00:01adminsetnosy: + WorldMaker, - maxbattcher
2010-03-18 15:06:04koweysettopic: + Target-2.5
messages: + msg10278
2010-05-25 22:04:29adminsetnosy: + builes.adolfo
messages: + msg11113
2010-05-25 22:06:04adminsetfiles: + unnamed, log, log10
2010-05-25 22:07:12koweylinkissue1852 superseder
2010-05-29 14:25:04koweysetmessages: + msg11156
2010-06-03 16:54:51builes.adolfosetstatus: needs-implementation -> resolved
messages: + msg11217
2010-06-06 12:52:44adminsetnosy: + abuiles, - builes.adolfo
2010-06-15 20:52:10adminsetmilestone: 2.5.0
2010-06-15 20:59:49adminsettopic: - Target-2.5
2010-06-15 22:40:18abuilessetmessages: + msg11456
resolvedin: 2.5.0