darcs

Issue 390 wish: "darcs whatsnew somefile" lstats every file in the working copy and pristine/ directory

Title wish: "darcs whatsnew somefile" lstats every file in the working copy and pristine/ directory
Priority wishlist Status resolved
Milestone Resolved in
Superseder Nosy List SamB, darcs-devel, dmitry.kurochkin, jch, kowey, markstos, spookylukey, thorkilnaur, tommy, tux_rocker
Assigned To tux_rocker
Topics Confirmed, IncludesExampleOrTest, Performance

Created on 2007-01-12.16:10:58 by spookylukey, last changed 2009-08-27.14:20:57 by admin.

Messages
msg1406 (view) Author: spookylukey Date: 2007-01-12.16:10:56
I do "darcs whatsnew somefile" and the performance is terrible (this is working 
on a remote filesystem that is mounted locally, so it massively exaggerates any 
problems).  Doing an strace on this simple operation produces a huge file, 
which appears to be doing lstat64 on every file in the _darcs/pristine/ and the 
working copy.

I can't think of any reason why that would be necessary, especially since I 
specifically asked for just one file.

In a bash script:

#!/bin/bash
mkdir mytempdir || exit 1
cd mytempdir
darcs init
touch file1 file2 file3 file4
darcs add file1 file2 file3 file4
darcs record -am "test"
# Expect the following to produce no output:
strace darcs whatsnew file1 2>&1 > /dev/null | egrep 'file2|file3|file4'
msg1421 (view) Author: droundy Date: 2007-01-14.16:48:12
On Fri, Jan 12, 2007 at 04:10:58PM +0000, Luke Plant wrote:
> I do "darcs whatsnew somefile" and the performance is terrible (this is working 
> on a remote filesystem that is mounted locally, so it massively exaggerates any 
> problems).  Doing an strace on this simple operation produces a huge file, 
> which appears to be doing lstat64 on every file in the _darcs/pristine/ and the 
> working copy.
> 
> I can't think of any reason why that would be necessary, especially since I 
> specifically asked for just one file.

Just sloppy programming.  We compute the entire set of changes first, and
then select the ones affecting that file.  In our defense, there are
slightly tricky cases involving renames where simply looking at one file in
both pristine and working directory wouldn't give the right answer.  But
yeah, it's just sloppy (lazy) programming.
-- 
David Roundy
http://www.darcs.net
msg1425 (view) Author: jch Date: 2007-01-15.22:27:19
> I do "darcs whatsnew somefile" and the performance is terrible (this
> is working on a remote filesystem that is mounted locally, so it
> massively exaggerates any problems).  Doing an strace on this simple
> operation produces a huge file, which appears to be doing lstat64 on
> every file in the _darcs/pristine/ and the

Known issue.  The Slurpy code needs to be refined.

Note that there's no issue on a sane local filesystem, as the i-node
cache (or your OS's equivalent) will hide the problem.  However, it
will bite you badly on a remote filesystem that has synchronous
semantics for stat.

                                        Juliusz
msg3182 (view) Author: markstos Date: 2008-02-07.17:03:54
I'll retest this was the latest darcs2 build to confirm if this still an issue.
msg3226 (view) Author: markstos Date: 2008-02-08.03:33:42
I just sent a patch to add bugs/whatsnew.pl, which confirms that this bug still
exists. Using a hashed format presents the pristine stat calls to the un-used
files, but 'stat' is still called on extra files in the working directory.
msg4225 (view) Author: SamB Date: 2008-04-10.23:48:28
This isn't specific to "whatsnew", is it? It seems like every command that can 
be restricted to operate on certain files slurps in the whole pristine tree 
regardless (at least, based on the progresss message), at least in a hashed 
repo. For instance: "revert", "record"...
msg6473 (view) Author: tux_rocker Date: 2008-10-26.19:56:24
I now pushed a fix that is limited to whatsnew's scanning the working copy to
the sprint branch. Judging by what uses the
Darcs.Repository.Internal.get_unrecorded* family of functions, we can use it for
the "changes", "mark-conflicts", "whatsnew", "revert", "unrevert", "record",
"unrecord", "remove" and "amend-record" commands too. 

I have not done anything yet about the amount of stats on the pristine.
msg6475 (view) Author: kowey Date: 2008-10-26.22:00:30
For those following this, tux_rocker is referring to the temporary branch used
during the darcs hacking sprint of 2008-10-25

http://code.haskell.org/darcs/sprint

We'll be working on getting these patches into mainline darcs soon!
msg8001 (view) Author: tux_rocker Date: 2009-08-03.19:02:16
All my work on the working directory stats is now in, and the pristine access
should be taken care of by Petr's hashed-storage work.
History
Date User Action Args
2007-01-12 16:10:58spookylukeycreate
2007-01-14 16:48:13droundysetstatus: unread -> unknown
nosy: droundy, tommy, beschmi, kowey, spookylukey
messages: + msg1421
2007-01-15 22:27:21jchsetnosy: + jch
messages: + msg1425
2008-02-07 17:03:57markstossettopic: + Performance
nosy: + markstos
title: "darcs whatsnew somefile" lstats every file in the working copy and pristine/ directory -> wish: "darcs whatsnew somefile" lstats every file in the working copy and pristine/ directory
messages: + msg3182
assignedto: markstos
2008-02-08 03:33:45markstossetstatus: unknown -> deferred
nosy: droundy, jch, tommy, beschmi, kowey, markstos, spookylukey
topic: + Confirmed, IncludesExampleOrTest
messages: + msg3226
assignedto: markstos -> (no value)
2008-04-10 23:48:33SamBsetnosy: + SamB
messages: + msg4225
2008-05-14 14:00:04koweysetstatus: deferred -> unknown
nosy: + dagit
2008-10-25 10:34:06tux_rockersetnosy: + dmitry.kurochkin, tux_rocker, simon, thorkilnaur
assignedto: tux_rocker
2008-10-26 19:56:32tux_rockersetnosy: droundy, jch, tommy, beschmi, kowey, markstos, dagit, SamB, simon, spookylukey, thorkilnaur, tux_rocker, dmitry.kurochkin
messages: + msg6473
2008-10-26 19:56:54tux_rockersetstatus: unknown -> has-patch
nosy: droundy, jch, tommy, beschmi, kowey, markstos, dagit, SamB, simon, spookylukey, thorkilnaur, tux_rocker, dmitry.kurochkin
2008-10-26 22:00:37koweysetnosy: droundy, jch, tommy, beschmi, kowey, markstos, dagit, SamB, simon, spookylukey, thorkilnaur, tux_rocker, dmitry.kurochkin
messages: + msg6475
2009-08-03 19:02:24tux_rockersetstatus: has-patch -> resolved
nosy: droundy, jch, tommy, beschmi, kowey, markstos, dagit, SamB, simon, spookylukey, thorkilnaur, tux_rocker, dmitry.kurochkin
messages: + msg8001
2009-08-06 17:45:08adminsetnosy: + jast, Serware, darcs-devel, zooko, mornfall, - droundy, jch, SamB, spookylukey, tux_rocker
2009-08-06 20:41:16adminsetnosy: - beschmi
2009-08-10 22:00:57adminsetnosy: + tux_rocker, SamB, jch, spookylukey, - darcs-devel, zooko, jast, Serware, mornfall
2009-08-10 23:59:04adminsetnosy: - dagit
2009-08-25 17:32:22adminsetnosy: + darcs-devel, - simon
2009-08-27 14:20:57adminsetnosy: jch, tommy, kowey, markstos, darcs-devel, SamB, spookylukey, thorkilnaur, tux_rocker, dmitry.kurochkin