On a repository with a lot of patches, asking for darcs changes on a particular
file or directory can take a *long* time (30+ seconds of CPU time is common in
the GHC repository).

This is preventing us from using the darcs browser in Trac, because it
automatically runs darcs changes in this way.  Even the darcsweb browser isn't
really usable with GHC.

If I could wish for only one thing in darcs, faster 'darcs changes <file>' would
be it.  I imagine that in order to do this you have to store the files changed
along with each patch in the inventory, but once you have this a lot of other
operations could be sped up too (darcs annotate would be my #2 wish).  Also,
wouldn't this let you do 'darcs changes <file>' on a partial repository, as long
as you don't also say -v?

On Fri, Feb 03, 2006 at 10:54:59AM +0000, Simon Marlow wrote:
> On a repository with a lot of patches, asking for darcs changes on a
> particular file or directory can take a *long* time (30+ seconds of CPU
> time is common in the GHC repository).
> 
> This is preventing us from using the darcs browser in Trac, because it
> automatically runs darcs changes in this way.  Even the darcsweb browser
> isn't really usable with GHC.
> 
> If I could wish for only one thing in darcs, faster 'darcs changes
> <file>' would be it.  I imagine that in order to do this you have to
> store the files changed along with each patch in the inventory, but once
> you have this a lot of other operations could be sped up too (darcs
> annotate would be my #2 wish).

Actually, I'd prefer to use a separate file to list the files and
directories touched by each patch (or perhaps listing the patches touching
each file or directory?).  That way older darcs would still be able to pull
from a new-format repository (but would be unable to write to it, so the
file-touching cache would stay up-to-date).

This is definitely a good idea, that's been procrastinated because darcs is
already fast enough for most repositories (i.e. the ones that I use for
darcs itself, or at work).

It'd be moderately easy to do this.  We'd just need to make
Repository.updateInventory write to the new file (or files) also.  You'd
need to look inside the PatchToken (which will hold the filename of the
actual patch, which has been written, and which you can then read).  We'd
also need to add to RepoFormat a new RepoProperty, which would be
FilesModifiedCache or something.  Then a repo that has such a cache would
have a _darcs/format file containing something like

"darcs-1.0|files-modified-cache"

which would mean that any darcs that understands darcs-1.0 can read the
repo, but only those that understand files-modified-cache could write to
it.  So then updateInventory would check if we have the FilesModifiedCache
format and if so would write to the cache which files are modified by this
patch.

Of course, then changes, etc., would all need to be rewritten to take
advantage of this info, which could be tedious, but could be done gradually
once the data is actually stored.  I think I might actually lean towards
storing a list of patches for each file or directory rather than the other
way around, but it's hard to say.

We'd also need a way to add this cache to a repo, presumably via darcs
optimize.  And perhaps the same flag for darcs get and initialize.  Or
would this be something we'd always want by default? It really doesn't take
up much space or time, so perhaps all new repos should include this data?

> Also, wouldn't this let you do 'darcs changes <file>' on a partial
> repository, as long as you don't also say -v?

Indeed, your way of doing it would allow this.  If we did it using my
caching trick, I'm not sure whether this would work.  It would depend if we
grab the remote files-touched information on darcs get --partial.
-- 
David Roundy
http://www.darcs.net

Simon Marlow:

>> I imagine that in order to do this you have to store the files
>> changed along with each patch in the inventory,

David Roundy:

> Actually, I'd prefer to use a separate file to list the files and
> directories touched by each patch (or perhaps listing the patches touching
> each file or directory?).

Why's that?  We are planning to rework the inventory format, we might
as well make it extensible.

I'm thinking of an inventory format that starts with a line of the form

  darcs-inventory-1.0 keyword1 keyword2 keyword3

which contains all the keywords that are stored in the inventory.
Then the inventory itself will contain entries of the form

  patch-id keyword1=... keyword2=... keyword3=...

and we could initially have keywords such as

  patch-hash                   (discussed at FOSDEM)
  contents-hash                (   --         --   )
  touched-files                (proposed by Simon)

all of which would be optional.

There's just one thing that's not clear to me: how would you deal with
rename patches?

                                        Juliusz

Darcs should index and/or cache its metadata so that commands such as 'darcs
changes [file]' and 'darcs annotate [file]' are fast.  These commands should run
near-instantaneously.

I have a large darcs repo (> 500 MB including a few binaries), and running
either of the commands means darcs uses 100% of the CPU and takes anywhere
between 5 and 10 minutes. It also happens to consume the better part of 2gig of RAM.

It should be noted that darcs-server and darcsweb are both essentially built on
top of those commands and would therefore benefit from having them sped up.

I'm also interested in the possibility of using caching to improve performance
in these cases.

Setting status deferred, as this is a wishlist issue.

Best regards
Thorkil

Sorry, deferring was a mistake. Setting status need-info instead, asking for a 
discussion of whether we should work towards supporting this feature.

Thanks and best regards
Thorkil

On Tue, 13 Jan 2009 11:17:10 -0000
Thorkil Naur <bugs@darcs.net> wrote:

> 
> Thorkil Naur <naur@post11.tele.dk> added the comment:
> 
> Sorry, deferring was a mistake. Setting status need-info instead, asking for a 
> discussion of whether we should work towards supporting this feature.

I would like to to see this supported, and I believe it is one of the places
that is a common spot for users to observe that darcs is "slow".

I use this command somewhat regularly. (Although it is usually "fast enough"
for me).

I would like to see it addressed as part of a broader focus on performance of
key areas.

    Mark

If I understand correctly, this would be addressed by the current filecache work
in issue984

Re-setting dependency specifically to a filecache ticket. 'Deferred' here just
means "we'll fix issue1566 first and then come and look at this".

Per kowey's last comment this issue depends on issue1566.  However, this
issue currently has an earlier milestone (2.5.0) than issue1566 (which
has 2.6.0).  That seems inconsistent.

Milestone bumped accordingly (but we're going to try hard to get the 
patchindex stuff in as soon after Darcs 2.5 is out as possible)

I think this is resolved now, since the PatchIndex work has landed.

Yes.