Patch 173 Work in progress: patch index

Title Work in progress: patch index
Superseder Nosy List beschmi, bsrkaditya, galbolle, ganesh
Related Issues
Status obsoleted Assigned To beschmi

Created on 2010-03-09.07:12:25 by beschmi, last changed 2013-02-18.09:34:22 by gh. Tracked on DarcsWatch.

File name Status Uploaded Type Edit Remove
add-ghci-target-_do-not-push_.dpatch beschmi, 2010-03-21.17:00:59 text/x-darcs-patch
add-support-for-filecache-_optimize-and-changes-for-now-only_.dpatch beschmi, 2010-03-09.07:12:24 text/x-darcs-patch
added-files-filemodmonad_-filemod_-filemodtypes.dpatch bsrkaditya, 2012-02-29.00:07:45 application/octet-stream
added-files-filemodmonad_-filemod_-filemodtypes.dpatch bsrkaditya, 2012-03-01.04:32:27 application/octet-stream
changes.dpatch bsrkaditya, 2012-03-31.15:42:53 application/octet-stream
sync.dpatch bsrkaditya, 2012-04-01.10:39:20 application/octet-stream
unnamed beschmi, 2010-03-09.07:12:24 text/plain
unnamed beschmi, 2010-03-21.17:00:59 text/plain
See mailing list archives for discussion on individual patches.
msg10134 (view) Author: beschmi Date: 2010-03-09.07:12:24
This is a preview of the filecache work I'm doing. You can test it
with "darcs optimize --filecache" and then use the cache with
"darcs changes --filecache GNUmakefile". There are still some things
I have to investigate.

1 patch for repository http://www.darcs.net/repos/unstable:

Mon Mar  8 21:54:50 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * add support for filecache (optimize and changes for now only)
msg10304 (view) Author: ganesh Date: 2010-03-19.22:45:12
I'm marking this as "amend-in-progress" as it's not intended to be pushed 
yet. Someone else jump in and correct me if that's an incorrect use of 
the bugtracker.
msg10401 (view) Author: beschmi Date: 2010-03-21.17:00:59
Just a snapshot for anyone interested.

13 patches for repository /Users/beschmi/darcs/unstable:

Tue Mar  9 20:28:39 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * add ghci target (do not push)

Tue Mar  9 21:52:02 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * add support for filecache (optimize and changes for now only)

Wed Mar 17 01:06:30 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * fix comment

Wed Mar 17 01:08:03 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * add ghci target to Makefile

Wed Mar 17 01:10:31 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * add preliminary support for filecache to annotate

Sat Mar 20 16:40:46 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * remove unneeded module import

Sat Mar 20 16:41:39 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * improve FileMod modules

Sat Mar 20 17:30:55 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * fixes for filecache^Wpatch index

Sat Mar 20 23:57:56 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * disable trace in FileModMonad

Sat Mar 20 23:59:10 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * various improvements to patch index (sets instead of lists, directory name)

Sun Mar 21 01:11:55 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * some more filecache to patch index replacements

Sun Mar 21 01:35:49 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * commit before modifying the storage format

Sun Mar 21 16:30:08 CET 2010  Benedikt Schmidt <beschmi@gmail.com>
  * use Data.Binary
msg12367 (view) Author: dagit Date: 2010-08-28.23:28:59
I started reviewing this (I'm also happy to finish it off if Benedikt is busy).

I noticed this comment in the code:
-- | The latest Patch that touches the file is always the first in the list
type TouchingMap = Map FileId (Set PatchId)

I was skeptical that (Set PatchId) would keep things sorted correctly in all cases.  I looked at how 
Set would be storing things:
-- | The PatchId identifies a patch and can be created from a PatchInfo with make_filename
type PatchId = String

So the sort order comes down to string sorting.  The next question is, how are the PatchIds created?  
The comment in other places say to use make_filename, which is now renamed to makeFilename:
-- This makes darcs-1 (non-hashed repos) filenames, and is also generally used in both in
-- hashed and non-hashed repo code for making patch "hashes"
makeFilename :: PatchInfo -> String
makeFilename pi =
    showIsoDateTime d++"-"++sha1_a++"-"++sha1PS sha1_me++".gz"
        where b2ps True = BC.pack "t"
              b2ps False = BC.pack "f"
              sha1_me = B.concat [_piName pi,
                                  _piAuthor pi,
                                  _piDate pi,
                                  B.concat $ _piLog pi,
                                  b2ps $ isInverted pi]
              d = readPatchDate $ _piDate pi
              sha1_a = take 5 $ sha1PS $ _piAuthor pi

The interesting and relevant bit is that the patchids start with an ISO Date/Time.

Because patches can commuted and stored in a different order than the order they are created in, the 
type of TouchingMap means sometimes it may be possible for the patch index will store touching files 
in the wrong order.

I'll see if I can come up with an example that would be wrong with (Set PatchId) and correct with 
[PatchId].  Of course, the "trivial" example is when patches that touch the same file are created at 
nearly the same time on different machines.  In that scenario, one machine could have clock that is 
behind the other.  If it the clocks were enough different you could imagine the patches being in the 
wrong order in the index.
msg12374 (view) Author: beschmi Date: 2010-08-30.00:35:19
Hi Jason,

On 8/29/10 1:28 , Jason Dagit wrote:

> Jason Dagit<no-reply@example.com>  added the comment:
> I started reviewing this (I'm also happy to finish it off if Benedikt is busy).

I'm happy that someone is looking into this and it is a good motivation
to publish a new patch that is easier to consume for reviewers. I can
publish my darcs-repo which has been forked from unstable in march if
you (or someone else) wants to look at the code before I get around to
rebasing everything to current unstable.

 > I noticed this comment in the code:
 > -- | The latest Patch that touches the file is always the first in 
the list
 > type TouchingMap = Map FileId (Set PatchId)

The code changed quite a bit since I sent the last patch, so I don't
remember all the details anymore. Nevertheless, I'm pretty sure that
the code never assumed that the alphabetic order of patchids has any
relation to the order of patches in the repo. So the comment is clearly
wrong and might be left over from an older version of the code.

The main idea at the time was that you have to read the inventory
anyways, so that's where you get the list of patch-infos in the right
order and the TouchingMap is just used to filter out the relevant

msg14542 (view) Author: galbolle Date: 2011-06-17.10:27:46
Benedikt, can you publish the repository with your work in progress? Are
you ok with someone else porting it to the current HEAD of darcs?
msg15191 (view) Author: bsrkaditya Date: 2012-02-29.00:07:45
I have ported patch index to the current screened.
Create a patch index with darcs optimize --patch-index,
and show the patch index with darcs show patch-index-all.
You can also view the files in patch index with darcs show patch-index-
msg15195 (view) Author: mndrix Date: 2012-02-29.23:45:21
On Tue, Feb 28, 2012 at 5:07 PM, BSRK Aditya <bugs@darcs.net> wrote:
> BSRK Aditya <bsrkaditya@gmail.com> added the comment:
> I have ported patch index to the current screened.

Thanks for porting this patch.  It applied cleanly to a fresh copy of
screened.  However, I get the following build error:

    Could not find module `Darcs.Commands.ShowPatchIndex'

It looks like src/Darcs/Commands/ShowPatchIndex.hs is missing from the
patch bundle.

msg15197 (view) Author: bsrkaditya Date: 2012-03-01.04:32:27
Ah! I missed that file. Use the patch I attached here instead. I 
confirmed that it builds.
msg15455 (view) Author: bsrkaditya Date: 2012-03-31.15:42:53
The last patch(dated 2012-03-01.04:32:27) now has conflicts with latest
screened. I am uploading a new patch with the conflict resolved.
msg15472 (view) Author: bsrkaditya Date: 2012-04-01.10:38:27
This patch again removes the conflicts introduced to screened
msg15473 (view) Author: bsrkaditya Date: 2012-04-01.10:39:20
Sorry, I mailed the wrong patch.
msg16691 (view) Author: gh Date: 2013-02-18.09:34:22
This has been integrated in darcs since the end of BSRK's summer of code 
Date User Action Args
2010-03-09 07:12:25beschmicreate
2010-03-13 20:25:49darcswatchsetdarcswatchurl: http://darcswatch.nomeata.de/repo_http:__darcs.net_.html#bundle-b0e56ebe2bce5ebdcf0f984e3a3daca67ed9611b
2010-03-19 22:45:12ganeshsetstatus: needs-review -> followup-in-progress
nosy: + ganesh
messages: + msg10304
2010-03-21 17:01:03beschmisetfiles: + add-ghci-target-_do-not-push_.dpatch, unnamed
messages: + msg10401
title: add support for filecache (optimize and changes for no... -> Work in progress: patch index
2010-03-21 17:03:57darcswatchsetdarcswatchurl: http://darcswatch.nomeata.de/repo_http:__darcs.net_.html#bundle-b0e56ebe2bce5ebdcf0f984e3a3daca67ed9611b -> http://darcswatch.nomeata.de/repo_http:__darcs.net_.html#bundle-f9ef1a22192c3cd61def990dfc604334ec0041a6
2010-03-22 17:51:46koweysetassignedto: beschmi
2010-08-28 23:28:59dagitsetmessages: + msg12367
2010-08-29 10:03:20koweysetnosy: - darcs-users
2010-08-30 00:35:19beschmisetmessages: + msg12374
2011-05-10 19:36:14darcswatchsetdarcswatchurl: http://darcswatch.nomeata.de/repo_http:__darcs.net_.html#bundle-f9ef1a22192c3cd61def990dfc604334ec0041a6 -> http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-f9ef1a22192c3cd61def990dfc604334ec0041a6
2011-05-10 22:05:42darcswatchsetdarcswatchurl: http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-f9ef1a22192c3cd61def990dfc604334ec0041a6 -> http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-b0e56ebe2bce5ebdcf0f984e3a3daca67ed9611b
2011-06-17 10:27:46galbollesetmessages: + msg14542
2011-06-17 10:29:35galbollesetnosy: + galbolle
2012-02-29 00:07:45bsrkadityasetfiles: + added-files-filemodmonad_-filemod_-filemodtypes.dpatch
nosy: + bsrkaditya
messages: + msg15191
2012-02-29 00:09:04darcswatchsetdarcswatchurl: http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-b0e56ebe2bce5ebdcf0f984e3a3daca67ed9611b -> http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-c52e323788300c17839c037f72b44dd8cc3bf31f
2012-02-29 23:45:22mndrixsetmessages: + msg15195
2012-03-01 04:32:27bsrkadityasetfiles: + added-files-filemodmonad_-filemod_-filemodtypes.dpatch
messages: + msg15197
2012-03-01 04:32:56darcswatchsetdarcswatchurl: http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-c52e323788300c17839c037f72b44dd8cc3bf31f -> http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-aaf65916af47694c025c301cd1d7458922bb1693
2012-03-15 23:15:32mndrixlinkpatch428 superseder
2012-03-31 15:42:53bsrkadityasetfiles: + changes.dpatch
messages: + msg15455
2012-03-31 15:43:19darcswatchsetdarcswatchurl: http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-aaf65916af47694c025c301cd1d7458922bb1693 -> http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-9d2ac4c019686feac4f14a35648fe601289bfb95
2012-04-01 10:38:27bsrkadityasetfiles: + changes.dpatch
messages: + msg15472
2012-04-01 10:38:55bsrkadityasetfiles: - changes.dpatch
2012-04-01 10:39:20bsrkadityasetfiles: + sync.dpatch
messages: + msg15473
2012-04-01 10:40:21darcswatchsetdarcswatchurl: http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-9d2ac4c019686feac4f14a35648fe601289bfb95 -> http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-ef26d73814fba69c7d70a1e619d0aecd1f50d553
2013-02-18 09:34:23ghsetstatus: followup-in-progress -> obsoleted
messages: + msg16691