This patch implement --look-for-moves flag for amend-record, record and whatsnew commands. The resolve dependencies code is a little too messy and i don't know if it is completely right. The test cases are incomplete but i'm confident it should work in most if not all of the test cases that are missing(i have to do more testing).
I don't know if the same code could function without unsafeCoerceP.
To make amend-record work right when used twice in a row i have to delete the pending patch generated when the patch is modified. I don't know if this is desirable.
I will love to hear any corrections you have about the code.
It's needed a patch to hashed-storage too.
1 patch for repository http://darcs.net/screened:

Sat Aug 10 01:37:54 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * resolve issue642: Automatic detection of file renames
  Implement --look-for-moves flag for amend-record, record and whatsnew commands.

Attachments

• patch-preview.txt
• resolve-issue642_-automatic-detection-of-file-renames.dpatch
• unnamed

I'm having problems sending the patch because mornfall repository is
down. The repository with the patch needed to compile hashed-storage
with support for fileids is in
http://hub.darcs.net/jlneder/hashed-storage-look-for-moves.
The support for fileid in windows is lacking in unix-compat. I would
see how to implement it when i can get a windows machine to try.

2013/8/10 Jose Luis Neder <bugs@darcs.net>:
>
> New submission from Jose Luis Neder <jlneder@gmail.com>:
>
> This patch implement --look-for-moves flag for amend-record, record and whatsnew commands. The resolve dependencies code is a little too messy and i don't know if it is completely right. The test cases are incomplete but i'm confident it should work in most if not all of the test cases that are missing(i have to do more testing).
> I don't know if the same code could function without unsafeCoerceP.
> To make amend-record work right when used twice in a row i have to delete the pending patch generated when the patch is modified. I don't know if this is desirable.
> I will love to hear any corrections you have about the code.
> It's needed a patch to hashed-storage too.
> 1 patch for repository http://darcs.net/screened:
>
> Sat Aug 10 01:37:54 ART 2013  Jose Luis Neder <jlneder@gmail.com>
>   * resolve issue642: Automatic detection of file renames
>   Implement --look-for-moves flag for amend-record, record and whatsnew commands.
>
> ----------
> files: patch-preview.txt, resolve-issue642_-automatic-detection-of-file-renames.dpatch, unnamed
> messages: 16974
> nosy: jlneder
> status: needs-screening
> title: resolve issue642: Automatic detection of file renames
>
> __________________________________
> Darcs bug tracker <bugs@darcs.net>
> <http://bugs.darcs.net/patch1093>
> __________________________________

There is the hashed storage patch... I'm fixing a bug I found in the dependency code :/ in the darcs patch...
1 patch for repository http://repos.mornfall.net/hashed-storage/0.5:

Fri Aug  9 21:26:08 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * add fileid info to the index format
  added some utility functions in anchoredPath too.

Attachments

• add-fileid-info-to-the-index-format.dpatch
• patch-preview.txt
• unnamed

These two patches implement --look-for-moves flag for record, amend-record, and whatsnew, and show index with fileid information.

Use cases are explained on the wiki in Using/LookForMoves.

2 patches for repository http://darcs.net/screened:

Thu Aug 29 09:45:41 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * Show fileid info with show index
  Update timestamp index with new adds and moves in pending
  (darcs mv, darcs add)

Mon Sep  2 07:02:28 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * resolve issue642: Automatic detection of file renames
  Implement --look-for-moves flag for amend-record, record and whatsnew commands.

Attachments

• patch-preview.txt
• show-fileid-info-with-show-index.dpatch
• unnamed

This two patches add fileid info to the timestamp index format.
The first patch fix compile warnings.
The second patch implement the mentioned functionality, and bump the hashed-storage version to 0.5.11.
2 patches for repository http://hub.darcs.net/jlneder/hashed-storage-look-for-moves:

Wed Aug 28 17:44:21 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * add fileid info to the index format
  add some utility functions in AnchoredPath too for managing fileids.
  bump version to 0.5.11

Wed Aug 28 18:04:02 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * Fix warnings for ghc >= 7.4.1
  * redundant imports
  * non-exhaustive pattern matches
  * name clashes
  * missing top-level signatures
  * unused variables

Attachments

• add-fileid-info-to-the-index-format.dpatch
• patch-preview.txt
• unnamed

These two patches add fileids info to hashed-storage.
The first patch fix all compile warnings.
The second patch implement support for fileids for linux(posix) and windows and bump version to 0.5.11
2 patches for repository http://hub.darcs.net/jlneder/hashed-storage-look-for-moves:

Wed Aug 28 17:44:21 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * add fileid info to the index format
  add some utility functions in AnchoredPath too for managing fileids.
  bump version to 0.5.11

Wed Aug 28 18:04:02 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * Fix warnings for ghc >= 7.4.1
  * redundant imports
  * non-exhaustive pattern matches
  * name clashes
  * missing top-level signatures
  * unused variables

Attachments

• add-fileid-info-to-the-index-format.dpatch
• patch-preview.txt
• unnamed

Review of the first patch:

> [Show fileid info with show index
> Jose Luis Neder <jlneder@gmail.com>**20130829124541
>  Ignore-this: 3b7600ba1ec1e976d5ab863ec9ab9d3
>  Update timestamp index with new adds and moves in pending
>  (darcs mv, darcs add)
> ] hunk ./src/Darcs/Repository/Internal.hs 79
>  
>  import Darcs.Repository.State ( readRecorded
>                                , readWorking
> +                              , updateIndex
>                                )
>  import Darcs.Repository.LowLevel
>      ( readPending
> hunk ./src/Darcs/Repository/Internal.hs 826
>                   finalizePending repository updateWorking
>              debugMessage "Done finalizing changes..."
>              doesPatchIndexExist dir >>= (`when`
createOrUpdatePatchIndexDisk repository)
> +            updateIndex repository
>      | otherwise = fail Old.oldRepoFailMsg


OK, so every call to `finalizeRepositoryChanges` now calls updateIndex,
ie, commands
record, tag, amend, rollback, pull, etc.

>  -- TODO: rename this and document the transaction protocol
(revert/finalize)
> hunk ./src/Darcs/Repository/State.hs 34
>      , readRecorded, readUnrecorded, readRecordedAndPending, readWorking
>      , readPendingAndWorking
>      -- * Index.
> -    , readIndex, invalidateIndex, UseIndex(..), ScanKnown(..)
> +    , readIndex, updateIndex, invalidateIndex, UseIndex(..),
ScanKnown(..)
>      -- * Utilities
>      , filterOutConflicts ) where
>  
> hunk ./src/Darcs/Repository/State.hs 55
>  
>  import Darcs.Patch ( RepoPatch, PrimOf, sortCoalesceFL, fromPrim,
fromPrims
>                     , effect, anonymous )
> -import Darcs.Patch.Apply ( ApplyState, applyToTree, effectOnFilePaths )
> +import Darcs.Patch.Apply ( ApplyState, applyToTree,
effectOnFilePaths, Apply )
>  import Darcs.Patch.Witnesses.Ordered ( RL(..), FL(..), (+>+), mapFL_FL
>                                       , (:>)(..) )
>  import Darcs.Patch.Witnesses.Eq ( EqCheck(IsEq, NotEq) )
> hunk ./src/Darcs/Repository/State.hs 313
>               return idx
>       else I.readIndex "_darcs/index" darcsTreeHash
>  
> +updateIndex :: (RepoPatch p, ApplyState p ~ Tree) => Repository p wR
wU wT -> IO ()
> +updateIndex repo = do
> +    invalid <- doesFileExist "_darcs/index_invalid"
> +    exist <- doesFileExist "_darcs/index"
> +    format_valid <- if exist
> +                     then I.indexFormatValid "_darcs/index"
> +                     else return True
> +    when (exist && not format_valid) $
> +#if mingw32_HOST_OS
> +       renameFile "_darcs/index" "_darcs/index.old"
> +#else
> +       removeFile "_darcs/index"
> +#endif
> +    pris <- readRecordedAndPending repo
> +    _ <- I.updateIndexFrom "_darcs/index" darcsTreeHash pris
> +    when invalid $ removeFile "_darcs/index_invalid"
> +
 
s/exist/exists
s/format_valid/formatValid

> hunk ./src/Darcs/UI/Commands/Add.hs 71
>  import Darcs.Repository.Flags ( UpdateWorking(..) )
>  import Darcs.Patch ( Patchy, PrimPatch, applyToTree, addfile, adddir,
listTouchedFiles )
>  import Darcs.Patch.Apply ( ApplyState )
> -import Darcs.Repository.State ( readRecordedAndPending )
> +import Darcs.Repository.State ( readRecordedAndPending, updateIndex )
>  import Darcs.Repository
>      (
>        withRepoLock
> hunk ./src/Darcs/UI/Commands/Add.hs 180
>          fail "No files were added"
>      unless gotDryRun $
>        do addToPending repository YesUpdateWorking ps
> +         updateIndex repository
>           when notQuiet $
>             putStr $ unlines $ ["Adding:"] ++ listTouchedFiles ps
>    where
 
OK, so that the index gets the fileid of the added file.

> hunk ./src/Darcs/UI/Commands/Move.hs 41
>  import Darcs.Repository.Flags ( UpdateWorking (..) )
>  import System.FilePath.Posix ( (</>), takeFileName )
>  import System.Directory ( renameDirectory )
> -import Darcs.Repository.State ( readRecordedAndPending, readRecorded )
> +import Darcs.Repository.State ( readRecordedAndPending, readRecorded,
updateIndex )
>  import Darcs.Repository
>      ( Repository
>      , withRepoLock
> hunk ./src/Darcs/UI/Commands/Move.hs 205
>          Nothing -> addToPending repository YesUpdateWorking
(movePatch :>: NilFL)
>          Just (Sealed p) -> addToPending repository YesUpdateWorking
(p :>: movePatch :>: NilFL)
>        moveFileOrDir work old_fp new_fp
> +      updateIndex repository
>        unless (Quiet `elem` opts) $
>          putStrLn $ unwords $ ["Moved:", old_fp, "to:", new_fp]

This seems to be to update the file name in the index.

>  
> hunk ./src/Darcs/UI/Commands/Move.hs 226
>      withSignalsBlocked $ do
>        unseal (addToPending repository YesUpdateWorking) $ toFL $
catMaybes addpatches ++ movepatches
>        zipWithM_ (moveFileOrDir work) moved movetargets
> +      updateIndex repository
>        unless (Quiet `elem` opts) $
>          putStrLn $ unwords $ ["Moved:"] ++ moved ++ ["to:", finaldir]

Same thing here.
 
> hunk ./src/Darcs/UI/Commands/ShowIndex.hs 29
>      , showPristineCmd -- for alias
>      ) where
>  
> +import Control.Applicative ( (<$>) )
>  import Control.Monad ( (>=>) )
>  import Darcs.UI.Arguments ( DarcsFlag(..), workingRepoDir,
>                          files, directories, nullFlag )
> hunk ./src/Darcs/UI/Commands/ShowIndex.hs 41
>  import Storage.Hashed( floatPath )
>  import Storage.Hashed.Hash( encodeBase16, Hash( NoHash ) )
>  import Storage.Hashed.Tree( list, expand, itemHash, Tree, TreeItem(
SubTree ) )
> -import Storage.Hashed.Index( updateIndex )
> +import Storage.Hashed.Index( updateIndex, listFileIDs )
>  import Darcs.Util.Path( anchorPath, AbsolutePath )
>  
> hunk ./src/Darcs/UI/Commands/ShowIndex.hs 44
> +import System.Posix.Types ( FileID )
> +
>  import qualified Data.ByteString.Char8 as BS
> hunk ./src/Darcs/UI/Commands/ShowIndex.hs 47
> +import Data.Maybe ( fromJust )
> +import qualified Data.Map as M ( Map, lookup, fromList )
>  
>  showIndex :: DarcsCommand
>  showIndex = DarcsCommand {
> hunk ./src/Darcs/UI/Commands/ShowIndex.hs 69
> -dump :: [DarcsFlag] -> Tree IO -> IO ()
> -dump opts tree = do
> +dump :: [DarcsFlag] -> Maybe (M.Map FilePath FileID) -> Tree IO -> IO ()
> +dump opts fileids tree = do

ok


> hunk ./src/Darcs/UI/Commands/ShowIndex.hs 81
>              isdir = case i of
>                        SubTree _ -> "/"
>                        _ -> ""
> -        line $ hash ++ " " ++ path ++ isdir
> +            fileid = case fileids of
> +                       Nothing -> ""
> +                       Just fileids' -> " " ++ (show $ fromJust $
M.lookup path fileids')
> +        line $ hash ++ fileid ++ " " ++ path ++ isdir
>    x <- expand tree
>    mapM_ output $ (floatPath ".", SubTree x) : list x

ok

> hunk ./src/Darcs/UI/Commands/ShowIndex.hs 90
>  showIndexCmd :: (AbsolutePath, AbsolutePath) -> [DarcsFlag] ->
[String] -> IO ()
>  showIndexCmd _ opts _ = withRepository (useCache opts) $ RepoJob $
\repo ->
> -  readIndex repo >>= updateIndex >>= dump opts
> +  do index <- readIndex repo
> +     index_tree <- updateIndex index
> +     fileids <- (M.fromList . map (\((a,_),b) -> (anchorPath ""
a,b))) <$> listFileIDs index
> +     dump opts (Just fileids) index_tree
>  
>  showPristineCmd :: (AbsolutePath, AbsolutePath) -> [DarcsFlag] ->
[String] -> IO ()
>  showPristineCmd _ opts _ = withRepository (useCache opts) $ RepoJob $
> hunk ./src/Darcs/UI/Commands/ShowIndex.hs 97
> -  readRecorded >=> dump opts
> +  readRecorded >=> dump opts Nothing

ok.

Your patch is missing a dependency bump to hashed-storage 0.5.11 .

I'd like to know about the performance hit of maintaining index with fileids
on various commands (add, record) on repositories with many versioned files.
For instance the Agda repository has almost 4000 files.
You could just time a bash script that "touches" many empty files
and add them to a repo, comparing between with filesids and without.
(see http://hub.darcs.net/gh/bench for instance).

Fixed.
I moved the getFileID and getFileID' functions to Storage/Hashed/Index.hs.
I think replacePrefixPath is similar in kind with appendPath or catPaths so i
leave it in AnchoredPath.hs.
I added a haddock explaination about listFileIDs, getFileID, Result and
ResultF.
I changed the 7.6 ghc warnings but it means that it could not be build with
ghc 7.4 anymore(for example it could not be build in Ubuntu 12.04 because it
has haskell-platform 2012.1.0.0). It needs the last haskell platform version
(2013.2.0.0). I'm not sure it is good to do this now.
Next come the darcs patch

2 patches for repository jlneder@hub.darcs.net:hashed-storage-look-for-moves:

Wed Sep 11 10:07:26 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * Fix warnings for ghc >= 7.6
  * Bump base dependency version to 4.6
  * redundant imports
  * non-exhaustive pattern matches
  * name clashes
  * missing top-level signatures
  * unused variables

Wed Sep 11 10:40:37 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * add fileid info to the index format
  add some utility functions in AnchoredPath too for managing fileids.
  bump version to 0.5.11

Attachments

• fix-warnings-for-ghc-__-7_6.dpatch
• patch-preview.txt
• unnamed

"show index" patch fixed. (I changed the name of the variables in readIndex
too)
1 patch for repository http://darcs.net/screened:

Wed Sep 11 10:52:39 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * Show fileid info with show index
  Update timestamp index with new adds and moves in pending
  (darcs mv, darcs add)

Attachments

• patch-preview.txt
• show-fileid-info-with-show-index.dpatch
• unnamed

I forgot the dependency bump.

1 patch for repository http://darcs.net/screened:

Wed Sep 11 10:57:36 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * Show fileid info with show index
  Update timestamp index with new adds and moves in pending
  (darcs mv, darcs add)

Attachments

• patch-preview.txt
• show-fileid-info-with-show-index.dpatch
• unnamed

Some superficial comments about the second patch:

* the comment of readPendingAndWorkingWithMoves is outdated (comes from
a copy-paste)
* the function addMovesToChanges should live somewhere inside of
Darcs.Repository instead of Darcs.UI. This is for the (potential) users
of libdarcs to avoid importing stuff from Darcs.UI for features that are
not, well, UI features. Granted, a lot of code that now lives in
Darcs.UI could be moved to Darcs.Repository, but the idea is to avoid
adding more to it.
* in the shell test, there is a commented part. Either you should
uncomment it, or you should add it as a failing test (naming it
failing-look-for-moves.sh for instance)

A further comment, it would be nice to add a test script that checks
that boring files are correctly handled (suggested by Ganesh at
<http://lists.osuosl.org/pipermail/darcs-users/2013-August/026929.html>).

The case I'm thinking about is: moving a non-boring file to a boring
one (--look-for-moves should not suggest it).

g.

This is a brief summary of the changes:
* Update comment of readPendingAndWorkingWithMoves.
* Added Darcs.Repository.Util with the function getMovesPs(before was addMovesToChanges).
* Added failing-look_for_moves.sh with the failing test.
* Added boring file test.
* Changed some types and names because it makes more sense with the --look-for-replaces patch.

Some minor changes in the "show index" patch.
2 patches for repository http://darcs.net/screened:

Sun Sep 15 20:34:58 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * Show fileid info with show index
  Update timestamp index with new adds and moves in pending
  (darcs mv, darcs add)

Sun Sep 15 20:41:26 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * resolve issue642: Automatic detection of file renames

Attachments

• patch-preview.txt
• show-fileid-info-with-show-index.dpatch
• unnamed

I'm accepting Jose's patch and add a patch to have the code of
hashed-storage 0.5.11 inside of darcs' repository.

Why I accept Jose's patch:

I've already reviewed the code, the feature works well in practice,
and has shell tests.
There is one failing test for a very cornerish
case "exchange of dirs with contents and exchange filenames inside"
which shows that it can be perfected, but I find the patches good
enough to get in.

Why I include hashed-storage 0.5.11 (with changes from Jose's bundle
at http://bugs.darcs.net/patch1101 ):

hashed-storage was initially written by Petr Rockai but is not developed
anymore. The code is not expected to evolve, save from fixes to avoid
code rot. I could upload an "official" repository for hashed-storage
somewhere, but I can't guarantee I will maintain it personally.
So a safe approach is to copy the code into darcs' repository and say
we are collectively responsible for it.
Petr may continue working on fslib, the successor of hashed-storage,
at some point in the future, and we may choose to port darcs to it.

I'll upload hashed-storage 0.5.11 to hackage soon.

3 patches for repository http://darcs.net:

Thu Jan  9 20:00:42 ART 2014  Jose Luis Neder <jlneder@gmail.com>
  * Show fileid info with show index
  Update timestamp index with new adds and moves in pending
  (darcs mv, darcs add)
  Require hashed-storage 0.5.11

Sun Sep 15 20:41:26 ART 2013  Jose Luis Neder <jlneder@gmail.com>
  * resolve issue642: Automatic detection of file renames

Thu Jan  9 21:34:02 ART 2014  Guillaume Hoffmann <guillaumh@gmail.com>
  * add hashed-storage 0.5.11

Attachments

• patch-preview.txt
• show-fileid-info-with-show-index.dpatch
• unnamed

I'm having trouble reproducing a hashed-storage repository whose code 
matches that imported into darcs. I've grabbed 
http://hub.darcs.net/jlneder/hashed-storage-look-for-moves and also 
tried applying patches from http://bugs.darcs.net/file4493/add-fileid-
info-to-the-index-format.dpatch without success. Do you have one? I'm 
having some build problems on Windows and a proper history of the code 
would be very useful (it would also be useful later on even if we do end 
up merging the code into Darcs properly).

Hi Ganesh,

> I'm having trouble reproducing a hashed-storage repository whose code
> matches that imported into darcs. I've grabbed
> http://hub.darcs.net/jlneder/hashed-storage-look-for-moves and also
> tried applying patches from http://bugs.darcs.net/file4493/add-fileid-
> info-to-the-index-format.dpatch without success.

Those are the right changes but I fixed more warnings when copying the
hashed-storage code to darcs' repository. Apart from that there is no
important change (AFAIR), but indeed I did not consider the case of
Windows, sorry! :(

To track down the build problems on Windows you could take Jose's
repository + patch bundle and work from this. He did test it under
Windows.