Three patches for optimize subcommands plus one cleanup dependency.

4 patches for repository http://darcs.net/screened:

patch 83f6afeba5d5f9da1320d34aeec61f86f766e857
Author: Ben Franksen <ben.franksen@online.de>
Date:   Fri Jun 25 19:36:00 CEST 2021
  * optimize reorder: add --deep/--shallow option

  With --shallow (the default) the behavior is as before: we make the latest
  tag clean and create a Tagged section for it (which will become an inventory
  when saved to disk). Note that this may make the latest clean tag dirty. It
  may also increase the size of the head inventory and therefore fail to be an
  optimization, since the latest tag may cover less patches than an earlier
  clean tag.

  With --deep, we traverse all patches in order, trying to make every tag
  clean as long as that doesn't make any previous clean tag dirty, and create
  Tagged sections (and thus inventories) for all of them. Since this never
  makes a clean tag dirty, the size of the head inventory is guaranteed to be
  smaller or equal to the original one. This operation is idempotent.

patch 5d6bdbf9a695247173d9bbc6ff9afa66a1cc73d7
Author: Ben Franksen <ben.franksen@online.de>
Date:   Thu Nov 24 23:54:43 CET 2022
  * optimize (un)compress: also handle pristine

patch 4afe38f0a53cc3551436be01a4ebae3be1c02e22
Author: Ben Franksen <ben.franksen@online.de>
Date:   Mon Jun 27 19:18:05 CEST 2022
  * use System.Directory.withCurrentDirectory instead of Darcs.Util.File.<same>

  This does not catch exceptions and has no special treatment of "". Catching
  expceptions such as "does not exist" in such a generic way is bad, so this
  is not just simpler but also more informative to the end user. The special
  treatment of "" seems to be no longer needed.

patch fcf29158c008dcf67ad115e69078c508f25dd48d
Author: Ben Franksen <ben.franksen@online.de>
Date:   Fri Nov 25 22:43:56 CET 2022
  * optimize cache: remove traversal of darcs repos and clean the cache properly

  Previously you could pass repositories as arguments (and if you didn't, it
  would search your whole HOME for darcs repos) and then it would do the
  equivalent of `darcs clean` on them, too. This functionality is is out of
  scope for darcs and can be emulated with a simple shell command e.g.

  > find -type d -name _darcs -execdir darcs optimize clean \;

  What remains is to clean the global cache. There is a much better solution
  for this and it is already implemented in Darcs.Util.Cache.cleanCaches:
  simply remove all files in the cache that have a hard-link count < 2. This
  is what the command now does (for all three types of hashed dirs).

Attachments

• optimize-reorder_-add-__deep___shallow-option.dpatch
• patch-preview.txt

One more semantic dependency (a cleanup patch) that I failed to explicitly
depend on; without this building fails.

1 patch for repository http://darcs.net/screened:

patch 931e3c8a6342487b4e05ffd1e9ac8e141d8f2542
Author: Ben Franksen <ben.franksen@online.de>
Date:   Sat Mar  6 13:01:04 CET 2021
  * clean up getRecordedUpToMatch and the commands that use it

  The procedure now returns a tree and has been renamed to
  getPristineUpToMatch. The commands that use it (dist, show files, show
  contents) now work directly with the resulting tree, rather than a freshly
  created temporary working directory. This is a lot simpler and more
  efficient. Incidentally, it allows to easily support match options for
  --zip.

  However, the predist command expects to be run inside a working tree. So we
  write out the tree to a temporary directory before the predist and read it
  back in afterwards. The extra cost of this is payed only if a predist
  command has been set (using setpref predist).

Attachments

• clean-up-getrecordeduptomatch-and-the-commands-that-use-it.dpatch
• patch-preview.txt

>   * clean up getRecordedUpToMatch and the commands that use it

OK - I only skimmed the code but the high level description sounds good.

>   * optimize reorder: add --deep/--shallow option

Nice - should perhaps have a shell test?

>   * optimize (un)compress: also handle pristine

OK - again might need a test? There don't seem to be any existing ones
for optimize compress :-(

>   * use System.Directory.withCurrentDirectory instead of 
Darcs.Util.File.<same>

OK, nice cleanup

>  * optimize cache: remove traversal of darcs repos and clean the cache 
properly

OK, seems like a sensible removal of unnecessary functionality.

I started working on tests as you suggested and then got 
sidetracked... it may turn out that I will send some other patches 
first.

Here is the the first test patch.

While working on a test for compress/uncompress I realized that uncompress
and then compress destroys file sharing with hard links. Should we always do
a relink pass after compress?

BTW, I also noted that optimize relink doesn't work on Windows, we even have
a ticket for this (issue1702). Will send patches to fix that.

1 patch for repository http://darcs.net/screened:

patch c69c475f532413bb922997950f81bc24944d96ca
Author: Ben Franksen <ben.franksen@online.de>
Date:   Sat Jul 15 21:07:19 CEST 2023
  * add a test for optimize reorder --deep

Attachments

• add-a-test-for-optimize-reorder-__deep.dpatch
• patch-preview.txt

> While working on a test for compress/uncompress I realized that 
> uncompress and then compress destroys file sharing with hard links.
> Should we always do a relink pass after compress?

Yes, I think so. I guess we don't care about sharing uncompressed 
files?

Actually, I wonder why we even bother to keep supporting
uncompressed?

> Actually, I wonder why we even bother to keep supporting uncompressed?

I guess this question is answered by this patch in patch2315:

  * always store hashed files in compressed form

  This removes the --[un-]compress option from most commands. The only
  remaining command that supports it is `push` since there it is used to
  control whether the bundle is sent in compressed or uncompressed from.
  Sending uncompressed is needed for compatibility with older (pre 2.5) remote
  darcs versions. The `optimize uncompress` subcommand is still available, as
  an uncompressed repository may be nice to have for debugging. Note that we
  are still able to detect and read uncompressed hashed files.

Here is the second test.

Regarding a relink pass for optimize compress, this is currently still WIP
because I am not sure that the current algorithm for relink is guaranteed to
preserve sharing for other repos. I have also asked myself if it wouldn't be
simpler to just replace hashed files with the (supposedly compressed) ones
from the cache. These are questions we should discuss in a separate ticket.

1 patch for repository http://darcs.net/screened:

patch 83b8731d305b7f0c1ce4cfb95b8638000ecc526b
Author: Ben Franksen <ben.franksen@online.de>
Date:   Mon Jul 17 05:23:35 CEST 2023
  * add a tests for optimize compress/uncompress

Attachments

• add-a-tests-for-optimize-compress_uncompress.dpatch
• patch-preview.txt