Hi,

there go couple more patches removing usage of SlurpDirectory. After these, I
think there are only two categories of SlurpDirectory use left:

- record, amend-record, rollback (maybe some others) use them to announce the
  various cases of commandline-given files not existing, not being in repo etc.
  the code for this is duplicated and wants refactoring
- add uses slurps to do some checking

When these are sorted out (hopefully this week, or maybe the next) we should be
able to retire SlurpDirectory and all related code (eg. slurp_* in Repository).

Yours,
   Petr.

4 patches for repository http://darcs.net:

Mon Nov 16 09:02:25 CET 2009  Petr Rockai <me@mornfall.net>
  * Remove unused force_replace_slurpy.

Mon Feb  8 19:20:27 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace Slurpy usage in Commands.Diff.

Mon Feb  8 23:47:04 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove implementation of --store-in-memory, simplifying matcher code.

Tue Feb  9 00:11:25 CET 2010  Petr Rockai <me@mornfall.net>
  * Avoid use of SlurpDirectory in Commands.ShowFiles.

Attachments

• remove-unused-force_replace_slurpy_.dpatch
• unnamed

Kablaam!

22 patches for repository http://darcs.net:

Mon Nov 16 09:02:25 CET 2009  Petr Rockai <me@mornfall.net>
  * Remove unused force_replace_slurpy.

Mon Feb  8 19:20:27 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace Slurpy usage in Commands.Diff.

Mon Feb  8 23:47:04 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove implementation of --store-in-memory, simplifying matcher code.

Tue Feb  9 00:11:25 CET 2010  Petr Rockai <me@mornfall.net>
  * Avoid use of SlurpDirectory in Commands.ShowFiles.

Wed Feb 10 13:26:01 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace SlurpDirectory usage in Commands.Add with Tree-based code.

Thu Feb 11 00:47:11 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement the Slurp-based file/dir existence-check functions in terms of Trees.

Thu Feb 11 00:48:20 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-work Commands.Add (simplify, use the new treeHas* functions).

Thu Feb 11 00:50:00 CET 2010  Petr Rockai <me@mornfall.net>
  * Generalize announceFiles used by whatsnew and use it in record as well.

Thu Feb 11 00:50:39 CET 2010  Petr Rockai <me@mornfall.net>
  * Add a missing Functor constraint on ReadableDirectory TreeMonad instance.

Thu Feb 11 00:57:54 CET 2010  Petr Rockai <me@mornfall.net>
  * Purge Slurpy usage in Commands.Rollback (use announceFiles from whatsnew).

Thu Feb 11 00:58:47 CET 2010  Petr Rockai <me@mornfall.net>
  * Purge unused fileExists from Commands.Add.

Thu Feb 11 01:13:26 CET 2010  Petr Rockai <me@mornfall.net>
  * Use stock setScriptsExecutable from Darcs.Repository in Commands.Convert.

Thu Feb 11 01:13:53 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement optimize --relink using Trees instead of Slurpies.

Thu Feb 11 01:14:35 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement setScriptsExecutable using Trees instead of Slurpies.

Thu Feb 11 10:31:53 CET 2010  Petr Rockai <me@mornfall.net>
  * Port Commands.Move from Slurpy to Tree.

Fri Feb 12 11:17:51 CET 2010  Petr Rockai <me@mornfall.net>
  * Cannot "darcs remove" non-existent files.

Fri Feb 12 11:18:44 CET 2010  Petr Rockai <me@mornfall.net>
  * Use a more canonic way to create empty hashed pristine in optimize (--upgrade).

Fri Feb 12 11:19:46 CET 2010  Petr Rockai <me@mornfall.net>
  * Purge unused bits of checkpoint reading.

Fri Feb 12 11:28:42 CET 2010  Petr Rockai <me@mornfall.net>
  * Reimplement applyHashed in terms of hashedTreeIO (Storage.Hashed.Monad).

Fri Feb 12 11:32:58 CET 2010  Petr Rockai <me@mornfall.net>
  * Use a more canonic way to create empty hashed pristine.

Fri Feb 12 11:33:22 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace slurp_recorded with readRecorded in make_new_pending.

Fri Feb 12 11:36:41 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove SlurpDirectory.

Attachments

• remove-unused-force_replace_slurpy_.dpatch
• unnamed

I've started reviewing this. Here are some partial comments. Petr, are
you ok for me to any patches I'm already happy with (so long as darcs
compiles/the testsuite passes, of course) while the rest of the review
is still ongoing?

> Fri Feb 12 10:32:58 GMT 2010  Petr Rockai <me@mornfall.net> 
>   * Use a more canonic way to create empty hashed pristine. 

> Fri Feb 12 10:18:44 GMT 2010  Petr Rockai <me@mornfall.net> 
>   * Use a more canonic way to create empty hashed pristine in optimize
(--upgrade). 

hardcoded "_darcs/pristine.hashed" is not a good standard, IMO 

> Fri Feb 12 10:17:51 GMT 2010  Petr Rockai <me@mornfall.net> 
>   * Cannot "darcs remove" non-existent files. 
 
What's the logic behind this? I agree that removing a file that's in
pristine but not working is redundant, but I don't see why there should
be a complaint. BTW while scratching my head at the remove code I found
a bug which I'll raise as a separate ticket.

> Thu Feb 11 00:14:35 GMT 2010  Petr Rockai <me@mornfall.net> 
>   * Re-implement setScriptsExecutable using Trees instead of Slurpies. 
 
> +    tree <- expand =<< readPlainTree "." -- TODO filter out _darcs 
 
This TODO needs doing 

> Wed Feb 10 23:50:39 GMT 2010  Petr Rockai <me@mornfall.net> 
>   * Add a missing Functor constraint on ReadableDirectory TreeMonad
instance. 
 
OK 

> Mon Feb  8 23:11:25 GMT 2010  Petr Rockai <me@mornfall.net> 
>   * Avoid use of SlurpDirectory in Commands.ShowFiles. 
>  
> +import Prelude hiding ( all ) 
> ... 
> +all <- ... 
 
Just call the variable a different name instead of breaking hiding
standard Prelude functions in the rest of the module 
 
Rest looks ok. 

> Mon Feb  8 22:47:04 GMT 2010  Petr Rockai <me@mornfall.net> 
>   * Remove implementation of --store-in-memory, simplifying matcher code. 
 
This seems to be a live option. If you're dumping it, (a) there needs to
be some discussion and (b) you haven't actually removed it properly,
you're just ignoring it. 
 
> Mon Feb  8 18:20:27 GMT 2010  Petr Rockai <me@mornfall.net> 
>   * Replace Slurpy usage in Commands.Diff. 
 
OK 
 
> Mon Nov 16 08:02:25 GMT 2009  Petr Rockai <me@mornfall.net> 
>   * Remove unused force_replace_slurpy. 
 
OK

Hi,

Ganesh Sittampalam <bugs@darcs.net> writes:
> I've started reviewing this. Here are some partial comments. Petr, are
> you ok for me to any patches I'm already happy with (so long as darcs
> compiles/the testsuite passes, of course) while the rest of the review
> is still ongoing?
Sure, go ahead.

>> Fri Feb 12 10:32:58 GMT 2010  Petr Rockai <me@mornfall.net> 
>>   * Use a more canonic way to create empty hashed pristine. 
>
>> Fri Feb 12 10:18:44 GMT 2010  Petr Rockai <me@mornfall.net> 
>>   * Use a more canonic way to create empty hashed pristine in optimize
> (--upgrade). 
>
> hardcoded "_darcs/pristine.hashed" is not a good standard, IMO 
Dunno. It's easier to write this way and it can't be changed most of the
time (only when we break backward compatibility) and is easy to grep
anyway. It also avoids an extra lookup when reading the code.

>> Fri Feb 12 10:17:51 GMT 2010  Petr Rockai <me@mornfall.net> 
>>   * Cannot "darcs remove" non-existent files. 
>  
> What's the logic behind this? I agree that removing a file that's in
> pristine but not working is redundant, but I don't see why there should
> be a complaint. BTW while scratching my head at the remove code I found
> a bug which I'll raise as a separate ticket.
Hm. The logic is that old darcs would write out a pending patch with the
removal even if the file was not there, which would then alter behaviour
of darcs show files --pending. Or somesuch (maybe the other way
around). The same patch should have a change to the testsuite to a
similar effect. I am not convinced either way, just seemed a sane thing
to do.

>> Thu Feb 11 00:14:35 GMT 2010  Petr Rockai <me@mornfall.net> 
>>   * Re-implement setScriptsExecutable using Trees instead of Slurpies. 
>  
>> +    tree <- expand =<< readPlainTree "." -- TODO filter out _darcs 
>  
> This TODO needs doing 
Well, it should be safe, even if less optimal. It could be argued boring
files should be ignored as well here. But then, this is only called in
contexts where we are constructing a new working tree
(get/convert/test/trackdown) and we have an untouched _darcs (which by
darcs action alone cannot contain anything starting with #! even if
there was any harm in +x-ing such files). Ok ok I'll do that.

>> Mon Feb  8 23:11:25 GMT 2010  Petr Rockai <me@mornfall.net> 
>>   * Avoid use of SlurpDirectory in Commands.ShowFiles. 
>>  
>> +import Prelude hiding ( all ) 
>> ... 
>> +all <- ... 
>  
> Just call the variable a different name instead of breaking hiding
> standard Prelude functions in the rest of the module 
: - ( I hate the shadowing warning. All concise and reasonable names I
want to use are always taken...

>> Mon Feb  8 22:47:04 GMT 2010  Petr Rockai <me@mornfall.net> 
>>   * Remove implementation of --store-in-memory, simplifying matcher code. 
>  
> This seems to be a live option. If you're dumping it, (a) there needs to
> be some discussion and (b) you haven't actually removed it properly,
> you're just ignoring it. 
Yeah, I dumped it because I didn't think it was of any use. Discussion
can of course happen, my vote is to drop it. :) I agree it should be
removed completely if that's the decision.

Yours,
   Petr.

On Sat, 20 Feb 2010, Petr Rockai wrote:

> Hi,
>
> Ganesh Sittampalam <bugs@darcs.net> writes:
>> I've started reviewing this. Here are some partial comments. Petr, are
>> you ok for me to any patches I'm already happy with (so long as darcs
>> compiles/the testsuite passes, of course) while the rest of the review
>> is still ongoing?
> Sure, go ahead.

I've now pushed a few patches. Note that some of the remaining patches are 
in conflict with current HEAD.

The rest of my review is at the bottom.

>>> Fri Feb 12 10:32:58 GMT 2010  Petr Rockai <me@mornfall.net>
>>>   * Use a more canonic way to create empty hashed pristine.
>>
>>> Fri Feb 12 10:18:44 GMT 2010  Petr Rockai <me@mornfall.net>
>>>   * Use a more canonic way to create empty hashed pristine in optimize
>> (--upgrade).
>>
>> hardcoded "_darcs/pristine.hashed" is not a good standard, IMO
> Dunno. It's easier to write this way and it can't be changed most of the
> time (only when we break backward compatibility) and is easy to grep
> anyway. It also avoids an extra lookup when reading the code.

I think we need to have a separate discussion on darcs-users to establish 
the standard one way or another.

>>> Fri Feb 12 10:17:51 GMT 2010  Petr Rockai <me@mornfall.net>
>>>   * Cannot "darcs remove" non-existent files.
>>
>> What's the logic behind this? I agree that removing a file that's in
>> pristine but not working is redundant, but I don't see why there should
>> be a complaint. BTW while scratching my head at the remove code I found
>> a bug which I'll raise as a separate ticket.
> Hm. The logic is that old darcs would write out a pending patch with the
> removal even if the file was not there, which would then alter behaviour
> of darcs show files --pending. Or somesuch (maybe the other way
> around). The same patch should have a change to the testsuite to a
> similar effect. I am not convinced either way, just seemed a sane thing
> to do.

Thinking about it again, I can also see a use case for when you want to 
make sure the file stays marked as removed but actually put it back later. 
So I'm against this patch without further evidence on the other side.

>>> Thu Feb 11 00:14:35 GMT 2010  Petr Rockai <me@mornfall.net>
>>>   * Re-implement setScriptsExecutable using Trees instead of Slurpies.
>>
>>> +    tree <- expand =<< readPlainTree "." -- TODO filter out _darcs
>>
>> This TODO needs doing
> Well, it should be safe, even if less optimal. It could be argued boring
> files should be ignored as well here. But then, this is only called in
> contexts where we are constructing a new working tree
> (get/convert/test/trackdown) and we have an untouched _darcs (which by
> darcs action alone cannot contain anything starting with #! even if
> there was any harm in +x-ing such files). Ok ok I'll do that.

OK, this is awaiting amendment/a follow-up.

>>> Mon Feb  8 23:11:25 GMT 2010  Petr Rockai <me@mornfall.net>
>>>   * Avoid use of SlurpDirectory in Commands.ShowFiles.
>>>
>>> +import Prelude hiding ( all )
>>> ...
>>> +all <- ...
>>
>> Just call the variable a different name instead of breaking hiding
>> standard Prelude functions in the rest of the module
> : - ( I hate the shadowing warning. All concise and reasonable names I
> want to use are always taken...

I'd be in favour of turning off the shadowing warnings, because they block 
legitimate deliberate uses of shadowing where you want to avoid 
accidentally reusing the shadowed value.

But actually I'm still against using standard Prelude function names as 
variables in anything but a very limited scope, because of the potential 
for confusion for someone else editing the code later.

>>> Mon Feb  8 22:47:04 GMT 2010  Petr Rockai <me@mornfall.net>
>>>   * Remove implementation of --store-in-memory, simplifying matcher code.
>>
>> This seems to be a live option. If you're dumping it, (a) there needs to
>> be some discussion and (b) you haven't actually removed it properly,
>> you're just ignoring it.
> Yeah, I dumped it because I didn't think it was of any use. Discussion
> can of course happen, my vote is to drop it. :) I agree it should be
> removed completely if that's the decision.

The right way to remove features is to explicitly start a discussion, not 
slip in a patch and hope it gets past the reviewer :-)

> Fri Feb 12 10:36:41 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Remove SlurpDirectory.

OK

> Fri Feb 12 10:33:22 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Replace slurp_recorded with readRecorded in make_new_pending.

OK

> Fri Feb 12 10:28:42 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Reimplement applyHashed in terms of hashedTreeIO (Storage.Hashed.Monad).

Why does this ignore the cache parameter?

Note that we are switching the monad apply uses from SlurpMonad to TreeIO,
and as a result fail inside apply turns directly into fail in IO, which is
what the old code did explicitly.

I still don't like hardcoded directories.

Otherwise OK.

> Fri Feb 12 10:19:46 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Purge unused bits of checkpoint reading.

OK

Note that there were two separate copies of read_checkpoints!

> Thu Feb 11 09:31:53 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Port Commands.Move from Slurpy to Tree.

OK

> Thu Feb 11 00:13:53 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Re-implement optimize --relink using Trees instead of Slurpies.

More hardcoded paths

otherwise OK

> Thu Feb 11 00:13:26 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Use stock setScriptsExecutable from Darcs.Repository in Commands.Convert.

The old code worked on an explicit repository, while the new code relies 
on getCurrentDirectory. Are we sure this is ok?

otherwise OK

> Wed Feb 10 23:58:47 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Purge unused fileExists from Commands.Add.

description wrong - should be Commands.Record

> Wed Feb 10 23:57:54 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Purge Slurpy usage in Commands.Rollback (use announceFiles from whatsnew).

OK

> Wed Feb 10 23:50:00 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Generalize announceFiles used by whatsnew and use it in record as well.

This extracts options from the repository rather than having them passed 
down to it. What difference does this make?

otherwise OK

> Wed Feb 10 23:48:20 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Re-work Commands.Add (simplify, use the new treeHas* functions).

Replacing a chain of tests with a whole set at once - could any of the 
later tests fail when previously they'd have been skipped?

otherwise OK

> Wed Feb 10 23:47:11 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Re-implement the Slurp-based file/dir existence-check functions in 
terms of Trees.

I think we should be trying to break up Darcs.Utils or restrict it to 
really generic utility code, rather than dumping more stuff in it.
Can this stuff go elsewhere or in a new module?

otherwise OK

> Wed Feb 10 12:26:01 GMT 2010  Petr Rockai <me@mornfall.net>
>   * Replace SlurpDirectory usage in Commands.Add with Tree-based code.

Likewise for Darcs.Utils

otherwise OK

Hi,

Ganesh Sittampalam <ganesh@earth.li> writes:
> I think we need to have a separate discussion on darcs-users to establish the
> standard one way or another.
(about hardcoded _darcs-paths... I'll post a separate mail later)

> Thinking about it again, I can also see a use case for when you want to make
> sure the file stays marked as removed but actually put it back later. So I'm
> against this patch without further evidence on the other side.
Well, this will then need some tracking down, since darcs remove of
nonexistent file is a no-op with the current code, for whatever
reason. Might be my remove restructuring that broke something. I'll look
into it later.

>> : - ( I hate the shadowing warning. All concise and reasonable names I
>> want to use are always taken...
>
> I'd be in favour of turning off the shadowing warnings, because they block
> legitimate deliberate uses of shadowing where you want to avoid accidentally
> reusing the shadowed value.
>
> But actually I'm still against using standard Prelude function names as
> variables in anything but a very limited scope, because of the potential for
> confusion for someone else editing the code later.
:( ... amended

>> Fri Feb 12 10:28:42 GMT 2010  Petr Rockai <me@mornfall.net>
>>   * Reimplement applyHashed in terms of hashedTreeIO (Storage.Hashed.Monad).
>
> Why does this ignore the cache parameter?
The question should probably stand "why it's there at all". I'd say
that's an omission, I'll remove the parameter.

>> Thu Feb 11 00:13:26 GMT 2010  Petr Rockai <me@mornfall.net>
>>   * Use stock setScriptsExecutable from Darcs.Repository in Commands.Convert.
>
> The old code worked on an explicit repository, while the new code relies on
> getCurrentDirectory. Are we sure this is ok?

>
> otherwise OK
>
>> Wed Feb 10 23:58:47 GMT 2010  Petr Rockai <me@mornfall.net>
>>   * Purge unused fileExists from Commands.Add.
>
> description wrong - should be Commands.Record
amended

>> Wed Feb 10 23:50:00 GMT 2010  Petr Rockai <me@mornfall.net>
>>   * Generalize announceFiles used by whatsnew and use it in record as well.
>
> This extracts options from the repository rather than having them passed down
> to it. What difference does this make?
It avoids extra parameter in an already long list. The opts are those
that are passed to withRepository used to obtain the repo object, which
is, IIUIC, the same.

>> Wed Feb 10 23:48:20 GMT 2010  Petr Rockai <me@mornfall.net>
>>   * Re-work Commands.Add (simplify, use the new treeHas* functions).
>
> Replacing a chain of tests with a whole set at once - could any of the later
> tests fail when previously they'd have been skipped?
>
> otherwise OK
>
>> Wed Feb 10 23:47:11 GMT 2010  Petr Rockai <me@mornfall.net>
>>   * Re-implement the Slurp-based file/dir existence-check functions in 
> terms of Trees.
>
> I think we should be trying to break up Darcs.Utils or restrict it to really
> generic utility code, rather than dumping more stuff in it.
> Can this stuff go elsewhere or in a new module?
>> Wed Feb 10 12:26:01 GMT 2010  Petr Rockai <me@mornfall.net>
>>   * Replace SlurpDirectory usage in Commands.Add with Tree-based code.
>
> Likewise for Darcs.Utils

Well, existsAnycase could probably go to hashed-storage. For the rest, I
guess Darcs.Utils is as good place as any. We could create TreeUtils for
the 4 functions but sounds like overkill. I'd eventually like to flip
more of the code to use AnchoredPath directly, and the 4 helpers can go
to hashed-storage in AnchoredPath variant. Or maybe just move them right
away and put up with the extra floatPath ugliness on client side of
code. At least it'll provide further incentive to convert things about.

For those *Really* bits, I have no idea why they are there at
all. Should be removed eventually.

Yours,
   Petr.

Could we please get a status update?  Looks like this ticket hasn't been 
updated for a while.

Thanks,
Jason

This is my current version of the slurpkill patch. It still needs work, but at
least it solves the big roadblock of producing useless pristine caches (by
converting all the way when needed).

I'll go through the review points and re-submit again in a while.

19 patches for repository http://darcs.net:

Mon Feb  8 23:47:04 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove implementation of --store-in-memory, simplifying matcher code.

Wed Feb 10 13:26:01 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace SlurpDirectory usage in Commands.Add with Tree-based code.

Thu Feb 11 00:47:11 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement the Slurp-based file/dir existence-check functions in terms of Trees.

Thu Feb 11 00:48:20 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-work Commands.Add (simplify, use the new treeHas* functions).

Thu Feb 11 00:50:00 CET 2010  Petr Rockai <me@mornfall.net>
  * Generalize announceFiles used by whatsnew and use it in record as well.

Thu Feb 11 00:57:54 CET 2010  Petr Rockai <me@mornfall.net>
  * Purge Slurpy usage in Commands.Rollback (use announceFiles from whatsnew).

Thu Feb 11 01:13:26 CET 2010  Petr Rockai <me@mornfall.net>
  * Use stock setScriptsExecutable from Darcs.Repository in Commands.Convert.

Thu Feb 11 01:13:53 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement optimize --relink using Trees instead of Slurpies.

Thu Feb 11 01:14:35 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement setScriptsExecutable using Trees instead of Slurpies.

Thu Feb 11 10:31:53 CET 2010  Petr Rockai <me@mornfall.net>
  * Port Commands.Move from Slurpy to Tree.

Fri Feb 12 11:17:51 CET 2010  Petr Rockai <me@mornfall.net>
  * Cannot "darcs remove" non-existent files.

Fri Feb 12 11:18:44 CET 2010  Petr Rockai <me@mornfall.net>
  * Use a more canonic way to create empty hashed pristine in optimize (--upgrade).

Fri Feb 12 11:28:42 CET 2010  Petr Rockai <me@mornfall.net>
  * Reimplement applyHashed in terms of hashedTreeIO (Storage.Hashed.Monad).

Fri Feb 12 11:32:58 CET 2010  Petr Rockai <me@mornfall.net>
  * Use a more canonic way to create empty hashed pristine.

Fri Feb 12 11:33:22 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace slurp_recorded with readRecorded in make_new_pending.

Fri Feb 12 11:36:41 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove SlurpDirectory.

Thu Feb 25 07:30:37 CET 2010  Petr Rockai <me@mornfall.net>
  * Avoid use of SlurpDirectory in Commands.ShowFiles.

Thu Feb 25 07:35:13 CET 2010  Petr Rockai <me@mornfall.net>
  * Purge unused fileExists from Commands.Record.

Fri Mar 19 15:37:31 CET 2010  Petr Rockai <me@mornfall.net>
  * Automatically drop size prefixes from all of pristine when needed.

Attachments

• remove-implementation-of-__store_in_memory_-simplifying-matcher-code_.dpatch
• unnamed

This adds two new patches on top of the previous stack, fixing things pointed
out in review.

What mostly remains is Darcs.Utils which I would defer for myself, since I
can't be currently bothered to look into that. And the hardcoded paths, about
which in my attempt at preventing a flamewar I mostly also prevented a
discussion. (Hm.) Not yet decided. I'll patch it up on top, anyway, somehow.

Yours,
   Petr.

5 patches for repository http://darcs.net:

Thu Feb 11 01:14:35 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement setScriptsExecutable using Trees instead of Slurpies.

Fri Feb 12 11:28:42 CET 2010  Petr Rockai <me@mornfall.net>
  * Reimplement applyHashed in terms of hashedTreeIO (Storage.Hashed.Monad).

Fri Mar 19 15:37:31 CET 2010  Petr Rockai <me@mornfall.net>
  * Automatically drop size prefixes from all of pristine when needed.

Fri Mar 19 16:04:05 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove a couple of unused Cache parameters.

Fri Mar 19 16:12:42 CET 2010  Petr Rockai <me@mornfall.net>
  * Filter out _darcs in setScriptsExecutable.

Attachments

• re_implement-setscriptsexecutable-using-trees-instead-of-slurpies_.dpatch
• unnamed

Status update: I need to look at the latest round of updates properly.

We also need to deal with the issue of the hashed filenames in pristine, 
which Petr raised on the mailing list some weeks back. In the past, 
darcs has varied between using "size-hash" and "hash", the most recent 
choice being size-hash which means that almost all existing repositories 
will use that. With the advent of hashed-storage, there's no need for 
the size-hash format, and the ambiguity leads to there not being any 
unique hash for directories. Hashed-storage also doesn't write size-hash 
format, in the interests of code simplicity.

The problem with directories not having unique hashes is that diff 
becomes less efficient, because it can't detect unchanged directories as 
often. So ideally we'd migrate all repos to completely use hash.

Obviously this will involve some pain. The situation will be worst for 
people using different versions of darcs (pre-2.5 and 2.5) on the same 
repo via direct filesystem access - e.g. NFS mounts etc. I believe this 
is a rare but not unknown situation.

As I understand it the current submission eagerly switches to hash 
format when any write is made to pristine and a size-hash file is found. 
It does this in a relatively inefficient but simple way, just writing 
out the whole of pristine again. A smarter implementation that renames 
files as needed would be possible.

Whether we use a more efficient conversion or not, I believe we should 
at least print out a warning to the user explaining why things are slow. 

The alternative to automatically converting on every write is to just 
print out a warning recommending that the user run darcs optimize --
pristine.

Petr, please correct me if any of this is wrong.

Hi,

this is the current incarnation of the noslurps patchset. It should work with
current darcs HEAD (it also needs hashed-storage HEAD). I have resolved
conflicts and I have added a warning about pristine conversion.

As soon as this is merged, I will move on to the newset patch (it will need
some amending both due to noslurps and other intervening work). Also, I will
probably start working on the Cache code, since it is in a desperate need of
repair. I have mostly ignored Cache so far in the new pristine handling code,
but this comes with one downside: pristine hardlinking is no longer
automatically maintained. I'll try to restore that feature during the cache
refactor.

Yours,
   Petr.

PS: IIRC, there was some slight collateral damage from noslurps, but I can't
recall it anymore and the testsuite does not seem to complain. If you notice
something, please shout ASAP.

32 patches for repository http://darcs.net:

Mon Feb  8 23:47:04 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove implementation of --store-in-memory, simplifying matcher code.

Wed Feb 10 13:26:01 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace SlurpDirectory usage in Commands.Add with Tree-based code.

Thu Feb 11 00:47:11 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement the Slurp-based file/dir existence-check functions in terms of Trees.

Thu Feb 11 00:48:20 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-work Commands.Add (simplify, use the new treeHas* functions).

Thu Feb 11 00:50:00 CET 2010  Petr Rockai <me@mornfall.net>
  * Generalize announceFiles used by whatsnew and use it in record as well.

Thu Feb 11 00:57:54 CET 2010  Petr Rockai <me@mornfall.net>
  * Purge Slurpy usage in Commands.Rollback (use announceFiles from whatsnew).

Thu Feb 11 01:13:26 CET 2010  Petr Rockai <me@mornfall.net>
  * Use stock setScriptsExecutable from Darcs.Repository in Commands.Convert.

Thu Feb 11 01:13:53 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement optimize --relink using Trees instead of Slurpies.

Thu Feb 11 01:14:35 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement setScriptsExecutable using Trees instead of Slurpies.

Thu Feb 11 10:31:53 CET 2010  Petr Rockai <me@mornfall.net>
  * Port Commands.Move from Slurpy to Tree.

Fri Feb 12 11:17:51 CET 2010  Petr Rockai <me@mornfall.net>
  * Cannot "darcs remove" non-existent files.

Fri Feb 12 11:18:44 CET 2010  Petr Rockai <me@mornfall.net>
  * Use a more canonic way to create empty hashed pristine in optimize (--upgrade).

Fri Feb 12 11:28:42 CET 2010  Petr Rockai <me@mornfall.net>
  * Reimplement applyHashed in terms of hashedTreeIO (Storage.Hashed.Monad).

Fri Feb 12 11:32:58 CET 2010  Petr Rockai <me@mornfall.net>
  * Use a more canonic way to create empty hashed pristine.

Fri Feb 12 11:33:22 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace slurp_recorded with readRecorded in make_new_pending.

Fri Feb 12 11:36:41 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove SlurpDirectory.

Thu Feb 25 07:30:37 CET 2010  Petr Rockai <me@mornfall.net>
  * Avoid use of SlurpDirectory in Commands.ShowFiles.

Thu Feb 25 07:35:13 CET 2010  Petr Rockai <me@mornfall.net>
  * Purge unused fileExists from Commands.Record.

Fri Mar 19 15:37:31 CET 2010  Petr Rockai <me@mornfall.net>
  * Automatically drop size prefixes from all of pristine when needed.

Fri Mar 19 16:04:05 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove a couple of unused Cache parameters.

Wed Apr 28 22:34:46 CEST 2010  Petr Rockai <me@mornfall.net>
  * Make the automatic pristine conversion a bit nicer.

Wed Apr 28 22:36:29 CEST 2010  Petr Rockai <me@mornfall.net>
  * Filter out _darcs in setScriptsExecutable.

Wed Apr 28 22:48:11 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Commands.Remove.

Wed Apr 28 22:50:30 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Commands.Convert.

Wed Apr 28 22:52:31 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Commands.ShowFiles.

Wed Apr 28 22:53:29 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Repository.Internal.

Wed Apr 28 22:53:42 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflict in Commands.Move.

Wed Apr 28 23:04:41 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Match.

Wed Apr 28 23:05:05 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Commands.Add.

Wed Apr 28 23:05:10 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflict in Commands.Record.

Wed Apr 28 23:53:37 CEST 2010  Petr Rockai <me@mornfall.net>
  * Make clean_hashdir work on repository-local dirs even without caches.

Wed Apr 28 23:54:46 CEST 2010  Petr Rockai <me@mornfall.net>
  * Clean up unused bits in Repository.HashedIO.

Attachments

• remove-implementation-of-__store_in_memory_-simplifying-matcher-code_.dpatch
• unnamed

Hi,

I have amended some patches (mostly to reduce conflictness). This also means
that some of the patches from previous bundles have disappeared. This bundle is
the complete branch now and nothing else from this ticket should be applied.

I have mainline + this bundle as my current main darcs and I am about to run
some benchmarks on it. It also passes the testsuite. I am using hashed-storage
HEAD along with it (and it won't compile with any released version). So this is
something that will need to be addressed.

I'd probably release something like h-s 0.5 so we can continue the 0.4.x line
for darcs 2.4.x. The changes are not backwards-incompatible per se, but I'd
rather err on the safe side. When this set is reviewed, we just need to release
h-s, bump the dependencies on HEAD (there's another patch to bump the mmap
dependency upper bound, since h-s HEAD requires 0.5+).

Yours,
   Petr.

29 patches for repository http://darcs.net:

Mon Feb  8 23:47:04 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove implementation of --store-in-memory, simplifying matcher code.

Wed Feb 10 13:26:01 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace SlurpDirectory usage in Commands.Add with Tree-based code.

Thu Feb 11 00:47:11 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement the Slurp-based file/dir existence-check functions in terms of Trees.

Thu Feb 11 00:48:20 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-work Commands.Add (simplify, use the new treeHas* functions).

Thu Feb 11 00:50:00 CET 2010  Petr Rockai <me@mornfall.net>
  * Generalize announceFiles used by whatsnew and use it in record as well.

Thu Feb 11 01:13:53 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement optimize --relink using Trees instead of Slurpies.

Thu Feb 11 01:14:35 CET 2010  Petr Rockai <me@mornfall.net>
  * Re-implement setScriptsExecutable using Trees instead of Slurpies.

Thu Feb 11 10:31:53 CET 2010  Petr Rockai <me@mornfall.net>
  * Port Commands.Move from Slurpy to Tree.

Fri Feb 12 11:18:44 CET 2010  Petr Rockai <me@mornfall.net>
  * Use a more canonic way to create empty hashed pristine in optimize (--upgrade).

Fri Feb 12 11:28:42 CET 2010  Petr Rockai <me@mornfall.net>
  * Reimplement applyHashed in terms of hashedTreeIO (Storage.Hashed.Monad).

Fri Feb 12 11:32:58 CET 2010  Petr Rockai <me@mornfall.net>
  * Use a more canonic way to create empty hashed pristine.

Fri Feb 12 11:33:22 CET 2010  Petr Rockai <me@mornfall.net>
  * Replace slurp_recorded with readRecorded in make_new_pending.

Fri Feb 12 11:36:41 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove SlurpDirectory.

Thu Feb 25 07:30:37 CET 2010  Petr Rockai <me@mornfall.net>
  * Avoid use of SlurpDirectory in Commands.ShowFiles.

Thu Feb 25 07:35:13 CET 2010  Petr Rockai <me@mornfall.net>
  * Purge unused fileExists from Commands.Record.

Fri Mar 19 15:37:31 CET 2010  Petr Rockai <me@mornfall.net>
  * Automatically drop size prefixes from all of pristine when needed.

Fri Mar 19 16:04:05 CET 2010  Petr Rockai <me@mornfall.net>
  * Remove a couple of unused Cache parameters.

Wed Apr 28 22:34:46 CEST 2010  Petr Rockai <me@mornfall.net>
  * Make the automatic pristine conversion a bit nicer.

Wed Apr 28 22:36:29 CEST 2010  Petr Rockai <me@mornfall.net>
  * Filter out _darcs in setScriptsExecutable.

Wed Apr 28 22:52:31 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Commands.ShowFiles.

Wed Apr 28 22:53:29 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Repository.Internal.

Wed Apr 28 22:53:42 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflict in Commands.Move.

Wed Apr 28 23:04:41 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Match.

Wed Apr 28 23:05:05 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflicts in Commands.Add.

Wed Apr 28 23:05:10 CEST 2010  Petr Rockai <me@mornfall.net>
  * Resolve conflict in Commands.Record.

Wed Apr 28 23:53:37 CEST 2010  Petr Rockai <me@mornfall.net>
  * Make clean_hashdir work on repository-local dirs even without caches.

Wed Apr 28 23:54:46 CEST 2010  Petr Rockai <me@mornfall.net>
  * Clean up unused bits in Repository.HashedIO.

Wed May  5 21:00:59 CEST 2010  Petr Rockai <me@mornfall.net>
  * Use stock setScriptsExecutable from Darcs.Repository in Commands.Convert.

Wed May  5 21:52:35 CEST 2010  Petr Rockai <me@mornfall.net>
  * Purge Slurpy usage in Commands.Rollback (use announceFiles from whatsnew).

Attachments

• remove-implementation-of-__store_in_memory_-simplifying-matcher-code_.dpatch
• unnamed

ok, I think this looks fine. Eric has OK'd removal of --store-in-memory 
on the basis that it's unused and not worth going through a huge process 
for.

What's in hashed-storage HEAD that's not in a hashed-storage release? I 
only see this patch which seems inconsequential:

Sun May  2 14:29:44 EDT 2010  Petr Rockai <me@mornfall.net>
  * Fix build on GHC 6.12 on Windows.

Let me know if there are any other hashed-storage changes I should 
review. Otherwise please commit the bundle yourself when ready along 
with any version bumps needed.

Petr sent an email that didn't make it to the bug tracker, pointing out 
that hashed-storage does have more changes, but the 0.4.12 tag isn't 
clean so I missed them. 

I've now read the changes and they look fine, apart from an error 
message "Boo" or similar, which will be very confusing if it ever makes 
it to a darcs user.

This patch bundle (with 29 patches) was just applied to the repository http://darcs.net/.
This message was brought to you by DarcsWatch
http://darcswatch.nomeata.de/repo_http:__darcs.net_.html#bundle-a3ba5ab4a2c40b06c999bfa67c8de3fb68e19616

This patch bundle (with 29 patches) was just applied to the repository http://darcs.net/reviewed.
This message was brought to you by DarcsWatch
http://darcswatch.nomeata.de/repo_http:__darcs.net_reviewed.html#bundle-a3ba5ab4a2c40b06c999bfa67c8de3fb68e19616