Completely at odds with norma Unix behaviour, Darcs has a habit of
creating temporary files in the working tree, instead of in $TMPDIR or
/tmp.  Nor do they have mkstemp-style names (-0 instead of .XXXXXX).

This confuses our users, e.g.

13:35 <dino-> I just did a darcs tag with long comment. What is the 'darcs-tag-0' file that it left in the project root dir about?
13:36 <dino-> Is that normal?
13:36 <twb> dino-: it's a backup file :-(
13:36 <twb> Darcs is stupid and doesn't know how to name backup files properly, or to put them in $TMPDIR
13:36 <dino-> The tag otherwise seemed to work, looks fine in cha output
13:37 <dino-> So harmless bug?
13:37 <twb> dino-: basically you can just delete that file

AFAIK these files are also not in the default boring list (so darcs
add and darcs record -l want to version control them), and often they
are not deleted.  Presumably the excuse for this is to avoid data
loss, but IMO the current behaviour is excessive.

Currently my repo has:

    darcs-amend-record-0
    darcs-amend-record-0.~1~
    darcs-amend-record-1
    darcs-amend-record-1.~1~
    darcs-amend-record-2
    darcs-amend-record-2.~1~
    darcs-record-0 darcs-temp-mail-0
    darcs-temp-mail-0.~1~
    darcs-temp-mail-0.~2~
    darcs-temp-mail-0.~3~
    darcs-temp-mail-1
    darcs-temp-mail-1.~1~
    darcs.cabal-darcs-backup0
    darcs9599-new_3c63f32f823ff92f99d802c48a9ddd0a
    darcs9603-new_fe335076796c541830bc54310d35cec7

So incidentally, I think that -darcs-backupN should instead be .~N+1~,
the standard GNU and CVS style for numbered backups.

On Mon, Apr 06, 2009 at 03:37:09 -0000, Trent Buck wrote:
> Completely at odds with norma Unix behaviour, Darcs has a habit of
> creating temporary files in the working tree, instead of in $TMPDIR or
> /tmp.

So darcs knows about $TMPDIR and other similar paths (see tempdir_loc).
The question is when it should be using these locations.

In commands which generate a log, the current code is

 make_log = world_readable_temp "darcs-amend-record"

Perhaps there needs to be a variant of world_readable_temp that
generates files in that location.

Also, we should be slightly careful to consider (a) reasons that we may
want to keep these logs around (darcs record --logfile?) and (b) cases
where we really do want them to be in the working directory (e.g.
backups)

I'll mark this ProbablyEasy in any case

Marking as a feature and ProbablyEasy.

Also, perhaps the discoverability of these log files is a good thing?

Is there a deeper problem like - the log files are actually being created in the
right place (we really do want them in working), but we're just not cleaning
them up in the cases where darcs record is successful?  Maybe the real behaviour
we're aiming for is for the log files to stick around only if there is a
problem, like a test that fails.

On Tue, Apr 07, 2009 at 04:38:29PM -0000, Eric Kow wrote:
> Also, perhaps the discoverability of these log files is a good
> thing?

I concede that's a point for the status quo, but I think matters could
be improved by making them LOOK like backup files.  Having said that,
tools like mutt and reportbug happily create their analogous working
files in /tmp, and that doesn't seem to bother anybody.

> Is there a deeper problem like - the log files are actually being
> created in the right place (we really do want them in working), but
> we're just not cleaning them up in the cases where darcs record is
> successful?

No, I think Darcs is pretty good in that respect.  But for me the
problem with this is that I run "darcs record" and "darcs amend"
*expecting* the tests to fail.  That's the only way I know to run the
commit test!  (I could use "darcs check", but that will take
deciminutes or even hours because I'm working on a lazy darcs-2
repository and there is a log of history to download and validate.)

> Maybe the real behaviour we're aiming for is for the log files to
> stick around only if there is a problem, like a test that fails.

That's what mutt and reportbug do, but they use /tmp (or TMPDIR, if
it's defined).  Then, if there's a problem, they actually print on
stdout

    I had a problem.  The file you were editing is available at
    /tmp/mutt-XXXXXX-XXX-XXX

On Wed, Apr 08, 2009 at 07:40:35 -0000, Trent Buck wrote:
> > Maybe the real behaviour we're aiming for is for the log files to
> > stick around only if there is a problem, like a test that fails.
> 
> That's what mutt and reportbug do, but they use /tmp (or TMPDIR, if
> it's defined).  Then, if there's a problem, they actually print on
> stdout
> 
>     I had a problem.  The file you were editing is available at
>     /tmp/mutt-XXXXXX-XXX-XXX

Sounds like a much nicer plan (assuming we can always know when we've
had a problem, which we do for failing tests, at least)

On Wed, Apr 08, 2009 at 11:03:39AM -0000, Eric Kow wrote:
> On Wed, Apr 08, 2009 at 07:40:35 -0000, Trent Buck wrote:
>>> Maybe the real behaviour we're aiming for is for the log files to
>>> stick around only if there is a problem, like a test that fails.
>>
>> That's what mutt and reportbug do, but they use /tmp (or TMPDIR, if
>> it's defined).  Then, if there's a problem, they actually print on
>> stdout
>>
>>     I had a problem.  The file you were editing is available at
>>     /tmp/mutt-XXXXXX-XXX-XXX
>
> Sounds like a much nicer plan (assuming we can always know when we've
> had a problem, which we do for failing tests, at least)

Also I should mention that last time I asked for this (or something
similar?), David vetoed it on the basis that the working directory was
guaranteed to be writable, but the temp area might not... or something
equally strange but vaguely plausible.  It ought to be in the
darcs-users archive if someone wants to dig it up.

I need somebody to track down David's objection on the mailing list, link to it
and comment on plausibility of his objection and solutions.  

This would be a good time to compare tradeoffs: how likely is this to fail?
would a backing off strategy make the code pointlessly ugly? Note that users
could always tell darcs what directory to use as temp.

Seems like you could start with temp and back off to the working dir as
a last resort.  Note Trent's suggestion to emit a message about the log
file's location.

PS. I've copied Trent's suggestion about the name of backup files to
issue1812

I wonder if this is related: http://lists.osuosl.org/pipermail/darcs-
users/2008-May/012049.html

kowey's link had a line break in the middle, so here's another copy:
http://lists.osuosl.org/pipermail/darcs-users/2008-May/012049.html

What do other VCSes do in this case, or do they just not create 
temporary files at all? I tend to agree with David's concerns over 
security. Perhaps we should store these files in _darcs/tmp ?

Note that the backup file naming issue is now fixed and I personally 
think they should remain next to the original files in the working 
directory.

Hi!


> Perhaps we should store these files in _darcs/tmp ?
>
It seems a good idea, I go for it?

Something I was thinking is that store all the temps files together can
be a little messy. I mean, dont know when or how(if exist, for some
reason, a darcs command) erase the temps files, but it can happen
that temp dir have ~400 files (till ~100 for every command, for the
commands send, tag, record and amendrecord)
Again, I dont know if this can be annoying. I suppose, if it is, we
can have temps dirs for every command (_darcs/temp/temp_*cmdName*)
and store there the corresponding temp files.

I know I dont know many things :)
Sorry if what I'm saying doesnt make sense jeje.

cheers.
-- 
Ale Gadea

Attachments

• unnamed

One crude strategy may be to bucket them by date. Although end users
should not be poking around in _darcs, it also gives a natural sense
of what you can delete, and can serve for future forensics.

It's also one of these situations where because we aren't necessarily
too sure how to proceed, the best thing may be to submit the patch and
see what discussion it sparks.  Concreteness tends to be a great
motivator :-)

(needless to say, this may create unnecessary work for you, but if you
are in the code-to-think school of thought, it may be a useful thing
to do anyway)


On 26 March 2014 17:15, Ale Gadea <alex.aegf@gmail.com> wrote:
> Hi!
>
>>
>> Perhaps we should store these files in _darcs/tmp ?
>
> It seems a good idea, I go for it?
>
> Something I was thinking is that store all the temps files together can
> be a little messy. I mean, dont know when or how(if exist, for some
> reason, a darcs command) erase the temps files, but it can happen
> that temp dir have ~400 files (till ~100 for every command, for the
> commands send, tag, record and amendrecord)
> Again, I dont know if this can be annoying. I suppose, if it is, we
> can have temps dirs for every command (_darcs/temp/temp_cmdName)
> and store there the corresponding temp files.
>
> I know I dont know many things :)
> Sorry if what I'm saying doesnt make sense jeje.
>
> cheers.
> --
> Ale Gadea
>
> _______________________________________________
> darcs-devel mailing list
> darcs-devel@darcs.net
> http://lists.osuosl.org/mailman/listinfo/darcs-devel
>



-- 
Eric Kow <http://erickow.com>

A final question... (really?)

One crude strategy may be to bucket them by date. Although end users
> should not be poking around in _darcs, it also gives a natural sense
> of what you can delete, and can serve for future forensics.
>
Oh right, so the format for a temp file could be
_darcs/temp/DDMMYY/cmdName-N,
may be the DD it's to much?


> It's also one of these situations where because we aren't necessarily
> too sure how to proceed, the best thing may be to submit the patch and
> see what discussion it sparks.  Concreteness tends to be a great
> motivator :-)
>
;) I'm on it.

-- 
Ale Gadea

Attachments

• unnamed

About bucketting: I don't get where the figure of 100 temporary files
per command comes from (Ale's comment). The files created with
worldReadableTemp are either the long comment descriptions (for commands
record, amend, tag) or the patch bundle mail message (for send). They
remain if the command does not end properly (failing test, misconfigured
sendmail, CTRL+C...). So that's a maximum of 1 file per command
execution.  Bucketting would be overkill.

Also I think the first two comments of this thread should be taken with
a grain of salt.  First, we are not talking about backup files but files
*needed* to let you use a third party text editor to edit the long
comment of a patch or the body of a mail to send a patch bundle.

And second, many of the files of the 2nd comment (msg7581) are *not* the
files we are talking about. All the ones ending with .~n~ were extra
backups made by some text editor; then darcs.cabal-darcs-backup0 is the
result of applying a conflicting patch (this file should be now named
darcs.cabal.~1~ since http://bugs.darcs.net/issue1812 was fixed).
Finally darcs9599-new_3c63... comes from another place (getting?
pulling?) and should be investigated separately.

Here is an excerpt from the help of hg commit:

   If no commit message is specified, Mercurial starts your
   configured editor where you can enter a message. In case your
   commit fails, you will find a backup of your message in
   ``.hg/last-message.txt``.

On 27/03/2014 04:08, Ale Gadea wrote:
> A final question... (really?) 
> 
>     One crude strategy may be to bucket them by date. Although end users
>     should not be poking around in _darcs, it also gives a natural sense
>     of what you can delete, and can serve for future forensics.
> 
> Oh right, so the format for a temp file could be
> _darcs/temp/DDMMYY/cmdName-N,
> may be the DD it's to much?

I think if you're using the date you might as well include the day.

A minor tweak would be to put it in an order that sorts nicely, e.g.
YYMMDD or perhaps go down the ISO format route (YYYY-MM-DD).

Ganesh

Great dissection, very useful reminder that users don't care about the
subte conceptual distinctions (one should they have to); they just see the
symptom and go bleh. It's up to us to tease thins apart like Guillaume did

Comments inline

On Thursday, 27 March 2014, Guillaume Hoffmann <bugs@darcs.net> wrote:
>
> About bucketting: I don't get where the figure of 100 temporary files
> per command comes from (Ale's comment). The files created with
> worldReadableTemp are either the long comment descriptions (for commands
> record, amend, tag) or the patch bundle mail message (for send). They
> remain if the command does not end properly (failing test, misconfigured
> sendmail, CTRL+C...). So that's a maximum of 1 file per command
> execution.  Bucketting would be overkill.


Perhaps worthwhile to just try the simplest thing. If you want to bucket,
bucket with ISO dates, but simpler would be a good start

One motivation maybe is that temp files can build up over multiple
executions of darcs if the user does not delete them



> Finally darcs9599-new_3c63... comes from another place (getting?
> pulling?) and should be investigated separately.


This is where I see the most clutter


> Here is an excerpt from the help of hg commit:
>
>    If no commit message is specified, Mercurial starts your
>    configured editor where you can enter a message. In case your
>    commit fails, you will find a backup of your message in
>    ``.hg/last-message.txt``.
>

Sort of thing that should only happen after a failure


-- 
Eric Kow <http://erickow.com>

Attachments

• unnamed

>>
>> Here is an excerpt from the help of hg commit:
>>
>>    If no commit message is specified, Mercurial starts your
>>    configured editor where you can enter a message. In case your
>>    commit fails, you will find a backup of your message in
>>    ``.hg/last-message.txt``.
>
>
> Sort of thing that should only happen after a failure

Maybe. Mercurial leaves it there all the time. In fact, it first edits
some random-named file in /tmp/ then copies it to
.hg/last-message.txt.

Git, on the other hand, directly edits (and always leaves) the file
.git/COMMIT_EDITMSG .


> Perhaps worthwhile to just try the simplest thing. If you want to bucket,
> bucket with ISO dates, but simpler would be a good start

Simpler is what we have, with increasing numbers :-)

Even simpler is (like hg or git) overwriting the same file over and over,

> One motivation maybe is that temp files can build up over multiple
> executions of darcs if the user does not delete them

A point for overwriting a single file.

(In fact I'm feeling currently more inclined towards that solution.)

>> Finally darcs9599-new_3c63... comes from another place (getting?
>> pulling?) and should be investigated separately.
>
> This is where I see the most clutter

I've created a new ticket for them: http://bugs.darcs.net/issue2362

On 27 March 2014 at 19:25:50, Guillaume Hoffmann (guillaumh@gmail.com) wrote:
> >>
> >> Here is an excerpt from the help of hg commit:
> >>
> >> If no commit message is specified, Mercurial starts your
> >> configured editor where you can enter a message. In case your
> >> commit fails, you will find a backup of your message in
> >> ``.hg/last-message.txt``.
> >
> >
> > Sort of thing that should only happen after a failure
>
> Maybe. Mercurial leaves it there all the time. In fact, it first edits
> some random-named file in /tmp/ then copies it to
> .hg/last-message.txt.

OH, as for the file creation, no comment.
I was just thinking about the wall of text we might emit, my thinking
being that we should be fairly terse (while still trying to be
helpful) and not say stuff if it's just going to be ignored, or
confuse people

> > One motivation maybe is that temp files can build up over multiple
> > executions of darcs if the user does not delete them
>
> A point for overwriting a single file.
>
> (In fact I'm feeling currently more inclined towards that solution.)

Is that safe? For example, if I am running multiple darcsen
concurrently (due to absent-mindedness)…

On 27 March 2014 20:56, Guillaume Hoffmann <bugs@darcs.net> wrote:
>
> Guillaume Hoffmann <guillaumh@gmail.com> added the comment:
>
>>>
>>> Here is an excerpt from the help of hg commit:
>>>
>>>    If no commit message is specified, Mercurial starts your
>>>    configured editor where you can enter a message. In case your
>>>    commit fails, you will find a backup of your message in
>>>    ``.hg/last-message.txt``.
>>
>>
>> Sort of thing that should only happen after a failure
>
> Maybe. Mercurial leaves it there all the time. In fact, it first edits
> some random-named file in /tmp/ then copies it to
> .hg/last-message.txt.
>
> Git, on the other hand, directly edits (and always leaves) the file
> .git/COMMIT_EDITMSG .
>
>
>> Perhaps worthwhile to just try the simplest thing. If you want to bucket,
>> bucket with ISO dates, but simpler would be a good start
>
> Simpler is what we have, with increasing numbers :-)
>
> Even simpler is (like hg or git) overwriting the same file over and over,
>
>> One motivation maybe is that temp files can build up over multiple
>> executions of darcs if the user does not delete them
>
> A point for overwriting a single file.
>
> (In fact I'm feeling currently more inclined towards that solution.)
>
>>> Finally darcs9599-new_3c63... comes from another place (getting?
>>> pulling?) and should be investigated separately.
>>
>> This is where I see the most clutter
>
> I've created a new ticket for them: http://bugs.darcs.net/issue2362
>
> __________________________________
> Darcs bug tracker <bugs@darcs.net>
> <http://bugs.darcs.net/issue1416>
> __________________________________
> _______________________________________________
> darcs-devel mailing list
> darcs-devel@darcs.net
> http://lists.osuosl.org/mailman/listinfo/darcs-devel



-- 
Eric Kow <http://erickow.com>

> OH, as for the file creation, no comment.
> I was just thinking about the wall of text we might emit, my thinking
> being that we should be fairly terse (while still trying to be
> helpful) and not say stuff if it's just going to be ignored, or
> confuse people

This text is part of the output of "hg help commit", it's not a piece
of text that appears when one does a commit (if that's what you
meant). I'd say it's okay to have it there.

>> A point for overwriting a single file.
>>
>> (In fact I'm feeling currently more inclined towards that solution.)
>
> Is that safe? For example, if I am running multiple darcsen
> concurrently (due to absent-mindedness)…

"record", "amend" and "tag" take the repository lock so with them we're safe.

"send", on the other hand, does not, so…

1) either we keep on using the same system of increasing suffixes (ie,
more than one file)
2) or we detect if there's already a darcs-send file and either a)
refuse to run  or b) print a warning and go ahead overwriting it

If you look at git's source code, they implement 2) b) but using two
filenames: one (say) `darcs-send` when you're currently editing the
file, and one `darcs-send.final` that is used after the editor process
has stopped. I *guess* that's what's going on, because it's Perl code:

<https://github.com/git/git/blob/master/git-send-email.perl#L255>

I was reading and compiling all the comments and recommendations, and
I almost have the following:

1. Temps files saved in "_darcs/tmp/"
2. Format of temps files darcs-cmd-N (ie, no changes)

I think that is the most conservative change and could be a little step in
case of implement the overwriting of the same file over and over (like hg
or git)

Now, a problem with this is that I have to create by hand the tmp dir in
"_darcs/", I supose that one can implement the creation in case of that
doesn't
exist but the right way should be create the folder with "darcs init".

Another problem, this much more related with the implementation, is that
because the naive implementation of, prefixing "_darcs/tmp" to the result of
worldReadableTemp doesn't work(*). The solution I have is, say, not-pretty
and consist of make a general function,

*worldReadableTemp' :: TempType -> String -> IO String*

and redefine

*worldReadableTemp = worldReadableTemp' NormalTmp* (this has the normal
semantics)

and make the new function for create temp command files

*worldReadableCmdTemp = worldReadableTemp' CmdTmp* (this prefix
"_darcs/temp")

Why I comment all this?; Well, first the correct solution I think it be
that the prefix
of "_darcs/tmp" doesn't break all(*), but I don't sure if that can be made.
Second,
maybe there are a simplier solution.

(*) Doesn't work because, prefixing without being careful breaks the
creation of
others temp files like this,

"_darcs/tmp/_darcs/patches/pending.tentative-0"

Maybe using only "_darcs", allows a prettier solution using the "older"
worldReadableTemp with the prefix of  "_darcs", again I not sure because I
don't know much of the code.

So, in summary i will be sending a patch with the implementation that
briefly
I comment above. If that seems ok :)

cheers!


2014-03-27 20:15 GMT-03:00 Guillaume Hoffmann <bugs@darcs.net>:

>
> Guillaume Hoffmann <guillaumh@gmail.com> added the comment:
>
> > OH, as for the file creation, no comment.
> > I was just thinking about the wall of text we might emit, my thinking
> > being that we should be fairly terse (while still trying to be
> > helpful) and not say stuff if it's just going to be ignored, or
> > confuse people
>
> This text is part of the output of "hg help commit", it's not a piece
> of text that appears when one does a commit (if that's what you
> meant). I'd say it's okay to have it there.
>
> >> A point for overwriting a single file.
> >>
> >> (In fact I'm feeling currently more inclined towards that solution.)
> >
> > Is that safe? For example, if I am running multiple darcsen
> > concurrently (due to absent-mindedness)...
>
> "record", "amend" and "tag" take the repository lock so with them we're
> safe.
>
> "send", on the other hand, does not, so...
>
> 1) either we keep on using the same system of increasing suffixes (ie,
> more than one file)
> 2) or we detect if there's already a darcs-send file and either a)
> refuse to run  or b) print a warning and go ahead overwriting it
>
> If you look at git's source code, they implement 2) b) but using two
> filenames: one (say) `darcs-send` when you're currently editing the
> file, and one `darcs-send.final` that is used after the editor process
> has stopped. I *guess* that's what's going on, because it's Perl code:
>
> <https://github.com/git/git/blob/master/git-send-email.perl#L255>
>
> __________________________________
> Darcs bug tracker <bugs@darcs.net>
> <http://bugs.darcs.net/issue1416>
> __________________________________
>



-- 
Ale Gadea

Attachments

• unnamed

The following patch sent by Ale Gadea <alex.aegf@gmail.com> updated issue Issue1416 with
status=resolved;resolvedin=2.10.0 HEAD

* Resolve Issue1416: put log files in tempdir instead of in working dir. 
Ignore-this: 645ddf9ddd197e306a0c5209ed321b9
  
The temporary files are found in _darcs.
For the commands amend-record, record and tag, the temp file is patch_description.txt
and is persistent, ie always have the last message whether failed or not the command.
For send, we use two files, darcs-send for the editing and darcs-send.final for the 
backup. If all is ok, the files are removed.