Previously I was doing web scraping to create a useful mbox of an
arbitrary roundup issue.  This morning Simon "sm" Michael set up an
extension found on the roundup wiki, to allow extracting the mbox
directly from the database backend.

This implementation is still imperfect (and being server-side,
requires an admin to fix :-( ).  Particularly:

There needs to be a Reply-To: bugs@darcs.net header, so people viewing
the mbox can just hit "reply" and have their comments go to the right
place.

The subject line should contain a meaningful summary.  Apparently
roundup does not preserve the original subject line, so it should be
recreted to contain

- the string [issueNNN] at the start.  This is CRITICAL because
  otherwise replying to the email in mutt will create a new bug.

- the issue title when the email was sent (or if that's not kept, the
  current issue title).

- the property changes associated with the email, in a similar format
  to the inbound email.

For example, this message

    http://bugs.darcs.net/msg130

would have

    Subject: [issue33] wish: improve "darcs --xml" [nosy=+droundy,tommy,bortzmeyer]

Similarly because this history change did not have an associated
comment, it should become an mbox message with an empty (or stub)
body.  Otherwise just looking at the mbox, commentless changes made
via the web interface will be invisible to me.

  2005-11-30 14:01:01 droundy    set    nosy: droundy, tommy, bortzmeyer
                                        superseder: - wish: improve "darcs --xml"
  2005-11-30 14:01:01 droundy    unlink issue33 superseder

Thanks. The extension installed is 
http://www.mechanicalcat.net/tech/roundup/wiki/MessagesAsMbox which is activated 
using the "(mbox)" link on the right, above the first message of an issue, when 
displayed in the bug tracker.

We discussed this briefly on #darcs and you indicated that this issue is low 
priority (http://irclog.perlgeek.de/darcs/2008-10-31).

Best regards
Thorkil

I'm in a position to work on this in the near future.
Please supply me with the mbox.py you are using (I think it has been modified
from the one on mechanicalcat).  If it's easy to do so, please supply me with
a copy of the data that's actually in use, as an sqlite3 database file.

That will save me having to generate dummy data in my roundup-demo instance,
and will also help me catch edge cases.  For example, I've noticed that
messages created using the web interface have a message-id "None" in the
original mbox.py version.

Is this action on my part that's required?  If so, I'd appreciate a recipe for
doing it, or hints.

On Sun, Dec 28, 2008 at 10:48:18AM -0000, Eric Kow wrote:
> Is this action on my part that's required?

Yes, or someone else with appropriate admin access to the server.

> If so, I'd appreciate a recipe for doing it, or hints.

Firstly, look for extensions/mbox.py, and send me a copy.  It might
not be called "mbox", but extensions/ and .py should be the same.

As to dumping the database to sqlite, I don't know how to do that.
I'll manage without it until Simon is around.

I'm attaching messages_as_mbox.py

Attachments

• messages_as_mbox.py

Regarding email attachments, roundup.roundupdb.IssueClass().send_message() does everything just right -- except that the last line pushes the content to an SMTP server instead of to the HTTP daemon.  Probably the kind of copy-and-paste hack job that typifies real-world Python will be necessary, but it sure beats writing all that code myself.

Status update before bedtime.

Attached is the version of mbox.py that I have been tweaking today.
mbox.py,v is an RCS file that contains the revision history, too.

Simon, if you could integrate these changes into the copy of mbox.py
running on bugs.darcs.net sometime, that'd be awesome.  Especially
these lines mean that I can just hit "reply to" in mutt, and it will
do the right thing.  Unfortunately there is no way to have any
information in the Subject: line after [issueN], because it would
change the ticket's title.  (Personally I think that's a REALLY STUPID
thing of Roundup's, but there you have it.)

    a('Reply-To: ' + self.db.config.TRACKER_EMAIL)
    a('Subject: [issue' + self.nodeid + ']')

PS: hiding the email addresses as I've done also has a good
side-effect: in mutt, if I hit "reply to" on an email with my email
address in To:, mutt will not know where to send it.  But for "From:
Trent W. Buck:;", mutt will just use Reply-To :-)

Attachments

• mbox.py
• mbox.py,v

The current script apparently has a bug, I just found this case that breaks it:
http://bugs.darcs.net/issue904?@action=mbox
I should look into this sometime.

Now that the roundup repository is exposed to me, I have imported my
RCS change history as Darcs patches.  I did not preserve the
timestamps, because using --pipe is a fucking nightmare (see
issue1276).

This patch bundle includes at least one patch that was not applied
even as a bulk import, the one that provides X-Roundup-* headers.

Sat Mar 28 15:17:27 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Rollback bulk RCS import.

Sat Mar 28 15:25:25 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Rollback miscellaneous tweaks.

Sat Mar 28 15:26:32 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Messages submitted from the web UI have a Message-ID of None.

Sat Mar 28 15:26:53 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * A blank line is needed between the header and body.

Sat Mar 28 15:27:08 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Bind everything first.

Sat Mar 28 15:28:06 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Don't abuse the printf (%) DSL for simple concatenation.

Sat Mar 28 15:29:49 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Simplify message body loop.

Sat Mar 28 15:30:17 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Use the asctime() format dictated by mbox.

Sat Mar 28 15:34:38 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Suppress the recipient's email addresses, as replies should go to the ticket.
  Also list (in hidden format) all the people the message was sent to.

Sat Mar 28 15:39:02 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Set the Reply-To header field.

Sat Mar 28 15:39:16 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Set the Subject header field.
  This (and Reply-To) mean that I can just hit "reply to" in mutt, and
  it will do the right thing.  Unfortunately there is no way to have any
  information in the Subject: line after [issueN], because it would
  change the ticket's title.  IMO it's a huge misfeature in Roundup to
  always set the ticket title, but there you have it.

Sat Mar 28 15:39:58 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Typo: s/name/username/.
  Fixes the backtrace when trying to get an mbox with message(s) from
  people without GECOS information.

Sat Mar 28 15:41:10 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Improved address(), but left disabled as mutt breaks it.

Sat Mar 28 15:43:05 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Include important metadata as extended header fields.

Sat Mar 28 15:43:53 EST 2009  Trent W. Buck <trentbuck@gmail.com>
  * Stub for attachments.

Attachments

• rollback-bulk-rcs-import_.dpatch
• unnamed

Hi Trent,

On Sat, Mar 28, 2009 at 04:49:10 -0000, Trent Buck wrote:
> Now that the roundup repository is exposed to me, I have imported my
> RCS change history as Darcs patches.  I did not preserve the
> timestamps, because using --pipe is a fucking nightmare (see
> issue1276).
> 
> This patch bundle includes at least one patch that was not applied
> even as a bulk import, the one that provides X-Roundup-* headers.

Is my job here to apply these patches and push them to darcs.net?
They apply cleanly, for what it's worth.

Thanks!

I expect there'll be more like this. It seems odd to be filling up the  
darcs bug tracker's configuration repo with the history of Trent's  
roundup add-on. What about maintaining this in its own repo and  
periodically importing to the darcs roundup repo with single larger  
changes ("import twb's roundup mbox exporter, spring edition") ?

On Sat, Mar 28, 2009 at 03:16:23PM -0000, Eric Kow wrote:
> 
> Eric Kow <kowey@darcs.net> added the comment:
> 
> Hi Trent,
> 
> On Sat, Mar 28, 2009 at 04:49:10 -0000, Trent Buck wrote:
> > Now that the roundup repository is exposed to me, I have imported my
> > RCS change history as Darcs patches.  I did not preserve the
> > timestamps, because using --pipe is a fucking nightmare (see
> > issue1276).
> > 
> > This patch bundle includes at least one patch that was not applied
> > even as a bulk import, the one that provides X-Roundup-* headers.
> 
> Is my job here to apply these patches and push them to darcs.net?
> They apply cleanly, for what it's worth.

Yes, please.  I sent the bundle here instead of darcs-users on the
basis that most darcs-users denizens won't care about this feature.

On Sat, Mar 28, 2009 at 05:02:46PM -0000, Simon Michael wrote:
> I expect there'll be more like this. It seems odd to be filling up
> the darcs bug tracker's configuration repo with the history of
> Trent's roundup add-on.

Why is it odd?

> What about maintaining this in its own repo and periodically
> importing to the darcs roundup repo with single larger changes
> ("import twb's roundup mbox exporter, spring edition") ?

Why is that better than pulling the patches themselves?  Isn't that
what a DVCS is all about?

On Sat, Mar 28, 2009 at 22:04:51 -0000, Trent Buck wrote:
> Yes, please.  I sent the bundle here instead of darcs-users on the
> basis that most darcs-users denizens won't care about this feature.

Done

I ran into some trouble doing this because the pristine cache was not
group writeable in this directory. But it seems to be fine now.

> Why is it odd?

You are working on a general roundup feature. Roundup users and  
developers will not expect to find that work in the darcs bug tracker  
repo. Folks working on the darcs bug tracker will not expect that  
history to be obscured by a lot of roundup development commits. Makes  
sense ?

Eric didn't see my objection and has applied your patches, so no  
worries, but let's please reconsider this when you next work on that  
stuff.

On Tue, Apr 07, 2009 at 05:49:13PM -0000, Simon Michael wrote:
>> Why is it odd?
>
> You are working on a general roundup feature. Roundup users and
> developers will not expect to find that work in the darcs bug
> tracker repo.

Sure, but even if I had the canonical version of that script stored in
another repo (e.g. http://code.haskell.org/~twb/roundup-mbox/), you'd
still want to pull those patches into your repo.

If you just threw away the version history and re-recorded it as
patches like "everything that changed in mbox.py in 2009", you would
break your ability to exchange patches with my repo.  That is, you
would not be able to blacklist my patches that you didn't want, nor
would you be able to created patches to send back to me.

> Folks working on the darcs bug tracker will not expect that history
> to be obscured by a lot of roundup development commits. Makes sense
> ?

--match 'not touch extensions/messages_as_mbox.py' or similar.  I
don't see that as a big issue, but I'm prepared to debate it with you.

I have no doubt you are! :) I find the mbox stuff distracting, but you  
have a point and I will deal with it.

After darcs.net was upgraded from Etch to Lenny, the newest mbox.py
work was applied.  This added X-Roundup-Foo headers and support for
attachments.  The former is working, but the latter is not.  I need
to investigate why attachments aren't being imported correctly at
darcs.net, since they were working correctly in my test instance.

This tracker's mbox extension now implements attachments and supports patch objects.