darcs

Issue 26 Darcs needs real MIME parsing, fails with Mail.app, Courier

Title Darcs needs real MIME parsing, fails with Mail.app, Courier
Priority bug Status needs-implementation
Milestone 2.10.0 HEAD Resolved in
Superseder Nosy List dmitry.kurochkin, kowey, thorkilnaur, tux_rocker
Assigned To
Topics

Created on 2005-11-28.13:44:46 by jast, last changed 2011-10-13.13:11:02 by markstos.

Files
File name Uploaded Type Edit Remove
darcs-apply.log.txt.bz2 zooko, 2009-01-08.18:28:48 application/octet-stream
patch-fiddling.txt kowey, 2009-01-08.17:53:41 text/plain
patch.txt.bz2 zooko, 2009-01-08.17:32:56 application/octet-stream
Messages
msg95 (view) Author: jast Date: 2005-11-28.13:44:45
Hi,

apparently, Darcs does not expect MTAs to rewrite the MIME boundaries of
mails created by `darcs send' (but, for instance, Courier does this), and
therefore neglects to interpret the "boundary" value from the mail's
"Content-type" header. Mails that have been rewritten in this way are not
unquoted by Darcs, so they fail signature verification by GnuPG when running
`darcs apply'.

For reference, the boundary value appears like this:

> Content-Type: multipart/mixed; boundary="=_something"

and then has to be understood when it appears later on in the mail:

> --=_something--

Even if this is a fairly exotic bug, I'd really love this issue to be
addressed, since I don't really want to keep relying on my workaround script.

Thanks!
Best regards
Jan
msg106 (view) Author: droundy Date: 2005-11-29.12:54:23
On Mon, Nov 28, 2005 at 01:44:46PM +0000, Jan Krueger wrote:
> apparently, Darcs does not expect MTAs to rewrite the MIME boundaries of
> mails created by `darcs send' (but, for instance, Courier does this), and
> therefore neglects to interpret the "boundary" value from the mail's
> "Content-type" header. Mails that have been rewritten in this way are not
> unquoted by Darcs, so they fail signature verification by GnuPG when running
> `darcs apply'.

Indeed, darcs' handling of MIME is pretty limited.  Adding proper MIME
support would definitely be a Good Thing.
-- 
David Roundy
http://www.darcs.net
msg200 (view) Author: tommy Date: 2005-12-11.15:50:33
Darcs also assumes the Content-Description header exists and
is the last one.  A simple fix could be to look for the
Content-Type header line and search for a blank line from
there.

Pasting related mail from Daniel Bünzli to darcs-users:

Hello,

I'm using Mail.app under macosx. I have a problem applying the
patches of an email, I just save the contents of the email to the
file 'patch2' and applying results in :

>> darcs apply ~/tmp/patch2
>darcs failed:  Malformed patch bundle: '[X001. Added base example.
>Extracted two sections to work in parallel' is not 'Context:'

However if I go through my web interface and save the message to the
file 'patch' and apply it everything works fine. The difference
between patch and patch2 are :

>> diff patch patch2
>1c1
>< From X@epfl.ch dim déc 11 13:27:48 2005
>---
>> From X@epfl.ch Sun Dec 11 13:27:48 2005
>41a42
>>
>43d43
>< Content-Type: text/plain
>44a45
>> Content-Type: text/plain
>54d54
>< Content-Type: text/x-darcs-patch
>56c56
>< Content-Description: A darcs patch for your repository!
>---
>> Content-Type: text/x-darcs-patch
>466,469d465
><
>< .
><
><

Relevant seems that Mail.app drops Content-Description and the final
dot and swaps Content-Type and Content-Transfer-Encoding. Is this a
bug for Mail.app or for darcs ? I don't know much about MIME.

Daniel
msg2568 (view) Author: markstos Date: 2008-01-19.02:21:35
I looked into this some. 

The current meager mail parsing happens in Darcs/Email.hs in the "read_mail"
routine around line 122. 

However, I suspect it would be fairly easy to take advantage of one of the
Haskell MIME parsing libraries which have appeared since the darcs project
started. For example, here's one from 2006, which appears to have a compatible
license:

http://www.mail-archive.com/haskell@haskell.org/msg19533.html

Maybe it wouldn't be so hard to "plug in it" ? 

My co-worker uses Mail.app and can't detach darcs patches, so my team is
directly affected by this.
msg2651 (view) Author: droundy Date: 2008-01-22.14:30:28
On Sat, Jan 19, 2008 at 02:21:36AM -0000, Mark Stosberg wrote:
> My co-worker uses Mail.app and can't detach darcs patches, so my team is
> directly affected by this.

This sounds like it's an issue of darcs generating correct MIME emails,
rather than parsing them correctly? Or is it that when he attaches the
darcs patches manually to his email, the resulting email can't be parsed by
darcs?
-- 
David Roundy
Department of Physics
Oregon State University
msg2653 (view) Author: markstos Date: 2008-01-22.14:59:26
David Roundy wrote:
> 
> On Sat, Jan 19, 2008 at 02:21:36AM -0000, Mark Stosberg wrote:
>> My co-worker uses Mail.app and can't detach darcs patches, so my team is
>> directly affected by this.
> 
> This sounds like it's an issue of darcs generating correct MIME emails,
> rather than parsing them correctly? Or is it that when he attaches the
> darcs patches manually to his email, the resulting email can't be parsed by
> darcs?

My understanding is that the patches generated are valid MIME.... it is 
the parser on the receiving end that is naive:

- It expects the boundary to be a particular thing, but it could be 
rewritten, as Courier does.

- Darcs also assumes the Content-Description header exists and
is the last one.  Tommy says a simple fix could be to look for the
Content-Type header line and search for a blank line from
there. (But real MIME parsing would be better).

Also, it seems the final dot might be getting mangled. (which might be a 
  considered a Mail.app bug, but would be nice to be robust against).

This prior message has a good summary:
http://bugs.darcs.net/msg200

    Mark
msg5464 (view) Author: markstos Date: 2008-08-13.02:24:42
I would like to see this fixed for Darcs 2.1, although the release manager may
disagree and remove the release-critical tag.
msg6949 (view) Author: mornfall Date: 2008-12-31.10:20:07
Let's target this for 2.3. Hopefully it's not too hard to fix.
msg7015 (view) Author: zooko Date: 2009-01-08.17:32:56
I just learned that darcs patches submitted to my mailing list by a contributor
are not usable.  The new information -- to me -- is that it has nothing to do
with the mail user agent used on the receiving side (I currently use Apple
Mail.app, unfortunately).  Even if I log into the server which hosts the mailing
list and copy the mail out of its /var/lib/mailman file, the patch still doesn't
apply.  Here is the patch I wanted to apply, attached as patch.txt.bz2, and here
is the output from darcs apply:

$ darcs apply -v -v -v patch.txt

darcs failed:  Patch bundle failed hash!
This probably means that the patch has been corrupted by a mailer.
The most likely culprit is CRLF newlines.
Attachments
msg7016 (view) Author: kowey Date: 2009-01-08.17:53:56
Zooko, does the attached version of the patch apply?

The two main differences are that I have removed the trailing diff
(not from darcs) and that I have deleted some curly braces,
notably around the hunk.
Attachments
msg7017 (view) Author: markstos Date: 2009-01-08.18:06:41
On Thu, 08 Jan 2009 17:32:59 -0000
Zooko <bugs@darcs.net> wrote:

> 
> Zooko <zooko@zooko.com> added the comment:
> 
> I just learned that darcs patches submitted to my mailing list by a contributor
> are not usable.  The new information -- to me -- is that it has nothing to do
> with the mail user agent used on the receiving side (I currently use Apple
> Mail.app, unfortunately).  Even if I log into the server which hosts the mailing
> list and copy the mail out of its /var/lib/mailman file, the patch still doesn't
> apply.  Here is the patch I wanted to apply, attached as patch.txt.bz2, and here
> is the output from darcs apply:
> 
> $ darcs apply -v -v -v patch.txt
> 
> darcs failed:  Patch bundle failed hash!
> This probably means that the patch has been corrupted by a mailer.
> The most likely culprit is CRLF newlines.

It seems useful in this case to have the send use "darcs send -O" to output the
patch outside of the mail system, and then gzip it to give it you, and compare
that with the result you found in /var/lib/mailman.

Then you can see for certain if the issue originated in darcs, or if the patch
was modified by another tool, and what exactly the modification was.

    Mark
msg7018 (view) Author: zooko Date: 2009-01-08.18:25:48
Okay, I updated the Tahoe wiki to request bzip2 of patches:

http://allmydata.org/trac/tahoe/wiki/Patches

This made me wonder if darcs shouldn't (optionally) do something like that --
compress the patch output.
msg7019 (view) Author: zooko Date: 2009-01-08.18:28:48
Thanks for looking at this, kowey.  I tried to darcs apply the
"patch-fiddling.txt" and got a (different) error.  Attached is
darcs-apply.log.txt.bz2.  By the way if you want to see what happens without
waiting for me to try it and report back, you can get the repository from
http://allmydata.org/source/tahoe/trunk-hashedformat and try applying the patch
to that.
Attachments
msg8382 (view) Author: kowey Date: 2009-08-23.10:23:05
I think we should farm this out to a third-party library like
http://hackage.haskell.org/package/mime
msg11607 (view) Author: tux_rocker Date: 2010-06-27.18:30:04
Doesn't look like it's going to happen before 2.6.
msg13303 (view) Author: kowey Date: 2010-12-09.17:00:23
Another package we could potentially use: 
http://hackage.haskell.org/package/mime-mail
msg13304 (view) Author: kowey Date: 2010-12-09.17:13:37
Doh, that (mime-mail) is rendering, not parsing (for now)
msg14764 (view) Author: markstos Date: 2011-10-13.13:11:01
It's not a regression since 2.5, so bumping to 2.10.
History
Date User Action Args
2005-11-28 13:44:46jastcreate
2005-11-29 12:53:51droundysetnosy: droundy, tommy, jast
2005-11-29 12:54:24droundysetstatus: unread -> unknown
nosy: droundy, tommy, jast
messages: + msg106
2005-12-11 15:50:33tommysetnosy: droundy, tommy, jast
messages: + msg200
2008-01-19 02:21:36markstossetnosy: + markstos, darcs-devel, kowey, beschmi
messages: + msg2568
title: Darcs fails to treat rewritten MIME mails correctly -> Darcs needs real MIME parsing, fails with Mail.app, Courier
2008-01-22 14:30:30droundysetnosy: droundy, tommy, beschmi, kowey, markstos, darcs-devel, jast
messages: + msg2651
2008-01-22 14:59:27markstossetnosy: droundy, tommy, beschmi, kowey, markstos, darcs-devel, jast
messages: + msg2653
2008-02-14 04:24:32zookosetnosy: + zooko
2008-02-16 23:01:12markstossetstatus: unknown -> has-patch
nosy: droundy, tommy, beschmi, kowey, markstos, darcs-devel, zooko, jast
2008-08-13 02:24:48markstossettopic: + Target-2.0
nosy: + Serware, dagit, simon
messages: + msg5464
2008-12-31 10:20:13mornfallsettopic: + Target-2.3, - Target-2.0
nosy: + dmitry.kurochkin, mornfall, thorkilnaur
messages: + msg6949
2009-01-08 17:32:59zookosetfiles: + patch.txt.bz2
nosy: droundy, tommy, beschmi, kowey, markstos, darcs-devel, zooko, jast, dagit, simon, thorkilnaur, dmitry.kurochkin, Serware, mornfall
messages: + msg7015
2009-01-08 17:54:00koweysetfiles: + patch-fiddling.txt
nosy: droundy, tommy, beschmi, kowey, markstos, darcs-devel, zooko, jast, dagit, simon, thorkilnaur, dmitry.kurochkin, Serware, mornfall
messages: + msg7016
2009-01-08 18:06:44markstossetnosy: droundy, tommy, beschmi, kowey, markstos, darcs-devel, zooko, jast, dagit, simon, thorkilnaur, dmitry.kurochkin, Serware, mornfall
messages: + msg7017
2009-01-08 18:25:51zookosetnosy: droundy, tommy, beschmi, kowey, markstos, darcs-devel, zooko, jast, dagit, simon, thorkilnaur, dmitry.kurochkin, Serware, mornfall
messages: + msg7018
2009-01-08 18:28:52zookosetfiles: + darcs-apply.log.txt.bz2
nosy: droundy, tommy, beschmi, kowey, markstos, darcs-devel, zooko, jast, dagit, simon, thorkilnaur, dmitry.kurochkin, Serware, mornfall
messages: + msg7019
2009-08-06 16:06:26koweysettopic: + Target-2.4, - Target-2.3
nosy: droundy, tommy, beschmi, kowey, markstos, darcs-devel, zooko, jast, dagit, simon, thorkilnaur, dmitry.kurochkin, Serware, mornfall
2009-08-06 17:31:52adminsetnosy: - droundy
2009-08-06 20:46:30adminsetnosy: - beschmi
2009-08-10 21:55:45adminsetnosy: - tommy, markstos, darcs-devel, zooko, jast, dagit, Serware, mornfall
2009-08-23 10:23:07koweysetstatus: has-patch -> needs-implementation
nosy: kowey, simon, thorkilnaur, dmitry.kurochkin
messages: + msg8382
2009-08-25 17:23:57adminsetnosy: + darcs-devel, - simon
2009-08-25 19:09:27koweysetnosy: kowey, darcs-devel, thorkilnaur, dmitry.kurochkin
2009-08-27 14:30:14adminsetnosy: kowey, darcs-devel, thorkilnaur, dmitry.kurochkin
2009-09-14 10:45:43koweysettopic: + Target-2.5, - Target-2.4
nosy: kowey, darcs-devel, thorkilnaur, dmitry.kurochkin
2010-03-10 15:09:48koweylinkissue689 superseder
2010-06-15 20:51:47adminsetmilestone: 2.5.0
2010-06-15 20:58:59adminsettopic: - Target-2.5
2010-06-27 18:30:05tux_rockersetnosy: + tux_rocker
messages: + msg11607
milestone: 2.5.0 -> 2.8.0
2010-12-09 17:00:23koweysetnosy: - darcs-devel
messages: + msg13303
2010-12-09 17:13:38koweysetmessages: + msg13304
2011-10-13 13:11:02markstossetmessages: + msg14764
milestone: 2.8.0 -> 2.10.0 HEAD