darcs

Issue 1676 bad interaction between darcs-devel being nosy and rejecting messages

Title bad interaction between darcs-devel being nosy and rejecting messages
Priority bug Status resolved
Milestone Resolved in
Superseder Nosy List Serware, darcs-devel, dmitry.kurochkin, kowey, stephen
Assigned To
Topics BugTracker

Created on 2009-11-05.15:21:47 by kowey, last changed 2010-05-06.12:44:06 by kowey.

Messages
msg9210 (view) Author: kowey Date: 2009-11-05.15:21:44
1. darcs-devel is set to reject non-automated messages (ie. messages that don't
come from the issue tracker)
2. darcs-devel is nosy
3. people sometimes reply-all to bugs and get rebuffed by darcs-devel

I need to rethink this interaction.  One possibility is to see if we can make
darcs-devel a roundup BCC (I suspect this won't be easy).  A simpler possibility
is just to relax the restriction and never again try to enforce by automation
what can be adequately achieved by mere nagging.
msg9212 (view) Author: stephen Date: 2009-11-06.01:10:24
Eric Kow writes:

 > I need to rethink this interaction.  One possibility is to see if we can make
 > darcs-devel a roundup BCC (I suspect this won't be easy).

This should be almost trivial.  Track the nosy reactor down to where
it sends the mail, and add a BCC to the message created with

    msg['Bcc'] = "darcs-devel@darcs.net"

just before sending.

You can test by putting "kowey" in there instead.  Also add

    msg['X-Darcs-Test'"] = "automagic bcc"

so that you can distinguish that particular message from any others.
(Watch out for personal dupe filtering!)

But is that really the problem?  I suspect the problem is at the other
end.  I *think* you can probably win at the darcs-devel side by
making "bugs@darcs.net" a member of the list and setting it to
no-mail.  If you're worried about spam (and who isn't? ;), you can add
this Handler to the Mailman pipeline right after other validity checks:

-------------------------------- cut --------------------------------
# DarcsSource.py -- verify that a post was sent by a Darcs.net agent
# Copyright (C) 1998-2004 by the Free Software Foundation, Inc.
# Copyright (C) 2004, 2009 by Stephen J. Turnbull
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

"""Checks whether this message was sent by Roundup.

The MTA adds a Received header.  Find a reliable host "near" Roundup
(preferably the Roundup host), and determine what the Received header
it adds looks like, then figure a regexp to match the contents.  For
spam fighting, the more precise the better.  You may want to match
the date/time, and add a check for reasonably fresh to avoid replay
attacks.

If this header is not present, the bugs@darcs.net address is being
abused, and the message can be discarded.  We can improve authenticity
by adding a local cookie, but that should be unnecessary.

You may wish to hold such messages instead.  When the message is held
for approval, then the hold database is updated and any administrator
notification messages are sent.  Finally an exception is raised to let
the pipeline machinery know that further message handling should stop.

Questions about this Handler to "Stephen J. Turnbull" <stephen@xemacs.org>.

"""

# I expect we don't need all these.
import email
from email.MIMEText import MIMEText
from email.MIMEMessage import MIMEMessage
import email.Utils
import re
from types import ClassType

from Mailman import mm_cfg
from Mailman import Utils
from Mailman import Errors
from Mailman import Message
from Mailman import i18n
from Mailman import Pending
from Mailman.Logging.Syslog import syslog

# this should match the content of the relevant received header
darcs_received_regexp = ""
darcs_received_regexp = re.compile(darcs_received_regexp)

# First, play footsie with _ so that the following are marked as translated,
# but aren't actually translated until we need the text later on.
def _(s):
    return s


#
# Change Errors.DiscardMessage to Errors.HoldMessage to hold for
# moderation.
#
# Note that this is DiscardMessage so the members aren't actually used!
class NotDarcsSource(Errors.DiscardMessage):
    reason = _('The message attempted to bypass filtering')
    rejection = _('You are forbidden to post messages via this address.')

# Note that this is DiscardMessage so the members aren't actually used!
class DarcsSourceMismatch(Errors.DiscardMessage):
    reason = _('Message headers are inconsistent with this list')
    rejection = _('Posts to this list must be addressed to the list.')

# And reset the translator
_ = i18n._



def process(mlist, msg, msgdata):
    # DON'T check for approval!

    # Get the name of the list, message-id, and sender for later use
    listname = mlist.internal_name()
    sender = msgdata.get('sender', msg.get_sender())
    message_id = msg.get('message-id', 'n/a')

    # Check for our header
    found = False
    for received in msg.getall('Received'):
        if darcs_received_regexp.match(received):
            found = True
            break

    # Handle the message
    if not found:
	    # Log the discarded message
	    syslog('vette', '%s post from %s discarded, message-id=%s: %s',
        	   listname, sender, message_id, reason)
            # to Hold, uncomment the following and comment the raise stmt
            # hold_for_approval(mlist, msg, msg_data, NotDarcsSource
	    raise NotDarcsSource
    elif not 'darcs-devel' == listname:
	# Log the discarded message
	syslog('vette', '%s post from %s discarded, message-id=%s: %s',
        	listname, sender, message_id, reason)
            # to Hold, uncomment the following and comment the raise stmt
            # hold_for_approval(mlist, msg, msg_data, DarcsSourceMismatch
	raise DarcsSourceMismatch
    else:
	# found header and it matches
	pass

# except for comments everything below here is identical to Hold.py
# #### import them directly?


# this is a bad idea if we're pretty sure it's spam ...
def ackp(msg):
    ack = msg.get('x-ack', '').lower()
    precedence = msg.get('precedence', '').lower()
    if ack <> 'yes' and precedence in ('bulk', 'junk', 'list'):
        return 0
    return 1



def hold_for_approval(mlist, msg, msgdata, exc):
    # BAW: This should really be tied into the email confirmation system so
    # that the message can be approved or denied via email as well as the
    # web.
    if type(exc) is ClassType:
        # Go ahead and instantiate it now.
        exc = exc()
    listname = mlist.real_name
    sender = msgdata.get('sender', msg.get_sender())
    usersubject = msg.get('subject')
    charset = Utils.GetCharSet(mlist.preferred_language)
    if usersubject:
        usersubject = Utils.oneline(usersubject, charset)
    else:
        usersubject = _('(no subject)')
    message_id = msg.get('message-id', 'n/a')
    owneraddr = mlist.GetOwnerEmail()
    adminaddr = mlist.GetBouncesEmail()
    requestaddr = mlist.GetRequestEmail()
    # We need to send both the reason and the rejection notice through the
    # translator again, because of the games we play above
    reason = Utils.wrap(exc.reason_notice())
    msgdata['rejection_notice'] = Utils.wrap(exc.rejection_notice(mlist))
    id = mlist.HoldMessage(msg, reason, msgdata)
    # Now we need to craft and send a message to the list admin so they can
    # deal with the held message.
    d = {'listname'   : listname,
         'hostname'   : mlist.host_name,
         'reason'     : _(reason),
         'sender'     : sender,
         'subject'    : usersubject,
         'admindb_url': mlist.GetScriptURL('admindb', absolute=1),
         }
    # We may want to send a notification to the original sender too
    fromusenet = msgdata.get('fromusenet')
    # Since we're sending two messages, which may potentially be in different
    # languages (the user's preferred and the list's preferred for the admin),
    # we need to play some i18n games here.  Since the current language
    # context ought to be set up for the user, let's craft his message first.
    #
    # This message should appear to come from <list>-admin so as to handle any
    # bounce processing that might be needed.
    cookie = mlist.pend_new(Pending.HELD_MESSAGE, id)
    if not fromusenet and ackp(msg) and mlist.respond_to_post_requests and \
           mlist.autorespondToSender(sender):
        # Get a confirmation cookie
        d['confirmurl'] = '%s/%s' % (mlist.GetScriptURL('confirm', absolute=1),
                                     cookie)
        lang = msgdata.get('lang', mlist.getMemberLanguage(sender))
        subject = _('Your message to %(listname)s awaits moderator approval')
        text = Utils.maketext('postheld.txt', d, lang=lang, mlist=mlist)
        nmsg = Message.UserNotification(sender, adminaddr, subject, text, lang)
        nmsg.send(mlist)
    # Now the message for the list owners.  Be sure to include the list
    # moderators in this message.  This one should appear to come from
    # <list>-owner since we really don't need to do bounce processing on it.
    if mlist.admin_immed_notify:
        # Now let's temporarily set the language context to that which the
        # admin is expecting.
        otranslation = i18n.get_translation()
        i18n.set_language(mlist.preferred_language)
        try:
            lang = mlist.preferred_language
            charset = Utils.GetCharSet(lang)
            # We need to regenerate or re-translate a few values in d
            d['reason'] = _(reason)
            d['subject'] = usersubject
            # craft the admin notification message and deliver it
            subject = _('%(listname)s post from %(sender)s requires approval')
            nmsg = Message.UserNotification(owneraddr, owneraddr, subject,
                                            lang=lang)
            nmsg.set_type('multipart/mixed')
            text = MIMEText(
                Utils.maketext('postauth.txt', d, raw=1, mlist=mlist),
                _charset=charset)
            dmsg = MIMEText(Utils.wrap(_("""\
If you reply to this message, keeping the Subject: header intact, Mailman will
discard the held message.  Do this if the message is spam.  If you reply to
this message and include an Approved: header with the list password in it, the
message will be approved for posting to the list.  The Approved: header can
also appear in the first line of the body of the reply.""")),
                            _charset=Utils.GetCharSet(lang))
            dmsg['Subject'] = 'confirm ' + cookie
            dmsg['Sender'] = requestaddr
            dmsg['From'] = requestaddr
            nmsg.attach(text)
            nmsg.attach(MIMEMessage(msg))
            nmsg.attach(MIMEMessage(dmsg))
            nmsg.send(mlist, **{'tomoderators': 1})
        finally:
            i18n.set_translation(otranslation)
    # Log the held message
    syslog('vette', '%s post from %s held, message-id=%s: %s',
           listname, sender, message_id, reason)
    # raise the specific MessageHeld exception to exit out of the message
    # delivery pipeline
    raise exc
-------------------------------- cut --------------------------------
msg10569 (view) Author: kowey Date: 2010-03-30.09:41:19
[I'm resending this because I made a mistake which was causing the mail
to be rejected]

Note that my efforts seem to have yielded no fruit (magic bcc messages).
I'm going to leave darcs-devel off nosy-to-all until I get another round
tuit.

On Fri, Nov 06, 2009 at 10:18:06 +0900, Stephen J. Turnbull wrote:
>  > I need to rethink this interaction.  One possibility is to see if we can make
>  > darcs-devel a roundup BCC (I suspect this won't be easy).
> 
> This should be almost trivial.  Track the nosy reactor down to where
> it sends the mail, and add a BCC to the message created with
> 
>     msg['Bcc'] = "darcs-devel@darcs.net"
> 
> just before sending.

Thanks! It was almost trivial as you said.

With some studying of the roundupdb.py file, it turns out that the
'nosymessage' function in Roundup has an optional bcc argument.

So I've tacked on bcc='darcs-devel@darcs.net' to our call to
nosymessage, like so

 cl.nosymessage(nodeid, msgid, oldvalues,bcc=['E.Y.Kow@brighton.ac.uk'])

EDIT: actually, that needs to be a userid, I think, so
      db.user.lookup('darcs-devel') instead of just 'darcs-devel'

> You can test by putting "kowey" in there instead.  Also add
> 
>     msg['X-Darcs-Test'"] = "automagic bcc"

Ah, thanks for the two reminders :-)
 
> But is that really the problem?  I suspect the problem is at the other
> end.  I *think* you can probably win at the darcs-devel side by
> making "bugs@darcs.net" a member of the list and setting it to
> no-mail.  If you're worried about spam (and who isn't? ;), you can add
> this Handler to the Mailman pipeline right after other validity checks:
 
To be honest, I'm a bit scared of messing with the Mailman pipeline :-/
I'm going to see what effect this has.

I'll note that in between your posting and my reply, I disabled the auto-reject
in darcs-devel so that people could at least send messages.  Then I got a
different problem of some people only replying to list, and the BTS not seeing
anything.

So to sum up (I hope), I'll repeat the comment I wrote into that file

# We BCC darcs-devel because we want BTS traffic to be visible to
# interested folks, and we want them to be able to provide feedback
#
# We use BCC instead of just making darcs-devel nosy because when
# we make darcs-devel nosy, either
#
#    i)  people would reply just to darcs-devel and the BTS does
#        not see the message
#    ii) people would reply-all (which may interfere with any
#        no-mail policies the list may have)

This message is also my first test of the new BCC approach...

-- 
Eric Kow <http://www.nltg.brighton.ac.uk/home/Eric.Kow>
PGP Key ID: 08AC04F9
msg10977 (view) Author: kowey Date: 2010-05-06.12:44:05
This appears to be solved by the approach I mentioned in msg10569.  I
was confused initially because darcs-devel (nor my test accounts) seemed
to be receiving the BCC messages.  But that was due to [i] darcs-devel
rejecting non To/Cc messages (mailman) and [ii] errors on my test
account ends.

I hope this works!  Now people on darcs-devel should be able to reply to
bugs and it'll all just automagically wind up in the right place
(tracker then => darcs-devel).

Now that we know how to play with Bcc/CC in roundup, we can start
thinking about fancier things like telling darcswatch not to spam
darcs-users.
History
Date User Action Args
2009-11-05 15:21:47koweycreate
2009-11-06 01:10:39stephensetnosy: + stephen
messages: + msg9212
2009-12-18 08:31:32twbsetstatus: unknown -> needs-implementation
2010-03-30 09:41:20koweysetmessages: + msg10569
2010-05-06 12:44:06koweysetstatus: needs-implementation -> resolved
messages: + msg10977