NAME

Mystery Girl - A Bounce Handler For Dada Mail


DESCRIPTION

Mystery Girl intelligently handles bounces from Dada Mail list messages. Each message is first parsed. The parsed email will then be examined and an action will be taken. The examination and action are set in a collection of rules. These rules can be tweaked, added, removed and generally mucked about with.

The usual action that is taken is to apply a, score to the offending email address, everytime the address bounces back a message. Once the, Threshold is reached, the email address is unsubscribed from the list.

This usually means that it takes a few bounces from a particular email address to get it removed from a list. This gives a bit of wiggle room and makes sure an email address that is bouncing is bouncing for a fairly good reason, for example: it no longer exists.


OBTAINING A COPY OF THIS PROGRAM

Mystery Girl is located in the, dada/plugins directory of the main Dada Mail distribution, under the name, dada_bounce_handler.pl


REQUIREMENTS

These points are absolutely necessary. Please make sure you have them before you try to install this plugin:


RECOMMENDED

These points are not required, but recommended to have to use Mystery Girl:


Configuration

There's a few things you need to configure in this script, they're all at the top.

As far as required changes, that's it. We'll get to interesting optional things further down the line.


Installation

Setting up Mystery Girl to run as a CGI script

(Optional)

Originally, Mystery Girl was a purely command line tool, to be used either on the command line, or run via a crontab.

Now, you may run Mystery Girl via your web browser. Mystery Girl acts like a Dada Mail plugin.

Usually, you'll set up Dada Mail in your cgi-bin: in your cgi-bin, there's a directory called, ``dada''. Inside the, ``dada'' directory, there are at least two directories, one called, ``DADA'' (uppercase) and the mail.cgi script.

In the, ``dada'' directory, create a new directory called, ``plugins''. Upload the dada_bounce_handler.pl script, already configured, into this directory. Change its permissions to, ``755''. Visit the script in your web browser. To run the bounce handler on your bounced messages, click the, Parse Bounces... button.

Before the bounce handler will work, Dada Mail has to know to use it. Jump down to, Telling Dada Mail to use the Bounce Handler for instructions on how to do that.

If you would like have a link on the left hand side of the list control panel, find the following line in the Config.pm:

 #                                      {-Title      => 'Bounce Handler',
 #                                       -Title_URL  => $PLUGIN_URL."/dada_bounce_handler.pl",
 #                                       -Function   => 'dada_bounce_handler',
 #                                       -Activated  => 1,
 #                                      },

And uncomment it (Take off the, ``#'' on each line).

Setting up Mystery Girl to run via the command line.

If you are not going to set up this program as a cgi script, I would just make a directory in your home directory and place the script there. If you've set up Dada Mail as outlined in the Magic Book, you may want to make another directory in the .dada_files directory, called .scripts, and install this script in there. You can still install the bounce handler in the, cgi-bin/dada/plugins/ directory as it's a nice convenience to run the program as both a command line/cronjob and also as a Dada Mail plugin.

chmod 755 dada_bounce_handler.pl

That's it as far as installation of the script.

Running the script via the command line.

Running the program without any arguments will have it check the mailbox for bounces, parse the messages and handle the bounces. ie:

 prompt>./dada_bounce_handler.pl

I suggest before you do that, you test the dada_bounce_handler.pl.

Testing

You can pass the --test argument to dada_bounce_handler.pl to make sure everything is workings as it should. The --test argument needs to take one of a few paramaters:


Setting The Schedule

You could run dada_bounce_handler.pl every now and again from the command line, but you'd get very sick of it and I spent an entire weekend in May to write this script to be lazy.

To accomplish that, you want to set this script to execute via a cron or scheduled, job. Here's what a theoretical cron tab for this script may look like:

 0  1   *   *  * /usr/bin/perl /home/myaccount/cgi-bin/dada/plugins/dada_bounce_handler.pl >/dev/null 2>&1

This will run the script every day around 1am. You can run this script as often as you want, just be logical. I wouldn't run this script every five minutes, that's a bit overkill.

Different hosts may have a control panel to set up crontabs, my host gives me the pleasure of the contrab command. I type in:

 prompt> crontab -e

and am launched into my favorite text editor to type in the crontab.


Telling Dada Mail to use the Bounce Handler.

You're going to have to tell Dada Mail explicitly that you want bounces to go to the bounce handler. The first step is to set the Dada List Administrator to your bounce email address. You set this in the list control panel, under Change List Information

Once you do that, you need to tell Dada Mail that you want the correct headers in your list messages to say, ``use the admin address for bounces''

Usually, this means that the Return-path header needs to be set. There are a few ways to accomplish this, some more preferable than others.

To test out any of these configurations, Send yourself a test message and view the source of the message itself, in your mail reader. In the mail headers, you should see the Return-Path header:

 Return-Path: <dadabounce@myhost.com>
 Delivered-To: justin@myhost.com
 Received: (qmail 75721 invoked from network); 12 May 2003 04:50:01 -0000
 Received: from myhost.com (208.10.44.140)
   by hedwig.myhost.com with SMTP; 12 May 2003 04:50:01 -0000
 Date:Sun, 11 May 2003 23:50:01 -0500
 From:justin <justin@myhost.com>
 Subject:Test, Test, Test
 To:justin@myhost.com
 Sender:dadabounce@myhost.com
 Reply-To:justin <justin@myhost.com>
 Precedence:list
 Content-type:text/plain; charset=iso-8859-1

Notice that the first line has the Return-Path header, correctly putting my bounce email address. My List Owner address, justin@myhost.com still occupies the To: and Reply-To headers, so whoever replies to my message will reply to me, not the bounce handler.

Once you've dialed in your list to use the bounce handler, you should be all set.


Optional Fun Things

There's a slew of optional arguments you can give to this script:

Rules, Rule!

dada_bounce_handler.pl figures out what to do with the bounce messages receives by consulting a group of rules. These rules are highly configurable, so if you need to change the behavior of this script, you don't have to change the code.

These rules are stored in the $Rules hashref. An example rule:

     {
        exim_user_unknown => { 
            Examine => { 
                Message_Fields => { 
                    Status      => [qw(5.x.y)], 
                    Guessed_MTA => [qw(Exim)],  
                }, 
                Data => { 
                    Email       => 'is_valid',
                    List        => 'is_valid', 
                }
            },
                Action => { 
                     add_to_score => $Default_Hard_Bounce_Score,
                }, 
            }
    },

exim_user_unknown is the title of the rule - just a label, nothing else.

Examine holds a set of parameters that the handler looks at when trying to figure out what to do with a bounced message. This example has a Message_Fields entry and inside that, a Status entry. The Status entry holds a list of status codes. The ones in shown there all correspond to hard bounces; the mailbox probably doesn't exist. Message_Fields also hold a, Guessed_MTA entry - it's explicitly looking for a bounce back from the, Exim mail server.

Examine also holds a Data entry, which holds the Email or List entries, or both. Their values are either 'is_valid', or 'is_invalid'.

So, to sum this all up, this rule will match a message that has Status: Message Field contaning a user unknown error code, (5.1.1, etc) and also a Guessed_MTA Message Field containing, Exim. The message also has to be parsed to have found a valid email and list name.

Pretty Slick, eh?

If this all matches, the Action is... acted upon. In this case, the offending email address will be appended a, Bounce Score of, whatever, $Default_Hard_Bounce_Score, which is by default, 4.

If you would like to have the bounced address automatically removed, without any sort of scoring happening, change the action from,

    add_to_score => $Default_Hard_Bounce_Score

to:

    unsubscribe_bounced_email => 'from_list'

Also, changing from_list, to from_all_lists will do the trick.

I could change the line:

 unsubscribe_bounced_email => 'from_list',

to:

 mail_list_owner => 'user_unknown_message'

This will, instead of deleting the email automatically, send a message to the list owner, stating that, ``Hey, the message bounced, what do you want to do?''

Another example:

 {
 over_quota => {
         Examine => {
                Message_Fields => {
                        Status => [qw(5.2.2)]
                },
                Data => { 
                        Email => 'is_valid', 
                        List  => 'is_valid',
                }
        },
        Action => { 
                mail_list_owner => 'over_quota_message', 
        },
 }

This time, I created a list for messages that get bounced because the mailbox is full. This is still considered a hard bounce, but I don't want the subscriber removed because they haven't check their inbox during the week. In this case, the Action has been set to:

 mail_list_owner => 'over_quota_message',

Which will do what it sounds like, it'll mail the list owner a message explaining the circumstances.

Here's a schematic of all the different things you can do:

 {
 rule_name => {
         Examine => {
                Message_Fields => {
                        Status               => qw([    ]), 
                        Last-Attempt-Date    => qw([    ]), 
                        Action               => qw([    ]), 
                        Status               => qw([    ]), 
                        Diagnostic-Code      => qw([    ]), 
                        Final-Recipient      => qw([    ]), 
                        Remote-MTA           => qw([    ]), 
                        # etc, etc, etc
                        
                },
                Data => { 
                        Email => 'is_valid' | 'is_invalid' 
                        List  => 'is_valid' | 'is_invalid' 
                }
        },
        Action => { 
                   add_to_score             =>  $x, # where, "$x" is a number
                           mail_list_owner           => 'user_unknown_message', 
                           mail_list_owner           => 'email_not_found_message', 
                           mail_list_owner           => 'over_quota_message', 
                           unsubscribe_bounced_email => 'from_list' | 'from_all_lists',
        },
 },

Mystery Girl also supports the use of regular expressions for matching any of the Message_Fields. To tell the parser that you're using a regular expression, make the Message_Field key end in '_regex':

 'Final-Recipient_regex' => [(qr/RFC822/)], 

Setting rules is sort of the super advanced part of the configuration, but it may come in handy.


More on Scores, Thresholds, etc

We talked about scoring, but not in great detail, so let's do that:

By default, The Bounce Handler assigns a particular score to each email address that bounces back a message. These scores are tallied each time an email address bounces a message.

Since Dada Mail understands the differences between Hard Bounces and Soft Bounces, it'll append a smaller score for soft bounces, and a larger score for hard bounces.

Once the email address's Bounce Score reaches the Threshold, the email address is then removed from the list.

You can manipulate the Soft and Hard Bounce Scores and Threshold pretty easily. On the top of this script, you'll see the necessary variables to tweak,

Fairly self-explanitory.

If you want even greater control over what kind of bounces give what scores, you can manipulate the Bounce Rules themselves, as described above.

Some things to understand:

Currently, Scores are Global - they work for all lists at once. Once an email address reaches the Threshold, they will be removed from all lists that are handled by Dada Mail. In one sense, this seems limiting, but if an email address is not receiving mail from one list, or doesn't exist, it doesn't have much worth on any list of yours.

If you would like to periodically erase the saved scores, you may do so, by running this script via the command line, like so:

    ./dada_bounce_handler.pl --erase_score_card


FAQs


History

NOTE: Most of the history of this program is located in the main Dada Mail changelog.


To Do

Perhaps think about making filters specifically for Postfix. They seem to have their own way of doing things, like Qmail.

Add onto that custom a filter for AOL/Compuserve/Netscape


Thanks

Thanks to: Jake Ortman Henry Hughes for some prelim bounce examples.

Thanks to Eryq ( http://www.zeegee.com ) for the amazing MIME-tools collection. It's a gnarly group of modules.


COPYRIGHT

Copyright (c) 1999 - 2006 Justin Simoni http://justinsimoni.com All rights reserved.

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

Parts of this script were swiped from Mail::Bounce::Qmail module, fetched from here:

http://mikoto.sapporo.iij.ad.jp/cgi-bin/cvsweb.cgi/fmlsrc/fml/lib/Mail/Bounce/Qmail.pm

The copyright of that code stated:

Copyright (C) 2001,2002,2003 Ken'ichi Fukamachi All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

Thanks Ken'ichi