Mystery Girl - A Bounce Handler For Dada Mail
Mystery Girl intelligently handles bounces from Dada Mail list messages. Each message is first parsed. The parsed email will then be examined and an action will be taken. The examination and action are set in a collection of rules. These rules can be tweaked, added, removed and generally mucked about with.
The usual action that is taken is to apply a, score to the offending email address, everytime the address bounces back a message. Once the, Threshold is reached, the email address is unsubscribed from the list.
This usually means that it takes a few bounces from a particular email address to get it removed from a list. This gives a bit of wiggle room and makes sure an email address that is bouncing is bouncing for a fairly good reason, for example: it no longer exists.
Mystery Girl is located in the, dada/plugins directory of the main Dada Mail distribution, under the name, dada_bounce_handler.pl
These points are absolutely necessary. Please make sure you have them before you try to install this plugin:
You're safer using an older version of Mystery Girl, with a newer version of Dada Mail.
You will need to setup a new email address for Mystery Girl to check. I usually set up an account named, ``bounces@yourdomain.com'', where, ``yourdomain.com'' is the name of the domain Dada Mail is installed on.
Currently, Mystery Girl only can use non-SSL-encrypted connections to a POP3 server listening to port 110.
Some things to consider:
These points are not required, but recommended to have to use Mystery Girl:
If you do not know how to set up a cron job, attempting to set one up for Dada Mail will result in much aggravation. Please read up on the topic before attempting!
crontab -e
command. You may also be able to set up a cron tab using a web-based control panel tool, like Cpanel.
Shell access also facilitates testing of the program.
There's a few things you need to configure in this script, they're all at the top.
If you are planning on running Mystery Girl via a cron tab, you will have to change the Path to Dada Mail's Perl Libraries.
NOTE This is not the same as your path to Perl (which is usually #!/usr/bin/perl). I get this asked frequently.
You will need to explicitly state where both your path to the regular Perl libs are, and the Dada Mail libraries are. For example:
use lib qw( /home/myaccount/www/cgi-bin/dada /home/myaccount/www/cgi-bin/dada/DADA /home/myaccount/www/cgi-bin/dada/DADA/perllib /usr/local/lib/perl5/site_perl/5.8.0/mach /usr/local/lib/perl5/site_perl/5.8.0 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.0/BSDPAN /usr/local/lib/perl5/5.8.0/mach /usr/local/lib/perl5/5.8.0 );
If you don't know where your Perl library is, trying running this via the command line:
perl -e 'print $_ ."\n" foreach @INC';
If you do not know how to run the above command, visit Dada Mail in a web browser, log into your list and on the left hand menu and: click, About Dada Mail
Under Script Information, click the, More... link and under the, Perl Library Locations, select each point that begins with a, ``/'' and use those as your site-wide path to your perl libraries.
You need to change the $Server, $Username and $Password variables to reflect the permissions for the email address you're going to use for the bounce handler.
As far as required changes, that's it. We'll get to interesting optional things further down the line.
(Optional)
Originally, Mystery Girl was a purely command line tool, to be used either on the command line, or run via a crontab.
Now, you may run Mystery Girl via your web browser. Mystery Girl acts like a Dada Mail plugin.
Usually, you'll set up Dada Mail in your cgi-bin: in your cgi-bin, there's a directory called, ``dada''. Inside the, ``dada'' directory, there are at least two directories, one called, ``DADA'' (uppercase) and the mail.cgi script.
In the, ``dada'' directory, create a new directory called, ``plugins''. Upload the dada_bounce_handler.pl script, already configured, into this directory. Change its permissions to, ``755''. Visit the script in your web browser. To run the bounce handler on your bounced messages, click the, Parse Bounces... button.
Before the bounce handler will work, Dada Mail has to know to use it. Jump down to, Telling Dada Mail to use the Bounce Handler for instructions on how to do that.
If you would like have a link on the left hand side of the list control panel, find the following line in the Config.pm:
# {-Title => 'Bounce Handler', # -Title_URL => $PLUGIN_URL."/dada_bounce_handler.pl", # -Function => 'dada_bounce_handler', # -Activated => 1, # },
And uncomment it (Take off the, ``#'' on each line).
If you are not going to set up this program as a cgi script, I would just make a directory in your home directory and place the script there. If you've set up Dada Mail as outlined in the Magic Book, you may want to make another directory in the .dada_files directory, called .scripts, and install this script in there. You can still install the bounce handler in the, cgi-bin/dada/plugins/ directory as it's a nice convenience to run the program as both a command line/cronjob and also as a Dada Mail plugin.
chmod 755 dada_bounce_handler.pl
That's it as far as installation of the script.
Running the program without any arguments will have it check the mailbox for bounces, parse the messages and handle the bounces. ie:
prompt>./dada_bounce_handler.pl
I suggest before you do that, you test the dada_bounce_handler.pl.
You can pass the --test argument to dada_bounce_handler.pl to make sure everything is workings as it should. The --test argument needs to take one of a few paramaters:
prompt>./dada_bounce_handler.pl --test pop3
This will test only your POP3 login. If it's successful, it'll return the number of messages waiting:
prompt>./dada_bounce_handler.pl --test pop3 POP3 Login succeeded. Message count: 5
If the login failed, you'll get back a message that reads:
prompt>./dada_bounce_handler.pl --test pop3 POP3 login failed.
dada_bounce_handler.pl won't act on these test messages, but will do everything until that point. You'll get back a verbose message of the going's on of the script:
prompt> perl dada_bounce_handler.pl --test message8.txt test #1: message8.txt ------------------------------------------------------------ ------------------------------------------------------------------------ Content-type: multipart/report Effective-type: multipart/report Body-file: NONE Subject: Returned mail: see transcript for details Num-parts: 3 -- Content-type: text/plain Effective-type: text/plain Body-file: NONE -- Content-type: message/delivery-status Effective-type: message/delivery-status Body-file: NONE -- Content-type: message/rfc822 Effective-type: message/rfc822 Body-file: NONE Num-parts: 1 -- Content-type: text/plain Effective-type: text/plain Body-file: NONE Subject: Simoni Creative - Dada Mail Mailing List Confirmation -- ------------------------------------------------------------------------ List: skazat_design_newsletter Email: de4est@centurytel.net
Last-Attempt-Date: Sun, 13 Apr 2003 20 Action: failed Status: 5.1.1 Diagnostic-Code: SMTP; 550 5.1.1 <de4est@centurytel.net>... User unknown Final-Recipient: RFC822; de4est@centurytel.net Remote-MTA: DNS; [209.142.136.158]
Using Rule: default
The first chunk of output is a skeleton of the bounced message. If it looks similar to what's above, you most likely gave the bounce handler a real email message.
After that, will be listed the findings of the bounce handler. The List and Email address will be listed, followed by some diagnostic code.
The last thing printed out is the rule, and we'll get to that shortly.
You could run dada_bounce_handler.pl every now and again from the command line, but you'd get very sick of it and I spent an entire weekend in May to write this script to be lazy.
To accomplish that, you want to set this script to execute via a cron or scheduled, job. Here's what a theoretical cron tab for this script may look like:
0 1 * * * /usr/bin/perl /home/myaccount/cgi-bin/dada/plugins/dada_bounce_handler.pl >/dev/null 2>&1
This will run the script every day around 1am. You can run this script as often as you want, just be logical. I wouldn't run this script every five minutes, that's a bit overkill.
Different hosts may have a control panel to set up crontabs, my host gives me the pleasure of the contrab command. I type in:
prompt> crontab -e
and am launched into my favorite text editor to type in the crontab.
You're going to have to tell Dada Mail explicitly that you want bounces to go to the bounce handler. The first step is to set the Dada List Administrator to your bounce email address. You set this in the list control panel, under Change List Information
Once you do that, you need to tell Dada Mail that you want the correct headers in your list messages to say, ``use the admin address for bounces''
Usually, this means that the Return-path header needs to be set. There are a few ways to accomplish this, some more preferable than others.
In the list control panel, go to: Sending Options - SMTP settings> and check the box labeled: Set the Sender of SMTP mailings to the list administration email address
This should set the sending to the admin email, and in turn, set the Return-Path header.
To test out any of these configurations, Send yourself a test message and view the source of the message itself, in your mail reader. In the mail headers, you should see the Return-Path header:
Return-Path: <dadabounce@myhost.com> Delivered-To: justin@myhost.com Received: (qmail 75721 invoked from network); 12 May 2003 04:50:01 -0000 Received: from myhost.com (208.10.44.140) by hedwig.myhost.com with SMTP; 12 May 2003 04:50:01 -0000 Date:Sun, 11 May 2003 23:50:01 -0500 From:justin <justin@myhost.com> Subject:Test, Test, Test To:justin@myhost.com Sender:dadabounce@myhost.com Reply-To:justin <justin@myhost.com> Precedence:list Content-type:text/plain; charset=iso-8859-1
Notice that the first line has the Return-Path header, correctly putting my bounce email address. My List Owner address, justin@myhost.com still occupies the To: and Reply-To headers, so whoever replies to my message will reply to me, not the bounce handler.
Once you've dialed in your list to use the bounce handler, you should be all set.
There's a slew of optional arguments you can give to this script:
But anyways:
prompt>./dada_bounce_handler \ --server mail.myhost.com\ --username dadabounce\ --password secretgodmoney
All three of these options are optional and you can use them with any of the tests, discussed above.
prompt>./dada_bounce_handler --test bounces But bounce handling will go through to completion.
Mystery Girl version: 1.6 Dada Mail version: 2.10.9
[Sun May 11 16:57:23 2003] justin unsubscribe_bounced_email from_list \ fdsafsa890sadf89@hotmail.com Status: 5.x.y, Action: ,
The format is:
time \t list \t action \t email \t diagnostics
If you don't want to pass the log each time, you can set a log in the $Log variable -
my $Log = $LOGS . '/bounces.txt';
If you're using the Log Viewer plugin (part of the MagicBook), the plugin will automatically find this file and add it to the logs it will show.
I like to use this as a final test; I can test one real message towards completion and make sure everything is OK.
If you do want to handle, say 1000 messages at a day, I would suggest to set the number of messages it handles to something like 100 and set your cronjob to run 10 times, perhaps 15 minutes apart. Your call, though.
dada_bounce_handler.pl figures out what to do with the bounce messages receives by consulting a group of rules. These rules are highly configurable, so if you need to change the behavior of this script, you don't have to change the code.
These rules are stored in the $Rules hashref. An example rule:
{ exim_user_unknown => { Examine => { Message_Fields => { Status => [qw(5.x.y)], Guessed_MTA => [qw(Exim)], }, Data => { Email => 'is_valid', List => 'is_valid', } }, Action => { add_to_score => $Default_Hard_Bounce_Score, }, } },
exim_user_unknown is the title of the rule - just a label, nothing else.
Examine holds a set of parameters that the handler looks at when trying to figure out what to do with a bounced message. This example has a Message_Fields entry and inside that, a Status entry. The Status entry holds a list of status codes. The ones in shown there all correspond to hard bounces; the mailbox probably doesn't exist. Message_Fields also hold a, Guessed_MTA entry - it's explicitly looking for a bounce back from the, Exim mail server.
Examine also holds a Data entry, which holds the Email or List entries, or both. Their values are either 'is_valid', or 'is_invalid'.
So, to sum this all up, this rule will match a message that has Status: Message Field contaning a user unknown error code, (5.1.1, etc) and also a Guessed_MTA Message Field containing, Exim. The message also has to be parsed to have found a valid email and list name.
Pretty Slick, eh?
If this all matches, the Action is... acted upon. In this case, the offending email address will be appended a, Bounce Score of, whatever, $Default_Hard_Bounce_Score, which is by default, 4.
If you would like to have the bounced address automatically removed, without any sort of scoring happening, change the action from,
add_to_score => $Default_Hard_Bounce_Score
to:
unsubscribe_bounced_email => 'from_list'
Also, changing from_list, to from_all_lists will do the trick.
I could change the line:
unsubscribe_bounced_email => 'from_list',
to:
mail_list_owner => 'user_unknown_message'
This will, instead of deleting the email automatically, send a message to the list owner, stating that, ``Hey, the message bounced, what do you want to do?''
Another example:
{ over_quota => { Examine => { Message_Fields => { Status => [qw(5.2.2)] }, Data => { Email => 'is_valid', List => 'is_valid', } }, Action => { mail_list_owner => 'over_quota_message', }, }
This time, I created a list for messages that get bounced because the mailbox is full. This is still considered a hard bounce, but I don't want the subscriber removed because they haven't check their inbox during the week. In this case, the Action has been set to:
mail_list_owner => 'over_quota_message',
Which will do what it sounds like, it'll mail the list owner a message explaining the circumstances.
Here's a schematic of all the different things you can do:
{ rule_name => { Examine => { Message_Fields => { Status => qw([ ]), Last-Attempt-Date => qw([ ]), Action => qw([ ]), Status => qw([ ]), Diagnostic-Code => qw([ ]), Final-Recipient => qw([ ]), Remote-MTA => qw([ ]), # etc, etc, etc }, Data => { Email => 'is_valid' | 'is_invalid' List => 'is_valid' | 'is_invalid' } }, Action => { add_to_score => $x, # where, "$x" is a number mail_list_owner => 'user_unknown_message', mail_list_owner => 'email_not_found_message', mail_list_owner => 'over_quota_message', unsubscribe_bounced_email => 'from_list' | 'from_all_lists', }, },
Mystery Girl also supports the use of regular expressions for matching any of the Message_Fields. To tell the parser that you're using a regular expression, make the Message_Field key end in '_regex':
'Final-Recipient_regex' => [(qr/RFC822/)],
Setting rules is sort of the super advanced part of the configuration, but it may come in handy.
We talked about scoring, but not in great detail, so let's do that:
By default, The Bounce Handler assigns a particular score to each email address that bounces back a message. These scores are tallied each time an email address bounces a message.
Since Dada Mail understands the differences between Hard Bounces and Soft Bounces, it'll append a smaller score for soft bounces, and a larger score for hard bounces.
Once the email address's Bounce Score reaches the Threshold, the email address is then removed from the list.
You can manipulate the Soft and Hard Bounce Scores and Threshold pretty easily. On the top of this script, you'll see the necessary variables to tweak,
Fairly self-explanitory.
If you want even greater control over what kind of bounces give what scores, you can manipulate the Bounce Rules themselves, as described above.
Some things to understand:
Currently, Scores are Global - they work for all lists at once. Once an email address reaches the Threshold, they will be removed from all lists that are handled by Dada Mail. In one sense, this seems limiting, but if an email address is not receiving mail from one list, or doesn't exist, it doesn't have much worth on any list of yours.
If you would like to periodically erase the saved scores, you may do so, by running this script via the command line, like so:
./dada_bounce_handler.pl --erase_score_card
Dada Mail basically works by saying, After x amount of bounces, just remove from the list.
You'll need to do a few things:
Notice there are no quotes around 0777.
http://sourceforge.net/tracker/?group_id=13002&atid=113002
http://sourceforge.net/tracker/?group_id=13002&atid=113002
And we'll see if we can't get that kind of bounce in a new version.
When the bounce handler emails a list owner, you can do nothing but answer back to it. Yeah Yeah Yeah.
(colophon)
Actually, the lyrics I'm thinking of aren't from the song, Mystery Girl, but from the song, ``Bang!'' off of the YYY's self titled release. Mystery Girl is the next song on that album. The song after that is one called, ``Art Star'', which is what I am in the daytime! The next song is called, ``Miles Away'', which is where you probably are to me. All this in, ``Our Time'' (the last song) See? it's like this was all written in the stars.
Here's a small clip of the YYY's performing ``Mystery Girl'' at the Gothic on 11.20.03 that I took:
http://mojo.skazat.com/media/YYYs_Mystery_Girl_Clip.mov
hot!
NOTE: Most of the history of this program is located in the main Dada Mail changelog.
VERP support was added to Dada Mail. This should make finding what email bounced the message easier.
The MIME-Tools CPAN Perl Module collection is included with Dada Mail now! You do NOT have to install it manually anymore!
Can't use an undefined value as an ARRAY reference at dada_bounce_handler.pl
There was also a small issue were either the list or email were found in the bounce, but this information was thrown away if both weren't found in some instances. This should be fixed and allow some better bounce handling for Exim and Postfix users.
More Rules have added
How humiliating. That should be fixed.
I also added a few more rules, the delivery_error_550 was much too strict, as not all bounced messages have a Diagnostic Code; so I made another rule that's very similar, but doesn't have the <Diagnostic-Code_regex>
Emails sent from Mystery Girl to a list owner now should have a description of what the status of a bounce means, if there is a Status. This should allow someone to have a better idea on what they should do with a report from Mystery Girl. All descriptions have been taken right out of rfc1893:
http://www.ietf.org/rfc/rfc1893.txt
...
Why not? Well, Mystery Girl really didn't know about anything * but * the ``Status'' and ``Action'' Message Fields. Furthermore, I didn't follow my own scheme for the bounce rules and put Qmail and Exim as a scalar, not an array ref.
So, now Mystery Girl knows about the Guessed_MTA message field, and should know about every other one as well.
If you were having trouble having your own rules work, above is why. Everything should be patched up and fixed. The new code is actually half the size and works much better. Go... stuff!
Furthermore, I've changed the format of the rules, The Rules themselves are a array ref, instead of a hash ref, which means that the rules are tried in order.
I've also added regular expression function to Examine, if you have a message field, say, Status, that you want to do a regular expression on, you can say this:
Status_regex => [(qr/^5(\.0\.0|\.1\.1)$/)],
instead of:
Status => [qw(5.0.0 5.1.1)];
This version was introduced in Dada Mail 2.8.8, it should work for any version of Dada Mail from 2.8.5 on.
Thanks to Tracy Gibson (sf: tntmom5) and Adam Henry hank _at_ marinar.com for the exim reports.
I also added a separate rule for both qmail and Exim, since both don't produce real status codes, just '5.x.y' or '4.x.y', you may, for some reason, treat these as special cases.
I also added a new flag, --version, so you can report just exactly what version you have of the proggy.
THAT is fixed. No even amusingly funny comments about how this script should work now.
It is a common experience that a problem difficult at night is resolved in the morning after a committee of sleep has worked on it.
- John Steinbeck
I also took the -w flag off, since it was creating some line noise i'll deal with sooner than later.
Script should work now
should
work...
No for real.
Perhaps think about making filters specifically for Postfix. They seem to have their own way of doing things, like Qmail.
Add onto that custom a filter for AOL/Compuserve/Netscape
Thanks to: Jake Ortman Henry Hughes for some prelim bounce examples.
Thanks to Eryq ( http://www.zeegee.com ) for the amazing MIME-tools collection. It's a gnarly group of modules.
Copyright (c) 1999 - 2006 Justin Simoni http://justinsimoni.com All rights reserved.
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
Parts of this script were swiped from Mail::Bounce::Qmail module, fetched from here:
http://mikoto.sapporo.iij.ad.jp/cgi-bin/cvsweb.cgi/fmlsrc/fml/lib/Mail/Bounce/Qmail.pm
The copyright of that code stated:
Copyright (C) 2001,2002,2003 Ken'ichi Fukamachi All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Thanks Ken'ichi