Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

X-Spam-Score parsing should be better

#228

It is a somewhat common practice to format the X-Spam-Score header as suggested by the MIMEDefang "milter" (widely used with Sendmail and Postfix) like this:

X-Spam-Score: $hits ($score) $names

Where $hits is a numeric score (e.g. from SpamAssassin), $score is a string of asterisks as long as the integer part of $hits, and $names is a string containing a comma-delimited list of the specific rules matches.

MM tries to parse that header, but it only works when $hits is a negative number. Any message with a positive value for $hits is shown in the message list "Spam Score" column as 0.000. (Parsing works correctly with mail filters that only put a number in X-Spam-Score.)

Reported by Bill Cole · January 31st, 2012 @ 10:39 PM

State: fixreleased
Milestone: none
Assigned to: benny benny

Activity

  1. benny
    benny
    • State changed from new to accepted

    Actually, MailMate does not parse X-Spam-Score. It parses X-Spam-Status which in one of my messages look like this (added by SpamSieve I believe):

    X-Spam-Status: No, score=-2.532 tagged_above=-100 required=6.31 tests=[BAYES_00=-2.599, RCVD_BY_IP=0.067]

    That message also has:

    X-Spam-Score: -2.532

    So, it may work to base the column on that header instead, but I'm not sure that is true in general. To be sure, I could base it on X-Spam-Score and then fall back to X-Spam-Status.

    Could you provide me with some examples of your X-Spam-Score headers, so I can be sure it'll work for you?

    (Not sure what is going on with the the positive values. I have both positive and negative values in my X-Spam-Status headers and they seem to work fine in the Spam Score column.)

    February 1st, 2012 @ 09:16 AM

  2. Bill Cole
    Bill Cole

    Amazing! It appears I've had a bug in some filter code that has gone completely unnoticed for years. Apparently it is adding a X-Spam-Status header similar to the SpamSieve one (less [] around he test list) only to mail scoring below -1. How very odd! I have some work to do...

    Below are 20 examples of X-Spam-Score headers generated by the MIMEDefang default pattern. I've trimmed them to 79 characters to prevent wrapping: the trimmed-off parts were just more of the comma-delimited rule name list. Note that I can't provide samples for scores over 4.3 because I reject those messages in SMTP, but a system that accepts and delivers obvious spam might add headers with scores that are very high. I've seen spam scores over 100, but not often. It is also possible to see negative scores of that magnitude, usually because of manual whitelisting. It should be simple enough to just extract the first token of the header value and confirm that it looks like a number.

    X-Spam-Score: -8.7 () BAYES_50,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RCVD_IN_RP_RNBL
X-Spam-Score: 3.826 (***) BAYES_99,DATE_IN_PAST_03_06,HTML_FONT_SIZE_HUGE,HTML_
X-Spam-Score: -0.009 () BAYES_50,HTML_MESSAGE,SCC_DEBUG,T_RP_MATCHES_RCVD
X-Spam-Score: 2.7 (**) BAYES_50,HTML_MESSAGE,SCC_DEBUG,TVD_FROM_1
X-Spam-Score: 1.061 (*) BAYES_50,NO_DNS_FOR_FROM,RCVD_IN_DNSWL_NONE,RCVD_IN_SCC
X-Spam-Score: -1.06 () AWL,BAYES_00,HTML_MESSAGE,SCC_DEBUG,T_RP_MATCHES_RCVD
X-Spam-Score: 0.011 () BAYES_50,DKIM_SIGNED,DKIM_VALID,RCVD_IN_DNSWL_NONE,SCC_D
X-Spam-Score: 2.141 (**) BAYES_50,RCVD_IN_DNSWL_MED,SCC_DEBUG,SINGLE_HEADER_1K,
X-Spam-Score: 0.937 () BAYES_50,HTML_IMAGE_RATIO_02,HTML_MESSAGE,SCC_DEBUG,SCC_
X-Spam-Score: 1.3 (*) BAYES_40,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SCC_DEBUG,SCC_RCV
X-Spam-Score: 2.772 (**) BAYES_50,DKIM_ADSP_CUSTOM_MED,FORGED_YAHOO_RCVD,FREEMA
X-Spam-Score: 2.449 (**) BAYES_60,HTML_MESSAGE,RCVD_IN_BRBL_LASTEXT,SCC_DEBUG
X-Spam-Score: -2.001 () BAYES_60,HTML_MESSAGE,RCVD_IN_DNSWL_HI,SCC_DEBUG,SPF_PA
X-Spam-Score: 1.735 (*) BAYES_50,FREEMAIL_FROM,HTML_MESSAGE,MIME_QP_LONG_LINE,R
X-Spam-Score: -0.712 () BAYES_20,SCC_DEBUG,SPF_PASS,T_RP_MATCHES_RCVD
X-Spam-Score: 0.002 () BAYES_50,HTML_MESSAGE,LOTS_OF_MONEY,SCC_DEBUG,SCC_SHORTN
X-Spam-Score: 3.758 (***) BAYES_50,RCVD_IN_BRBL_LASTEXT,RDNS_NONE,SCC_DEBUG,SPF
X-Spam-Score: 0.089 () BAYES_50,SCC_DEBUG,SCC_RCVD_FORMAT_NOT_SENDMAIL,SPF_PASS
X-Spam-Score: 2.252 (**) BAYES_50,DKIM_ADSP_CUSTOM_MED,FREEMAIL_ENVFROM_END_DIG
X-Spam-Score: 2.252 (**) BAYES_50,DKIM_ADSP_CUSTOM_MED,FREEMAIL_ENVFROM_END_DIG

    February 2nd, 2012 @ 04:33 AM

  3. benny
    benny
    • State changed from accepted to fixcommitted

    Based on your examples I am now also parsing X-Spam-Score and I improved the parser for X-Spam-Status. Only problem is that I only know that it'll work for my messages and your examples. These headers really need to be standardized :-)

    The Spam Score column is based on X-Spam-Status if a score is found within that and otherwise it uses the X-Spam-Score header. (Unfortunately sorting only works for X-Spam-Status.)

    And now the fun part. I also added parsing of the test names (for both headers). So now it is possible to use the Statistics layout and, e.g., see what it comes up with if one selects “X-Spam-Status ▸ Tests ▸ Test”. Here is my top 10 for some mailing list messages not marked as spam:

    %     Name (count)
40.6: BAYES_00 (5029)
22.4: AWL (2779)
16.6: RCVD_BY_IP (2062)
 8.3: HTML_MESSAGE (1032)
 1.8: HTML_30_40 (219)
 1.8: HTML_40_50 (218)
 1.2: HTML_50_60 (145)
 1.0: RATWARE_GECKO_BUILD (122)
 0.9: HTML_20_30 (113)
 0.7: HTML_10_20 (86)

    Maybe not very useful :-)

    February 2nd, 2012 @ 09:13 AM

  4. benny
    benny
    • State changed from fixcommitted to fixreleased

    March 21st, 2012 @ 04:40 PM

  5. Michael Tsai
    Michael Tsai

    Actually, MailMate does not parse X-Spam-Score. It parses X-Spam-Status which in one of my messages look like this (added by SpamSieve I believe)

    I just came across this old thread and wanted to clarify that SpamSieve does not add or change the X-Spam-Status (or any other header).

    March 29th, 2016 @ 05:55 PM

  6. benny
    benny

    @Michael:I have no idea why I would have written that since it makes no sense :-) Most likely I meant to write SpamAssassin.

    March 29th, 2016 @ 06:35 PM

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.