#228 ✓fixreleased
Bill Cole

X-Spam-Score parsing should be better

Reported by Bill Cole | January 31st, 2012 @ 10:39 PM

It is a somewhat common practice to format the X-Spam-Score header as suggested by the MIMEDefang "milter" (widely used with Sendmail and Postfix) like this:

X-Spam-Score: $hits ($score) $names

Where $hits is a numeric score (e.g. from SpamAssassin), $score is a string of asterisks as long as the integer part of $hits, and $names is a string containing a comma-delimited list of the specific rules matches.

MM tries to parse that header, but it only works when $hits is a negative number. Any message with a positive value for $hits is shown in the message list "Spam Score" column as 0.000. (Parsing works correctly with mail filters that only put a number in X-Spam-Score.)

Comments and changes to this ticket

  • benny

    benny February 1st, 2012 @ 09:16 AM

    • State changed from “new” to “accepted”

    Actually, MailMate does not parse X-Spam-Score. It parses X-Spam-Status which in one of my messages look like this (added by SpamSieve I believe):

    X-Spam-Status: No, score=-2.532 tagged_above=-100 required=6.31 tests=[BAYES_00=-2.599, RCVD_BY_IP=0.067]
    

    That message also has:

    X-Spam-Score: -2.532
    

    So, it may work to base the column on that header instead, but I'm not sure that is true in general. To be sure, I could base it on X-Spam-Score and then fall back to X-Spam-Status.

    Could you provide me with some examples of your X-Spam-Score headers, so I can be sure it'll work for you?

    (Not sure what is going on with the the positive values. I have both positive and negative values in my X-Spam-Status headers and they seem to work fine in the Spam Score column.)

  • Bill Cole

    Bill Cole February 2nd, 2012 @ 04:33 AM

    Amazing! It appears I've had a bug in some filter code that has gone completely unnoticed for years. Apparently it is adding a X-Spam-Status header similar to the SpamSieve one (less [] around he test list) only to mail scoring below -1. How very odd! I have some work to do...

    Below are 20 examples of X-Spam-Score headers generated by the MIMEDefang default pattern. I've trimmed them to 79 characters to prevent wrapping: the trimmed-off parts were just more of the comma-delimited rule name list. Note that I can't provide samples for scores over 4.3 because I reject those messages in SMTP, but a system that accepts and delivers obvious spam might add headers with scores that are very high. I've seen spam scores over 100, but not often. It is also possible to see negative scores of that magnitude, usually because of manual whitelisting. It should be simple enough to just extract the first token of the header value and confirm that it looks like a number.

    X-Spam-Score: -8.7 () BAYES_50,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,RCVD_IN_RP_RNBL
    X-Spam-Score: 3.826 (***) BAYES_99,DATE_IN_PAST_03_06,HTML_FONT_SIZE_HUGE,HTML_
    X-Spam-Score: -0.009 () BAYES_50,HTML_MESSAGE,SCC_DEBUG,T_RP_MATCHES_RCVD
    X-Spam-Score: 2.7 (**) BAYES_50,HTML_MESSAGE,SCC_DEBUG,TVD_FROM_1
    X-Spam-Score: 1.061 (*) BAYES_50,NO_DNS_FOR_FROM,RCVD_IN_DNSWL_NONE,RCVD_IN_SCC
    X-Spam-Score: -1.06 () AWL,BAYES_00,HTML_MESSAGE,SCC_DEBUG,T_RP_MATCHES_RCVD
    X-Spam-Score: 0.011 () BAYES_50,DKIM_SIGNED,DKIM_VALID,RCVD_IN_DNSWL_NONE,SCC_D
    X-Spam-Score: 2.141 (**) BAYES_50,RCVD_IN_DNSWL_MED,SCC_DEBUG,SINGLE_HEADER_1K,
    X-Spam-Score: 0.937 () BAYES_50,HTML_IMAGE_RATIO_02,HTML_MESSAGE,SCC_DEBUG,SCC_
    X-Spam-Score: 1.3 (*) BAYES_40,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SCC_DEBUG,SCC_RCV
    X-Spam-Score: 2.772 (**) BAYES_50,DKIM_ADSP_CUSTOM_MED,FORGED_YAHOO_RCVD,FREEMA
    X-Spam-Score: 2.449 (**) BAYES_60,HTML_MESSAGE,RCVD_IN_BRBL_LASTEXT,SCC_DEBUG
    X-Spam-Score: -2.001 () BAYES_60,HTML_MESSAGE,RCVD_IN_DNSWL_HI,SCC_DEBUG,SPF_PA
    X-Spam-Score: 1.735 (*) BAYES_50,FREEMAIL_FROM,HTML_MESSAGE,MIME_QP_LONG_LINE,R
    X-Spam-Score: -0.712 () BAYES_20,SCC_DEBUG,SPF_PASS,T_RP_MATCHES_RCVD
    X-Spam-Score: 0.002 () BAYES_50,HTML_MESSAGE,LOTS_OF_MONEY,SCC_DEBUG,SCC_SHORTN
    X-Spam-Score: 3.758 (***) BAYES_50,RCVD_IN_BRBL_LASTEXT,RDNS_NONE,SCC_DEBUG,SPF
    X-Spam-Score: 0.089 () BAYES_50,SCC_DEBUG,SCC_RCVD_FORMAT_NOT_SENDMAIL,SPF_PASS
    X-Spam-Score: 2.252 (**) BAYES_50,DKIM_ADSP_CUSTOM_MED,FREEMAIL_ENVFROM_END_DIG
    X-Spam-Score: 2.252 (**) BAYES_50,DKIM_ADSP_CUSTOM_MED,FREEMAIL_ENVFROM_END_DIG
    
  • benny

    benny February 2nd, 2012 @ 09:13 AM

    • State changed from “accepted” to “fixcommitted”

    Based on your examples I am now also parsing X-Spam-Score and I improved the parser for X-Spam-Status. Only problem is that I only know that it'll work for my messages and your examples. These headers really need to be standardized :-)

    The Spam Score column is based on X-Spam-Status if a score is found within that and otherwise it uses the X-Spam-Score header. (Unfortunately sorting only works for X-Spam-Status.)

    And now the fun part. I also added parsing of the test names (for both headers). So now it is possible to use the Statistics layout and, e.g., see what it comes up with if one selects “X-Spam-Status ▸ Tests ▸ Test”. Here is my top 10 for some mailing list messages not marked as spam:

    %     Name (count)
    40.6: BAYES_00 (5029)
    22.4: AWL (2779)
    16.6: RCVD_BY_IP (2062)
     8.3: HTML_MESSAGE (1032)
     1.8: HTML_30_40 (219)
     1.8: HTML_40_50 (218)
     1.2: HTML_50_60 (145)
     1.0: RATWARE_GECKO_BUILD (122)
     0.9: HTML_20_30 (113)
     0.7: HTML_10_20 (86)
    

    Maybe not very useful :-)

  • benny

    benny March 21st, 2012 @ 04:40 PM

    • State changed from “fixcommitted” to “fixreleased”
  • Michael Tsai

    Michael Tsai March 29th, 2016 @ 05:55 PM

    Actually, MailMate does not parse X-Spam-Score. It parses X-Spam-Status which in one of my messages look like this (added by SpamSieve I believe)

    I just came across this old thread and wanted to clarify that SpamSieve does not add or change the X-Spam-Status (or any other header).

  • benny

    benny March 29th, 2016 @ 06:35 PM

    @Michael:I have no idea why I would have written that since it makes no sense :-) Most likely I meant to write SpamAssassin.

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

Mac OS X email client.

Shared Ticket Bins

People watching this ticket

Pages