#2511 ✓fixcommitted
Ari Goldmith

MailMate running a fetch with a negative ID - how do I reset?

Reported by Ari Goldmith | March 22nd, 2020 @ 04:26 AM

There was some hiccup with the mail server today. I'm not sure if that caused some corrupt message or not.

Currently, when I try to take online or synchronize it comes back with an error:

D8 BAD Error in IMAP command UID FETCH: Invalid uidset (0.000 + 0.000 secs).

This is the response from:

D8 UID FETCH -2088730488:* (UID)

It looks like MailMate thinks there is an id range starting at that negative number. How do I reset that?

Here is the log leading up to the error:
03:43:58 Handling reply
03:43:58 Running action
03:43:58 Sending request (145)
03:43:58 Handling request
03:43:58 Ready to run action (retry count: 0)
03:43:58 Synchronizing mailbox (attempting to do it quickly): INBOX
03:43:58 Selecting mailbox: INBOX
03:43:58 S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk $Forwarded $MailFlagBit0 Forwarded)
03:43:58 S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk $Forwarded $MailFlagBit0 Forwarded *)] Flags permitted.
03:43:58 S: * 6913 EXISTS
03:43:58 S: * 0 RECENT
03:43:58 S: * OK [UNSEEN 7] First unseen.
03:43:58 S: * OK [UIDVALIDITY 1534701063] UIDs valid
03:43:58 S: * OK [UIDNEXT 2206236812] Predicted next UID
03:43:58 S: * OK [HIGHESTMODSEQ 25012] Highest
03:43:58 S: D7 OK [READ-WRITE] Select completed (0.000 + 0.000 secs).
03:43:58 Attempting fast synchronization.
03:43:58 C: D8 UID FETCH -2088730488:* (UID)
03:43:58 S: D8 BAD Error in IMAP command UID FETCH: Invalid uidset (0.000 + 0.000 secs).
03:43:58 Error: Server response: “D8 BAD Error in IMAP command UID FETCH: Invalid uidset (0.000 + 0.000 secs).”.
03:43:58 Error code: 1000
03:43:58 Failed action (1003). Reset observed read/write timeouts: 8/8

Comments and changes to this ticket

  • Ari Goldmith

    Ari Goldmith March 22nd, 2020 @ 12:59 PM

    Is the UID in MailMate field too small to hold the large UIDs generated by the server? ( a signed int, instead of unsigned int or something?)

  • Ari Goldmith

    Ari Goldmith March 22nd, 2020 @ 03:42 PM

    It's Nov 9th 2022, and I'm still trying to track down this issue...

    in which when MailMate tries to sync with the IMAP server it does a UID FETCH of a negative number.

    I’m guessing that number is being generated from the last UIDNEXT and stored in a variable that is a Int32 but should be an UInt32 as evidenced below.

    Some lines from my log:

    22:51:19 S: * 0 RECENT
    22:51:19 S: * OK [UNSEEN 28] First unseen.
    22:51:19 S: * OK [UIDVALIDITY 1584965498] UIDs valid
    22:51:19 S: * OK [UIDNEXT 2206296242] Predicted next UID
    22:51:19 S: * OK [HIGHESTMODSEQ 7] Highest
    22:51:19 S: D3 OK [READ-WRITE] Select completed (0.007 + 0.000 + 0.006 secs).
    22:51:19 Condstore-based check (7 > 7) means fast synchronization is enabled
    22:51:19 Attempting fast synchronization.
    22:51:19 C: D4 UID FETCH -2088671054:* (UID)
    22:51:19 S: D4 BAD Error in IMAP command UID FETCH: Invalid uidset (0.001 + 0.000 secs).
    22:51:19 Error: Server response: “D4 BAD Error in IMAP command UID FETCH: Invalid uidset (0.001 + 0.000 secs).”.

    Here the server states the next UID would be 2206296242.

    Mailmate tries to do a UID FETCH -2088671054:* (UID)

    2206296242 | 0 happens to equal -2088671054 which isn’t a coincidence, so there’s your overflow.

    I’d assume it is when Mailmate stores the UIDNEXT from the previous session (it gets mangled) and then uses the mangled value in the new session to fetch new messages since the last session.

  • Zivago Lee

    Zivago Lee September 18th, 2020 @ 03:10 PM

    I also get this every so often. The server is a dovecot server and the only way to "fix" this is to delete all the dovecot files in my Maildir (dovecot-uidlist, dovecot.index.cache, dovecot.index.*, dovecot-uidvalidity*). Let me know I can get logs from Mailmate and I can provide them.

  • Ari Goldmith

    Ari Goldmith November 9th, 2022 @ 11:05 PM

    Could it be that the variable that stores UIDNEXT is a signed 32bit int instead of an unsigned 32bit int, and therefore when it processes the UIDNEXT and stores it in the variable it gets turned into a negative number. Then when it queries the server, it does a UID FETCH -UIDNEXT:* which then errors out by the server?

  • benny

    benny November 10th, 2022 @ 11:14 AM

    • State changed from “new” to “reproduced”

    Sorry about the far too late reply on this. (I've explicitly marked you as a non-spammer now, but that wasn't related to my lack of reply.)

    The original IMAP “standard” is very explicit about UID values being 32 bit and (I just checked) this hasn't actually changed RFC9501. I think I originally thought this ticket was about a server allowing 64 bit values, but as is clearly stated it's actually incorrect handling of values larger than 31 bit. That should certainly be fixable and I should have done that a long time ago. I'm very sorry about that and I'll take a look at it today.

    Looking forward, a switch to 64 bit might some day be necessary, but that's going to be harder to change in MailMate (but possible of course).

  • Ari Goldmith

    Ari Goldmith November 10th, 2022 @ 04:41 PM

    Thank you, Benny!

    I had checked the IMAP RFC to see if the server was improperly generating UIDs https://datatracker.ietf.org/doc/html/rfc3501#section- It turned out that technically the IDs were out of range for a 32 signed int, but in range for an unsigned 32 int which is the definition of a "number" at the end of the rfc (0 <= n < 4,294,967,296)

    Also, I've been harassing the ESP too because though they claim they reindex the INBOX, but the UIDs don't get reset, which also would have temporarily fixed the issue, or kicked the can until the next time it popped up.

    I'm glad this item is fixable, and am happy I was able to contribute something to help make MailMate better.


  • benny

    benny November 11th, 2022 @ 08:16 AM

    • State changed from “reproduced” to “fixcommitted”

    Hold down ⌥ when clicking “Check Now” in the Software Update preferences pane to get the latest test release.

    This was a “blind” fix, that is, I did not really try to reproduce the issue, but I do believe I found the issue.

    MailMate was definitely to blame here, but it's also pretty wild to have more than 2 billion messages pass through an IMAP mailbox. If my math is correct then it's about 5000 messages every hour for the past 50 years. If that can happen then 4 billion is likely not much better.

    The UIDs of a mailbox can only be reset if the server also resets the UIDVALIDITY value. Unfortunately, I'm afraid some email clients won't handle that correctly though (but that would be a much worse bug compared to the above). MailMate does handle this, but it'll result in all emails of the mailbox being refetched from the server.

  • Ari Goldmith

    Ari Goldmith November 11th, 2022 @ 12:12 PM

    Indeed, there wasn't that many messages that passed through that mailbox. Though my guess is that they may have generated the starting UID from a date string YYMMDDHHMM or something similar, and then kept counting from there, instead of bumping the UIDVALIDITY up.

    Thank you so much for the quick fix!

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

Mac OS X email client.

Shared Ticket Bins

People watching this ticket