Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

MailMate running a fetch with a negative ID - how do I reset?

#2511

There was some hiccup with the mail server today. I'm not sure if that caused some corrupt message or not.

Currently, when I try to take online or synchronize it comes back with an error:

D8 BAD Error in IMAP command UID FETCH: Invalid uidset (0.000 + 0.000 secs).

This is the response from:

D8 UID FETCH -2088730488:* (UID)

It looks like MailMate thinks there is an id range starting at that negative number. How do I reset that?

Here is the log leading up to the error:
03:43:58 Handling reply
03:43:58 Running action
03:43:58 Sending request (145)
03:43:58 Handling request
03:43:58 Ready to run action (retry count: 0)
03:43:58 Synchronizing mailbox (attempting to do it quickly): INBOX
03:43:58 Selecting mailbox: INBOX
03:43:58 C: D7 SELECT INBOX (CONDSTORE)
03:43:58 S: * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk $Forwarded $MailFlagBit0 Forwarded)
03:43:58 S: * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk $Forwarded $MailFlagBit0 Forwarded *)] Flags permitted.
03:43:58 S: * 6913 EXISTS
03:43:58 S: * 0 RECENT
03:43:58 S: * OK [UNSEEN 7] First unseen.
03:43:58 S: * OK [UIDVALIDITY 1534701063] UIDs valid
03:43:58 S: * OK [UIDNEXT 2206236812] Predicted next UID
03:43:58 S: * OK [HIGHESTMODSEQ 25012] Highest
03:43:58 S: D7 OK [READ-WRITE] Select completed (0.000 + 0.000 secs).
03:43:58 Attempting fast synchronization.
03:43:58 C: D8 UID FETCH -2088730488:* (UID)
03:43:58 S: D8 BAD Error in IMAP command UID FETCH: Invalid uidset (0.000 + 0.000 secs).
03:43:58 Error: Server response: “D8 BAD Error in IMAP command UID FETCH: Invalid uidset (0.000 + 0.000 secs).”.
03:43:58 Error code: 1000
03:43:58 Failed action (1003). Reset observed read/write timeouts: 8/8

Reported by Ari Goldmith · March 22nd, 2020 @ 04:26 AM

State: fixreleased
Milestone: none
Assigned to: benny benny

Activity

  1. Ari Goldmith
    Ari Goldmith

    Is the UID in MailMate field too small to hold the large UIDs generated by the server? ( a signed int, instead of unsigned int or something?)

    March 22nd, 2020 @ 12:59 PM

  2. Ari Goldmith
    Ari Goldmith

    It's Nov 9th 2022, and I'm still trying to track down this issue...

    in which when MailMate tries to sync with the IMAP server it does a UID FETCH of a negative number.

    I’m guessing that number is being generated from the last UIDNEXT and stored in a variable that is a Int32 but should be an UInt32 as evidenced below.

    Some lines from my log:

    22:51:19 S: * 0 RECENT
    22:51:19 S: * OK [UNSEEN 28] First unseen.
    22:51:19 S: * OK [UIDVALIDITY 1584965498] UIDs valid
    22:51:19 S: * OK [UIDNEXT 2206296242] Predicted next UID
    22:51:19 S: * OK [HIGHESTMODSEQ 7] Highest
    22:51:19 S: D3 OK [READ-WRITE] Select completed (0.007 + 0.000 + 0.006 secs).
    22:51:19 Condstore-based check (7 > 7) means fast synchronization is enabled
    22:51:19 Attempting fast synchronization.
    22:51:19 C: D4 UID FETCH -2088671054:* (UID)
    22:51:19 S: D4 BAD Error in IMAP command UID FETCH: Invalid uidset (0.001 + 0.000 secs).
    22:51:19 Error: Server response: “D4 BAD Error in IMAP command UID FETCH: Invalid uidset (0.001 + 0.000 secs).”.

    Here the server states the next UID would be 2206296242.

    Mailmate tries to do a UID FETCH -2088671054:* (UID)

    2206296242 | 0 happens to equal -2088671054 which isn’t a coincidence, so there’s your overflow.

    I’d assume it is when Mailmate stores the UIDNEXT from the previous session (it gets mangled) and then uses the mangled value in the new session to fetch new messages since the last session.

    March 22nd, 2020 @ 03:42 PM

  3. Zivago Lee
    Zivago Lee

    I also get this every so often. The server is a dovecot server and the only way to "fix" this is to delete all the dovecot files in my Maildir (dovecot-uidlist, dovecot.index.cache, dovecot.index.*, dovecot-uidvalidity*). Let me know I can get logs from Mailmate and I can provide them.

    September 18th, 2020 @ 03:10 PM

  4. Ari Goldmith
    Ari Goldmith

    Could it be that the variable that stores UIDNEXT is a signed 32bit int instead of an unsigned 32bit int, and therefore when it processes the UIDNEXT and stores it in the variable it gets turned into a negative number. Then when it queries the server, it does a UID FETCH -UIDNEXT:* which then errors out by the server?

    November 9th, 2022 @ 11:05 PM

  5. benny
    benny
    • State changed from new to reproduced

    Sorry about the far too late reply on this. (I've explicitly marked you as a non-spammer now, but that wasn't related to my lack of reply.)

    The original IMAP “standard” is very explicit about UID values being 32 bit and (I just checked) this hasn't actually changed RFC9501. I think I originally thought this ticket was about a server allowing 64 bit values, but as is clearly stated it's actually incorrect handling of values larger than 31 bit. That should certainly be fixable and I should have done that a long time ago. I'm very sorry about that and I'll take a look at it today.

    Looking forward, a switch to 64 bit might some day be necessary, but that's going to be harder to change in MailMate (but possible of course).

    November 10th, 2022 @ 11:14 AM

  6. Ari Goldmith
    Ari Goldmith

    Thank you, Benny!

    I had checked the IMAP RFC to see if the server was improperly generating UIDs https://datatracker.ietf.org/doc/html/rfc3501#section-2.3.1.1 It turned out that technically the IDs were out of range for a 32 signed int, but in range for an unsigned 32 int which is the definition of a "number" at the end of the rfc (0 <= n < 4,294,967,296)

    Also, I've been harassing the ESP too because though they claim they reindex the INBOX, but the UIDs don't get reset, which also would have temporarily fixed the issue, or kicked the can until the next time it popped up.

    I'm glad this item is fixable, and am happy I was able to contribute something to help make MailMate better.

    Cheers!

    November 10th, 2022 @ 04:41 PM

  7. benny
    benny
    • State changed from reproduced to fixcommitted

    Hold down ⌥ when clicking “Check Now” in the Software Update preferences pane to get the latest test release.

    This was a “blind” fix, that is, I did not really try to reproduce the issue, but I do believe I found the issue.

    MailMate was definitely to blame here, but it's also pretty wild to have more than 2 billion messages pass through an IMAP mailbox. If my math is correct then it's about 5000 messages every hour for the past 50 years. If that can happen then 4 billion is likely not much better.

    The UIDs of a mailbox can only be reset if the server also resets the UIDVALIDITY value. Unfortunately, I'm afraid some email clients won't handle that correctly though (but that would be a much worse bug compared to the above). MailMate does handle this, but it'll result in all emails of the mailbox being refetched from the server.

    November 11th, 2022 @ 08:16 AM

  8. Ari Goldmith
    Ari Goldmith

    Indeed, there wasn't that many messages that passed through that mailbox. Though my guess is that they may have generated the starting UID from a date string YYMMDDHHMM or something similar, and then kept counting from there, instead of bumping the UIDVALIDITY up.

    Thank you so much for the quick fix!

    November 11th, 2022 @ 12:12 PM

  9. benny
    benny
    • State changed from fixcommitted to fixreleased

    December 25th, 2024 @ 04:11 PM

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.