MailMate running a fetch with a negative ID - how do I reset?
-
Ari Goldmith
Is the UID in MailMate field too small to hold the large UIDs generated by the server? ( a signed int, instead of unsigned int or something?)
-
Ari Goldmith
It's Nov 9th 2022, and I'm still trying to track down this issue...
in which when MailMate tries to sync with the IMAP server it does a UID FETCH of a negative number.
I’m guessing that number is being generated from the last UIDNEXT and stored in a variable that is a Int32 but should be an UInt32 as evidenced below.
Some lines from my log:
22:51:19 S: * 0 RECENT
22:51:19 S: * OK [UNSEEN 28] First unseen.
22:51:19 S: * OK [UIDVALIDITY 1584965498] UIDs valid
22:51:19 S: * OK [UIDNEXT 2206296242] Predicted next UID
22:51:19 S: * OK [HIGHESTMODSEQ 7] Highest
22:51:19 S: D3 OK [READ-WRITE] Select completed (0.007 + 0.000 + 0.006 secs).
22:51:19 Condstore-based check (7 > 7) means fast synchronization is enabled
22:51:19 Attempting fast synchronization.
22:51:19 C: D4 UID FETCH -2088671054:* (UID)
22:51:19 S: D4 BAD Error in IMAP command UID FETCH: Invalid uidset (0.001 + 0.000 secs).
22:51:19 Error: Server response: “D4 BAD Error in IMAP command UID FETCH: Invalid uidset (0.001 + 0.000 secs).”.Here the server states the next UID would be 2206296242.
Mailmate tries to do a UID FETCH -2088671054:* (UID)
2206296242 | 0 happens to equal -2088671054 which isn’t a coincidence, so there’s your overflow.
I’d assume it is when Mailmate stores the UIDNEXT from the previous session (it gets mangled) and then uses the mangled value in the new session to fetch new messages since the last session.
-
Zivago Lee
I also get this every so often. The server is a dovecot server and the only way to "fix" this is to delete all the dovecot files in my Maildir (dovecot-uidlist, dovecot.index.cache, dovecot.index.*, dovecot-uidvalidity*). Let me know I can get logs from Mailmate and I can provide them.
-
Ari Goldmith
Could it be that the variable that stores UIDNEXT is a signed 32bit int instead of an unsigned 32bit int, and therefore when it processes the UIDNEXT and stores it in the variable it gets turned into a negative number. Then when it queries the server, it does a UID FETCH -UIDNEXT:* which then errors out by the server?
-
benny
- State changed from new to reproduced
Sorry about the far too late reply on this. (I've explicitly marked you as a non-spammer now, but that wasn't related to my lack of reply.)
The original IMAP “standard” is very explicit about UID values being 32 bit and (I just checked) this hasn't actually changed RFC9501. I think I originally thought this ticket was about a server allowing 64 bit values, but as is clearly stated it's actually incorrect handling of values larger than 31 bit. That should certainly be fixable and I should have done that a long time ago. I'm very sorry about that and I'll take a look at it today.
Looking forward, a switch to 64 bit might some day be necessary, but that's going to be harder to change in MailMate (but possible of course).
-
Ari Goldmith
Thank you, Benny!
I had checked the IMAP RFC to see if the server was improperly generating UIDs https://datatracker.ietf.org/doc/html/rfc3501#section-2.3.1.1 It turned out that technically the IDs were out of range for a 32 signed int, but in range for an unsigned 32 int which is the definition of a "number" at the end of the rfc (0 <= n < 4,294,967,296)
Also, I've been harassing the ESP too because though they claim they reindex the INBOX, but the UIDs don't get reset, which also would have temporarily fixed the issue, or kicked the can until the next time it popped up.
I'm glad this item is fixable, and am happy I was able to contribute something to help make MailMate better.
Cheers!
-
benny
- State changed from reproduced to fixcommitted
Hold down ⌥ when clicking “Check Now” in the Software Update preferences pane to get the latest test release.
This was a “blind” fix, that is, I did not really try to reproduce the issue, but I do believe I found the issue.
MailMate was definitely to blame here, but it's also pretty wild to have more than 2 billion messages pass through an IMAP mailbox. If my math is correct then it's about 5000 messages every hour for the past 50 years. If that can happen then 4 billion is likely not much better.
The UIDs of a mailbox can only be reset if the server also resets the
UIDVALIDITYvalue. Unfortunately, I'm afraid some email clients won't handle that correctly though (but that would be a much worse bug compared to the above). MailMate does handle this, but it'll result in all emails of the mailbox being refetched from the server. -
Ari Goldmith
Indeed, there wasn't that many messages that passed through that mailbox. Though my guess is that they may have generated the starting UID from a date string YYMMDDHHMM or something similar, and then kept counting from there, instead of bumping the UIDVALIDITY up.
Thank you so much for the quick fix!
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.