Freron MailMate

Comments and changes to this ticket

• You flagged this item as spam.
  

  Ari Goldmith March 22nd, 2020 @ 12:59 PM

  Is the UID in MailMate field too small to hold the large UIDs generated by the server? ( a signed int, instead of unsigned int or something?)

• You flagged this item as spam.
  

  Ari Goldmith March 22nd, 2020 @ 03:42 PM

  It's Nov 9th 2022, and I'm still trying to track down this issue...

  in which when MailMate tries to sync with the IMAP server it does a UID FETCH of a negative number.

  I’m guessing that number is being generated from the last UIDNEXT and stored in a variable that is a Int32 but should be an UInt32 as evidenced below.

  Some lines from my log:

  22:51:19 S: * 0 RECENT
  22:51:19 S: * OK [UNSEEN 28] First unseen.
  22:51:19 S: * OK [UIDVALIDITY 1584965498] UIDs valid
  22:51:19 S: * OK [UIDNEXT 2206296242] Predicted next UID
  22:51:19 S: * OK [HIGHESTMODSEQ 7] Highest
  22:51:19 S: D3 OK [READ-WRITE] Select completed (0.007 + 0.000 + 0.006 secs).
  22:51:19 Condstore-based check (7 > 7) means fast synchronization is enabled
  22:51:19 Attempting fast synchronization.
  22:51:19 C: D4 UID FETCH -2088671054:* (UID)
  22:51:19 S: D4 BAD Error in IMAP command UID FETCH: Invalid uidset (0.001 + 0.000 secs).
  22:51:19 Error: Server response: “D4 BAD Error in IMAP command UID FETCH: Invalid uidset (0.001 + 0.000 secs).”.

  Here the server states the next UID would be 2206296242.

  Mailmate tries to do a UID FETCH -2088671054:* (UID)

  2206296242 | 0 happens to equal -2088671054 which isn’t a coincidence, so there’s your overflow.

  I’d assume it is when Mailmate stores the UIDNEXT from the previous session (it gets mangled) and then uses the mangled value in the new session to fetch new messages since the last session.

• You flagged this item as spam.
  

  Zivago Lee September 18th, 2020 @ 03:10 PM

  I also get this every so often. The server is a dovecot server and the only way to "fix" this is to delete all the dovecot files in my Maildir (dovecot-uidlist, dovecot.index.cache, dovecot.index.*, dovecot-uidvalidity*). Let me know I can get logs from Mailmate and I can provide them.

• You flagged this item as spam.
  

  Ari Goldmith November 9th, 2022 @ 11:05 PM

  Could it be that the variable that stores UIDNEXT is a signed 32bit int instead of an unsigned 32bit int, and therefore when it processes the UIDNEXT and stores it in the variable it gets turned into a negative number. Then when it queries the server, it does a UID FETCH -UIDNEXT:* which then errors out by the server?

• 

  benny November 10th, 2022 @ 11:14 AM

  • State changed from “new” to “reproduced”

  Sorry about the far too late reply on this. (I've explicitly marked you as a non-spammer now, but that wasn't related to my lack of reply.)

  The original IMAP “standard” is very explicit about UID values being 32 bit and (I just checked) this hasn't actually changed RFC9501. I think I originally thought this ticket was about a server allowing 64 bit values, but as is clearly stated it's actually incorrect handling of values larger than 31 bit. That should certainly be fixable and I should have done that a long time ago. I'm very sorry about that and I'll take a look at it today.

  Looking forward, a switch to 64 bit might some day be necessary, but that's going to be harder to change in MailMate (but possible of course).

• You flagged this item as spam.
  

  Ari Goldmith November 10th, 2022 @ 04:41 PM

  Thank you, Benny!

  I had checked the IMAP RFC to see if the server was improperly generating UIDs https://datatracker.ietf.org/doc/html/rfc3501#section-2.3.1.1 It turned out that technically the IDs were out of range for a 32 signed int, but in range for an unsigned 32 int which is the definition of a "number" at the end of the rfc (0 <= n < 4,294,967,296)

  Also, I've been harassing the ESP too because though they claim they reindex the INBOX, but the UIDs don't get reset, which also would have temporarily fixed the issue, or kicked the can until the next time it popped up.

  I'm glad this item is fixable, and am happy I was able to contribute something to help make MailMate better.

  Cheers!

• 

  benny November 11th, 2022 @ 08:16 AM

  • State changed from “reproduced” to “fixcommitted”

  Hold down ⌥ when clicking “Check Now” in the Software Update preferences pane to get the latest test release.

  This was a “blind” fix, that is, I did not really try to reproduce the issue, but I do believe I found the issue.

  MailMate was definitely to blame here, but it's also pretty wild to have more than 2 billion messages pass through an IMAP mailbox. If my math is correct then it's about 5000 messages every hour for the past 50 years. If that can happen then 4 billion is likely not much better.

  The UIDs of a mailbox can only be reset if the server also resets the UIDVALIDITY value. Unfortunately, I'm afraid some email clients won't handle that correctly though (but that would be a much worse bug compared to the above). MailMate does handle this, but it'll result in all emails of the mailbox being refetched from the server.

• You flagged this item as spam.
  

  Ari Goldmith November 11th, 2022 @ 12:12 PM

  Indeed, there wasn't that many messages that passed through that mailbox. Though my guess is that they may have generated the starting UID from a date string YYMMDDHHMM or something similar, and then kept counting from there, instead of bumping the UIDVALIDITY up.

  Thank you so much for the quick fix!