#3087 ✓bluesky
Gavin Eadie

IMAP timeouts while reading huge mailboxes

Reported by Gavin Eadie | June 17th, 2022 @ 01:36 PM

I decided to use MailMate as a mailstore for many years of old emails. I used Mailchemy to combine various scattered mbox files and eliminate duplicate emails. The result is a huge repository of emails (950,000 emails in a 10GB file). Mailchemy provides an IMAP server implementation in the event that the client cannot import files directly.

I readily admit that I'm doing some guessing here - I've been wrong before!

All that being said, the process works very well when MailMate connects to the server (via localhost) but appears to be impatient when interrogating the server! In the initial discovery process some iterations are called for but are not successful. My belief is that MailMate is timing out because it doesn't get the IMAP OK soon enough. Though it is getting a stream of incoming data, that OK (which is behind all that data) is what it is really waiting for. Here's some log:

12:11:10 S: * 16715 FETCH (UID 16715 INTERNALDATE "10-Nov-2010 17:26:42 -0500")
12:11:10 S: * 16716 FETCH (UID 16716 INTERNALDATE "28-Sep-2010 10:48:53 -0400")
12:11:10 S: * 16717 FETCH (UID 16717 INTERNALDATE "27-Sep-2010 10:50:04 -0400")
12:11:10 S: * 16718 FETCH (UID 16718 INTERNALDATE "21-Oct-2010 08:14:16 -0400")
12:14:05 S: * 16719 FETCH (UID 16719 INTERNALDATE 
12:14:05 Error code: 9
12:14:05 New timeout values (157/8): 262/81
12:14:05 Failed action (1003). Reset observed read/write timeouts: 131/8

12:14:05 Handling reply
12:14:05 Error: Failed multiple retries (4). Final error code was 9.
12:14:05 Terminating non-running connection...
12:16:05 Running action
12:16:05 Handling request
12:16:05 Trying to disconnect nicely (30)...
12:16:05 C: R7 LOGOUT
12:16:35 S: 
12:16:35 Clearing connection to localhost
12:16:35 Ready to run action (type: 3, retry count: 0)
12:16:35 Disconnecting
12:16:35 Clearing connection to localhost
12:16:35 Completed action (3). Observed read/write timeouts: 131/8

It looks like MailMate increases the wait time after timeouts but maybe not enough for this perverse use case.

While I was typing this, the above process transferred a few thousand messages (that's happening on another Mac, with no screen, dedicated to this task which may take days) so maybe I'm the entity that should be patient! Anyway, I'm going to leave this ticket here in case some comments on it, or, suggests a better method.

PS: IMAP doesn't need to gather that initial info about the what the server is holding as one massive transaction .. it could ask for smaller chunks (10k?) when faced with a deluge like mine.

Comments and changes to this ticket

  • Gavin Eadie

    Gavin Eadie June 17th, 2022 @ 01:41 PM

    (a) the other product is Emailchemy

    (b) huge transfers do eventually succeed:

    12:45:06 S: * 434134 FETCH (UID 434134 INTERNALDATE "01-Jan-1970 00:00:00 -0500")
    12:45:06 S: * 434135 FETCH (UID 434135 INTERNALDATE "01-Jan-1970 00:00:00 -0500")
    12:45:06 S: D6 OK UID FETCH
    12:45:06 S: * BYE
    12:45:06 S: D7 OK LOGOUT
  • benny

    benny June 24th, 2022 @ 02:29 PM

    • State changed from “new” to “bluesky”

    MailMate wasn't really designed for huge email stores and you might run into other performance issues. It's not that I wouldn't want to improve that, but there are simply too many other things to focus on (of more benefit for the average user).

    (The bottleneck in MailMate will be the number of messages and not the size of messages.)

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

Mac OS X email client.

Shared Ticket Bins

People watching this ticket