Comments and changes to this ticket

• 

  benny January 24th, 2012 @ 01:48 PM

  • State changed from “new” to “accepted”
  • Assigned user set to “benny”

  Yes, I agree that it should be possible to use IMAP IDLE with more than the INBOX, but it is not straightforward how this should work. I don't quite agree with your notes, but we can probably work something out that'll work for most users.

  The easiest GUI approach would be a menu item for each IMAP mailbox. A menu allowing one to set something like “IDLE, 5 minutes, 10 minutes, 30 minutes, ...”.

  But I don't really like that. Keep in mind that most servers have restrictions on the number of simultaneous connections. MailMate has a hidden setting for that and the default is currently 3 (it used to be larger, but users reported problems with “too many connections”). I would like to make this setting visible and then use the following strategy:

  • The number of allowed connections is also the number of IDLE connections used when all mailboxes are synchronized.
  • The IDLE connections are used for the mailboxes which most frequently receive new messages.
  • All other mailboxes are checked using an interval based on how frequently new messages are received. Maybe with some maximum time between checks.
  • As now, IMAP mailboxes are checked for new messages when they are selected (also implicitly via smart mailboxes).

  The only weakness of the above I can think of is an IMAP mailbox which rarely receives new messages, but when it does, the user would like to know quickly. I think such an example is a bit contrived, but it could perhaps be solved with a hidden setting for forcing a mailbox to use an IDLE connection.

  The tricky part is a good “function” to compute the frequency of checks for new messages. I'll think about that.

  Comments are welcome.

• You flagged this item as spam.
  

  lollipop January 26th, 2012 @ 06:00 PM

  True, the number of connections can be a problem for the server and there should be a setting for the max number with a reasonable default. However, I would not let an algo decide how to make use of these connections, especially not one based on the mail frequency.

  IMHO, the frequency of new mails has not always something to do with the frequency of new mail checks/alerts. A not so contrived example is a special mailbox used for trouble monitoring which should be monitored in near-time, but hopefully sees rarely new mails. Another one is a mailbox used for a high volume mailing list which may only be checked now and then. Thus, I'd prefer to be able to assign the idling connections to those mailboxes I want to have my focus on, but that's a very subjective selection.

  I propose the following:

  • Globally configure the max number of connections N.

  • Globally configure a polling interval.

  • Allow the user to flag up to N (if you can re-use the idling connections for the polling of the other mailboxes) mailboxes to be checked using IDLE. Per default, INBOX is flagged. If the user flags the (N+1)th mailbox, display an error msgs pointing to the max number setting and the N already flagged mailboxes, so the user can decide to increase the number of connections or to remove a flag from another mailbox.

  • Mailboxes configured to display any kind of counter but not to use IDLE are polled according to the globally configured interval.

  • All mailboxes (polled or not polled, maybe not the ones using IDLE) are checked for new mails when they are selected.

• You flagged this item as spam.
  

  leeb October 15th, 2012 @ 02:50 PM

  I'd like to give this a bump and second it as an issue. Right now as far as I can tell MailMate effectively never synchronizes on its own. With Mail, reading a message on one client will very rapidly result in it being marked as read on all other clients. MailMate however will show "Unread Message" indefinitely, at least until a manual sync is triggered (or the message itself is clicked of course). At the very least being able to specify some scheduled "Synchronize All Sources" would be appreciated if IDLE can't be done.

• 

  benny October 16th, 2012 @ 06:14 AM

  You write “rapidly” which is not possible by regularly querying the
  server. Therefore I think either MailMate does not pick up some
  information sent while running the IDLE command or your other email
  clients use some kind of different push
  method not supported by
  MailMate.

  Could you try looking at the logs in the Activity Viewer to see if
  anything happens after running IDLE (when marking a message as unread)?
  Also note the email clients and IMAP servers used.

• You flagged this item as spam.
  

  leeb October 16th, 2012 @ 06:09 PM

  You write “rapidly” which is not possible by regularly querying the server.

  While IDLE does seem to get used for near instantaneous results, there also seems to be a manual fallback. Looking at my own server as well as Google's API use limits/quotas I see no indication that a sync once per minute (or 5 minutes, or 15 for that matter) would cause any trouble.

  At any rate I can reproduce relatively easily, there seems to be some issue where MailMate gets "stuck" in terms of not recognizing a message as read even though it can normally handle it via IDLE immediately. I sent my self a series of messages. Both Mail and MailMate received them at the same time, Activity Viewer showed S39 IDLE/D13 UID FETCH etc. In Mail I clicked on the third, and MailMate immediately recognized it as read. I sent a new one, clicked on the second (in Mail.app, both changes recognized by MailMate). But when I finally clicked the first in Mail, this time MailMate didn't change it to read. In fact even after receiving new messages, MailMate still leaves that one as unread. It appears it'll stay that way indefinitely, until either the message is read in MailMate itself or a manual "Synchronize All Sources" is performed.

  So in general this seems to come up frequently when I have a series of unread messages, and check the most recent one first, then get to the older ones later. This happens both on Gmail and on other servers. I'll see if I can't dig into it a bit more.

• 

  benny October 16th, 2012 @ 07:07 PM

  Thanks for the thorough description. Note that the bottom part of the Activity Viewer allows you to see the log for the selected connection. It may reveal more information.

  When I have time for it I'll also try to reproduce the problem myself. Based on your description, it does sound like MailMate somehow gets stuck with regard to noticing IMAP keyword changes. If I remember correctly I have once examined how Googles server behaves and I thought it simply did not report when only keywords changed, but that might be wrong.

  Let me know if you find any interesting patterns or if you can see in the logs that MailMate ignores something sent during IDLE by Googles (or some other) server.

• You flagged this item as spam.
  

  lollipop January 4th, 2013 @ 11:33 AM

  Just want to set the focus back on the original subject: multiple IMAP IDLE connections to the same server. Could be done as follows:

  • enable by hidden preference (@benny: never understood, why all the cool stuff needs to be hidden ;-)
  • add a menu item to the context sensitive menu of an IMAP mailbox which can put the mailbox in a connected state (using IDLE)
  • draw this mailbox with bold letters like the INBOXes.
• 

  benny January 9th, 2013 @ 10:48 AM

  Sorry about the late reply, but it was because I was already working on this issue :-) And very much like you described (@lollipop).

  Fetch a test release by holding down ⌥ when clicking “Check Now” in the Software Update preferences pane. As always, remember that test releases are not always stable versions of MailMate. The feature is briefly described in the release notes.

  And it's not a hidden feature ;-)

• You flagged this item as spam.
  

  John Man January 11th, 2013 @ 01:22 AM

  I just bought MailMate and discovered that it has the same problem as mentioned above by leeb:

  1 My ISP delivers e-mail to an IMAP account.

  2 My Mac mini running Mail + SpamSieve weeds out the spam every minute.

  3 My MBP running MailMate (connected using IMAP) does not notice when spam has been deleted from the inbox, i.e. the inbox is never updated to reflect that spam has been deleted.
  BUT
  On the other hand, if I have my MBP running Mail (connected using IMAP), it does notice very quickly when the inbox has spam deleted.

• 

  benny January 11th, 2013 @ 07:54 AM

  @John: I think the problem described by leeb is “only” about the read/unread state not being synchronized. Your problem seems to be worse since deleted messages are not detected. Could you verify that your MBP is running IDLE on the INBOX connection (it'll be in bold font in the mailbox outline) and see if anything happens at all for this connection in the Activity Viewer (⌘0)? I need to know whether it is a MailMate bug or a server bug. If it's a server bug then you could try to see if a '5 minute schecule' works (in the new test version). But in that case what you really need is a combination of idle and fixed schedule...

  @John: Which kind of IMAP server are you using? (Alternatively, what is the server address?)

  @John: If you regret your purchase then write me by email.

  @leeb: There is a workaround for Gmail keyword synchronization:

  defaults write com.freron.MailMate MmEnableKeywordsCheckInIDLEForGmail -bool YES
  

  And I can see in my code that I also added an extra variable to enable this in general in case other servers had the same problem.

  defaults write com.freron.MailMate MmEnableKeywordsCheckInIDLE -bool YES
  

  But these variables are certainly going to be replaced by some per-account/mailbox setting. I'm interested in hearing from any users who need these variables to make synchronization work with their server (to get an impression of the extent of this problem).

• You flagged this item as spam.
  

  John Man January 11th, 2013 @ 09:46 AM

  Of course I don't regret my purchase! :) I want to use MailMate (it's most similar to Mailsmith which I've been using for many years) once this little issue is sorted out.

  I'm using the IMAP server - pop3.gradwell.net (79.135.125.10). From MailMate Activity Viewer:

  "Clearing connection to pop3.gradwell.net Trying to connect to pop3.gradwell.net on port 993
  Successful connection.
  Initiating SSL/TLS...
  Setup of SSL/TLS connection complete.
  S: * OK IMAP4 Ready v-imap-proxy-2.gradwell.net 00020a79
  C: A0 CAPABILITY
  S: * CAPABILITY IMAP4 IMAP4REV1
  S: A0 OK CAPABILITY
  C: A1 LOGIN ••••••••••
  S: A1 OK You are so in
  C: A2 CAPABILITY
  S: * CAPABILITY IMAP4rev1 SASL-IR SORT THREAD=REFERENCES MULTIAPPEND UNSELECT LITERAL+ IDLE CHILDREN NAMESPACE LOGIN-REFERRALS
  S: A2 OK Capability completed."

  Here are more details about the problem:

  1 I send a test e-mail (containing "jojo") to myself.

  2 On my MBP, Mail picks it up and shows it appearing in my inbox.

  3 On my MBP, MailMate also picks it up and shows it appearing in my inbox. Activity Viewer shows the following afterwards:

  "S22 IDLE [BOLD LINE] D31 UID FETCH 110:* (UID)
  D4 UID FETCH 5:* (UID)"

  4 Clicking on "S22 IDLE", there is the following in the box below:

  "[...] C: G20 UID FETCH 75102 (UID BODY.PEEK[])
  S: * 152 FETCH (UID 75102 BODY[] {2231}
  S: Return-Path: <====@====.org>
  S: X-Gradwell-POP3: delivered to <====@====.org> at Fr ...
  <2191 bytes> S: ... rding-Rule: 77947
  S: X-Gradwell-Edge-Server: inbound-edge-3.mail.thdo.gradwell.net
  S:
  S: jojo
  S: )
  S: G20 OK Fetch completed.
  C: S21 NOOP
  S: S21 OK NOOP completed.
  Running IDLE (keeping a live connection)
  C: S22 IDLE
  S: + idling
  S: * OK Still here"

  5 On my Mac mini, I start the program which moves the "jojo" e-mail to my spam folder.

  6 On my MBP, Mail properly shows the "jojo" e-mail disappearing from my inbox.

  7 On my MBP, MailMate still shows the "jojo" e-mail in my inbox. Activity Viewer does show activity. The line "S22 IDLE [BOLD LINE]" has become "S24 IDLE [BOLD LINE]". (I didn't take note of what happened to the two D lines - is this relevant ?)

  8 Clicking on "S24 IDLE", there is the following additional text (continuing from what is set out above) in the box below:

  "[...] S: * OK Still here
  S: * 152 FETCH (FLAGS (\Deleted \Recent))
  C: DONE
  S: S22 OK Idle completed.
  C: S23 NOOP
  S: S23 OK NOOP completed.
  Running IDLE (keeping a live connection)
  C: S24 IDLE
  S: + idling"

  9 Now (many minutes later), the "jojo" message is still in MailMate's inbox - unread - BUT MailMate has picked up the fact that the "jojo" message is in the spam folder - also unread. In other words, MailMate is showing 1 message unread in inbox ("jojo") and also 1 message unread in spam (the same "jojo").

  10 I quit and restart MailMate. Activity Viewer shows activity but the situation is the same - 1 message unread in inbox ("jojo") and also 1 message unread in spam (the same "jojo").

  11 I quit MailMate. I make some changes using Mail, e.g. deleting "jojo" from spam and sent. I restart MailMate - now it properly shows "jojo" gone from both inbox and spam.

• 

  benny January 11th, 2013 @ 08:17 PM

  @John: Item number 8 is the interesting one and it made me realize what the problem is. Apple Mail moves messages by first making a COPY to the new mailbox and then marking the message as deleted in the source mailbox. But the message is not expunged (IMAP lingo). This is shown by MailMate using a red dot. I don't know exactly when Apple Mail decides to expunge the message, but it'll happen later on. Maybe when deselecting the mailbox. Does this makes sense to you? (It is all related to the delete+expunge model of IMAP which neither Mail or MailMate fully respects.) One workaround would be to let MailMate talk with SpamSieve instead. Then both Apple Mail and MailMate would reflect the changes (since MailMate expunges immediately after the successful copying of the message).

• You flagged this item as spam.
  

  John Man January 13th, 2013 @ 09:53 PM

  @benny: yes, your explanation makes perfect sense. I think Apple Mail may only expunge when the program is quit. Others have had similar issues, e.g.

  https://discussions.apple.com/thread/1095340?start=0&tstart=0

  I prefer not to use MailMate+SpamSieve (rather than Mail+SpamSieve) because then I lose the drone functionality of SpamSieve (which only works under Mail and Outlook):

  http://c-command.com/spamsieve/manual-ah/setting-up-a-spam-filte

  However, I've fixed the problem (Mail's unusual expunge behaviour) by creating a Ruby script to (1) wait a few seconds, (2) login to IMAP and (3) expunge; and then setting up Mail+SpamSieve to run this whenever a spam is moved to the spam folder (or whenever the drone funtionality is used). I just tested this and it works fine. Thanks for your help!

  (I've encountered another - possible? - issue. I'll dig around to see if it's a FAQ but if not, will start a separate ticket.)

• You flagged this item as spam.
  

  lollipop January 22nd, 2013 @ 06:58 PM

  Added a new submenu to the Mailbox menu named “Synchronization Schedule”.

  Coool, tnx benny!

  increase the number of simultaneous connections (it can be configured manually though).

  How can I increase the number? Couldn't find the hidden pref for that.

• 

  benny January 22nd, 2013 @ 07:04 PM

  @lollipop: Quit MailMate and edit this file: ~/Library/Application Support/MailMate/Sources.plist. For each account you can add something like:

  maxNumberOfConnections = 10;
  

  The default is 3. I haven't closed this ticket yet, because this setting is not part of the GUI.