Freron MailMate

Comments and changes to this ticket

• 

  benny April 29th, 2011 @ 06:54 AM

  • State changed from “new” to “accepted”

  It is on the list, but it is not a high priority. I have been told that
  there is a Spotlight importer in EagleFiler which will work for MailMate
  as well, but EagleFiler is not a free application, so I know that is not
  really a fair solution.

  Have you sent any of the crash reports? How many messages do you have in
  “All Messages”? You are also welcome to provide details by private
  email.

  I have accepted this ticket as a request for Spotlight integration.

  [state:accepted]

• You flagged this item as spam.
  

  mariow April 29th, 2011 @ 08:01 AM

  EagleFiler seems to be similar to DevonThink (which I use), but I do not want to import all my mails there. Spotlight would really be high prio for me as I use it to navigate OSX all day. It's just the most intuitive way to find whatever I want without caring about the individual app my data is in.

  I did not send crash reports yet because I usually do not wait for the search to finish. It just takes several minutes where I cannot use MailMate and then I just kill the app to be able to work again. My "all messages" currently has approx 149.000 messages.

• 

  benny April 29th, 2011 @ 08:14 AM

  Ok, so it is a performance problem with searching for body text in large
  sets of messages (I would like to make optimizations in this area, but
  it'll have to wait for now).

  You do not need to use EagleFiler. You only need the Spotlight importer
  included with EagleFiler to be installed (and then Spotlight will use
  space to index your messages, but there is no way to avoid that).

  I understand that Spotlight integration is desired whether or not body
  text search is optimized.

• You flagged this item as spam.
  

  Erik M-H April 29th, 2011 @ 02:21 PM

  Posted to "Spotlight integration" ticket #118 on MailMate forum.

  This is really interesting to me. For me, MailMate has provided the very fastest mail searching I've ever seen -- in fact, this is what prompted me to purchase MailMate even though I bought a lifetime license for Postbox last year. Mail, PowerMail, Mailsmith, Postbox, Eudora, etc., etc. have all seemed much slower.

  I've even gone through a phase where I imported my mail into EagleFiler, hoping it would be quicker. It was, somewhat, but the ongoing importing chore was tedious.

  In desperation, I even exported all my messages into individual .eml files for Spotlight to index in the Finder. It wasn't too bad, but again the exporting chore was tedious.

  By contrast, MailMate with 10K or 20K messages seemed nearly instantaneous. I've now imported my all my e-mail from the last 10 years -- 75K messages. It's admittedly not quite instantaneous any longer, but it's still really fast.

  I just performed a new, nearly random search using "unquoted text contains" on the 75K message base. It took about 3 seconds for MailMate to whittle that down to 72 messages.

  I'm not using an amazingly powerful machine either: this is on a MacBook Pro, 2.93 GHz Intel Core 2 Duo. Quite nice, of course, but not dazzlingly fast.

  It just seemed such a huge contrast to mariow's experience of force-quitting after several minutes of searching. Benny, are there different ways we might have MailMate set up to account for the amazing difference in response time?

• 

  benny April 29th, 2011 @ 03:03 PM

  I have 14K messages myself which might explain why MailMate is
  instantaneous for that number of messages. (I'm on a MacMini.)

  The performance hit is currently linear with the number of messages.
  (For body texts there exists many faster methods, but currently the same method used for searching message headers is used for body text.) Note
  though that the first search on body text can be slower, because data is
  fetched from disk.

  Mario may have a larger average message size than you have, for example
  caused by some kind of automated messages (like “stupid” Subversion
  commit messages which include large file diffs). According to my change
  logs I changed the maximum length parsed from a message around February
  16 (one of the last betas before 1.0 I think). If messages were imported
  before this date then it could make a considerable difference (depending
  on the type of messages), but I checked my communications with Mario and
  I'm pretty sure he is a post-1.0 user. If not, he could try re-importing
  his account(s).

  Just to clarify:

  • I know Mario wants Spotlight for more than speed-purposes
  • I do want to make searching faster

  (Note that you can watch a ticket to receive emails when they are changed. I think this only happens automatically when creating a ticket.
  Alternatively, there is also an RSS feed for all ticket changes which I
  use myself because of a bug in lighthouse which means I do not always
  receive emails for new tickets.)

• You flagged this item as spam.
  

  Erik M-H April 29th, 2011 @ 04:19 PM

  Many thanks for the details, Benny!

  Note: I did make sure to search for a term I hadn't searched for before -- but, you're right, of course: it was not the first text search I'd ever done. My own massive message importing was done only a couple of weeks ago, using 1.1.1, I think.

  According to my change logs I changed the maximum length parsed from a message around February 16

  Should I infer that any text after that limit is not indexed? At first reading, that sounded like a horrible limitation -- but as I think about how e-mail messages tend to work, it actually makes a great deal of sense, as long as the limit isn't unreasonably low.

  And, though I'm pleased and happy with the current search speed, that doesn't mean that faster isn't better! If it were even faster, I'd import e-mail from 1988 - 2000, too (probably 50K more messages) -- and I'd be in heaven!

  Thanks for the suggestions for ticket-watching. I'd never "taken" to RSS, but now that I have an iPad, I've discovered that RSS feeds now work very well for me -- and I've already subscribed to the feed for Freron at Lighthouse. :-) Thanks!

• 

  benny April 29th, 2011 @ 09:20 PM

  There are actually limits on the number of paragraphs (1000), the length
  of a paragraph (4KB), and the total length of the text in a body part
  (40KB). The limits are arbitrary and are mainly intended to catch any ridiculously large messages. I believe it'll very rarely be a problem in
  practice.

• You flagged this item as spam.
  

  Thomas Hanke May 19th, 2011 @ 12:59 PM

  Since I haven't used Spotlight as much as I thought I would, I can't say how many features the plugin comprises.

  Just in case anyone's want to try, the inofficial hack for eml in Spotlight works as for any other plugin:

  1. Try out a recent version of EF.
     
  2. It installs a Spotlight plugin for eml files.
     
  3. The rest is system processing.
     
  4. No reason to do any anything else – isn't Spotlight defined as not requiring any 3rd-party processing? Benny's term 'Spotlight importing' was kind of a red herring back then ;-)
     (5. Buy EF or not.)

  I'm pretty sure EF's worth its money, just not (yet) for me.
  That practical hint aside, I fully agree with Benny, that devs shouldn't endorse any gray practice like that.

  From my own, user, perspective I can't say whether any Spotlight plugin requires a license or is part of the licensed app.

  As I guess I mentioned back then, it may be worthwhile to just ask EF's dev for some sub-licensing of the plugin.
  Actually, it's quite amazing that no 'bigger player' in the postbox… eh thunder… eh mailing business got that idea themselves.

  It is on the list, but it is not a high priority. I have been told that there is a Spotlight importer in EagleFiler which will work for MailMate as well, but EagleFiler is not a free application, so I know that is not really a fair solution.

  Btw, evasive-aggressive answers elsewhere to customer requests regarding Spotlight & QuickLook was a major reason I got here :-)

• You flagged this item as spam.
  

  mariow June 1st, 2012 @ 10:28 AM

  This ticket has been stale for a full year. Are there any chances mailmate will get spotlight indexing soon without experimenting with EF?
  MailMate works really fine for my daily tasks - besides searching. The ease with which you can search inside Apple Mail is far away from the time it takes me to make a fulltext search in MailMate.

• 

  benny June 1st, 2012 @ 09:49 PM

  Don't expect Spotlight integration soon, but it is still a feature I would like to add (if anyone reading this is running OS X 10.8, it would be nice to know if .eml files are still not parsed by the default Spotlight implementation).

  With regard to searching. I expect this to be much faster in the next update of MailMate (after rebuilding the database). Note that it may be a bit easier to do searches if you make the default search use “Common Headers and Body” (or something else). You can use “Edit ▸ Find ▸ Use as Default Search” to do that. Also note that you can setup multiple conditions for the default search since only the ones with non-empty text fields are actually used for the searching.

• You flagged this item as spam.
  

  mariow June 6th, 2012 @ 03:12 PM

  I just tried using "Common Headers and Body", typed in a search word and Mailmate has yet to show the third letter of the word I typed. It has been working for several minutes on searching for all emails containing "sa". In short: The current full-text search does not work at all for me, I never search in the body of mails, only in addresses and subject line because this is the only way to get a result in a good time.

• 

  benny June 6th, 2012 @ 08:08 PM

  Yes, searching bodies is currently very slow. You need the latest test release and a database rebuild to get the latest improvements, but I'm not sure how fast it'll be in your case. In any case, searching from/to/subject is also much faster. Write me privately if you want to give the test version a try.

• You flagged this item as spam.
  

  larrycz July 3rd, 2013 @ 02:30 PM

  It doesn't seem Spotlight integration is working by default in OS X 10.8. What is the status of Spotlight integration of MailMate?

• 

  benny July 5th, 2013 @ 10:58 AM

  @larrycz: Current status is that Apple still does not index standard .eml files and I haven't created a Spotlight plugin. I haven't yet checked whether or not OS X Mavericks indexes .eml files (probably not).

• You flagged this item as spam.
  

  mariow January 21st, 2014 @ 10:47 AM

  Unfortunately Mavericks also does not index .eml files by default. Any chance we'll see a spotlight integration in 2014? Frankly, it's the only feature I'm missing in Mailmate :)

• 

  benny January 22nd, 2014 @ 01:50 PM

  Sorry, it's still not high on my list. Best workaround is to find another Spotlight plugin.

• You flagged this item as spam.
  

  mariow January 23rd, 2014 @ 09:00 AM

  Too bad. Unfortunately I have yet to find another Spotlight plugin that works for me and then correctly links the found entries to MailMate.

• 

  benny January 23rd, 2014 @ 09:03 AM

  @mariow: Opening it in MailMate is independent of the plugin I believe. Try “File ▸ Get Info” on a .eml file in the Finder and then assign MailMate to be the default for this file type.

• You flagged this item as spam.
  

  mariow January 23rd, 2014 @ 09:16 AM

  True, that should work.

• You flagged this item as spam.
  

  larrycz May 2nd, 2014 @ 10:05 PM

  +1

  Would very much like to see Spotlight integration. Its lack is the number one reason I would consider switching back to Apple Mail.

• 

  benny June 7th, 2014 @ 02:35 PM

  Has anyone checked if Yosemite (OS X 10.10) indexes the .eml files?

• You flagged this item as spam.
  

  Thomas Floeren March 24th, 2015 @ 07:00 PM

  No, on my machine it only indexes .emlx.

  Heck, I wasn’t aware of the fact that .eml aren’t indexed because my Spotlight searches always showed me the mails versions in Apple Mail’s folder (which are .emlx of course) and I didn’t pay enough attention.

  This is really a huge issue. No so much as long as the messages remain in MailMate, but I won’t keep all my messages forever in my mail boxes. So, as far I can tell, the only useful way to export (Spotlight-searchable) mails to disk is via Apple Mail, right?

  The problem is, I already have exported lots of mails to disk from MailMate (.eml). These are all unsearchable now. I guess I’ll have to reimport them all to Apple Mail and then reexport as .emlx …, sigh. What a mess. They are all in their associated project folders on disk.

  Why doesn’t MailMate use .emlx?

  Please, please put the .eml Spotlight importer high on your todo list.

  Thanks.
  Tom

• 

  benny March 25th, 2015 @ 08:30 AM

  @Thomas: The .emlx format is undocumented and should be considered an implementation detail of Apple Mail. It seems though that it is basically the .eml format, but with message size prefixed on the first line and a property list appended. I don't think you could just rename the files, but I suspect that it would work if renaming AND inserting the file size on the first line.

  I cannot use the .emlx format, partly because it is non-standard/undocumented and partly because Apple Mail could change its definition at any time.

  (The frustrating part is that Quick Look works fine with .eml files.)

• You flagged this item as spam.
  

  Thomas Floeren March 26th, 2015 @ 01:29 PM

  Thanks for the hint. Renaming and prefixing with the size does the job. (In fact any size > 4000 or so will do it. The size prefix seems to affect only the display of the message in AppleMail (if it’s smaller than the actual size the mail will be truncated).

  But, probably better: I found a third-party mdimporter, that seems to work well. It is contained in an app named “EagleFiler” by C-Command Software. Download the trial, mount the dmg and go to /Volumes/EagleFiler-1.6.4/EagleFiler.app/Contents/Library/Spotlight/

  and copy the EagleFilerSpotlightImporter.mdimporter to ~/Library/Spotlight, then reimport with
  mdimport -r ~/Library/Spotlight/EagleFilerSpotlightImporter.mdimporter

  Spotlight now indexes all my exported .eml files (if they are in a Spotlight-visible directory, of course). Great!

  What I didn’t find out is a way to make Spotlight index the MailMate Messages folder in ~/Library/Application\ Support

  Any idea how to do this?

  But this is secondary since MailMate’s own search is pretty good. The crucial thing was to make my exported .eml mails searchable and that’s OK now.

  (Though I don’t know if I have the right to use the SL importer from a third-party app I didn’t purchase …? It surely would be more “clean” if you included an importer with MailMate …)

  best,

  Tom

• 

  benny March 26th, 2015 @ 09:00 PM

  The importer is also mentioned in my first reply in this ticket, but it's not something I advocate due to the same consideration you described.

  I'm not sure why the folder is not indexed. Are you sure it's not just delayed. It can take a while.

• You flagged this item as spam.
  

  Thomas Floeren March 27th, 2015 @ 12:24 AM

  Uhps, yes. It’s even mentioned 13 times in this thread. Embarrassing, I really should read a thread before posting my crap. Sorry for the redundancy.

  Take it as a sign that an eml importer for MM is badly needed ;-)

  (I know, it should be provided by Apple, especially if considering that when you drag individual messages from Apple Mail to the Finder they also get saved as eml)

  Are you sure it's not just delayed. It can take a while.

  The strange thing was that afterwards the MM content appeared in Tembo (a Spotlight GUI from Houdah Software) but not in Spotlight or Finder search.

  Currently all my emls disappeared again from the index, but that’s probably because I messed something up while trying to trick the OS’s Mail mdimporter into accepting eml ;-)

• You flagged this item as spam.
  

  klktrk April 19th, 2015 @ 06:27 PM

  It seems to me the easiest way forward to enable both searching for email content within MailMate and within Spotlight, would be to move the default location of the mail messages from Library/Application Support... (exluded from Spotlight indexing) to a Location in Documents/MailMate. Or, what Apple Mail does, locating it in Library/Mail, which is included in Spotlight.

  If the location of the Mail messages were configurable in a preference, that would allow a user to make the choice themselves.

  From there it's just an install of a spotlight plugin for .eml messages.

• 

  benny April 22nd, 2015 @ 01:17 PM

  @klktrk: Hmm, I would prefer not changing where emails are saved. I'll look into whether or not it is correct that indexing is not done in the support folder (that's not how I recall it worked last time I looked at it). Note that Spotlight does not index .eml files no matter where they are located. I've reported this to Apple as a bug many years ago, but it probably won't be fixed/changed. The good news is that I'm actually working on a Spotlight plugin now. Hopefully I'll get that completed and be able to finally close this ticket.

• You flagged this item as spam.
  

  Thomas Floeren April 22nd, 2015 @ 09:58 PM

  The good news is that I'm actually working on a Spotlight plugin now.

  Hey, this is really great news! You shouldn’t ‘hide’ this information in post No. 28 of a lousy ticket thread ;-)

  (I had already begun to switch back to Apple Mail as main client last week …)

• 

  benny April 23rd, 2015 @ 09:29 AM

  @Thomas: Quite the contrary. I shouldn't announce anything that isn't finished yet :-) But I'm glad if it can bring you back when/if it is completed (the most important currently missing parts of the plugin are parsing of address headers, parsing of text body parts, and some way to enable/disable Spotlight indexing).

• You flagged this item as spam.
  

  klktrk April 28th, 2015 @ 09:38 AM

  I'll look into whether or not it is correct that indexing is not done in the support folder (that's not how I recall it worked last time I looked at it).

  Here's the test I did (Mac OS X 10.6.8):

  1. Created plain text file in ~/Library/Application Support/spotlighttest.txt
     
  2. Added unique string to plain text file (78tytfoot)
     
  3. Searched in spotlight for unique string 78tytfoot.
     
  4. No results.

  As far as I can tell, nothing within the Application Support folder is indexed by Spotlight by default, and I have hacked myself to hell and back trying to get Spotlight to index folders it does not include by default. There are multiple ways to exclude folders, but I don't see any way to force inclusion of other folders.

  I might be wrong about this, or maybe later versions of Mac OS X allow inclusion, but my testing indicates that the Application Support folder is not indexable by Spotlight at all.

• 

  benny April 28th, 2015 @ 12:08 PM

  @klktrk: Thanks for making me look into this in more detail before putting more work into the plugin. Apparently it's all a bit messy and Apple does not seem to recommend anything. Here we go:

  • You have to use the mdimport and mdls commands when analyzing what Spotlight does and does not do.
  • Spotlight does index the app support folder, but it's not part of the default results for Spotlight. You can only see it if you use the command line or if you explicitly tell the Finder to search the folder.
  • I'm going to get a lot of complaints if I claim to support Spotlight, but the results are not shown when using ⌘+space. There seems to be no way to force Spotlight to do this.
  • It appears that Thunderbird and some other apps use ~/Library/Caches/Metadata/.../ to make Spotlight show the results. In the case of Thunderbird, only metadata is saved, but it can apparently take up a lot of space.
  • I cannot use the Metadata folder to save emails, because users often delete this folder when they need more space.
  • It appears I cannot symlink anything to get around this problem. This prevents Spotlight from showing any results.
  • It does appear that the best solution is to place metadata only in this folder (in some kind of private format with a private file extension).
  • Metadata would make the MailMate Spotlight importer very fast.
  • The metadata must be recreated if the cache is deleted -- performance-wise this is going to be very expensive.

  To sum it up, I don't like where this is going. It might be better to simply add the .eml Spotlight importer I'm working on now and then offer a setting which can move the Messages folder to a searchable location like “Documents”, “~/Library/Mail” (if that works), or a dedicated home folder like “~/MailMate” (if that works).

• You flagged this item as spam.
  

  Thomas Floeren April 28th, 2015 @ 10:56 PM

  It might be better to simply add the .eml Spotlight importer I'm working on now and then offer a setting which can move the Messages folder to a searchable location

  This sounds like a very good solution. The missing Spotlight-searchability of mails inside of MM is – for me – the minor problem anyway. (I seldomly use my mailboxes as persistent document storage.) The game breaker is the missing Spotlight coverage of the exported .eml mails (yes, thanks to Apple’s crummyness).

  And that problem would be perfectly solved with an eml mdimporter. The option to move the MM folder to a SL-enabled location being a welcome additional bonus, of course.

• You flagged this item as spam.
  

  Steph May 5th, 2015 @ 08:42 PM

  It might be better to simply add the .eml Spotlight importer I'm working on now and then offer a setting which can move the Messages folder to a searchable location

  For what it's worth, Spotlight search-ability is the one feature I'm waiting on before switching over to MM, and this solution would be good enough for me.

• 

  benny May 6th, 2015 @ 12:14 PM

  Current status: MailMate can now properly parse address related headers (from/to/cc/bcc) and find/decode any text/plain parts. This probably covers 99% of what is needed for the Spotlight plugin to work for most users. I plan to add 2 GUI settings: One for enabling/disabling Spotlight indexing the Messages folder and one for telling MailMate to place the Messages folder in a different location (to allow the use of the Spotlight Search panel).

  But I'll have to do more testing before releasing this and I also have to fix some related issues (how MailMate deals with .eml files in general).

• 

  benny May 8th, 2015 @ 04:57 PM

  A test release is available now. Hold down ⌥ when clicking “Check Now” in the Software Update preferences pane. Use with care since it involves some core changes to MailMate and I've done very little testing -- let me know if I should pull it :-)

  Make sure to read the release notes to see how to make indexed content available via the Spotlight interface.

• You flagged this item as spam.
  

  Jean-Luc Deladriere May 18th, 2015 @ 08:18 AM

  Indexation works but when I open the mail from Spotlight it opens in Mail not in Mailmate
  I've set Mailmate as the default mail app like here: https://support.apple.com/en-gb/HT201607

• 

  benny May 18th, 2015 @ 09:14 AM

  @jean-luc: You need to manually tell the OS to open all .eml files in MailMate. You can do this is in the “Get Info” (Finder) interface. I'm not sure I can (easily) make this work automatically without introducing a custom file extension, but I might be wrong. Thanks for the feedback on this feature!

• You flagged this item as spam.
  

  Jean-Luc Deladriere May 18th, 2015 @ 09:21 AM

  Ok I found the answer to my question : just use Finder to change the default app for all .eml files

• You flagged this item as spam.
  

  Jean-Luc Deladriere May 18th, 2015 @ 09:22 AM

  oops didn't refresh the page : same answer (^_^)

• 

  benny January 20th, 2016 @ 12:37 PM

  • State changed from “accepted” to “fixreleased”