Mailmate crashes when searching "Unquoted Body Text" or

Have you allowed MailMate to send crash reports (General preferences pane)?

Is it really crashing or is it hanging until you have to force quit?

I enabled the crash report option. I will send a report shortly.

It is actually crashing, not hanging.

I just initiated a search that caused a crash. Can you verify that you received the crash report?

I did. The crash happens when loading a database index (probably for the unquoted text). Three things to do:

1. Update MailMate to the latest test version by holding down ⌥ when clicking “Check Now” in the Software Update preferences pane.
   

2. Run this command and provide the output:

   ls -al ~/Library/Application\ Support/MailMate/Database.noindex/Headers/#unquoted*
   

3. Run MailMate from the Terminal to see any error messages before the crash:

   /Applications/MailMate.app/Contents/MacOS/MailMate
   

Here is the directory listing:

$ ls -al ~/Library/Application\ Support/MailMate/Database.noindex/Headers/#unquoted*
-rw-------+ 33 deighton  1031  1124946280 Jun  9 11:02 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted#lc.cache
-rw-------+ 38 deighton  1031    75953556 Jun  9 11:02 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted#lc.offsets
-rw-------+  1 deighton  1031          62 Jun  9 11:02 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted#lc.plist
-rw-------+ 33 deighton  1031  1789645154 Jun  9 11:02 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted.cache
-rw-------+ 38 deighton  1031   127588080 Jun  9 11:02 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted.offsets
-rw-------+  1 deighton  1031          63 Jun  9 11:02 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted.plist

Here is the Terminal output from the crash:

$ /Applications/MailMate.app/Contents/MacOS/MailMate
2014-06-09 11:04:02.425 MailMate[26009:507] Bartender: Loaded BartenderHelperSixtySeven
Problems while parsing reported for one or more messages. Details can be found in
 /tmp/mailmate_logs/mailmate_parser_problems.log
mmap: Cannot allocate memory
ENOMEM: Unsuccessfully tried to map '/Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted#lc.cache' (1124955524)
Abort trap: 6

Ok, the problem is that MailMate runs out of address space. Those database files are huge. They are roughly 20 times bigger than mine. That would correspond to around 1 million messages if I scale up my numbers. MailMate cannot handle that which means your emails might be very large on average? If you have 100K messages then they contain 11KB of unquoted text on average...

How many messages do you have (estimated)?

Have you deleted a large number of messages (or removed accounts)?

I have >500k messages across 3 different accounts. I haven't deleted a large number of messages nor have I removed accounts.

What can MailMate handle? Is it worth rebuilding the indexes to see if they should be smaller? I can do some cleanup, but I'm sure I will run into the problem down the road. Can I restrict the indexes to ignore messages over X number of days?

Any help would be appreciated.

You are close to the limit I think. (I know some other users have ~500K messages, but it wouldn't work on my hardware.)

Rebuilding the indexes is only going to help if it's a bug that made them so big. Is there any reason that you might have a very large average message size (unquoted text, not attachments)?

For the sake of comparison, I would also like to see the following:

ls -al ~/Library/Application\ Support/MailMate/Database.noindex/Headers/#quoted*

I'll think about possible workarounds, but first I would like to figure out if there is a bug in play as well.

Here is the output:

$ ls -al ~/Library/Application\ Support/MailMate/Database.noindex/Headers/#quoted*
-rw-------+ 15 deighton  1031  53468013 Jun  9 18:16 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#quoted#lc.cache
-rw-------+ 17 deighton  1031  35510052 Jun  9 18:16 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#quoted#lc.offsets
-rw-------+  1 deighton  1031        60 Jun  9 18:16 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#quoted#lc.plist
-rw-------+  1 deighton  1031        22 May 10 11:31 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#quoted-markup.cache
-rw-------+  1 deighton  1031     30096 Jun  9 18:16 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#quoted-markup.offsets
-rw-------+  1 deighton  1031        53 Jun  9 18:16 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#quoted-markup.plist
-rw-------+ 15 deighton  1031  53563307 Jun  9 18:16 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#quoted.cache
-rw-------+ 17 deighton  1031  35510064 Jun  9 18:16 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#quoted.offsets
-rw-------+  1 deighton  1031        60 Jun  9 18:16 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#quoted.plist

Offhand, I don't know why my average, unquoted size would be so high.

I cleaned up a bunch of messages. Now, when I start MailMate, I receive a "Segmentation fault:11" error. I can't do anything before it crashes. Any thoughts on this or should I just start from scratch with MailMate?

First try a manual rebuild if possible: http://manual.mailmate-app.com/rebuild

I'm still puzzled by the file sizes. If a rebuild is possible then it would be nice to see the result of the ls commands afterwards.

For how long have you used MailMate?

Do you ever delete messages permanently (empty the trash)? If yes, How many messages would you say are deleted per day (rough estimate)?

I initiated a manual rebuild by moving the #unquoted* files out of the Headers directory. The good news is that after the rebuild the #unquoted* indexes are much smaller:

$ ls -al ~/Library/Application\ Support/MailMate/Database.noindex/Headers/#unquoted*
-rw-------+ 9 deighton  STSCI\science  599400257 Jun 10 15:19 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted#lc.cache
-rw-------+ 5 deighton  STSCI\science   40428708 Jun 10 15:19 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted#lc.offsets
-rw-------+ 1 deighton  STSCI\science         61 Jun 10 15:19 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted#lc.plist
-rw-------+ 9 deighton  STSCI\science  599430688 Jun 10 15:19 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted.cache
-rw-------+ 5 deighton  STSCI\science   40428708 Jun 10 15:19 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted.offsets
-rw-------+ 1 deighton  STSCI\science         61 Jun 10 15:19 /Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted.plist

The bad news is that MailMate is still crashing:

mmap: Cannot allocate memory
ENOMEM: Unsuccessfully tried to map '/Users/deighton/Library/Application Support/MailMate/Database.noindex/Headers/#unquoted#lc.cache' (599353379)

BTW, I have been using MailMate since May 2013. (I love it BTW).

I permanently delete messages about once a month. When I do, it is probably 100s or even 1000s of messages at a time.

It's positive that MailMate was able to complete the rebuild, but that doesn't help when the files are still too large. How long did the rebuild take?

I have an experimental 64 bit version of MailMate which would probably not run into this problem. Unfortunately, it is even more memory hungry which means I don't think that is a viable solution before that is improved. But you probably cannot wait for that.

MailMate has some (large) upper limits on how much plain text is indexed from any given message. I could make these configurable in order for you to limit the size of these files (requiring another rebuild though). Would that be a viable solution for you?

Actually, you could also quit MailMate and then delete the files without the #lc suffix. This would prevent you from doing case sensitive search, but that might be acceptable.

I would estimate the rebuild took ~2 hours. I wasn't paying attention though.

Limiting the amount of indexed text would be useful. That would certainly be better than the current situation. I would try that. In the meantime, I can remove the non-#lc files. That would be another improvement for me.

Thanks!

There is a whole set of hidden preferences now to control this (untested though). The most interesting are probably these:

defaults write com.freron.MailMate MmMaximumTotalSizeIndexed -integer 40000
defaults write com.freron.MailMate MmMaximumQuoteLevelIndexed -integer 256

Those are also the default values. The first one means that at most 40000 bytes are indexed from a text body part (unquoted+quoted). The second one controls the number of quote levels indexed which is practically all quoted levels by default.

If your indexes are large because you receive a lot of long quote-all emails with large quote levels then it might be sufficient to change the second setting, e.g.,

defaults write com.freron.MailMate MmMaximumQuoteLevelIndexed -integer 3

There is unfortunately no way to only re-index these parts of the database (it would be nice though for this kind of debugging).

You need to fetch this update as a direct download: http://updates.mailmate-app.com/archives/MailMate_r4321.tbz

Oh, I forgot that we already saw that the unquoted files are the problem. In other words you should change the other setting, e.g.,

defaults write com.freron.MailMate MmMaximumTotalSizeIndexed -integer 10000

Alternatively, these are also available:

defaults write com.freron.MailMate MmMaximumTextLinesIndexed -integer 1000
defaults write com.freron.MailMate MmMaximumTextParagraphSizeIndexed -integer 4096

First one tells MailMate the maximum of lines (paragraphs) to be indexed from a body part. The second one puts a limit on the size of such a line (paragraph).

I installed r4321, set MmMaximumTotalSizeIndexed to 10000, then forced a database rebuild. Unfortunately, it crashed at some point in the night. I restarted and it did finish the rebuild. I was able to search on unquoted text while my accounts were offline. Then, I brought the accounts back online and MailMate crashed soon after. While rebuilding the database again, I received the following error

MailMate(25147,0xb0115000) malloc: *** mach_vm_map(size=67108864) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc

I will try some of the other settings.

Sounds like r4321 might be more unstable or memory-hungry than previous revisions. Have you enabled sending crash reports in the General preferences pane of MailMate? I'm mostly interested in the crash when taking accounts online. Did this happen repeatedly? (Is this why you chose rebuilding again?)

I've fixed a somewhat unrelated bug which should reduce the memory usage when importing. That might help with respect to the crash during rebuilding (crashes during rebuilding are not a problem with respect to database corruption - MailMate should be able to recover and continue rebuilding although the message count does not reflect that it doesn't start over).

Thanks for taking the time to do this.

I've had several more crashes. I've enabled the crash reports, so you should have some additional information. I also changed MmMaximumTotalSizeIndexed to 5000, but it didn't seam to help. Should I go back to the stable version of MailMate to try and get up and running or would it help to try other settings or another dev version?

Sorry about the late response. Could you update to the latest test release (hold down ⌥ when clicking Check Now in the Software Update preferences pane). You can let it crash once or twice such that I'll get some “fresh” crash reports. Some intermediate revisions of MailMate have been very buggy, but I don't think that was the case for r4321 which means I don't expect the latest test version to behave much better for you.

I'm now running r4351 and I haven't had any problems. I'll let you know if I run into any issues.