Generic discussion about Email archiving best practices

ozy

Hello,

I am a quite happy Grommunio user since a couple of years, and am currently wondering about archive options.
I know that Grommunio ships with mailpiler which is a very nice email journaling solution, but I wonder how people today handle user archives that keep end user's folder structures.

There are a couple of options I am aware of:

Integrated outlook archiving: local, without keeping folder structure
Mailstore.com: Seems a good commerciçal solution to me, works with EWS, keeps folder structure, has an Outlook plugin
Mail-archiver: Seems a good FOSS alternative, works with M365 Graph, keeps folder structure

So far, I would think that using IMAP as backend will have Grommunio create midb files which could become heavy in terms of resources, so solutions using MAPI/EWS/Graph would be a better fit.

I went away from M365 for obvious reasons, and they had a "fully integrated archive system" with storage tiering.

So, open question for the community, how do you people handle archives ?
I for myself just "copy paste" stuff into local PST folders now and then, and I do the same for my wife which has like 600GB of mails in multiple PST files, I create new PST files every now and then for her.

I cannot imagine that a bigger structure would have the IT guy run to users to create manual archives, nor can I imagine that users would do that job themselves without a "quota pressure".

As of today I administer a couple of small Grommunio servers, as well as some more bigger structures still with O365. I would like to know how people got the archiving sorted out on Grommunio.

Any tip would be appreciated.

Best regards,
Ozy.

crpb

Mailstore here. Just today i tried the EWS-Archiving again and it finally works. Only problem is that it will create duplicates once for all mails it finds in the mailbox.
Sadly it only works with storeowner and not with just readany (if i was just using a dedicated account to create a structured archive for a mailbox) aswell as readany+deleteany what should be enough 🙈 - But i think that is a 'mailstore' problem that they only work with that and we don't have any delegation.

For the list:

https://www.benno-mailarchiv.de (has a community version)
https://www.reddoxx.com (only ent.)

ozy

@crpb Thank you for your input.

I read about beeno, but they do email journaling without folder structure, just as mailpiler already does.
I also read about mailarchiva which looks quite like mailstore, about same price range.

I thought about dovecot + imapsync setup, but such a "personal" project would quickly become a beast to maintain.
In the end my shortlist was between https://mail-archiver.org and https://mailstore.com

Would you mind sharing your experience with mailstore ?
Does it work without IMAP for Grommunio ? Is the outlook plugin "polished" and use SSO ?
What's your general view/recommandation on this software ?

crpb

Mailstore

Exchange Archiving works now with Grommunio (as said, with storeowner or the user)
SSO Works in the Client + Outlook-Plugin when you configure it accordingly.
- GPO Templates for Mailstore
Outlook-Plugin Polished?
- Yeah, i guess so. For users with a gigantic folderhierarchy it works pretty good as you can quickly go through the folders and look for things where they are and not guess like a maniac what you should use a search term to find that "one" mail whic was definitely placed in the folder A/B/E/G/A/B but you can't remember who wrote or what the actual content was.
Attachment Indexing
- This is working pretty good if you setup all those additional filetypes.

A bad thing about it is that it only runs on windows :p

ozy

Huh, I don't get the "Exchange Archiving works now with Grommunio (as said, with storeowner or the user)" ?
Could you explain what you mean by this ?
I don't see anywhere where I would setup archive rules in Grommunio, so I guess that's the Mailstore part ?
Also ,does Mailstore fetch mails via EWS/MAPI or does it need to grow a midb IMAP file ?

Sorry if I am a pain the a** for all those questions, but I'm genuine interested in knowing about that solution before trying a PoC.

crpb

On the Mailstoreserver you create archive jobs.
And YES, no more IMAP! EWS! Thats what i meant with "Exchange Archiving".
Otherwise i would have called it IMAP Archiving 😛

I yet haven't tried multimailbox archiving but this should work aswell as it just does the same under the hood for one or multiple accounts.
https://help.mailstore.com/en/server/Archiving_Emails_from_Microsoft_Exchange_SE#Archiving_Individual_Mailboxes

EDIT: Also note, there is a python library for it available to play with
https://help.mailstore.com/en/server/Python_API_Wrapper_Tutorial
/me is using that to check for failed archive jobs and if there are updates available

ozy

Thank you, inded mailstore server looks like the perfect candidate for what I want to achieve.
I am still advocating for a reasonable open source solution (see https://github.com/s1t5/mail-archiver/discussions/558 ) but AFAIK there's no good open source solution as of today.

I'll go with a Mailstore poc and report back.
Thank you 😉

externa1

ozy
We also use Mailstore on all customers (Exchange, O365 and Grommunio), big and small ones as the pricing is verry good.
but its one of the best solutions on archiving - also on export with lot of options.
at mailstore we use it with imap - it also works with delegation on imap in grommunio for us

rg
Christian

ozy

@externa1 Thanks for the feedback.
Why go with IMAP when it implies MIDB sync ? Wasn't EWS a viable option ?
Also, With imap delegation, does the end user still get to see it's "owned" archive ?

Thanks.

crpb

ozy Wasn't EWS a viable option

NOTE: This only works since like half a year or so.

THE_FRICKLER

For my professional Items i also use Mailstore. But it sucks while on Windows. The legal aspects of eMail-Backups are the reason.
For privat use we are using "MailArchiver". It is OpenSource und has no limitations. Here it runs on 5 private / semi-professional installations with up to 20 user.
In tests between both there is no difference in stability or in complettnes. But it doesnt has a native Outlook plugin.

ozy

@THE_FRICKLER Thanks, MailArchiver looks good to me, but using IMAP to get those archives is still resource heavy. Anyway, I have small servers, so in my cases resources won't be the biggest problem, I just didn't want to "duplicate" all mail data into a midb store just for the sake of archiving, since adding a new layer opens possibilities for new errors.
Quick question, how does mailArchiver work in terms of reversibility ? Did you ever check if export EML files did correspond 100% to the import (eg no transliteration issues/missing headers/changed dates) ?

@crpb The mailstore guys don't want to talk about non Microsoft EWS at least:

alle Möglichkeiten bezüglich EWS und Microsoft Exchange, welche offiziell unterstützt werden, sind im folgenden Artikel beschrieben: [...] Ob so etwas auch mit Grommunio möglich ist, haben wir nicht getestet und können wir keine Aussage zu machen. Bitte wenden Sie sich an den Support von Grommunio, um weitere Informationen über Grommunio zu erhalten.

I guess the support is protecting themselves from unusual stuff, which is fair.
But I really think the Grommunio people should ask them to officially test their solution, as it would be a great way to expand their ecosystem, on both sides.

I'll still make a couple of mailstore tests. So far, I could get EWS to work. Having non-LDAP servers (small amount of users), I still wonder if I can setup a Grommunio EWS delegation account in order to "not know" end user passwords. Then again, how will they manage to login to their archive ? I guess mailstore is really Active Directory oriented.

crpb

FWIW, As i said before that by switching from an imap-based archival to EWS with Mailstore you will likely end up archiving "many" if not "all" mails a second time as it can't detect the already archived mails via imap .. (well, there was enough going on about the lost imap headers by just moving a mail to another folder and whatnot 😛).
So i went ahead and you configured the "since date" field in the Archive-Job to the day of this change. 🤞

I haven't tested the removal of mails that are older X DAYS/YEARS with that!

And if you need to set the permissions aswell

# i only added the removal of storeowner because of the current _icky_ changes.
# if the access still fails you might get things working by running the following
# command before removing/adding the storeowner rights:
# (if there are granular permissions for that user in some place!)
# grommunio-admin exmdb $username folder revoke --recursive 0x9 $ARCHIVEUSER [all]
ARCHIVEUSER="mailstoreaccount@clownflare.de";
gromox-mbop foreach.mb.here echo-username |grep -v $ARCHIVEUSER | while read -r username ; do 
 echo $username ;
 grommunio-admin user storeowner $username remove $ARCHIVEUSER ; 
 grommunio-admin user storeowner $username add $ARCHIVEUSER ; 
done