Increased (2x) disk usage and number of files (6x) after (imapsync) migration

mculibrk

Hi there!
I'm currently testing some migration scenarios from Exchange, Kopano and "generic" IMAP mail servers to Grommunio. I have some "advanced" users which tend to have "enormous" maiboxes (>60GB with >500K mail items)

After some test migrations I noticed a big "disproportion" of disk usage.
For example, the original "IMAP" mailbox has a size of _60GB while the same user on Grommunio uses _110GB of disk space. After all cleaner and indexer re-execution.
Another, more concerning issue, is the number of individual files increased 6x!! From original 530K files to 3M files...
Even worse, the original IMAP server (maildir based) have those 530K files dispersed in various folders while in Grommunio all files are basically in a "single" folder:
/var/lib/gromox/user/1/0/cid --> 1.9M files
/var/lib/gromox/user/1/0/eml --> 530K files
/var/lib/gromox/user/1/0/ext --> 530K files

I know Grommunio keeps indexes in various sqlite databases (exchange, midb...) but this large number of files in a single folder makes me "uncomfortable"...

Especially so after using Grommunio-web and keep getting HTTP-500 errors on searching

Is there any option to reduce the number of files in single folders? Some "foldering scheme/leveling" or hashing? For example to have 2 or more sub-folder levels used?

Also... is there any way to remove IMAP "copy" of the messages?
I know Kopano makes the "imap version" of a message when accessed via IMAP (as do (or did) Exchange) but there was a way to "purge" that.
This is because the plan is to use multiple passes of IMAPSYNC to migrate the user mailboxes in a "soft" way (first migrate the last few months, switch systems and then all the rest in the "background")

Using the appliance for this testing (SLES)
grommunio-release-2022.12.1-lp154.3.2.x86_64
grommunio-imapsync-2.178-lp154.2.1.noarch
system-user-gromox-2-lp154.13.2.noarch
grommunio-web-3.2.32.d44c667-lp154.211.3.noarch
gromox-2.7.125.9864e9d-lp154.52.1.x86_64
grommunio-index-0.1.36.4ebe9ad-lp154.39.1.x86_64

Any suggestions/hints more than welcome!
Regards,
M

Andy

I am not a maintainer for Grommunio, rather an ambitious user. I also manage various mailboxes, the largest is Outlook PST about 13 GB in size and includes not quite 60,000 items, so seen rather small compared to your explanations, but there are parallels.

My experience is that migrations with third party software, like Outlook or the like inflate the subfolders shown, especially the CID folder. CID is always larger, as is the EML, which ultimately contains all the items. There is a PST import sequence "gromox-pff2mt Name.pst | gromox-mt2exm -u target@mailbox.de", which in my case unfortunately caused Outlook not to open after the import, but using this command set the increase in memory usage was about the same as the PST. Unfortunately, I can not seem to apply it at the moment.

In this respect I also have a "disproportion" of about 3:1 and this is not unusual. My VM occupies a size of approx. 37 GB, for the installation approx. 2-3 GB, remain 34 GB for data, whereby the other mailboxes are in the MB range and thus rather negligible. So you can calculate that your storage capacities should be around 200 GB, if the mailboxes are around 60 GB.