Mailbox stored size is huge compared to it's content

ozy

I have a huge mailbox I am unable to shrink.

Currently, grommunio-admin reports the mailbox size to be roughly 23GiB

grommunio-admin exmdb myuser@domain.tld store get
messagesizeextended                    25166673735  23.4 GiB

Looking at the mailbox diretory, the above sums up:

/var/lib/gromox/user/1/0 # du --max-depth 1 -h
557M    ./exmdb
0       ./tmp
7,2G    ./eml
37M     ./ext
13G     ./cid
12K     ./config
21G     .

The problem is that the mailbox has about 7,5GB of data (which is shown in Outlook / WebApp when right clicking on mailbox and showing size).
So there's a huge difference between my user having 7,5GB of "real" data, and Grommunio reporting 23GiB.

What's the cid folder, and why does it consume 13GB ?
Is there any maintenance command that I should run that I didn't ?

Thanks for any insight.
I've already roamed the forum for this issue and found this thread, but it didn't give any answers.

WalterH

Did you run the cleaner task every night?

crpb

WalterH Did you run the cleaner task every night?

theres a cleaner task?!

ozy

Thank you for pointing that out. I didn't know that was a requirement, btw I wonder why this isn't setup by default.

Anyway, I did run gromox-mbop -d /my/user/mailboxdir purge-datafiles, and some quite interesting stuff happened:
Grommunio reported usage

grommunio-admin exmdb myuser@domain.tld store get
messagesizeextended                    25446383067  23.7 GiB

Real disk usage

localhost:/var/lib/gromox/user/1/0 # du --max-depth 1 -h
552M    ./exmdb
0       ./tmp
0       ./eml
0       ./ext
13G     ./cid
12K     ./config
13G     .

All the files in ./eml directory vanished (mails are still in the mailbox) which makes me think that eml dir contains temporary import files, and cid dir contains the actual data.

But event with the actual file size shrunken, Grommunio still reports that mailbox to be almost full. Running size recalculation didn't help:

/usr/sbin/gromox-mbop -d /var/lib/gromox/user/1/0 recalc-sizes
Running `Old: 25166673735 bytes (25166034840 normal, 638895 FAI)
New: 25446383067 bytes (25165920528 normal, 753207 FAI)

Getting more details about sizes does nowhere show anything near 24GB:

localhost:/var/lib/gromox/user/1/0 # gromox-mbsize /var/lib/gromox/user/1/0
                                 Apparent    On FS
                                ---------  ---------
== RFC5322/Mbox representation ==
Received                             0 MB       0 MB
Sent                                 0 MB       0 MB
FS directories                       0 MB       0 MB
midb.sqlite3                         0 MB       0 MB
Total                                0 MB       0 MB

== FS: Body analysis ==
Missing items                        0          0
Informational content              609 MB       -
After deduplication                581 MB       -
Dedup ratio                      1.048 x        -
Dedup savings                      4.6 %        -
After compression                  160 MB     456 MB
File compression ratio           3.614 x    1.275 x
Savings over dedup                72.3 %     21.6 %
IFC compression ratio            3.787 x    1.337 x
Savings over IFC                  73.6 %     25.2 %

== FS: Attachment analysis ==
Missing items                        0          0
Informational content            23270 MB       -
After deduplication              14101 MB       -
Dedup ratio                      1.650 x        -
Dedup savings                     39.4 %        -
After compression                12010 MB   12073 MB
File compression ratio           1.174 x    1.168 x
Savings over dedup                14.8 %     14.4 %
IFC compression ratio            1.938 x    1.927 x
Savings over IFC                  48.4 %     48.1 %

== MAPI Reported Sizes / Network Transfer Size ==
Store size                       24267 MB       -
... Bodies                         609 MB     456 MB
... Attachments                  23270 MB   12073 MB
... Other                          387 MB       -

== On-disk sizes ==
Sum of MAPI data                 12752 MB   13081 MB
... exchange.sqlite3               551 MB     551 MB
... Bodies                         160 MB     456 MB
... Attachments                  12010 MB   12073 MB
... FS directories                  29 MB       0 MB
NTS deviation                     90.3 %     85.5 %

Total MAPI+RFC                   12752 MB    13081 MB
Provisioning factor over NTS     0.525 x    0.539 x

From what I read above, it looks like attachments are counted twice.

I also ran gromox-mbop -d /var/gromox/user/1/0 vacuum but that didn't help either.

Anything else I miss here ?

crpb

ozy All the files in ./eml directory vanished

This doesn't happen here in a couple mailboxes i checked with du -sch * ; purge-datafiles ; du -sch..
Those files are essentially for imap/pop so i guess you currently don't have any client running which makes use of those but did in the past maybe? I even stopped gromox-midb/imap/pop ran an purge-datafiles and files still persisted, as they should in my believe because otherwise that would just add overhead to the system when it had to regenerate those "• .../a@d/eml/mid_string: RFC5322 representation for a message." (LESS=+/RFC5322 man gromox / man midb)
Do you actively use imap on this account?

ozy

crpb I think you guys discussed that in another thread, which basically ended with for mailboxdir in /var/lib/gromox/user/*/*; do /usr/sbin/gromox-mbop -d "${mailboxdir}" purge-datafiles; done to be added as a cron task.

crpb

ozy

Ahh ok, i just thought something new was added.
I still run my manual grom_cleanup and grom_empty_trashbin every now and then on the systems and everywhere where my backupscript runs it does an gromox-mbop vacuum & .. purge-datafiles before i take a snapshot of the filesystem and rsync it to the zfs-backed nfs store.

ozy

Do you actively use imap on this account?
Never have used IMAP, only Outlook clients. But this has been imported from Kopano, so maybe that's the reason eml files where still there.

Anyway, my problem here is that Grommunio reports the mailbox to be 23GB whereas there are only 13GB files on disk.

[OFFTOPIC]
My systems are VMs, which I backup using qemu snapshots which I transfer via npbackup (restic based) to a zfs store too.
Perhaps I will simply add a hook for qemu fs-thaw action which will run a purge-datafiles, this way I get to have a clean base just after a successful backup.
Would you mind sharing your backup script so I can take some inspiration ?
[/OFFTOPIC]

crpb

ozy Anyway, my problem here is that Grommunio reports the mailbox to be 23GB whereas there are only 13GB files on disk.

Well, space-on-disk is basically something completly different than mailboxsize.
Just think of the gromox integrated compression which is enabled by default for quite a while now and than theres filesystem differences on compression and whatnot..

I've seen your statement that outlook itself reports way less than it actually shows via 'exmdb' but is that an online outlook-profile or an cached one i wonder.

ozy Would you mind sharing your backup script so I can take some inspiration ?

All in my repo/grombak.[xfs|btrfs] and mostly duplicates/references in my notepad here.

WalterH

Run this script every night to cleanup soft delete and deleted files:

#!/bin/bash
#
TIMESTAMP=30d20h
echo "Purge softdelete"
/usr/sbin/grommunio-admin user query username maildir | grep "/var/lib/" | awk '{ print $1 }' | xargs -I {} sh -c "echo \"Purge softdelete for: {}\"; /usr/sbin/gromox-mbop -u {} purge-softdelete -r -t $TIMESTAMP IPM_SUBTREE"
echo "Clean user directories"
/usr/sbin/grommunio-admin user query username maildir | grep "/var/lib/" | awk '{ print $1 }' | xargs -I {} sh -c 'echo "Purge mailbox: {}"; /usr/sbin/gromox-mbop -u {} purge-datafiles'

30d20h is the soft delete storage time.

crpb

WalterH

yeah, i should write a .timer + .unit 😛.

i think this would be nicer in a loop instead of icky xargs IMHO 🫣

#!/bin/bash
#
exec > >(tee -i >(logger -t gromox-cleanup))
exec 2>&1
TIMESTAMP=30d20h
p=$(command grommunio-admin config get options.userPrefix)
for username in $(command grommunio-admin user query username maildir |
        awk --assign p="$p" '$2 ~ p {print $1}'); do
        echo "Purge softdelete for: $username"
        command gromox-mbop -u "$username" purge-softdelete -r -t "$TIMESTAMP" IPM_SUBTREE
        echo "Purge orphaned data for: $username"
        command gromox-mbop -u "$username" purge-datafiles
done

WalterH

crpb i think this would be nicer in a loop instead of icky xargs IMHO

A single line is easier to paste into the terminal - this is the reason.

ozy

@WalterH @crpb Thank you for sharing your scripts

After having run both purge-softdelete and purge-datafiles, I now have 8.6GB on disk, and 14,8GB as mailbox size, which looks okay since running gromox-mbsize shows that those 14GB are deduplicated and compresed to about 8GB.

I've made a quick script myself, that I now run as systemd timer as @crpb already suggested, everyday at 2am

#!/usr/bin/env bash

SCRIPT_BUILD=202411254

GROMOX_MAILDIR_PATH="/var/lib/gromox"
SOFTDELETE_TIMESTAMP="30d20h"

function TrapQuit {
        local exitcode=0

        if [ "$SCRIPT_GOOD" == true ]; then
                log "Cleanup operation finished with success after ${SECONDS} seconds"
                exitcode=0
        else
                log "Cleanup operation failed after ${SECONDS}" "ERROR"
                exitcode=1
        fi
        exit $exitcode
}


function log {
        local log_line="${1}"
        local level="${2}"

        if [ "${level}" != "" ]; then
                log_line="${level}: ${log_line}"
        fi

        if [ "${level}" == "ERROR" ]; then
                SCRIPT_GOOD=false
                echo "ERROR: ${log_line}" >> "${LOG_FILE}"
                (>&2 echo -e "$log_line")
        else
                echo "${log_line}" >> "${LOG_FILE}"
                echo "${log_line}"
        fi
}


function cleanup {
        for maildir in $(grommunio-admin user query maildir | awk '{if ($1~gromox_mail) {print $1}}' gromox="${GROMOX_MAILDIR_PATH}"); do
                if [ "$SOFTDELETE_TIMESTAMP" != "" ]; then
                        log "Purging soft deletions for ${maildir}"
                        start_time=${SECONDS}
                        gromox-mbop -d "${maildir}" purge-softdelete -r -t "$SOFTDELETE_TIMESTAMP" IPM_SUBTREE >> "${LOG_FILE}" 2>&1
                        if [ $? -eq 0 ]; then
                                log "Operation took $((${SECONDS}-${start_time})) for ${maildir}"
                        else
                                log "Failed to purge soft deletions for ${maildir}" "ERROR"
                        fi
                fi
                log "Purging datafiles for ${maildir}"
                start_time=${SECONDS}
                gromox-mbop -d "${maildir}" purge-datafiles >> "${LOG_FILE}" 2>&1
                if [ $? -eq 0 ]; then
                        log "Operation took $((${SECONDS}-${start_time}))  for ${maildir}"
                else
                        log "Failed to purge datafiles for ${maildir}" "ERROR"
                fi
        done
}

## ENTRY POINT

trap TrapQuit TERM EXIT HUP QUIT
set -o pipefail
set -o errtrace
SCRIPT_GOOD=true

## Default log file

SCRIPT_NAME=$(basename "$0")
if [ -w /var/log ]; then
        LOG_FILE="/var/log/${SCRIPT_NAME}.log"
elif ([ "${HOME}" != "" ] && [ -w "${HOME}" ]); then
        LOG_FILE="${HOME}/${SCRIPT_NAME}.log"
elif [ -w . ]; then
        LOG_FILE="./${SCRIPT_NAME}.log"
else
        LOG_FILE="/tmp/${SCRIPT_NAME}.log"
fi

cleanup

I wonder if we should make a PR for the next grommunio release, since I think this is pretty basic stuff that everybody should have.

What do you guys think ?

crpb

ozy I wonder if we should make a PR for the next grommunio release, since I think this is pretty basic stuff that everybody should have.

Sure thing, at least for that there is the repo gromox not like with grommunio-spam-run where my ticket just collects dust *whistle*

ozy

Sure thing, at least for that there is the repo gromox not like with grommunio-spam-run where my ticket just collects dust whistle

I too had my fixes for grommunio-archive collect dust for more than a year until I recreated new PRs... Small team, much work I guess.

ozy

Done here

crpb

ozy
aahhhhh too slow 😛

crpb

first draft: https://gist.github.com/crpb/3d389a230bac453e6a00ea9bea01b93b

ozy

Lool, you won 😉
Perhaps we should change grommunio-admin to /usr/sbin/grommunio-admin in order to not rely on path variable too.
Mind if I update my PR with your changes, especially the env ones ?

Also, I don't think that the systemd files go directly into gromox repo. Any ideas where I could put them ?

crpb

ozy Mind if I update my PR with your changes, especially the env ones ?

of course

ozy Also, I don't think that the systemd files go directly into gromox repo. Any ideas where I could put them ?

systemd stuff goes in ./system/ i believe and the script if not already into ./tools

the shebang i changed as i can remember that they want it like this because of less build-dep's w/o /usr/bin/env.