The Message Index Data Base (MIDB) midb.sqlite3 accelerates IMAP and POP3 access to grommunio.
Additionally there are flat mail files which also increase the speed. The MIDB references these flat mail files.
- The MIDB is located in: /var/lib/gromox/user/X/Y/exmdb/midb.sqlite3
- The flat mail files are stored in: /var/lib/gromox/user/X/Y/eml/*.midb
- Each mailbox has its own MIDB and associated flat mail files
- The primary mail database, the exchange.sqlite3 is located in /var/lib/gromox/user/X/Y/exmdb/exchange.sqlite3
Please note, the MIDB can be rebuilt without any issues, but not the exchange.sqlite3, if the exchange.sqlite3 is damaged or deleted, the mailbox is most likely lost. A recovery procedure for a damaged but not deleted Domain or mail database (exchange.sqlite3) is documented in the KB: https://docs.grommunio.com/kb/sqlite.html, use it on your own risk.
The MIDB is automatically synchronized with exchange.sqlite3 by the system. But sometimes it is necessary to resynchronize all MIDBs, for example after large updates, like May 2023 update.
The rebuild/resynchronization procedure is described here
Hint: if you want to resyncronize only one MIDB, read Annex A.
1. preparation
- Important: disable IMAP/POP3 access for all users who do not use IMAP/POP3 to conserve disk space.
- Open a SSH session like Putty to the grommunio server
- You can copy and paste the commands into the SSH session
- To follow the regeneration of the MIDBs, you may open a 2nd SSH session for logging
- At Annex C. of this tutorial there is a Python script, please create this script at the grommunio server as /tmp/trigger_midb.py and make it executable:
chmod +x /tmp/trigger_midb.py, the script is needed at step 5.
If you created the /tmp/trigger_midb.py file with an Windows editor, convert the file to Unix line feeds:
zypper in dos2unix
dos2unix /tmp/trigger_midb.py
- Please note, regenerating the MIDBs generates huge load on the server and interrupts POP3 and IMAP access
- Hint: if you want to clean up the system: shrink the MIDBs and delete all flat mail files, then exit at point 3.
2. recreate empty MIDBs
To do this we have to stop the MIDB daemon and then recreate the MIDBs, this interrupts IMAP and POP3 access.
systemctl stop gromox-midb
Recreate the MIDBs
grommunio-admin user query username | grep "@" | sort | while read line; do echo -e "\nRecreate MIDB for: $line"; gromox-mkmidb -vf $line; done
Start the MIDB daemon and empty the IMAP and POP3 cache.
systemctl restart gromox-midb gromox-imap gromox-pop3
3. run the cleaner
If the MIDB was deleted, all flat mail files referenced by the MIDB are orphaned and must be deleted to free the storage.
for i in /var/lib/gromox/user/*/*; do /usr/libexec/gromox/cleaner -d "$i"; done
As crpb stated, with gromox 2.10 cleaner was replaced by gromox-mbop, so we need a modified command to remove the user files:
grommunio-admin user query username maildir | grep "/var/lib/" | awk '{ print $1 }' | xargs -I {} /usr/sbin/gromox-mbop -u {} purge-datafiles
Hint: at this point all MIDBs are empty and the flat mail files are deleted. If you only want to clean up the system, you can exit at this point.
4. create the mail directory list to regenerate the MIDBs
The file /tmp/mail_dirs.txt lists every mailbox of this system, each line contains a path to a mailbox.
Only users/mailboxes where IMAP/POP3 is enabled, are included in the list file.
The long version of this command:
grommunio-admin user query pop3_imap maildir | grep "/" | grep True | sed -r 's/True//g' | sed -r 's/\s+//g'| sort | while read line; do echo "$line"; done > /tmp/mail_dirs.txt
The short version of this command:
grommunio-admin user query maildir --filter pop3_imap=True | grep "/" | sort > /tmp/mail_dirs.txt
Thanks to crpb for the short version.
5. rebuild the MIDBs and create the flat mail files
To do this, we use the Python script we created earlier. This process is CPU and I/O intensive, so it is best to run it at a time when there is little or no load on the system.
Optionally we may open a 2nd SSH session on the server and watch in the log how the MIDBs are built. Launch journalctl -fu gromox-midb in the 2nd SSH session.
We will see messages like:
May 12 22:20:31 server midb[nnnnn]: Running sync_contents for /var/lib/gromox/user/X/Y, folder nnnnnnn
To start the rebuild process, launch the Python script:
/tmp/trigger_midb.py
The script reads the file /tmp/mail_dirs.txt line by line and passes the commands to port 5555 which is the MIDB daemon. When the Python script is finished, the file /tmp/mail_dirs.txt is deleted.
Please note, rebuilding of the MIDBs takes several minutes per mailbox, depending on the mailbox size.
6. now all MIDBs are regenerated
IMAP and POP3 should work again and faster than before.
Remove the script /tmp/trigger_midb.py, it is no longer used.
rm /tmp/trigger_midb.py
7. about backup
We deleted and recreated all flat mail file, this is mostly a huge amount of date. Most differential backups have issues when so many files are new or deleted. If you use a differential backup like rdiff backup, consider to nuke the backup and recreate the backup now.
Enjoy grommunio
Annex
A. Rebuild/resyncronize only one MIDB
If you want to rebuild only one MIDB, the process is outlined here.
A.1. recreate empty MIDB
Replace MAIL@DOMAIN.TLD with the mailbox/user name you want or resyncronize.
To do this we have to stop the MIDP daemon and recreate the MIDB, this interrupts IMAP and POP3 access for all users.
systemctl stop gromox-midb
Recreate the one MIDB
gromox-mkmidb -vf <MAIL@DOMAIN.TLD>
Start the MIDB daemon and empty the IMAP and POP3 cache
systemctl restart gromox-midb gromox-imap gromox-pop3
A.2. run the cleaner
If the MIDB was deleted, all flat mail files referenced by the MIDB are orphaned and must be deleted to free the storage.
for i in /var/lib/gromox/user/*/*; do /usr/libexec/gromox/cleaner -d "$i"; done
As crpb stated, with gromox 2.10 cleaner was replaced by gromox-mbop, so we need a modified command to remove the user files:
grommunio-admin user query username maildir | grep "/var/lib/" | awk '{ print $1 }' | xargs -I {} /usr/sbin/gromox-mbop -u {} purge-datafiles
Hint: at this point the MIDB is empty and the flat mail files are deleted. If you only want to clean up the system, you can exit at this point.
A.3. rebuild the MIDB and create the flat mail files
Find the mail directory for this mailbox
grommunio-admin user query ID username maildir | grep <MAIL@DOMAIN.TLD>
The mail directory like /var/lib/gromox/user/X/Y/ we need for the resync command
Open a Telnet session to the MIDB daemon
telnet ::1 5555
Wait for OK to appear, than feed the rebuild command including the mailbox path to the MIDP daemon
X-RSYM /var/lib/gromox/user/X/Y
Exit Telnet when done - TRUE 2 is success
quit
B. Changes:
- moved the command generation into the Python script
- minor enhancements
- added information about backup
- Python script minor enhancements
- added Rebuild only one MIDB
C. The Python script:
#!/usr/bin/python3
#
import socket, time, telnetlib, os
#
DIR_FILE = '/tmp/mail_dirs.txt'
def main():
if not os.path.isfile(DIR_FILE):
print("Error: {} file not found".format(DIR_FILE))
quit()
# Staring a TCP socket.
telnet_client = telnetlib.Telnet('::1', 5555)
# Opening and reading the mail directories list
file = open(DIR_FILE, 'r')
LINES = file.readlines()
wcount = 0
mcount = 0
telnet_client.read_until(b"OK", timeout=10) # wait for OK
# Count mailboxes / MIDB commands
for line in LINES:
mcount += 1
print("{} mail directories for MIDB synchronization found.".format(mcount))
# Sending the commands to MIDB
for line in LINES:
time.sleep(1)
wcount += 1
print("Mailbox {:>3}: {}".format(wcount, line.strip()))
# Build the synchronization command
byte_obj = ( "X-RSYM " + line.strip() + "\r\n").encode("ascii")
telnet_client.write(byte_obj)
telnet_client.read_until(b"TRUE ", timeout=300) # wait max. 5 minutes for TRUE
print(f"[Closing connection]")
file.close()
# Closing the connection from MIDB daemon
telnet_client.write(b'exit')
# Remove the mail directories file
os.remove(DIR_FILE)
if __name__ == "__main__":
main()
#
