Possible assistance with IMAP please

kbekus

hi all, I'm wondering if anyone can possibly point toward issues in my setup: I'm trying to get IMAP going. I've got gromox talking LDAP to Authentik and that's working fine... Postfix using SASLAUTHD also via LDAP to Authentik... now I'm trying to enable IMAP for a user and get the following in my log:

Jun 20 21:20:03 something.com midb[140734]: sqlite3_prepare_v2(/var/lib/gromox/user/1/2/exmdb/midb.sqlite3) "SELECT config_value FROM configurations WHERE config_id=10": disk I/O error

Jun 20 21:20:03 something.com midb[140734]: dbop_sqlite: /var/lib/gromox/user/1/2/exmdb/midb.sqlite3: impossible to determine schemaversion

I've looked into file and folder permissions, they seem OK. I've even done the nasty and set chmod 777 . and chmod 777 * in /exmdb just to rule them out totally. Was also fortunate to find the hint to turn off archiving or IMAP flat out won't work.

WalterH

Try to recover the user database, use this procedure: https://docs.grommunio.com/kb/sqlite.html but execute in the user database directory: /var/lib/gromox/user/1/2/exmdb/ .
Is there enough free space on volume?

kbekus

thanks Walter.

There's definitely enough space on the volume:
mailboxes 915G 743G 126G 86% /var/lib/gromox/user

FWIW I run Grommunio in a Debian 11 VM. I mount /var/lib/gromox/user as a direct mount from an SSD on the host - it's a bit slower than direct access but it works, there are 3 other accounts on this system that work fine.

Starting from scratch and following this guide
mv midb.sqlite3 midb.sqlite3.bk
gromox-mkmidb -vf user@domain.com

returns
dbop_sqlite: Creating table "configurations"
dbop_sqlite: Creating table "folders"
dbop_sqlite: Creating table "messages"
dbop_sqlite: Creating table "mapping"

root@mail:/var/lib/gromox/user/1/2/exmdb# for i in /var/lib/gromox/user/1/2; do /usr/libexec/gromox/cleaner -d "$i"; done
NOTE: This program has been replaced by gromox-mbop -d /var/lib/gromox/user/1/2 purge-datafiles.
Forwarding execution...
Jun 21 10:46:11 mail.domain.com http[142006]: I-2393: Purged 0 files (0B) from /var/lib/gromox/user/1/2/cid
Jun 21 10:46:11 mail.domain.com http[142006]: I-2393: Purged 0 files (0B) from /var/lib/gromox/user/1/2/eml
Jun 21 10:46:11 mail.domain.com http[142006]: I-2393: Purged 0 files (0B) from /var/lib/gromox/user/1/2/ext

root@mail:/var/lib/gromox/user/1/2/exmdb# telnet ::1 5555
Trying ::1...
Connected to ::1.
Escape character is '^]'.
OK
X-RSYM /var/lib/gromox/user/1/2
TRUE 2

this output occurs:
Jun 21 10:47:51 mail.domain.com midb[148102]: sqlite3_exec(/var/lib/gromox/user/1/2/exmdb/midb.sqlite3) "DELETE FROM mapping": disk I/O error
Jun 21 10:47:51 mail.domain.com midb[148102]: Running sync_mailbox for /var/lib/gromox/user/1/2
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 15 skipped: PR_CONTAINER_CLASS not IPF.Note
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 16 skipped: PR_CONTAINER_CLASS not IPF.Note
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 17 skipped: PR_CONTAINER_CLASS not IPF.Note
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 18 skipped: PR_CONTAINER_CLASS not IPF.Note
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 19 skipped: PR_CONTAINER_CLASS not IPF.Note
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 20 skipped: PR_ATTR_HIDDEN=1
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 21 skipped: PR_ATTR_HIDDEN=1
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 22 skipped: PR_ATTR_HIDDEN=1
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 29 skipped: PR_ATTR_HIDDEN=1
Jun 21 10:47:51 mail.domain.com midb[148102]: sync_mailbox /var/lib/gromox/user/1/2 fld 2130545 skipped: PR_CONTAINER_CLASS not IPF.Note
Jun 21 10:47:51 mail.domain.com midb[148102]: sqlite3_step(/var/lib/gromox/user/1/2/exmdb/midb.sqlite3) "SELECT folder_id, parent_fid, commit_max, name FROM folders WHERE folder_id=10": disk I/O error
Jun 21 10:47:51 mail.domain.com midb[148102]: sqlite3_step(/var/lib/gromox/user/1/2/exmdb/midb.sqlite3) "INSERT INTO folders (folder_id, parent_fid, commit_max, name) VALUES (10, 9, 133316349914391756, 'sent')": disk I/O error

Something interesting is that the sqlite3 file gets created with gromox:gromox ownership, I believe it needs to be grommunio:gromox? Even if I change ownership prior to steps 2 and 3 it still errors. I also note ownership of the folder is gromox:gromox - I manually reset this to grommunio:gromox but issuing a gromox-mkmidb -vf user@domain command resets ownership to gromox:gromox

root@mail:/var/lib/gromox/user/1/2/exmdb# ls -la
total 543708
drwxrwx--- 2 gromox gromox 4096 Jun 21 10:47 .
drwxrwx--- 9 grommunio gromox 4096 Jun 21 10:15 ..
-rwxrwxrwx 1 grommunio gromox 556523520 Jun 21 10:43 exchange.sqlite3
-rw-rw---- 1 gromox gromox 77824 Jun 21 10:47 midb.sqlite3
-rwxrwx--- 1 grommunio gromox 77824 Jun 21 10:42 midb.sqlite3.bk
-rwxrwxrwx 1 grommunio gromox 32768 Jun 21 10:47 midb.sqlite3-shm
-rwxrwxrwx 1 grommunio gromox 0 Jun 20 20:58 midb.sqlite3-wal

WalterH

kbekus There is something wrong with your database:
Jun 21 10:47:51 mail.domain.com midb[148102]: sqlite3_step(/var/lib/gromox/user/1/2/exmdb/midb.sqlite3) "SELECT folder_id, parent_fid, commit_max, name FROM folders WHERE folder_id=10": disk I/O error
Jun 21 10:47:51 mail.domain.com midb[148102]: sqlite3_step(/var/lib/gromox/user/1/2/exmdb/midb.sqlite3) "INSERT INTO folders (folder_id, parent_fid, commit_max, name) VALUES (10, 9, 133316349914391756, 'sent')": disk I/O error

Is the current database schema installed?

# gromox-dbop -U
dbop: Current database schema: gx-122

crpb

kbekus I mount /var/lib/gromox/user as a direct mount from an SSD on the host

first of all, if you use an extra partition/disk/whatevs you should include /var/lib/gromox/** and not just "user".

and that direct mount of an disk into virtualization just sounds icky on so many levels.
i've done such things for recovery-procedures but never for any production use as those things can go wrong on multiple levels. doesn't matter which virtualization you are using.

kbekus

thanks Walter, yes it is:

root@mail:/var/lib/gromox/user/1/2/exmdb# gromox-dbop -U
dbop: Current database schema: gx-122

mwilliams

midb is used primarily by imap and seems to have had some issue at some point (I/O error), a good idea to regenerate all midb indexes.

Proposal:

systemctl stop gromox-midb
find /var/lib/gromox/user/ -name "midb.sqlite" -exec rm -f {} \;
systemctl start gromox-midb
connecting with the account (IMAP) or using grommunio-admin show will regenerate midb. alternatively, use X-RSYM via midb socket as you described.

No data is lost with this approach.

kbekus

mwilliams
thanks for that, unfortunately I've tried that approach several times and it doesn't work - I continue to get the disk IO errors even with a deleted and recreated midb.sqlite file.

WalterH

Did you reboot the server to clear all open file handles? This may help.

kbekus

Interesting possibility. OK, stopped gromox-midb and disabled it.
Deleted midb.sqlite.
Rebooted the VM
Followed the steps in my first reply.

Sadly still disk I/O errors during the sync:

Jun 22 16:50:20 mail.domain.com midb[1895]: sqlite3_step(/var/lib/gromox/user/1/2/exmdb/midb.sqlite3) "SELECT folder_id, parent_fid, commit_max, name FROM folders WHERE folder_id=10": disk I/O error
Jun 22 16:50:20 mail.domain.com midb[1895]: sqlite3_step(/var/lib/gromox/user/1/2/exmdb/midb.sqlite3) "INSERT INTO folders (folder_id, parent_fid, commit_max, name) VALUES (10, 9, 133316349914391756, 'sent')": disk I/O error

WalterH

Did you install current updates and rebootet the server?
Which virtualization you use?
Which storage you use?
Which file system is on /var/lib/gromox/user/1/2/?
How you mount /var/lib/gromox/user please explain the mount options - on host and VM?

kbekus

Thankyou for the replies and assistance. I'm trying to keep the VM immutable with all data stored elsewhere - that's my design intent so that I can snapshot prior to an update and roll back quickly if necessary. I'm aware that the database schema may update but it's easy enough to dump the SQL database and restore if necessary.

Thanks for the heads-up RE: mounting /var/lib/gromox instead of /var/lib/gromox/user - good point.

I appreciate the thought that direct mounting of folders into VMs sounds icky and with 25 years experience in corporate IT I've seen my share of strange problems that ought not exist - I do get your point. I'm not being closed minded to the thought that my design intent is a bad idea. I'd previously mounted /var/lib/gromox/user as an NFS share but performance was just terrible so I changed to a direct mount.

WalterH - yep, all current software wise, though there have been some new updates in the last 24 hours I've yet to apply. My host machine is Debian 10 and virtualisation engine is libvirt 5.0 with kernel 4.19. I really need to pull my finger out and update to Debian 11 at a minimum. The VM uses a qcow2 disk image mounted as an IDE drive, the /var/lib/gromox/user folder is a direct folder mount:

root@mail:/var/lib/gromox# cat /etc/fstab | grep mailbox
#192.168.2.2:/opt/grommunio/mailbox     /var/lib/gromox/user    nfs     x-systemd.automount,noatime,nolock,intr,tcp,actimeo=1800 0 0
mailboxes       /var/lib/gromox/user    9p      trans=virtio    0 0

Please ignore the first entry, it was the previous NFS mount method I used.

I'll try copying the data local to the VM and see how I go.... I'll post my result.

thanks all.

WalterH

The file system should be a btrfs, grommunio tests only with btrfs.

kbekus

hi all,
thanks for the responses and guidance. Just to close the loop on this I migrated my system from a Debian 11 platform running several VMs and docker containers to a Proxmox server running this mail VM at the top Proxmox level. In order to do that I had to copy /var/lib/gromox/user into the VM from the mapped folder and stopped using the mapping.

This achieved 2 things: 1 - it made webmail wayyy faster.... and 2 - it fixed the issue so that now I can successfully access my mailbox via IMAP... wooo!

So yeah, issue closed. This is good and bad. Good in that it works, bad in that now the VM contains the binaries, config files and mail files... which makes backing up a bit less nice. One last thing, BTRFS isn't really used much in the Debian world... I personally use it a lot but it's not super popular.