Mac Mail not updating Grommunio properly on deleting/moving emails.

Mister2

When I delete or move emails using Mac Mail everything now works as expected except that the emails are not removed from the inbox/source location.

Tests performed:-

New email opened in Mac Mail, marks email as read in all clients (Mac Mail, Gr Web, IOS, Android) - correct.
New email opened in any client marks email as read in all other clients including Mac Mail - correct.
Deleting or moving emails to another folder, in IOS, Outlook, Gr Web, Android client works as expected on all clients including Mac Mail - correct.
Deleting or moving emails to another folder using Mac Mail updates Mac Mail correctly, messages removed from source folder, and placed in destination folder, however, all other clients are unaware that the deletion or move has happened and the original emails are left in their original folders as if untouched - incorrect.
If the undeleted messages are moved or deleted from GR Web, IOS, Android they duplicate the messages previously moved/deleted by Mac Mail in the destination folders - expected but effectively incorrect because of 4 above.

It looks like Grommunio is not sending/applying (or receiving) the move/deletion request from Mac Mail to update the database, yet it Grommunio/Mac Mail are ovbious;lyt communicating when an email is read as per item 1 above, as that behaviour is processed correctly.

Currently running Mac OS Sonoma 14.2.1 on MacBook Air M2 (20/12/23 Apple Update), the same behaviour was experienced with the previous version - 14.1.2.

I have also tried creating a brand new user and sent a couple of 'internal emails' to see if the issues were related to one of my existing mailboxes, but no change in behaviour is seen.

My Mac mail version is Version 16.0 (3774.300.61.1.2) and my current Grommunio versions (as of 18:00 today - 20-12-23) are:-

rpm -qi gromox | head -n 3 && rpm -qa *grom* | sort Name : gromox Version : 2.20.6.g19e4a21 Release : lp155.15.1 grommunio-admin-api-1.14.2.63c43b5-lp155.87.1.noarch grommunio-admin-common-20.e87783f-lp155.16.1.noarch grommunio-admin-web-2.8.0.25.35a839f-lp155.71.1.noarch grommunio-antispam-3.7.4-lp155.3.1.x86_64 grommunio-common-23.7d1954c-lp155.35.1.x86_64 grommunio-cui-1.0.256.5c857a6-lp155.60.1.noarch grommunio-dav-1.3.18.e16f8a2-lp155.1.1.noarch grommunio-dbconf-1.1.1.da20a46-lp155.16.1.x86_64 grommunio-error-pages-1.0.10.bb2df37-lp155.17.1.noarch grommunio-files-25.0.13-lp155.2.1.noarch grommunio-imapsync-2.264-lp155.2.1.noarch grommunio-index-0.1.37.74e396b-lp155.41.1.x86_64 grommunio-release-2023.11.1-lp155.8.1.x86_64 grommunio-setup-1.0.108.f2ac66c-lp155.46.1.noarch grommunio-sync-2.0.49.fe860ff-lp155.144.1.noarch grommunio-web-3.6.7.20975ed1-lp155.8.3.noarch gromox-2.20.6.g19e4a21-lp155.15.1.x86_64 gromox-debuginfo-2.20.6.g19e4a21-lp155.15.1.x86_64 gromox-debugsource-2.20.6.g19e4a21-lp155.15.1.x86_64 grub2-theme-grommunio-1-lp155.17.1.noarch patterns-grommunio-1-lp155.9.1.x86_64 plymouth-theme-grommunio-1-lp155.17.1.noarch systemd-coredump-grommunio-1-lp155.3.1.noarch systemd-presets-branding-grommunio-2023.11-lp155.1.1.noarch system-user-grommunio-3-lp155.10.6.noarch system-user-gromox-2-lp155.15.1.noarch

Not found any obvious error to report, but open to any requests for info as required.

Best Regards

Mark

mwilliams

Mister2

First: Thanks for the details, however: cannot reproduce here.

Have you upgraded with your existing account setup or have you re-setup the account after the update? There have been quite some changes recently also concerning EntryIDs...

Mister2

mwilliams In current setup I built the current VM with the appliance ISO dated 4/12/23 (downloaded just before building new VM). I initially tried to migrate the mailboxes by the /var/lib/gromox migration method below:-

Update and reboot the source and target system: zypper refresh & & zypper update.
Build new server installation identical to current live server. Follow your build documentation unto the point of creating user accounts (call this ‘new server’ in steps below).
Stop inbound Internet email and user access to live server mailboxes to avoid missing emails during migration.
Optional step - If Grommunio is in Virtual Environment, clone (or take backup of) live server to avoid any damage/corruption to current live server during migration (call this ‘old server’ in steps below), just in case you need to revert to what was your current live server.
Stop services on both systems: systemctl --all --output json list-units| jq '.[]|select(.unit|test("(^grom.|nginx|.fpm).service")).unit' |xargs systemctl stop
Export of the MariaDB database on old server: mysql --execute="SHOW DATABASES" --skip-column-names --batch |grep -Ev '^(mysql|(performance|information)_schema)$' |while read -r DB; do mysqldump --single-transaction --routines --triggers --events --add-drop-database $DB > /usr/local/share/$DB.sql ; done
User Data transfer old server to new server: rsync -aH -essh --delete --numeric-ids -P --stats --inplace /var/lib/gromox/ root@192.168.xxx.xxx:/var/lib/gromox/
Move MariaDB backup (from step 4) from old system to new system. Run this on old server: rsync -aH -essh --delete --numeric-ids -P --stats --inplace /usr/local/share/ root@192.168.xxx.xxx:/usr/local/share/
Check folder permissions on new server: chown -Rf grommunio:gromox /var/ib/gromox/user//
Import of the MariaDB database on new server: mysql grommunio < /usr/local/share/grommunio.sql
Import of the MariaDB database on new server: mysql sys < /usr/local/share/sys.sql
Check import user /var/lib/gromox/user/x/y/: run grommunio-admin user query username maildir on both systems - make sure they match.
Reboot

However, I had some issues after the update so rebuilt the VM again and exported my mailbox data to PST's using Outlook, then created new Outlook profiles when connected to the new Grommunio server and imported the PST's into Outlook and waited for the Offline folder sync to do its business.

The general use of Grommunio has been fine since on all platforms bar Mac Mail. On 10th December the updates stopped the persistent crashing I had been getting on deleting emails in Mac Mail (no issue in any other client).

I have created a new user on the Grommunio server and sent a few test internal emails to the new mailbox. Creating a new Mac Mail profile to this new mailbox gives exactly the same behaviour described in my first post.

If there is anything you need me to look at or provide to help I am more than happy to provide. My experience is a lot of years in the Windows arena so may need a little guidance in this Linux world, but I am slowly learning.

EDIT: Forgot to mention that I have applied the updates released this morning with no change to the behaviour.

Best Regards
Mark

mwilliams

Mister2

Thanks. But I need this for clarification: Did you ever on your mac fully recreate the EWS account profile or did you always keep it (or only disabled features) in your Mac regarding your EWS account after updating?

Mister2

@mwilliams Yes I have completely deleted the Mac Mail profile on several occasions since 10/12/23, as I wondered if any settings may have been made when the profile is created. The current profile for my mailbox was last created on 16/12/23, however, the new 'test' mailbox was only first created yesterday.

When I recreate my Mac Mail profile I go to ~/Library/Mail and delete all the content in this folder so a completely clean profile is created. Hope that answers your query.

More than happy to repeat the above process you feel that will help. It only takes about 30 mins for the mail to fully sync over Wi-Fi.

EDIT: Currently only have Mail selected for sync in the profiles.

Best Regards

Mark

andreaslang

Hi Mark,

I was also not able to produce the behavior you've described in the first post. What I noticed however that in some cases Mac apps (Mail, Calendar, Reminders etc.) take unusually long time (e.g. 5 minutes) to process changes which are almost instant in Outlook for Windows or Grommunio web.

Mister2 More than happy to repeat the above process you feel that will help. It only takes about 30 mins for the mail to fully sync over Wi-Fi.

Why does it take so long? Do you have a lot of emails/folders in your account?

mwilliams

Mister2

Thank you, yes, I do understand now. It is not required to delete the entire contents of the folder. It is enough to delete the account from the account settings and re-create it.

The only thing we are aware of, is that sometimes Mac Mail does not immediately send its changes to the backend, we've seen delays of up to 5 Minutes (but no EWS requests on wire). Does waiting like a few minutes after deleting or moving the mails in step 4 change anything?

Mister2

@andreaslang @mwilliams Thanks both, however, emails I deleted yesterday in Mac Mail inbox (junk marketing messages) are still sitting in Gr Web, IOS and android inbox, so waiting does not help.

I have just completely reset Mac Mail and the profile is currently recreating. Will see if deleting email in Mac Mail is any better once sync is complete. Will report back in about 20 mins.

Just a thought, I have a number of Public folders that I use which work fine in Gr Web, however, I also have a public folder calendar that obviously I cannot access via the Mac as it doesn't support 'Exchange Public Folders' - could this be causing any issues? IOS accesses the public folder calendar without issue but that uses Activesync of course.

Best Regards

Mark

mwilliams

Mister2

Unlikely that you public folder calender changes anything here. Keep us posted.

Mister2

@mwilliams @andreaslang Right, I have now completely removed everything from the ~/Library/Mail folder again to make sure there is nothing left from the previous mail import.

I have created a new profile for my mailbox and it has just completed syncing (₁₅,000 items - 1.5GB). Profile selected to sync mail only. Have deleted a email in Mac Mail and waited 5 minutes.

The deleted message, as previously mentioned, immediately appeared in the 'bin' folder in Mac Mail and the 'deleted Items folder in Grommunio Web client (which is reflected in IOS and Android clients. Unfortunately, after 7 mins the 'deleted email' is still sitting in the inbox in all clients except Mac Mail.

Best Regards

Mark

Grummel

I can confirm the above mentioned behavior on my grommunio installation using Mac Mail Version 16.0 (3774.200.91.1.1) with MacOs 14.1.2 (23B92).
If I delete an E-Mail in the Web-UI this Mail is synchronized with Mac Mail and is moved into the deleted items bin in Mac Mail as it should.
The other way round doesn't work correctly as @Mister2 describes above.
It seems that the sync path from Mac Mail to grommunio is broken somewhere.
Many Thanks to the whole Grommunio team for the efforts and progress getting MacMail to work!
Merry Christmas to everyone!

rpm -qi gromox | head -n 3 && rpm -qa *grom* | sort Name : gromox Version : 2.20.6.g19e4a21 Release : lp155.15.1 grommunio-admin-api-1.14.2.63c43b5-lp155.87.1.noarch grommunio-admin-common-20.e87783f-lp155.16.1.noarch grommunio-admin-web-2.8.0.25.35a839f-lp155.71.1.noarch grommunio-antispam-3.7.4-lp155.3.1.x86_64 grommunio-common-23.7d1954c-lp155.35.1.x86_64 grommunio-cui-1.0.256.5c857a6-lp155.60.1.noarch grommunio-dav-1.3.18.e16f8a2-lp155.1.1.noarch grommunio-error-pages-1.0.10.bb2df37-lp155.10.1.noarch grommunio-imapsync-2.264-lp155.1.1.noarch grommunio-index-0.1.37.74e396b-lp155.41.1.x86_64 grommunio-setup-1.0.108.f2ac66c-lp155.46.1.noarch grommunio-sync-2.0.49.fe860ff-lp155.144.1.noarch grommunio-web-3.6.7.20975ed1-lp155.8.3.noarch gromox-2.20.6.g19e4a21-lp155.15.1.x86_64 gromox-debuginfo-2.20.6.g19e4a21-lp155.15.1.x86_64 gromox-debugsource-2.20.6.g19e4a21-lp155.15.1.x86_64 grub2-theme-grommunio-1-lp155.17.1.noarch patterns-grommunio-1-lp155.9.1.x86_64 plymouth-theme-grommunio-1-lp155.17.1.noarch system-user-grommunio-3-lp155.10.6.noarch system-user-gromox-2-lp155.15.1.noarch

Mister2

@mwilliams @andreaslang

I have built a brand new VM based on the 4/12/23 appliance ISO. Imported my mailbox using a PST and Outlook in 'online' mode (ie not cached). Exactly the same behaviour as I previously reported and confirmed by Grummel

Anything else I can try to isolate the issue.

As @Grummel said DEV's have done a great job getting Mac Mail this far, would be really great if this last niggle could be fixed. Wishing all the DEV's, Grommunio Support team and all others who help to push Grommunio forward, a great Xmas and New Year.

Best Regards

Mark

Mister2

Yesterday I installed another new VM (test box) using the new appliance ISO from 28.12. Built up as per my normal appliance VM, but this time instead of migrating my mailboxes, set up a forwarder on my spam filter front end to duplicate the incoming emails to both my live and test servers.

Overnight I have received about 100 new emails (internal and external mix). I then cleared all Mac Mail profiles and created a new one for my mailbox on the new test server.

I can confirm that the issue mentioned in post 1 is still present exactly as described there and confirmed by Grummel. However, on checking the Admin GUI Gromox http log I now receive the following errors on each first attempt to delete a mail in Mac Mail.

[2024-01-01 12:14:28.443281]: sqlite3_exec() "INSERT INTO t2 (folder_id) VALUES (1900545)": NOT NULL constraint failed: t2.depth (19) [2024-01-01 12:14:28.460968]: sqlite3_exec() "INSERT INTO t2 (folder_id) VALUES (1900545)": NOT NULL constraint failed: t2.depth (19) [2024-01-01 12:22:44.572647]: sqlite3_exec() "INSERT INTO t47 (folder_id) VALUES (1900545)": NOT NULL constraint failed: t47.depth (19) [2024-01-01 12:22:44.590748]: sqlite3_exec() "INSERT INTO t47 (folder_id) VALUES (1900545)": NOT NULL constraint failed: t47.depth (19)

What is now odd is that to delete an Item in Mac Mail, I now have to do it twice. The first delete removes the message from Mac Mail/Inbox and puts in the bin folder. On the next client refresh the mail re-appears in the inbox, so now both in the inbox and the bin folders. A second delete operation removes the email from the inbox and duplicates it in the bin folder, but the message does not now re-appear in the inbox. Checking the Gr web client shows the deleted messages in the 'deleted items' folder but the messages have never been removed from the inbox. The Gr Web deleted items folder now shows multiple copies of the deleted emails, ie a copy of the deleted email for each time it was deleted within Mac Mail.

Hopefully, the log error messages will help with further diagnosis of this issue.

Now off to migrate my mailboxes to the new appliance in full.

Happy New Year to all.

mjmabs

Am seeing the mention above issues with MAC Mail as well, running a commercial grommunio appliance version 2023.11.1 fully updated. Can login to both Gro Web and Microsoft Outlook 2021 and delete a message no problem but if I attempt to delete a message from MAC Mail running on Sonoma version 14.1.1 (Mac Mini) I get the same error's in Admin GUI Gromox http log. If I follow the above steps of double delete it follows the process outlined above. We have about 60 Users, 2 of which use MAC Mail and so far only those two users have reported an issue with deleting.

[2024-01-05 09:42:30.784730]: sqlite3_exec() "INSERT INTO t10 (folder_id) VALUES (2111618)": NOT NULL constraint failed: t10.depth (19)
[2024-01-05 09:42:30.903995]: sqlite3_exec() "INSERT INTO t10 (folder_id) VALUES (2111618)": NOT NULL constraint failed: t10.depth (19)

freeloadermatt

I am also seeing very similar behaviour to @Mister2, @mjmabs and others above.

Delete email on anything except mac mail = works; delete visible in mac mail as expected.
Delete email on mac mail = a) seems to work at first within the mac client; b) nothing happens in web client or others connected to same account; c) email REAPPEARS in mac mail inbox a few minutes after being deleted (now at least that is in sync with web client etc); d) BUT (weird) the email that reappeared in Mac Mail inbox has "Undelete" option when you right click it, instead of "Move to Bin". i.e. that email is showing in mac mail inbox, but the mac mail menus are behaving as if the mail is in the trash can
Move email from inbox to a folder on anything except mac mail = works; move is visible in mac mail as expected
Move email on mac mail = a) seems to work at first on the mac client just like above; b) in web client the mail appears to be COPIED to the destination folder (i.e. now there is one copy in inbox, one copy in the destination folder); c) email reappears in mac inbox a few minutes after being moved, but now there are two copies visible on mac mail as well with one in inbox and one in the target folder; d) BUT (weird) both emails will have read (non-bold) status, if you mark EITHER of them as unread then BOTH the copies turn to unread status... i.e. it looks like its a single object/item being represented simultaneously in two different folders

So as per previous users, move and delete broken in very similar ways. Kind of makes sense given that "delete" seems to be implemented as "move to destination folder trashcan" BUT also note the "weird" items 2(d) and 4(d) where in both of them you end up with two copies of the email and they both seem to be "linked" to each other. In 2d the email copy in the inbox responds to right click with "undelete" context option, as you would expect of the other copy in the trash can. In 4d making a property change to either copy of the email affects both of them.
It seems like requests from mac mail client to move are being interpreted by the server as "copy and link the two resulting copies together" instead of move.

My system: Macbook Pro with Sonoma 14.2.1; Debain 11 build with API 1.14.0; backend 1.14.2; database gx-131

mwilliams

@freeloadermatt @mjmabs @Mister2 @Grummel

community:gromox-2.21.23.g17c2047

freeloadermatt

mwilliams

Thanks, will try and feedback with results. That version is only available for Debian 12 in the community repo. Please can it be made available for Debian 11 also?

Deb 12 is doable, but annoying because the admin-ui doesn't work properly:
https://community.grommunio.com/d/1157-debian-12-grommunio-admin-api-uwsg-cant-create-domain/17

Mister2

mwilliams Thanks for the update, but it doesn't look like this update has made any difference. What I have done today:-

Shutdown and Snapshot VM for DB consistent Snapshot
Reboot and apply todays updates

Although not needed - rebooted VM
Opened GR Web and Mac Mail, located a marketing/sales email and deleted.
Checked Bin (Mac Mail) and message in Bin and removed from Inbox.
Checked Gr Web - mail not removed from inbox but now also present in Deleted Items as well
Deleted Mac mail profile on Mac and created fresh profile and waited to fully sync.
Opened Mac Mail, noted email deleted in '4' now back in Mac Mail inbox, and NOT in Bin, still shown in bin in Gr Web.
Deleted email again on Mac Mail - same behaviour as described above, except now have 2 copies of same email in GR Web Deleted Items folder.

So overall it looks like community:gromox-2.21.23.g17c2047 has not resolved the issue for me.

Current details of my VM:-

28/12/23 Appliance ISO installed on Proxmox 8.1
Mac Airlock 2022 (M2) running Mac OS 14.2.1 (23C71) with Mac Mail Version 16.0 (3774.300.61.1.2)

rpm -qi gromox | head -n 3 && rpm -qa grom | sort
Name : groom
Version : 2.21.23.g17c2047
Release : lp155.14.1
grommunio-admin-api-1.14.2.63c43b5-lp155.87.1.noarch
grommunio-admin-common-23.e396105-lp155.18.1.noarch
grommunio-admin-web-2.8.0.28.7ed9ad0-lp155.74.1.noarch
grommunio-antispam-3.7.4-lp155.5.1.x86_64
grommunio-common-23.7d1954c-lp155.35.1.x86_64
grommunio-cui-1.0.256.5c857a6-lp155.60.1.noarch
grommunio-dav-1.3.18.e16f8a2-lp155.1.1.noarch
grommunio-dbconf-1.1.1.da20a46-lp155.4.11.x86_64
grommunio-error-pages-1.0.10.bb2df37-lp155.10.1.noarch
grommunio-files-25.0.13-lp155.2.1.noarch
grommunio-imapsync-2.264-lp155.1.1.noarch
grommunio-index-1.0.1.gd50c1fd-lp155.54.1.x86_64
grommunio-release-2023.11.2-lp155.9.1.x86_64
grommunio-setup-1.0.109.048c9c5-lp155.48.1.noarch
grommunio-sync-2.0.49.fe860ff-lp155.144.1.noarch
grommunio-web-3.6.15.42597875-lp155.15.1.noarch
gromox-2.21.23.g17c2047-lp155.14.1.x86_64
gromox-debuginfo-2.21.23.g17c2047-lp155.14.1.x86_64
gromox-debugsource-2.21.23.g17c2047-lp155.14.1.x86_64
grub2-theme-grommunio-1-lp155.17.1.noarch
patterns-grommunio-1-lp155.9.1.x86_64
plymouth-theme-grommunio-1-lp155.17.1.noarch
systemd-coredump-grommunio-1-lp155.3.1.noarch
systemd-presets-branding-grommunio-2023.11-lp155.1.1.noarch
system-user-grommunio-3-lp155.10.6.noarch
system-user-gromox-2-lp155.15.1.noarch

I am more than happy to provide anything needed to help diagnosis and resolution. Hoping the Dev's can find and resolve the issue and therefore help myself, and the others who have contributed to this ticket to confirm it is not just me (ie possibly my system setup) get back to using Mac Mail as their main email client. Gr Web works well for me on the Mac, but Mac mail is just a bit more integrated into my workflow.

Keep up the good work.

Mark

Mister2

Great news - I have applied the updates made available today

Repository      | Name               | Current Version                                        | Available Version                                     | Arch
---------------+--------------+--------------------------------------+--------------------------------------+-------
grommunio       | gromox             | 2.21.29.g81a193a-lp155.20.1                 | 2.22.0.g25942bf-lp155.1.1                     | x86_64
grommunio       | gromox-debuginfo   | 2.21.29.g81a193a-lp155.20.1        | 2.22.0.g25942bf-lp155.1.1                     | x86_64
grommunio       | gromox-debugsource | 2.21.29.g81a193a-lp155.20.1     | 2.22.0.g25942bf-lp155.1.1                    | x86_64
repo-sle-update | kpartx             | 0.9.4+74+suse.f97cc59-150500.3.3.1 | 0.9.4+77+suse.1f7a63b-150500.3.6.1 | x86_64
repo-sle-update | libmpath0      | 0.9.4+74+suse.f97cc59-150500.3.3.1 | 0.9.4+77+suse.1f7a63b-150500.3.6.1 | x86_64

and testing shows that this issue 'Mac Mail not updating Grommunio properly on deleting/moving emails' now appears to be resolved. Deleting emails from any folder and moving emails from one folder to another now all appear to perform as they do in the other email clients. The Grommunio Web Client also now reflects these changes as it should. Excellent update

A great thank you to the Devs and the Grommunio Team from me, in getting this niggly little issue fixed, as I can now start using Mac Mail as my primary mail client.

Mark