Update issues on RHEL 9

Andrew

Latest round of updates failed with a few package-related issues. RHEL 9.5, latest updates, using the community repo. Looks like some changes on both the Red Hat and Grommunio ends have got in the way. All issues seem to be RHEL-specific.

Issue 1: Grommunio-Common / Redis

grommunio-common won't install due to a conflict with Redis:

Error: Transaction test error:
  file /etc/redis from install of grommunio-common-26.a6f127d-40.1.x86_64 conflicts with file from package redis-6.2.7-1.el9.x86_64

Using rpmrebuild to compare old (working) + new (not working) rpms, the problematic one appears to have an additional line:
%dir %attr(0755, root, root) "/etc/redis"
I assume that is causing the problem, but spec files aren't something I am particularly knowledgeable about.

Issue 2: NGINX

RHEL's version of NGINX has changed from 1:1.20.1-16 to 2:1.20.1-20.
nginx-module-brotli and nginx-module-vts in the Grommunio repo require 1:1.20.1 so the update will not install.
There is an alternative NGINX update stream for v1.24. The current version on that stream is 1:1.24.0-4, but it also does not meet the 1:1.20.1 requirement.
Example:

Error:
 Problem: problem with installed package nginx-module-vts-1:1.20.1.0.2.2-2.36.x86_64
  - package nginx-module-vts-1:1.20.1.0.2.2-2.36.x86_64 from @System requires nginx = 1:1.20.1, but none of the providers can be installed
  - package nginx-module-vts-1:1.20.1.0.2.2-2.36.x86_64 from grommunio requires nginx = 1:1.20.1, but none of the providers can be installed
  - cannot install both nginx-2:1.20.1-20.el9.x86_64 from rhel-9-for-x86_64-appstream-rpms and nginx-1:1.20.1-16.el9_4.1.x86_64 from @System
  - cannot install both nginx-2:1.20.1-20.el9.x86_64 from rhel-9-for-x86_64-appstream-rpms and nginx-1:1.20.1-10.el9.x86_64 from rhel-9-for-x86_64-appstream-rpms
  - cannot install both nginx-2:1.20.1-20.el9.x86_64 from rhel-9-for-x86_64-appstream-rpms and nginx-1:1.20.1-13.el9.x86_64 from rhel-9-for-x86_64-appstream-rpms
  - cannot install both nginx-2:1.20.1-20.el9.x86_64 from rhel-9-for-x86_64-appstream-rpms and nginx-1:1.20.1-14.el9.x86_64 from rhel-9-for-x86_64-appstream-rpms
  - cannot install both nginx-2:1.20.1-20.el9.x86_64 from rhel-9-for-x86_64-appstream-rpms and nginx-1:1.20.1-14.el9_2.1.x86_64 from rhel-9-for-x86_64-appstream-rpms
  - cannot install both nginx-2:1.20.1-20.el9.x86_64 from rhel-9-for-x86_64-appstream-rpms and nginx-1:1.20.1-16.el9_4.1.x86_64 from rhel-9-for-x86_64-appstream-rpms
  - cannot install the best update candidate for package nginx-1:1.20.1-16.el9_4.1.x86_64

Issue 3: SQLite

Not sure if this one is recent as I haven't had need to check for a while, but I noticed a lot of errors in the "Gromox http" log about SQLite schema upgrades failing with no such function: UNIXEPOCH.
Looks like the UNIXEPOCH function was added in SQLite 3.38 and RHEL 9 uses SQLite 3.34 so it doesn't have this function.
Example:

[2024-11-13 15:01:34.494016]: dbop_sqlite: /var/lib/gromox/user/4/2: current schema EV-14; upgrading to EV-17.
[2024-11-13 15:01:34.494099]: dbop_sqlite: upgrading /var/lib/gromox/user/4/2 to schema EV-15
[2024-11-13 15:01:34.494184]: sqlite3_exec(/var/lib/gromox/user/4/2/exmdb/exchange.sqlite3) "INSERT INTO allocated_eids (range_begin,range_end,allocate_time,is_system) VALUES ((SELECT MAX(range_end)+1 FROM allocated_eids), (SELECT MAX(range_end)+0x10000 FROM allocated_eids),UNIXEPOCH(),1)": no such function: UNIXEPOCH (1) at [<libgromox_common.so.0(_ZN6gromox11gx_sql_execEP7sqlite3PKcj+0xa4) [0x7f2a89d7ee4b]><libgromox_dbop.so.0(_ZN6gromox19dbop_sqlite_upgradeEP7sqlite3PKcNS_11sqlite_kindEj+0x430) [0x7f2a89039a97]><libgxs_exmdb_provider.so.0(+0x37c23) [0x7f2a8995ac23]><libgxs_exmdb_provider.so.0(+0x39149) [0x7f2a8995c149]><libgxs_exmdb_provider.so.0(+0x39a68) [0x7f2a8995ca68]><libgxs_exmdb_provider.so.0(+0xb3dde) [0x7f2a899d6dde]><libgxs_exmdb_provider.so.0(+0x55572) [0x7f2a89978572]><libgxs_exmdb_provider.so.0(+0x5a9d4) [0x7f2a8997d9d4]><libgxs_exmdb_provider.so.0(+0x5b095) [0x7f2a8997e095]><libgxs_exmdb_provider.so.0(+0x5d442) [0x7f2a89980442]><libc.so.6(+0x89d22) [0x7f2a88a89d22]><libc.so.6(+0x10ed40) [0x7f2a88b0ed40]>]
[2024-11-13 15:01:34.494251]: dbop_sqlite upgrade /var/lib/gromox/user/4/2: Input/output error
[2024-11-13 15:01:34.494436]: E-2105: autoupgrade /var/lib/gromox/user/4/2: -1

GeneralProbe

I consulted with my RHEL expert and he has (for now) an explanation for Issue 1.

"/etc/redis"

must not be packaged by Grommunio.

Andrew

A few notes from having a play with issues 1 and 3:

Issue 1: Grommunio-Common / Redis

Removing the offending line %dir %attr(0755, root, root) "/etc/redis" from grommunio-common using rpmrebuild allowed the resulting RPM to install fine.

Issue 3: SQLite

I compiled the latest version of SQLite from source, but instead of running make install I copied the resulting lib files to /opt/libsqlite3, then added an EnvironmentFile to a gromox-http.service override that sets LD_LIBRARY_PATH=/opt/libsqlite3. This had the intended effect of making the Gromox HTTP service load the updated version of SQLite from /opt/libsqlite3 when running Python, without messing with the distro Python. The log shows the schema upgrades having completed, with no more errors being logged.

Related Errors

I was seeing a lot of grommunio-index failures, and ActiveSync dropping out and requiring reauthentication. Both of these issues seem to be resolved after kludging in the updated SQLite library.

For now everything seems to be working (except the ability to install NGINX updates), although it feels like a kludge so I will happily revert to a more "supported" fix should it arrive later.

daniel

@Andrew Thanks for sharing your issues. I'm trying to upgrade my box to Rocky Linux 9.5 and I'm facing the nginx issue mentioned above. Is there a solution or a workaround?

`# dnf update
Error:
Problem 1: package nginx-module-brotli-1:1.0.0rc+g2-5.35.x86_64 from @System requires nginx = 1:1.20.1, but none of the providers can be installed

cannot install both nginx-2:1.20.1-20.el9.0.1.x86_64 from appstream and nginx-1:1.20.1-16.el9_4.1.x86_64 from @System
cannot install the best update candidate for package nginx-module-brotli-1:1.0.0rc+g2-5.35.x86_64
cannot install the best update candidate for package nginx-1:1.20.1-16.el9_4.1.x86_64
Problem 2: package nginx-1:1.20.1-16.el9_4.1.x86_64 from @System requires nginx-core = 1:1.20.1-16.el9_4.1, but none of the providers can be installed
package nginx-module-vts-1:1.20.1.0.2.2-2.36.x86_64 from @System requires nginx = 1:1.20.1, but none of the providers can be installed
cannot install both nginx-core-2:1.20.1-20.el9.0.1.x86_64 from appstream and nginx-core-1:1.20.1-16.el9_4.1.x86_64 from @System
cannot install the best update candidate for package nginx-module-vts-1:1.20.1.0.2.2-2.36.x86_64
cannot install the best update candidate for package nginx-core-1:1.20.1-16.el9_4.1.x86_64
Problem 3: problem with installed package nginx-module-vts-1:1.20.1.0.2.2-2.36.x86_64
package nginx-module-vts-1:1.20.1.0.2.2-2.36.x86_64 from @System requires nginx = 1:1.20.1, but none of the providers can be installed
package nginx-module-vts-1:1.20.1.0.2.2-2.36.x86_64 from grommunio requires nginx = 1:1.20.1, but none of the providers can be installed
package nginx-1:1.20.1-16.el9_4.1.x86_64 from @System requires nginx-filesystem = 1:1.20.1-16.el9_4.1, but none of the providers can be installed
cannot install both nginx-filesystem-2:1.20.1-20.el9.0.1.noarch from appstream and nginx-filesystem-1:1.20.1-16.el9_4.1.noarch from @System
cannot install the best update candidate for package nginx-filesystem-1:1.20.1-16.el9_4.1.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
`

Thanks & best regards
Daniel

daniel

Little addition - I found a RL forum thread:
https://bugs.rockylinux.org/view.php?id=8285

The epoch of the nginx packages have been changed. So the dependency of nginx-module-brotli should be updated also.

Andrew

My workaround for the NGINX issue for now:

Download the nginx-module-brotli and nginx-module-vts rpms from the Grommunio repo
Edit the rpm spec files using rpmrebuild -e -p downloadedfile.rpm
Replace the line Requires: nginx = 1:1.20.1 with Requires: (nginx = 1:1.20.1 or nginx = 2:1.20.1)
Reinstall nginx-module-brotli and nginx-module-vts using the modified rpm files sudo dnf reinstall ~/rpmbuild/RPMS/x86_64/editedfile.rpm
Now the installed packages will not conflict with the latest version of NGINX from RHEL/Rocky and updates can be installed as normal with dnf upgrade

It's a kludge, but I haven't encountered any problems thus far.

daniel

Thanks @Andrew!

I've opened an issue in Github right now:
https://github.com/grommunio/gromox/issues/113

Hope one of the devs will pick it soon. It's a very small change and might hopefully be implemented soon.

crpb

Thats why we in Debian build it ourself. Well since Bookworm one was added and my script only needs to take care of the traffic thingy.

Andrew

Issue 2B - Brotil/VTS

It looks like the newly updated Brotli/VTS modules that Grommunio has released to fix Issue 2 in this thread have themselves created a problem. After installing this update, NGINX will no longer start.
The module files look to have been moved from /usr/share/nginx/modules to /usr/lib64/nginx/modules in this latest update, but the main config file /etc/nginx/nginx.conf is still attempting to load these modules from the old location.
In the old /usr/share/nginx/modules location there are now a couple of conf files that reference the new location for these modules, so I believe the entries in /etc/nginx/nginx.conf are now redundant. After commenting them out, NGINX starts up fine.

daniel

I can confirm that the issue has been fixed. I also ran into the problem mentioned by Andrew and fixed it by commenting out the two lines in /etc/nginx/nginx.conf. The modules are loaded in an included file from the correct path.

Also another problem with /etc/redis path in grommunio-common has been fixed.

After these issues, Grommunio runs fine with RL 9.5.

Thanks @Andrew for sharing!

Andrew

Thanks daniel. Did you experience the Python/SQLite issue (Issue 3 in the original post)? Apart from creating error logs in the "Gromox http" log it seemed to cause issues with ActiveSync sessions dropping and needing to reauthenticate every few hours, but has been fine since the workaround of adding my own libraries to LD_LIBRARY_PATH.
I suspect that one would be a bit more effort to fix properly as it would require either a code change or Grommunio to distribute their own build of Python with an updated version of SQLite.

daniel

@Andrew Today I experienced some really strange behaviors with Thunderbird and imap. Folders aren't working as expected.
Coud be a sqlite issue:
sqlite3_exec(/var/lib/gromox/user/0/1/exmdb/midb.sqlite3) "UPDATE folders SET name=CONCAT('t',ROWID)": no such function: CONCAT (1) at [<libgromox_common.so.0(_ZN6gromox11gx_sql_execEP7sqlite3PKcj+0xa4) [0x7f1603144258]><midb(+0x14e53) [0x56376fa12e53]><midb(+0x1604a) [0x56376fa1404a]><midb(+0x1a248) [0x56376fa18248]><midb(+0xc97e) [0x56376fa0a97e]><midb(+0xc9d7) [0x56376fa0a9d7]><midb(+0xd058) [0x56376fa0b058]><libc.so.6(+0x89d22) [0x7f1602689d22]><libc.so.6(+0x10ed40) [0x7f160270ed40]>]

Currently I don't have a clue how to fix this

Andrew

daniel That sounds like a similar issue. Mine was with the UNIXEPOCH function that was added in SQLite 3.38. Yours seems to be with the CONCAT function that looks to have been added in SQLite 3.44. RHEL 9.5 uses SQLite 3.34 so neither of these functions are valid. Grommunio looks to have updated their code to rely on a version of SQLite not supported by RHEL and derivative distros.
Unfortunately, because SQLite is being called by Python, manually installing a later version of SQLite itself isn't sufficient; because the libraries Python uses are independent to the system version of SQLite and are compiled as part of Python itself. Messing with the system version of Python doesn't seem like a good idea given the number of things that depend on it, so the the workaround I used is as follows.

Download and build the latest version of SQLite from source:

curl https://www.sqlite.org/2024/sqlite-autoconf-3470200.tar.gz -o sqlite-autoconf-3470200.tar.gz
tar xvfz sqlite-autoconf-3470200.tar.gz
cd sqlite-autoconf-3470200
./configure
make

Copy the compiled libraries from the .libs subdirectory to one where they can be accessed by Grommunio:

mkdir /opt/libsqlite3
cp -P .libs/* /opt/libsqlite3
cp --remove-destination libsqlite3.la /opt/libsqlite3

You could also just cp .libs/* /opt/libsqlite3 but there are a few symlinks included that would result in duplicate files that way. I'm not sure if all of these files are required, but I copied them all to be on the safe side.

Create a file /opt/libsqlite3/env with content LD_LIBRARY_PATH=/opt/libsqlite3

Create an override for the gromox-http service with systemctl edit gromox-http.service and add the following between the start and end comments to have the service reference the newly created environment file.

[Service]
EnvironmentFile=/opt/libsqlite3/env

Save the override and restart gromox-http systemctl restart gromox-http. The service should now use the updated SQLite libraries created in /opt/libsqlite3.

This seems to work and I haven't yet encountered any issues within my use case (Outlook MAPI/HTTP, ActiveSync, Grommunio Web), but it does mean that any time an update is released for SQLite I'll have to go through the process to compile the new version all over again to keep it patched. I definitely put it into the category of a kludge, but it keeps my system stable for now. Hopefully Grommunio can work out a more permanent solution for the EL distros.

daniel

Thanks @Andrew, great workaround description.
As I hope the devs will find a permanent solution without self managed sqlite, I've filed an issue at Github:
https://github.com/grommunio/gromox/issues/114

mwilliams

daniel

Solved as per https://github.com/grommunio/gromox/issues/114

Andrew

Issue 2B Returns - Brotil/VTS

My server installed the latest set of updates last night and NGINX died again.
It looks like the Grommunio updates have re-added the faulty /etc/nginx/nginx.conf instructions to load the Brotil/VTS modules from /usr/share/nginx/modules instead of the /usr/lib64/nginx/modules location Grommunio has moved them to.
I have removed the load_module lines again, and NGINX is working, but if Grommunio keeps adding the faulty lines back in, it will only last until the next update.

GeneralProbe

@Andrew
you should not modify the service file itself.
Instead please create this folder:

/etc/systemd/system/gromox-http.service.d/

and then create a file (e.g.)

sqlite.conf

with the content:

[Service]
Environment=LD_LIBRARY_PATH=/opt/grommunio/lib64

(or where you placed it in the first place)

Andrew

GeneralProbe systemctl edit doesn't change the original service file. It creates an override file that takes precedence over the contents of the service file. In this case it would create /etc/systemd/system/gromox-http.service.d/override.conf. It's essentially the same thing as your suggestion, but a more official/supported method of creating the file.

daniel

@mwilliams I've applied the update but I'm still experiencing problems:

Dec 10 18:33:02 midb[1145473]: sqlite3_exec(/var/lib/gromox/user/0/1/exmdb/midb.sqlite3) "UPDATE folders SET name=CONCAT('t',ROWID)": no such function: CONCAT (1) at [<libgromox_common.so.0(_ZN6gromox11gx_sql_execEP7sqlite3PKcj+0xa4) [0x7fc454a63258]><midb(+0x14e53) [0x5604e51cce53]><midb(+0x1604a) [0x5604e51ce04a]><midb(+0x16358) [0x5604e51ce358]><midb(+0xc97e) [0x5604e51c497e]><midb(+0xc9d7) [0x5604e51c49d7]><midb(+0xd058) [0x5604e51c5058]><libc.so.6(+0x89d22) [0x7fc454089d22]><libc.so.6(+0x10ed40) [0x7fc45410ed40]>]
Maybe the imap error has something to do with german umlauts or ampersand in folder names. I have duplicate folders in the subscription list and greyed out folders in the current folder pane. All folders have umlauts or ampersand in its name.

daniel

@mwilliams Sorrry, just noticed I've applied gromox-2.38.4 to my box - .5 seems not to be available in the repo. I'll update here after updating.