Update issues on RHEL 9

AAndrew · Dec 24, 2024

FYI the NGINX config issue is still happening. I updated again last night and it re-added the bad config lines.

Ddaniel · Dec 25, 2024

Grommunio seems to ship an updated version of sqlite that breaks dnf.
Anyone else affected?

bye
Daniel

AAndrew · Dec 25, 2024

daniel just checked, and yes - DNF is broken on my machine after installing a version of SQLite from Grommunio. The offending package appears to be sqlite-libs. If I append --exclude=sqlite-libs then DNF commands succeed and I can install updates.

Ddaniel · Dec 26, 2024

@Andrew Yeah, I downgraded, too and it seems to work.
I was afraid that the sqlite error above (missing CONCAT function) will be reintroduced again, but gromox still seems to use the old statements.

Yesterday I created an issue on Github, if you'd like to follow:
https://github.com/grommunio/gromox/issues/118

AAndrew · Dec 31, 2024

Looks like the SQLite packages have been pulled from the Grommunio repo, but the current version of grommunio-index is dependent on them so dnf upgrade fails unless that package is excluded. I reverted back to the RHEL SQLite packages but started getting CONCAT errors and a crashing index, so I've went back to my workaround of compiling my own version of SQLite libraries, and applied the override to both the gromox-http and grommunio-index services.

Ddaniel · 3 Jan

Andrew Created https://github.com/grommunio/gromox/issues/121 today regarding the nginx.conf modification.

AAndrew · 14 Jan

Installed latest updates. Seems like it's still breaking NGINX, and grommunio-index won't update because of a broken dependency on the deleted sqlite-libs package.

AAndrew · 17 Jan

Looks like the grommunio-index dependency is now fixed. I removed my custom SQLite override on the gromox-http and grommunio-index services yesterday and haven't had any issues so far.

AAndrew · 28 Jan

SQLite issues seem to be resolved, but the NGINX config is still getting mangled by updates

WalterH · 28 Jan

Andrew but the NGINX config is still getting mangled by updates

Please explain (with examples) what you mean with: but the NGINX config is still getting mangled by updates ?

crpb · 28 Jan

Andrew
The only one outstanding I believe now is the rogue load_module lines getting added to NGINX. I updated again last night and they got added back in, breaking NGINX once more. Deleting these lines (they're trying to load the modules from the original /usr/share/nginx/modules location but Grommunio moved them to /usr/lib64/nginx/modules) fixes it temporarily, but every time an update gets installs it re-adds the offending lines and breaks NGINX again.
> load_module modules/ngx_http_brotli_static_module.so;
> load_module modules/ngx_http_brotli_filter_module.so;
> load_module modules/ngx_http_vhost_traffic_status_module.so;

i believe they mean those which of course get reset by the packages config files.
can't you set some hook in DNF? to run like a sed over a file if it's beein updated by a package?

WalterH · 28 Jan

Are this modules installed on your system? I guess not and nginx refuses to load about missing modules.

If the grommunio developers wants theses modules, the system needs the modules. Try to install these modules and see what's happen?

AAndrew · 28 Jan

WalterH it's referenced a few times above. Every time an update gets installs it adds the following lines to the NGINX config that are referencing the wrong location, which causes NGINX to crash on start-up. Removing the offending lines resolves the issue temporarily, but every time a grommunio-common update installs it breaks NGINX again.

load_module modules/ngx_http_brotli_static_module.so;
load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_vhost_traffic_status_module.so;

@daniel created an issue on GitHub for it: https://github.com/grommunio/gromox/issues/121

AAndrew · 28 Jan

WalterH messages crossed in the ether.
The modules are installed and working, but during a previous update Grommunio moved them from /usr/share/nginx/modules to /usr/lib64/nginx/modules. They are correctly loading from /usr/lib64/nginx/modules, but the rogue lines that keep getting added to NGINX config are trying to load them from the old location as well, where they do not exist.

AAndrew · 28 Jan

Recreating the problem, showing the problematic load_module lines coming from grommunio-common:

[user@host ~]$ sudo head /etc/nginx/nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;

error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

[user@host ~]$ sudo dnf reinstall grommunio-common -y
Updating Subscription Management repositories.
Last metadata expiration check: 2:27:50 ago on Tue 28 Jan 2025 09:29:54 GMT.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                           Architecture                            Version                                          Repository                                  Size
=============================================================================================================================================================================================
Reinstalling:
 grommunio-common                                  x86_64                                  26.a6f127d-44.3                                  grommunio                                   14 k

Transaction Summary
=============================================================================================================================================================================================

Total download size: 14 k
Installed size: 8.6 k
Downloading Packages:
grommunio-common-26.a6f127d-44.3.x86_64.rpm                                                                                                                   80 kB/s |  14 kB     00:00    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                         78 kB/s |  14 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                     1/1 
  Reinstalling     : grommunio-common-26.a6f127d-44.3.x86_64                                                                                                                             1/2 
  Running scriptlet: grommunio-common-26.a6f127d-44.3.x86_64                                                                                                                             1/2 
  Cleanup          : grommunio-common-26.a6f127d-44.3.x86_64                                                                                                                             2/2 
  Verifying        : grommunio-common-26.a6f127d-44.3.x86_64                                                                                                                             1/2 
  Verifying        : grommunio-common-26.a6f127d-44.3.x86_64                                                                                                                             2/2 
Installed products updated.

Reinstalled:
  grommunio-common-26.a6f127d-44.3.x86_64                                                                                                                                                    

Complete!
[user@host ~]$ sudo head /etc/nginx/nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
load_module modules/ngx_http_brotli_static_module.so;
load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_vhost_traffic_status_module.so;

[user@host ~]$

The modules themselves are getting loaded fine from their new locations. Grommunio added config files in /usr/share/nginx/modules that take care of it when they moved the modules:

[user@host ~]$ cat /usr/share/nginx/modules/mod-brotli.conf
load_module "/usr/lib64/nginx/modules/ngx_http_brotli_filter_module.so";
load_module "/usr/lib64/nginx/modules/ngx_http_brotli_static_module.so";
[user@host ~]$ cat /usr/share/nginx/modules/mod-vhost-traffic-status.conf
load_module "/usr/lib64/nginx/modules/ngx_http_vhost_traffic_status_module.so";

It's just the load_module lines getting added to nginx.conf directly that cause the problem.

WalterH · 28 Jan

I looked in some of the packages in EL9 repository, unfortunately I did not find the nginx.conf file in these packages.

I searched in: gromox, grommunio-dav, grommunio-common, grommunio-admin-web, grommunio-admin-common, grommunio-web, grommunio-sync and grommunio-common but did not find the nginx.conf.

Do you know which package overwrites the nginx.conf file?

AAndrew · 28 Jan

WalterH it looks to be grommunio-common.
In my last reply to this thread I've copy-pasted an example of the terminal output from checking nginx.conf (no load_module lines in the "head" output at this point), reinstalling grommunio-common, then re-checking nginx.conf. After installing grommunio-common the offending lines reappear.

WalterH · 28 Jan

Interestingly, in grommunio-common-26.a6f127d-44.3.x86_64.rpm there is no nginx.conf copied to /etc/nginx. The question is, are there any scripts running that might modify nginx.conf or restore an old backup after installing grommunio-common?

AAndrew · 28 Jan

Looking at the spec file for grommunio-common using rpmrebuild, this part looks like the issue:

if [ -f "/etc/nginx/nginx.conf" ]; then
        if ! grep -q "^load_module modules/ngx_http_vhost_traffic_status_module.so;" "/etc/nginx/nginx.conf"; then
                sed -i 's/\(worker_processes.*\)/\1\nload_module modules\/ngx_http_vhost_traffic_status_module.so;/' "/etc/nginx/nginx.conf"
        fi
    if ! grep -q "^load_module modules/ngx_http_brotli_filter_module.so;" "/etc/nginx/nginx.conf"; then
                sed -i 's/\(worker_processes.*\)/\1\nload_module modules\/ngx_http_brotli_filter_module.so;/' "/etc/nginx/nginx.conf"
        fi
        if ! grep -q "^load_module modules/ngx_http_brotli_static_module.so;" "/etc/nginx/nginx.conf"; then
                sed -i 's/\(worker_processes.*\)/\1\nload_module modules\/ngx_http_brotli_static_module.so;/' "/etc/nginx/nginx.conf"
        fi
fi

This was looking at the grommunio-common-26.a6f127d-44.3.x86_64.rpm package from the repo using rpmrebuild -e -p grommunio-common-26.a6f127d-44.3.x86_64.rpm

WalterH · 28 Jan

OK, it is a script.
Unfortunately I do not have a RHEL system to test.