Admin API fails to start after uwsgi upgrade (wrong plugin name in config)

Duese6

Summary:
After upgrading to the current grommunio Community packages (Leap 15.6), the grommunio-admin-api service fails to start due to an invalid uWSGI plugin name in the shipped configuration.

Steps to reproduce:

Update system with zypper up (switch to uwsgi-2.0.31 and python311-uwsgi-python3 from the grommunio repo).
Restart grommunio-admin-api service.
Observe service logs.

Observed behavior:

open("/usr/lib64/uwsgi/python3.11_plugin.so"): No such file or directory
!!! UNABLE to load uWSGI plugin: /usr/lib64/uwsgi/python3.11_plugin.so !!!

The file /usr/lib64/uwsgi/python311_plugin.so exists, but not python3.11_plugin.so.
As a result, the admin API fails to load any app and the web interface shows “Could not connect to backend”.

Cause:
The file /usr/share/grommunio-admin-api/api-config.ini (shipped with the package) contains

plugin = python3.11

but the correct plugin provided by python311-uwsgi-python3 is

/usr/lib64/uwsgi/python311_plugin.so

Workaround:
Edit /usr/share/grommunio-admin-api/api-config.ini and change

plugin = python3.11

plugin = python311

Then restart the service:

systemctl restart grommunio-admin-api

After this change, the admin API and web interface work normally.

Expected behavior:
The package should ship with the correct plugin name (python311) so that the service starts out of the box.

Suggested fix:

Update api-config.ini in the RPM to use plugin = python311, or

Add a symlink /usr/lib64/uwsgi/python3.11_plugin.so → python311_plugin.so in python311-uwsgi-python3.

Optionally, move api-config.ini to /etc/grommunio-admin-api/ as a %config file to prevent overwriting user edits.

Environment:

openSUSE Leap 15.6

grommunio-admin-api 1.17.32.021a23c-lp156.1.1

uwsgi 2.0.31-lp156.1.1

python311-uwsgi-python3 2.0.31-lp156.1.1

Duese6

Temporary workaround:
Creating a symbolic link also fixes ths issue immediately:

ln -sf /usr/lib64/uwsgi/python311_plugin.so /usr/lib64/uwsgi/python3.11_plugin.so
systemctl restart grommunio-admin-api

JuLeZ_cne

Hi,
In the log of grommunio-admin-api, I see this message:

mail:~ # journalctl -fu grommunio-admin-api
Nov 01 15:14:55 mail uwsgi[2682]: --- no python application found, check your startup logs for errors ---

When I check with Python, it seems that something is wrong with the version:

mail:~ # sudo zypper dist-upgrade
Repository data is being loaded...
Reading installed packages...
Warning: You are about to perform a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before continuing. For more information on this command, see 'man zypper'.
Processing distribution upgrades...

Problem: 1: the installed uwsgi-python3-2.0.19.1-150400.8.3.1.x86_64 requires 'uwsgi = 2.0.19.1', but this requirement cannot be provided

Unavailable providers: uwsgi-2.0.19.1-150400.6.11.x86_64[base]

Solution 1: Perform the following actions:
Deinstall grommunio-admin-api-1.17.32.021a23c-lp156.15.1.noarch
Deinstall patterns-grommunio-1-lp156.9.1.x86_64
Solution 2: Deinstall uwsgi-python3-2.0.19.1-150400.8.3.1.x86_64
Solution 3: Keep obsolete grommunio-admin-api-1.17.32.021a23c-lp156.15.1.noarch
Solution 4: Break uwsgi-python3-2.0.19.1-150400.8.3.1.x86_64 by ignoring some dependencies

Choose from the above solutions by number or cancel [1/2/3/4/a/d/?] (a):

As a result, I can no longer access the Admin UI – it only shows “Internal Server Error”.
Has anyone encountered this problem before and could help me with it?
I also tried updating to the Community version, but that didn’t work either.

Duese6

Possible fix (worked for me):

I ran into the same issue. The root cause was a mismatch between the old uwsgi-python3 (SUSE, 2.0.19.1) and the newer grommunio packages that use the Python 3.11 uWSGI plugin.

Align admin API to the current Community build line:

sudo zypper in --oldpackage grommunio-admin-api-1.17.32.021a23c-lp156.1.1.noarch

Install the new uWSGI + Python 3.11 plugin:

sudo zypper in --from grommunio --allow-vendor-change uwsgi=2.0.31-lp156.1.1 python311-uwsgi-python3=2.0.31-lp156.1.1

Fix the plugin name (package currently ships the wrong one).
Choose one of these:

Config fix (preferred):

sudo sed -ri 's/^{plugins?\s*=\s*}python3.11/\1python311/' /usr/share/grommunio-admin-api/api-config.ini

Symlink workaround (survives updates of the admin API):

sudo ln -sf /usr/lib64/uwsgi/python311_plugin.so /usr/lib64/uwsgi/python3.11_plugin.so

Restart services and test:

sudo systemctl restart grommunio-admin-api grommunio-api nginx
curl -sS http://127.0.0.1:8080/about || curl -k https://localhost/admin/api/about

After this, the Admin UI came back up for me.

(FYI for maintainers: api-config.ini currently contains plugin = python3.11, but the plugin file installed by python311-uwsgi-python3 is python311_plugin.so. Changing the config to plugin = python311 or providing a compatibility symlink would fix this.)

Duese6

Looks like this has now been fixed in version 1.17.32.021a23c-lp156.3.1.
Either someone noticed the mismatch or saw the discussion here 🙂

crpb

Duese6 Either someone noticed the mismatch or saw the discussion here 🙂

And probably here aswell 🙈

Duese6

just installed latest version of python311-systemd 234-lp156.4.1

Now I'm getting Bad-Gateway, when im clicking on Monitoring

Log shows

Nov 07 16:15:34 grommunio uwsgi[27833]: /usr/sbin/uwsgi(+0x82214) [0x5599df4aa214]
Nov 07 16:15:34 grommunio uwsgi[27833]: /usr/sbin/uwsgi(+0x2d58e) [0x5599df45558e]
Nov 07 16:15:34 grommunio uwsgi[27833]: /lib64/libc.so.6(+0x40e6c) [0x7f10daa40e6c]
Nov 07 16:15:34 grommunio uwsgi[27833]: /lib64/libc.so.6(__libc_start_main+0x87) [0x7f10daa40f35]
Nov 07 16:15:34 grommunio uwsgi[27833]: /usr/sbin/uwsgi(_start+0x21) [0x5599df4555b1]
Nov 07 16:15:34 grommunio uwsgi[27833]: *** end of backtrace ***
Nov 07 16:15:40 grommunio uwsgi[27676]: DAMN ! worker 1 (pid: 27833) died, killed by signal 11 🙁 trying respawn ...
Nov 07 16:15:40 grommunio uwsgi[27676]: Respawned uWSGI worker 1 (new pid: 27903)
Nov 07 16:15:44 grommunio uwsgi[27676]: DAMN ! worker 1 (pid: 27903) died, killed by signal 11 🙁 trying respawn ...
Nov 07 16:15:44 grommunio uwsgi[27676]: Respawned uWSGI worker 1 (new pid: 27941)

Mister2

Duese6 Confirm same behaviour on my server with the python311-systemd 234-lp156.4.1 update. No change post reboot either.

Mister2

Today's updates:-
grommunio-admin-common-44.9fd4fb3-lp156.47.1.noarch
grommunio-common-29.3-lp156.5.1.x86_64
uwsgi-2.0.31-lp156.1.2.x86_64
python311-uwsgi-python3-2.0.31-lp156.1.2.x86_64
grommunio-admin-api-1.17.33.0a2d18a-lp156.6.1.noarch

have made no change. Still have to run systemctl restart grommunio_admin_api after a reboot to allow login to Admin GUI, and selecting monitoring in the GUI still brings up the 'Bad Gateway' error on all log types.

WalterH

Ticket#814236

Duese6

After updates this morning Admin-API seems to be down again....

at the moment I have no idea what is going on again.

log shows only this

Nov 14 09:55:10 grommunio uwsgi[1397]: [ERROR] (grommunio Admin API) Traceback (most recent call last):
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request
Nov 14 09:55:10 grommunio uwsgi[1397]: rv = self.dispatch_request()
Nov 14 09:55:10 grommunio uwsgi[1397]: ^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request
Nov 14 09:55:10 grommunio uwsgi[1397]: return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
Nov 14 09:55:10 grommunio uwsgi[1397]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/share/grommunio-admin-api/api/core.py", line 204, in wrapper
Nov 14 09:55:10 grommunio uwsgi[1397]: return call()
Nov 14 09:55:10 grommunio uwsgi[1397]: ^^^^^^
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/share/grommunio-admin-api/api/core.py", line 169, in call
Nov 14 09:55:10 grommunio uwsgi[1397]: ret = func(*args, **kwargs)
Nov 14 09:55:10 grommunio uwsgi[1397]: ^^^^^^^^^^^^^^^^^^^^^
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/share/grommunio-admin-api/endpoints/misc.py", line 43, in login
Nov 14 09:55:10 grommunio uwsgi[1397]: success, val = loginUser(request.form["user"], request.form["pass"])
Nov 14 09:55:10 grommunio uwsgi[1397]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/share/grommunio-admin-api/api/security.py", line 208, in loginUser
Nov 14 09:55:10 grommunio uwsgi[1397]: user: Users = Users.query.join(Altnames, isouter=True)\
Nov 14 09:55:10 grommunio uwsgi[1397]: ^^^^^^^^^^^
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/share/grommunio-admin-api/orm/init.py", line 38, in get
Nov 14 09:55:10 grommunio uwsgi[1397]: mapper = class_mapper(type)
Nov 14 09:55:10 grommunio uwsgi[1397]: ^^^^^^^^^^^^^^^^^^
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/base.py", line 562, in class_mapper
Nov 14 09:55:10 grommunio uwsgi[1397]: mapper = inspect_mapped_class(class, configure=configure)
Nov 14 09:55:10 grommunio uwsgi[1397]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/base.py", line 533, in inspect_mapped_class
Nov 14 09:55:10 grommunio uwsgi[1397]: mapper.check_configure()
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/mapper.py", line 2386, in _check_configure
Nov 14 09:55:10 grommunio uwsgi[1397]: _configure_registries({self.registry}, cascade=True)
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/mapper.py", line 4198, in _configure_registries
Nov 14 09:55:10 grommunio uwsgi[1397]: _do_configure_registries(registries, cascade)
Nov 14 09:55:10 grommunio uwsgi[1397]: File "/usr/lib64/python3.11/site-packages/sqlalchemy/orm/mapper.py", line 4235, in _do_configure_registries
Nov 14 09:55:10 grommunio uwsgi[1397]: raise e
Nov 14 09:55:10 grommunio uwsgi[1397]: sqlalchemy.exc.InvalidRequestError: One or more mappers failed to initialize - can't proceed with initialization of other mappers. Triggering mapper: 'Mapper[MLists(mlists)]'. Original exception was: reverse_property 'mlist' on relationship MLists.user references relationship Users.mlist, which does not reference mapper Mapper[MLists(mlists)]

Duese6

Okay, it looks like the startup time behavior has changed for some reason. MariaDB starts too late. This “problem” has been discussed many times before. In this case, I used the ovrride.conf workaround.

Duese6

Hi all,

Quick update from my side:
I’m now running
grommunio-admin-api 1.17.33.0a2d18a-lp156.7.1 (uwsgi remains 2.0.31, python311-systemd version unchanged).

The previous segfault that occurred when clicking Monitoring → Logs is no longer happening — good news.
However, the monitoring log tab still fails: when I click e.g. “Admin API”, “Gromox midb” or “Gromox delivery”, I get a 500 error.

The journal shows:

Nov 25 13:33:45 grommunio uwsgi[16239]: [ERROR] (grommunio Admin API) Traceback (most recent call last):
Nov 25 13:33:45 grommunio uwsgi[16239]: File "/usr/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request
Nov 25 13:33:45 grommunio uwsgi[16239]: rv = self.dispatch_request()
Nov 25 13:33:45 grommunio uwsgi[16239]: ^^^^^^^^^^^^^^^^^^^^^^^
Nov 25 13:33:45 grommunio uwsgi[16239]: File "/usr/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request
Nov 25 13:33:45 grommunio uwsgi[16239]: return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
Nov 25 13:33:45 grommunio uwsgi[16239]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 25 13:33:45 grommunio uwsgi[16239]: File "/usr/share/grommunio-admin-api/api/core.py", line 204, in wrapper
Nov 25 13:33:45 grommunio uwsgi[16239]: return call()
Nov 25 13:33:45 grommunio uwsgi[16239]: ^^^^^^
Nov 25 13:33:45 grommunio uwsgi[16239]: File "/usr/share/grommunio-admin-api/api/core.py", line 169, in call
Nov 25 13:33:45 grommunio uwsgi[16239]: ret = func(args, **kwargs)
Nov 25 13:33:45 grommunio uwsgi[16239]: ^^^^^^^^^^^^^^^^^^^^^
Nov 25 13:33:45 grommunio uwsgi[16239]: File "/usr/share/grommunio-admin-api/endpoints/system/logs.py", line 35, in getLog
Nov 25 13:33:45 grommunio uwsgi[16239]: return jsonify(data=LogReader.tail(log.get("format", "journald"), log["source"], n, skip, after))
Nov 25 13:33:45 grommunio uwsgi[16239]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 25 13:33:45 grommunio uwsgi[16239]: File "/usr/share/grommunio-admin-api/tools/logs.py", line 59, in tail
Nov 25 13:33:45 grommunio uwsgi[16239]: return cls.rregsource.tail(args, **kwargs)
Nov 25 13:33:45 grommunio uwsgi[16239]: ^^^^^^^^^^^^^^^^^^^^^^^^
Nov 25 13:33:45 grommunio uwsgi[16239]: File "/usr/share/grommunio-admin-api/tools/logs.py", line 75, in init
Nov 25 13:33:45 grommunio uwsgi[16239]: self.reader.add_match(_SYSTEMD_UNIT=unit)
Nov 25 13:33:45 grommunio uwsgi[16239]: File "/usr/lib64/python3.11/site-packages/systemd/journal.py", line 244, in add_match
Nov 25 13:33:45 grommunio uwsgi[16239]: super(Reader, self).add_match(arg)
Nov 25 13:33:45 grommunio uwsgi[16239]: ValueError: Invalid match

Duese6

Has anyone ever seen such ERROR/WARNINGS when clicking on "Mobile Devices" ?

Nov 27 11:25:30 grom uwsgi[5847]: [ERROR] (grommunio Admin API) Response validation failed: [InvalidSchemaValue(value={'data': [{'addinfo': 'OK : lifetime 180s', 'asversion': '16.1', 'c>
Nov 27 11:25:30 grom uwsgi[5847]: [WARNING] (grommunio Admin API) GET /api/v1/system/sync/top?filterEnded=20&filterUpdated=120 from 192.168.167.2 -> 500 '{"message":"The server generate>
Nov 27 11:25:32 grom uwsgi[5847]: [ERROR] (grommunio Admin API) Response validation failed: [InvalidSchemaValue(value={'data': [{'addinfo': 'OK : lifetime 180s', 'asversion': '16.1', 'c>
Nov 27 11:25:32 grom uwsgi[5847]: [WARNING] (grommunio Admin API) GET /api/v1/system/sync/top?filterEnded=20&filterUpdated=120 from 192.168.167.2 -> 500 '{"message":"The server generate>
Nov 27 11:25:34 grom uwsgi[5847]: [ERROR] (grommunio Admin API) Response validation failed: [InvalidSchemaValue(value={'data': [{'addinfo': 'OK : lifetime 180s', 'asversion': '16.1', 'c>
Nov 27 11:25:34 grom uwsgi[5847]: [WARNING] (grommunio Admin API) GET /api/v1/system/sync/top?filterEnded=20&filterUpdated=120 from 192.168.167.2 -> 500 '{"message":"The server generate>
Nov 27 11:25:36 grom uwsgi[5847]: [ERROR] (grommunio Admin API) Response validation failed: [InvalidSchemaValue(value={'data': [{'addinfo': 'OK : lifetime 180s', 'asversion': '16.1', 'c>
Nov 27 11:25:36 grom uwsgi[5847]: [WARNING] (grommunio Admin API) GET /api/v1/system/sync/top?filterEnded=20&filterUpdated=120 from 192.168.167.2 -> 500 '{"message":"The server generate>
Nov 27 11:25:38 grom uwsgi[5847]: [ERROR] (grommunio Admin API) Response validation failed: [InvalidSchemaValue(value={'data': [{'addinfo': 'OK : lifetime 180s', 'asversion': '16.1', 'c>
Nov 27 11:25:38 grom uwsgi[5847]: [WARNING] (grommunio Admin API) GET /api/v1/system/sync/top?filterEnded=20&filterUpdated=120 from 192.168.167.2 -> 500 '{"message":"The server generate>

It seems to be only when an Android Client is syncing.

grommunio-admin-api
1.17.33.0a2d18a

grommunio-sync
2.3.5.306a709

gromox-3.2.2