After upgrade, two issue's occur with Grommunio Admin Interface

bsondermeijer

Hi All,

I'm using the Grommunio Appliance based on OpenSuse.
And it works like a charm, however, I just updated to the latest version and since then the Admin UI wouldn't let me login. So after a restart with "systemctl restart grommunio-admin-api", I was able to login again.
This seems related to the fact that MySQL database is not started and the grommunio-admin-api tries to connect.

This shows up in the journal logging, with a

journalctl -u grommunio-admin-api | grep 'Database connection failed'

You see this just after a reboot.

Database connection failed with error 2002: Can't connect to local MySQL server through socket '/run/mysql/mysql.sock' (2)

Then again with a

systemctl restart grommunio-admin-api

this error is gone and all starts to work fine.

However one more issue occurs after restart of grommunio-admin-api.
Under the Monitor option trying to open a log gives an error "The server encountered an error while processing the request".

With a

journalctl -u grommunio-admin-api

I get this:

uwsgi[8775]: [ERROR] (grommunio Admin API) Traceback (most recent call last):
uwsgi[8775]:   File "/usr/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request
uwsgi[8775]:     rv = self.dispatch_request()
uwsgi[8775]:          ^^^^^^^^^^^^^^^^^^^^^^^
uwsgi[8775]:   File "/usr/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request
uwsgi[8775]:     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
uwsgi[8775]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
uwsgi[8775]:   File "/usr/share/grommunio-admin-api/api/core.py", line 204, in wrapper
uwsgi[8775]:     return call()
uwsgi[8775]:            ^^^^^^
uwsgi[8775]:   File "/usr/share/grommunio-admin-api/api/core.py", line 169, in call
uwsgi[8775]:     ret = func(*args, **kwargs)
uwsgi[8775]:           ^^^^^^^^^^^^^^^^^^^^^
uwsgi[8775]:   File "/usr/share/grommunio-admin-api/endpoints/system/logs.py", line 35, in getLog
uwsgi[8775]:     return jsonify(data=LogReader.tail(log.get("format", "journald"), log["source"], n, skip, after))
uwsgi[8775]:                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
uwsgi[8775]:   File "/usr/share/grommunio-admin-api/tools/logs.py", line 59, in tail
uwsgi[8775]:     return cls.rreg[source](target).tail(*args, **kwargs)
uwsgi[8775]:            ^^^^^^^^^^^^^^^^^^^^^^^^
uwsgi[8775]:   File "/usr/share/grommunio-admin-api/tools/logs.py", line 75, in __init__
uwsgi[8775]:     self.reader.add_match(_SYSTEMD_UNIT=unit)
uwsgi[8775]:   File "/usr/lib64/python3.11/site-packages/systemd/journal.py", line 244, in add_match
uwsgi[8775]:     super(Reader, self).add_match(arg)
uwsgi[8775]: SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
uwsgi[8775]: [WARNING] (grommunio Admin API) GET /api/v1/system/logs/Gromox midb?n=100 from

The error "SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats" indicates that an obsolete(?) pycrypto library is used.
According to Stackoverflow https://stackoverflow.com/questions/70705404/systemerror-py-ssize-t-clean-macro-must-be-defined-for-formats,
a library pycryptodome should be used instead of pycrypto.

Is there a solution for both issue's ?

umgfoin

bsondermeijer a library pycryptodome should be used instead of pycrypto.

IMO, this is incorrect - that's another issue (based on the same cause).

WalterH

zypper ref and zypper dup solved this on one system by downgrading some packages.

FortyTwo

WalterH Thank you very much! This solved the login-problem for the admin-interface!

Mister2

@bsondermeijer Same issue here (Grommunio CE Appliance - OpenSuse install fully updated), although I haven't at this time checked journalctl output, but I am unable to view any logs in the Admin GUI. Also noted pop3 service had stopped and would not restart via Admin GUI or after reboot.

I have just installed todays updates:-

gromox 3.1.96.g5d1e4af-lp156.30.1
gromox-debuginfo 3.1.96.g5d1e4af-lp156.30.1
gromox-debugsource 3.1.96.g5d1e4af-lp156.30.1
libtiff6 4.7.1-150600.3.23.1

and can report no change in access to logs via AdminGUI, however, after a reboot the pop3 service is now working again.

FortyTwo

Regarding the second problem bsondermeijer encountered:

I can reproduce the problem when Looking at Monitoring->Logging. Whenever I try to access a log-file, I get:

Nov` 06 16:05:17 mail uwsgi[1346]: [ERROR] (grommunio Admin API) Traceback (most recent call last):
Nov 06 16:05:17 mail uwsgi[1346]: File "/usr/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request
Nov 06 16:05:17 mail uwsgi[1346]: rv = self.dispatch_request()
Nov 06 16:05:17 mail uwsgi[1346]: ^^^^^^^^^^^^^^^^^^^^^^^
Nov 06 16:05:17 mail uwsgi[1346]: File "/usr/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request
Nov 06 16:05:17 mail uwsgi[1346]: return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
Nov 06 16:05:17 mail uwsgi[1346]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 06 16:05:17 mail uwsgi[1346]: File "/usr/share/grommunio-admin-api/api/core.py", line 204, in wrapper
Nov 06 16:05:17 mail uwsgi[1346]: return call()
Nov 06 16:05:17 mail uwsgi[1346]: ^^^^^^
Nov 06 16:05:17 mail uwsgi[1346]: File "/usr/share/grommunio-admin-api/api/core.py", line 169, in call
Nov 06 16:05:17 mail uwsgi[1346]: ret = func(args, **kwargs)
Nov 06 16:05:17 mail uwsgi[1346]: ^^^^^^^^^^^^^^^^^^^^^
Nov 06 16:05:17 mail uwsgi[1346]: File "/usr/share/grommunio-admin-api/endpoints/system/logs.py", line 35, in getLog
Nov 06 16:05:17 mail uwsgi[1346]: return jsonify(data=LogReader.tail(log.get("format", "journald"), log["source"], n, skip, after))
Nov 06 16:05:17 mail uwsgi[1346]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 06 16:05:17 mail uwsgi[1346]: File "/usr/share/grommunio-admin-api/tools/logs.py", line 59, in tail
Nov 06 16:05:17 mail uwsgi[1346]: return cls.rregsource.tail(args, **kwargs)
Nov 06 16:05:17 mail uwsgi[1346]: ^^^^^^^^^^^^^^^^^^^^^^^^
Nov 06 16:05:17 mail uwsgi[1346]: File "/usr/share/grommunio-admin-api/tools/logs.py", line 75, in init
Nov 06 16:05:17 mail uwsgi[1346]: self.reader.add_match(_SYSTEMD_UNIT=unit)
Nov 06 16:05:17 mail uwsgi[1346]: File "/usr/lib64/python3.11/site-packages/systemd/journal.py", line 244, in add_match
Nov 06 16:05:17 mail uwsgi[1346]: super(Reader, self).add_match(arg)
Nov 06 16:05:17 mail uwsgi[1346]: SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
Nov 06 16:05:17 mail uwsgi[1346]: [WARNING] (grommunio Admin API) GET /api/v1/system/logs/Admin API?n=100 from x.x.x.x -> 500 '{"message":"The server encountered an error while processing the request."}\n'

I did a zypper ref and zypper dup before.

The installed versions are:
grommunio-cui-1.1.3.b0091e0-lp156.6.1.noarch
grommunio-dav-1.5.17.c1122c5-lp156.1.1.noarch
grommunio-admin-common-44.9fd4fb3-lp156.45.1.noarch
gromox-debugsource-3.1.96.g5d1e4af-lp156.30.1.x86_64
grommunio-web-3.15.54.g76374b1d0-lp156.348.2.noarch
grommunio-release-2025.1.2-lp156.3.1.x86_64
grommunio-antispam-3.12.1-lp156.1.1.x86_64
gromox-debuginfo-3.1.96.g5d1e4af-lp156.30.1.x86_64
grommunio-keycloak-26.3.4-lp156.2.1.noarch
grommunio-imapsync-2.290-lp156.1.1.noarch
grommunio-dbconf-1.1.1.da20a46-lp156.13.1.x86_64
grommunio-setup-1.1.14.768d394-lp156.71.1.noarch
grommunio-sync-2.3.5.306a709-lp156.18.1.noarch
grommunio-admin-api-1.17.32.021a23c-lp156.4.1.noarch
grommunio-auth-0.2.24.0555de0-lp156.43.1.noarch
grommunio-error-pages-1.0.10.bb2df37-lp156.22.1.noarch
grommunio-common-29.0-lp156.2.1.x86_64
gromox-3.1.96.g5d1e4af-lp156.30.1.x86_64
grommunio-index-1.3.0.g2bd8a8c-lp156.15.2.x86_64
grommunio-admin-web-4.0.0.21.0ad253e-lp156.144.1.noarch

bsondermeijer

Hi All,

Ah with a:

grommunio-update check [upl-image-preview uuid=b3957c4d-5451-4347-af26-6c4a04c4696c

grommunio-update update

Thanks for that suggestion.

The login issue is fixed, however the Monitoring option -> one of the logs still give an error.

I suspect the pycrypto library, which seems to be obsolete and replaced by pycryptodome?

umgfoin

I suspect the pycrypto library, which seems to be obsolete and replaced by pycryptodome?

What makes you consider this? None of them are dependencies of admin-api.
Do you have a stacktrace of the error?

Does the failing log contain blob-entries in the journal?
I observed an admin-api issue comparable to yours with processing journal-logged blob-data. Such kind of entries can be logged e.g. by gromox-imap as a result of certain SSL-targeted attacks or authentication-aborts.

Truncating the log would resolve the issue, then.

dneuhaeuser

after recent updates from supported repo, I get a "bad gateway" error while trying to open any log in admin GUI.

for every try I'm also seeing these messages in journalctl:

Nov 14 01:27:10 grommunio uwsgi[13175]: !!! uWSGI process 13175 got Segmentation Fault !!!
Nov 14 01:27:10 grommunio uwsgi[13175]: *** backtrace of 13175 ***
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(uwsgi_backtrace+0x2a) [0x5625978adf6a]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(uwsgi_segfault+0x23) [0x5625978ae353]
Nov 14 01:27:10 grommunio uwsgi[13175]: /lib64/libc.so.6(+0x57900) [0x7f0ef0657900]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libsystemd.so.0(sd_journal_add_match+0x64) [0x7f0eebd34c24]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/python3.11/site-packages/systemd/_reader.cpython-311-x86_64-linux-gnu.so(+0x44b8) [0x7f0eee43f4b8]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(+0x2414cb) [0x7f0ef02414cb]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyObject_MakeTpCall+0x160) [0x7f0ef0212df0]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0xab1) [0x7f0ef01f5391]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(+0x114dfd) [0x7f0ef0114dfd]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyObject_FastCallDictTstate+0xa5) [0x7f0ef02415c5]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(+0x247a8f) [0x7f0ef0247a8f]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(+0x218ab1) [0x7f0ef0218ab1]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyObject_MakeTpCall+0x160) [0x7f0ef0212df0]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0xab1) [0x7f0ef01f5391]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(+0x114dfd) [0x7f0ef0114dfd]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyObject_Call+0x14f) [0x7f0ef0200a2f]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x4b16) [0x7f0ef01f93f6]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(+0x114dfd) [0x7f0ef0114dfd]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyObject_Call+0x14f) [0x7f0ef0200a2f]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x4b16) [0x7f0ef01f93f6]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(+0x114dfd) [0x7f0ef0114dfd]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyObject_FastCallDictTstate+0xa5) [0x7f0ef02415c5]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(+0x247c3b) [0x7f0ef0247c3b]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/libpython3.11.so.1.0(_PyObject_Call+0xc0) [0x7f0ef02009a0]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/uwsgi/python311_plugin.so(python_call+0xf) [0x7f0ef099df0f]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/lib64/uwsgi/python311_plugin.so(uwsgi_request_wsgi+0x25b) [0x7f0ef09a043b]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(wsgi_req_recv+0xa0) [0x56259785df80]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(simple_loop_run+0xcc) [0x5625978a9d7c]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(simple_loop+0x10) [0x5625978a9ba0]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(uwsgi_ignition+0x291) [0x5625978ae6a1]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(uwsgi_worker_run+0x293) [0x5625978b1c73]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(+0x82214) [0x5625978b2214]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(+0x2d58e) [0x56259785d58e]
Nov 14 01:27:10 grommunio uwsgi[13175]: /lib64/libc.so.6(+0x40e6c) [0x7f0ef0640e6c]
Nov 14 01:27:10 grommunio uwsgi[13175]: /lib64/libc.so.6(__libc_start_main+0x87) [0x7f0ef0640f35]
Nov 14 01:27:10 grommunio uwsgi[13175]: /usr/sbin/uwsgi(_start+0x21) [0x56259785d5b1]
Nov 14 01:27:10 grommunio uwsgi[13175]: *** end of backtrace ***
Nov 14 01:27:11 grommunio uwsgi[12977]: DAMN ! worker 1 (pid: 13175) died, killed by signal 11 :( trying respawn ...
Nov 14 01:27:11 grommunio uwsgi[12977]: Respawned uWSGI worker 1 (new pid: 14526)

WalterH

As a work around, use the console to view the logs. There is a ticket open for this problem.

radza

Das Problem mit der Admin API ist bei mir eine klassische Race Condition gewesen..

Ich konnte es lösen indem ich eine override.conf in /etc/systemd/system/grommunio-admin-api.service.d/override.conf angelegt habe

Inhalt:

[Unit]
After=mariadb.service
Requires=mariadb.service

Danach mehrere Reboots. Problem trat nicht wieder auf.

WalterH

So wie ich das inzwischen sehe, gibt es 2 Probleme, Das eine ist eine Race Condition, das andere hat mit den Python 3 bzw. Python 3.11 Modulen zu tun. Das untersuche ich noch.

dneuhaeuser

just did a fresh install for a new customer and updated to current community packages.
all logs in admin GUI are working again.

grommunio-admin-api 1.17.33.0a2d18a-lp156.7.1

bsondermeijer

After updating to the lastest version the error from the monitoring option is gone, and from the admin interface it's possible to check logs again.

I did:

grommunio-update check
grommunio-update update

After these commands I got this version list. Zo this issue is solved, thanks for support.

Reading installed packages...
S | Repository | Name ---+------------+--------------- v | sle-update | glib2-tools v | grommunio | grommunio-admin-web v | grommunio | grommunio-chat v | grommunio | grommunio-office v | grommunio | grommunio-office-fonts v | grommunio | grommunio-web v | grommunio | gromox v | grommunio | gromox-debuginfo v | grommunio | gromox-debugsource v | sle-update | kmod v | sle-update | libgio-2_0-0 v | sle-update | libglib-2_0-0 v | sle-update | libgmodule-2_0-0 v | sle-update | libgobject-2_0-0 v | sle-update | libgpgme11 v | sle-update | libkmod2 v | sle-update | libpython3_11-1_0 v | sle-update | python311 v | sle-update | python311-base v | grommunio | python311-systemd | Current Version | Available Version | Arch
---------+--------------------------------+--------------------------------+-------
| 2.78.6-150600.4.16.1 | 2.78.6-150600.4.22.1 | x86_64
| 4.0.0.21.0ad253e-lp156.144.1 | 4.0.0.23.19cacc8-lp156.145.1 | noarch
| 9.11.17-lp156.10.1 | 9.11.17-lp156.11.1 | x86_64
| 7.4.1-lp156.6.34 | 7.4.1-lp156.6.35 | x86_64
| 7.4.1-lp156.6.34 | 7.4.1-lp156.6.35 | noarch
| 3.15.76.g25c4fb241-lp156.362.1 | 3.15.89.gd1efa6e8f-lp156.363.2 | noarch
| 3.2.2.mcf051aa-lp156.4.1 | 3.2.2.mcf051aa-lp156.12.1 | x86_64
| 3.2.2.mcf051aa-lp156.4.1 | 3.2.2.mcf051aa-lp156.12.1 | x86_64
| 3.2.2.mcf051aa-lp156.4.1 | 3.2.2.mcf051aa-lp156.12.1 | x86_64
| 29-150600.11.4 | 29-150600.13.3.1 | x86_64
| 2.78.6-150600.4.16.1 | 2.78.6-150600.4.22.1 | x86_64
| 2.78.6-150600.4.16.1 | 2.78.6-150600.4.22.1 | x86_64
| 2.78.6-150600.4.16.1 | 2.78.6-150600.4.22.1 | x86_64
| 2.78.6-150600.4.16.1 | 2.78.6-150600.4.22.1 | x86_64
| 1.23.0-150600.3.2.1 | 1.23.0-150600.3.5.1 | x86_64
| 29-150600.11.4 | 29-150600.13.3.1 | x86_64
| 3.11.13-150600.3.35.1 | 3.11.14-150600.3.38.1 | x86_64
| 3.11.13-150600.3.35.2 | 3.11.14-150600.3.38.1 | x86_64
| 3.11.13-150600.3.35.1 | 3.11.14-150600.3.38.1 | x86_64
| 234-lp156.4.2 | 235-lp156.2.1 | x86_64