Further to my post [https://community.grommunio.com/d/1525-unable-to-login-to-some-mailboxes-with-web-client-following-todays-updates] I have been doing some deeper investigation. The process I have done and logs follow:-
- Clear browser cache for website grommunio.domain.com
- Start Grommunio Server
- Login with SSH
- Run Journalctl -f
- Check log - nothing happenin g except incoming emails (periodically)
- Open browser Window to https://grommunio.domain.com/web
- Login to GR Web with a user that is working OK, no log output
- Logout of GR WEB
- Clear browser cache for website grommunio.domain.com
- Login to GR Web to a mailbox that does not open - immediately get a segfault in the Grommunio kernel is reported.
Relevant segment of Journalctl log follows:-
`Mar 07 11:45:41 Grommunio kernel: Code: 66 0f 6f 2d b3 ff 10 00 66 0f 6f 35 bb ff 10 00 66 0f 6f 3d 93 ff 10 00 83 f9 30 0f 87 9a 00 00 00 83 f8 30 0f 87 91 00 00 00 <66> 0f 12 0f 66 0f 12 16 66 0f 16 4f 08 66 0f 16 56 08 66 44 0f 6f
Mar 07 11:45:41 Grommunio systemd[1]: Started Process Core Dump (PID 27424/UID 0).
Mar 07 11:45:45 Grommunio systemd-coredump[27425]: [🡕] Process 26991 (zcore) of user 479 dumped core.
Found module linux-vdso.so.1 with build-id: 729fae8cf2c599ad71ec3d7824afaa84341e6761
Found module libnss_dns.so.2 with build-id: 95ed8b7a40c418275069c5fc415873bf7f879f1f
Found module UTF-16.so with build-id: 32049c470dabaa9939e7d6c3983ef963b4a0bdd3
Found module libnss_files.so.2 with build-id: 6a102cac5610980614a7a2d5ebdd3dfbc2b0d95e
Found module libnss_compat.so.2 with build-id: 8851286c2216b994c30be15718f79cfbf91190cb
Found module libgxs_timer_agent.so with build-id: 71ad3a0162986d844cecda03d6a3c3a7d28aed32
Found module libaudit.so.1 with build-id: b1c6f127ee91b082bc26e5bc7240550ef1fae944
Found module libpam.so.0 with build-id: 1826a253bde7797134c8d9a4dd66ed52aaf5a873
Found module libgxs_authmgr.so with build-id: affbd94d21fa79e179d4902f59752e7a644facf5
Found module libmariadb.so.3 with build-id: d2c5029181345eae49f4f5a6d66d7331901f5662
Found module libgromox_dbop.so.0 with build-id: cd9adc09109a415a9b64f2c61ca2b2236c47a324
Found module libgxs_mysql_adaptor.so with build-id: 3d7601734d718e31372f3fddfaec57d4acdd3f6f
Found module libsasl2.so.3 with build-id: bf79dd013a4cb1be1aa572f29325b42941dd431b
Found module libresolv.so.2 with build-id: 83e50ade76c4ef68eb7291ca9eef373d0e5477f9
Found module liblber-2.4.so.2 with build-id: 5c904612e6ed3e358836fbaedcb14c67c0025269
Found module libldap_r-2.4.so.2 with build-id: 14f9994f64260cbddbe8aa3926dc76e58e9310a5
Found module libgxs_ldap_adaptor.so with build-id: 2e9d597ab1daefb2b3e3b3ba606a573572525cac
Found module libffi.so.7 with build-id: 62ece4f953ed6d967d46a60ce7979803ce3f51c3
Found module libgmp.so.10 with build-id: 9e11b2a675e3fd8af6d9ae9328dc8f105f15292f
Found module libhogweed.so.6 with build-id: 39c63d6c9dfadc1fb63c8ec143dabcb8755c67b9
Found module libnettle.so.8 with build-id: 0a1b350f4092635c549f85079a7d53565616443e
Found module libtasn1.so.6 with build-id: ed921be4af4a8a0bcaff1ccb130e341786b6f9b9
Found module libunistring.so.2 with build-id: bd7d3b65d889344af06d7f3dce4fc1b5e813f377
Found module libidn2.so.0 with build-id: cc588deadecd7fd913121bf76becb93046c35be3
Found module libp11-kit.so.0 with build-id: b04920981d44f49643e1783949a2478a81612966
Found module liblzma.so.5 with build-id: 2d656c3bd393d5f9e95fdbbfe4fbbdc19e0cf0ba
Found module libanl.so.1 with build-id: 77cfff1db7c76c059d4c1aa711fa83409b19aa49
Found module libgnutls.so.30 with build-id: 771ca46f45b99f904ec0b606d453048f882431d0
Found module libjitterentropy.so.3 with build-id: 6cd5111426fb5f10ef2d5fb9109ef901dd6a67f5
Found module libz.so.1 with build-id: 0bd1b117f07eb8157bb2931e94d966fbb352f652
Found module libxml2.so.2 with build-id: 4ab2cd29f14baa814ae707a67b87253e71d98d6a
Found module libfmt.so.8 with build-id: 93fa4af0cecf48e3c92a3891ed44f7294c7d4f29
Found module libm.so.6 with build-id: a81ec982656c9f3b51d4d7ea5384e7c4fd9708de
Found module libzstd.so.1 with build-id: d1fb4855c1f72b5941faa9ffac3edff26a2da3bb
Found module libtinyxml2.so.9 with build-id: e00d752dcfae6bfe42db91a6aacc412e11f5b16e
Found module libssl.so.1.1 with build-id: 24c209915cbbfaade4c5e332c39c1c3bbcdfe9ae
Found module libsqlite3.so.0 with build-id: 8a277ab0ee2456d01e0a9f5a729d6716b72f3dcc
Found module libjsoncpp.so.19 with build-id: d79dd65389319d1e9a8b3e50a6dbc7dbbbf0dbce
Found module libcrypt.so.1 with build-id: 57930159d50d6edc1adffca45f268126b3724ebf
Found module ld-linux-x86-64.so.2 with build-id: be5d0c46cde7620968b30cd3c9a737179b4f8f5f
Found module libc.so.6 with build-id: f732026552f6adff988b338e92d466bc81a01c37
Found module libgcc_s.so.1 with build-id: 12f775ec4aeb94b749897b1b65638f18b61d1b1f
Found module libstdc++.so.6 with build-id: c74eca671e2dd0f063706372d103f8acef88f1e3
Found module libvmime-suse.so.6 with build-id: 65d73f3de52d713db52991822124b3a992e215ee
Found module libHX.so.32 with build-id: 3df8692372d559cf822699c007162a3bf85f178c
Found module libcrypto.so.1.1 with build-id: b61a8d70989fdd00494c15631ad05356193b8234
Found module libpthread.so.0 with build-id: 2f8d3c2d0f4d7888c2598d2ff6356537f5708a73
Found module libgromox_mapi.so.0 with build-id: 86241c7e63c7cf47427b2761a29a3afec700176e
Found module libgromox_exrpc.so.0 with build-id: a2d627fa0ff753c565317bd94c119d56bf5a58f5
Found module libgromox_email.so.0 with build-id: c5c12eb24b74f319625b000a59508b40ede43620
Found module libgromox_common.so.0 with build-id: 1a648434bb6d284b3bf5531bda00a5fcc16cb130
Found module libdl.so.2 with build-id: d217b228882341aeb761264b83415f7458617268
Found module zcore with build-id: aa98a67adf2ae8f438b5a64b6aa1ab12e221533d
Stack trace of thread 27000:
#0 0x00007f53bf99e78f __GI___strncasecmp_l_sse2 (libc.so.6 + 0xaf78f)
#1 0x000055c1a36c2269 _Z24common_util_essdn_to_uidPKcPi (zcore + 0x24269)
#2 0x000055c1a3706f0b _Z12zs_openstore4GUID6BINARYPj (zcore + 0x68f0b)
#3 0x000055c1a36e6088 rpc_parser_dispatch (zcore + 0x48088)
#4 0x000055c1a36e8fbb zcrp_thrwork (zcore + 0x4afbb)
#5 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#6 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 26991:
#0 0x00007f53bf9cb121 clock_nanosleep@@GLIBC_2.17 (libc.so.6 + 0xdc121)
#1 0x00007f53bf9d0e43 __nanosleep (libc.so.6 + 0xe1e43)
#2 0x00007f53bf9d0d5a sleep (libc.so.6 + 0xe1d5a)
#3 0x000055c1a36d9938 main (zcore + 0x3b938)
#4 0x00007f53bf92424d __libc_start_main (libc.so.6 + 0x3524d)
#5 0x000055c1a36b79ca _start (zcore + 0x199ca)
Stack trace of thread 26992:
#0 0x00007f53bf9cb121 clock_nanosleep@@GLIBC_2.17 (libc.so.6 + 0xdc121)
#1 0x00007f53bf9d0e43 __nanosleep (libc.so.6 + 0xe1e43)
#2 0x00007f53bf9d0d5a sleep (libc.so.6 + 0xe1d5a)
#3 0x00007f53bf839e7b tmrag_scanwork (libgxs_timer_agent.so + 0x2e7b)
#4 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#5 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 26994:
#0 0x00007f53bf9cb121 clock_nanosleep@@GLIBC_2.17 (libc.so.6 + 0xdc121)
#1 0x00007f53bf9d0e43 __nanosleep (libc.so.6 + 0xe1e43)
#2 0x00007f53bf9d0d5a sleep (libc.so.6 + 0xe1d5a)
#3 0x000055c1a36bd541 zcoreab_scanwork (zcore + 0x1f541)
#4 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#5 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 27005:
#0 0x00007f53bf9cb121 clock_nanosleep@@GLIBC_2.17 (libc.so.6 + 0xdc121)
#1 0x00007f53bf9d0e43 __nanosleep (libc.so.6 + 0xe1e43)
#2 0x00007f53bf9d0d5a sleep (libc.so.6 + 0xe1d5a)
#3 0x000055c1a370587f zcorezs_scanwork (zcore + 0x6787f)
#4 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#5 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 27006:
#0 0x00007f53bf9cb121 clock_nanosleep@@GLIBC_2.17 (libc.so.6 + 0xdc121)
#1 0x00007f53bf9d0e43 __nanosleep (libc.so.6 + 0xe1e43)
#2 0x00007f53bf9d0d5a sleep (libc.so.6 + 0xe1d5a)
#3 0x00007f53c0428161 cl_pinger (libgromox_exrpc.so.0 + 0x14161)
#4 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#5 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 27217:
#0 0x00007f53bf9faa09 __poll (libc.so.6 + 0x10ba09)
#1 0x00007f53c04270f4 poll (libgromox_exrpc.so.0 + 0x130f4)
#2 0x00007f53c042775b cl_notif_reader2 (libgromox_exrpc.so.0 + 0x1375b)
#3 0x00007f53c04277b2 cl_notif_reader (libgromox_exrpc.so.0 + 0x137b2)
#4 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#5 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 27220:
#0 0x00007f53bf9faa09 __poll (libc.so.6 + 0x10ba09)
#1 0x00007f53c04270f4 poll (libgromox_exrpc.so.0 + 0x130f4)
#2 0x00007f53c042775b cl_notif_reader2 (libgromox_exrpc.so.0 + 0x1375b)
#3 0x00007f53c04277b2 cl_notif_reader (libgromox_exrpc.so.0 + 0x137b2)
#4 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#5 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 26998:
#0 0x00007f53c037170c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0x1170c)
#1 0x000055c1a36e8d26 zcrp_thrwork (zcore + 0x4ad26)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 27001:
#0 0x00007f53c037170c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0x1170c)
#1 0x000055c1a36e8d26 zcrp_thrwork (zcore + 0x4ad26)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 27002:
#0 0x00007f53c037170c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0x1170c)
#1 0x000055c1a36e8d26 zcrp_thrwork (zcore + 0x4ad26)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 26996:
#0 0x00007f53c037170c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0x1170c)
#1 0x000055c1a36e8d26 zcrp_thrwork (zcore + 0x4ad26)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 27004:
#0 0x00007f53c037170c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0x1170c)
#1 0x000055c1a36e8d26 zcrp_thrwork (zcore + 0x4ad26)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 27003:
#0 0x00007f53c037170c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0x1170c)
#1 0x000055c1a36e8d26 zcrp_thrwork (zcore + 0x4ad26)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 26995:
#0 0x00007f53c037170c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0x1170c)
#1 0x000055c1a36e8d26 zcrp_thrwork (zcore + 0x4ad26)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 26999:
#0 0x00007f53c037170c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0x1170c)
#1 0x000055c1a36e8d26 zcrp_thrwork (zcore + 0x4ad26)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 26997:
#0 0x00007f53c037170c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0 + 0x1170c)
#1 0x000055c1a36e8d26 zcrp_thrwork (zcore + 0x4ad26)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Stack trace of thread 26993:
#0 0x00007f53bfa0937f accept4 (libc.so.6 + 0x11a37f)
#1 0x000055c1a36d889d zcls_thrwork (zcore + 0x3a89d)
#2 0x00007f53c036a6ea start_thread (libpthread.so.0 + 0xa6ea)
#3 0x00007f53bfa0750f __clone (libc.so.6 + 0x11850f)
Mar 07 11:45:46 Grommunio systemd[1]: gromox-zcore.service: Main process exited, code=killed, status=11/SEGV
Mar 07 11:45:46 Grommunio systemd[1]: gromox-zcore.service: Failed with result 'signal'.
Mar 07 11:45:46 Grommunio systemd[1]: systemd-coredump@17-27424-0.service: Deactivated successfully.
Mar 07 11:45:46 Grommunio systemd[1]: gromox-zcore.service: Scheduled restart job, restart counter is at 18.
Mar 07 11:45:46 Grommunio systemd[1]: Stopped Gromox zcore service.
Mar 07 11:45:46 Grommunio systemd[1]: Started Gromox zcore service.
Mar 07 11:45:46 Grommunio zcore[27435]: gromox-zcore 2.25.26.gf86b6ba (pid 27435 uid 0)
Mar 07 11:45:46 Grommunio zcore[27435]: system: maximum file descriptors: 524288
Mar 07 11:45:46 Grommunio zcore[27435]: system: SMTP server is smtp://[::1]:25
Mar 07 11:45:46 Grommunio zcore[27435]: Reexecing /usr/libexec/gromox/zcore
Mar 07 11:45:46 Grommunio zcore[27435]: gromox-zcore 2.25.26.gf86b6ba (pid 27435 uid 479)
Mar 07 11:45:46 Grommunio zcore[27435]: system: maximum file descriptors: 524288
Mar 07 11:45:46 Grommunio zcore[27435]: system: SMTP server is smtp://[::1]:25
Mar 07 11:45:47 Grommunio zcore[27435]: ldap_adaptor: default host <ldapi:///>, base <>, #conn=8, mailattr=mail
Mar 07 11:45:47 Grommunio zcore[27435]: mysql_adaptor: Current schema n131 is recent.
Mar 07 11:45:47 Grommunio zcore[27435]: timer_agent: timer address is [::1]:6666
Mar 07 11:45:47 Grommunio zcore[27435]: system: zcore is now running
`
Gromox-zcore is then killed by the 'systemd-coredump' process. Grommunio-zcore then reloads.
- Login as user whose mailbox opens OK, login progresses without issue, and no log entries.
- Performed same test using updates from gromox 2.25.8.g88f8149-lp155.5.1 (and associated updates), this works for all mailboxes with no errors on login, and mailboxes all open normally, with no log output.
- Clone VM and apply updates to gromox 2.25.26.gf86b6ba-lp155.19.1. Same errors start to occur again.
I have also built a new clean VM from the 28-12 Appliance ISO, migrated the data and repeated the tests above and get exactly the same errors. Must be something gone wrong in the gromox 2.25.26.gf86b6ba-lp155.19.1 (and associated updates installed at the same time) - or to is not liking something about the user, but further testing has indicated that the error occurs as soon as the browser connects to the Grommunio server.
Having to hold back running the latest updates as GR WEB is the problem mailbox users preferred client for their mail.
Also it appears that Outlook and mobile clients are unaffected.
Looking back through my server update logs it would seem issue first occurred on 05-03-24 when the GUI updater offered the following updates:-
v | grommunio | grommunio-sync | 2.0.51.0aa8178-lp155.145.1 | 2.0.62.079e1cf-lp155.146.1 | noarch
v | grommunio | grommunio-web | 3.7.4.9bf5f501-lp155.42.1 | 3.7.5.64e25710-lp155.43.1 | noarch
v | grommunio | gromox | 2.25.8.g88f8149-lp155.5.1 | 2.25.24.gf4d57a9-lp155.12.1 | x86_64
v | grommunio | gromox-debuginfo | 2.25.8.g88f8149-lp155.5.1 | 2.25.24.gf4d57a9-lp155.12.1 | x86_64
v | grommunio | gromox-debugsource | 2.25.8.g88f8149-lp155.5.1 | 2.25.24.gf4d57a9-lp155.12.1 | x86_64
the update on 06-03-24 did not resolve the issue for me (Community version).
Hope above gives the developers something to track down the issue. Keep up the good work.