EWS hangs while gromox-http stays active

souverain · 2026-05-22T08:31:39+00:00

Hello, We encountered a temporary issue with EWS access on a grommunio/gromox server. The webmail was working, but Apple Mail on macOS could no longer conn...

Mister2

Installed today's updates:

crypto-policies 20250714.cd6043a-160000.2.1 gromox 3.7.252.me4b061f-lp160.44.1 gromox-debuginfo 3.7.252.me4b061f-lp160.44.1 gromox-debugsource 3.7.252.me4b061f-lp160.44.1 libstorage-ng-ruby 4.5.314-160000.1.1 libstorage-ng1 4.5.314-160000.1.1 openSUSE-build-key 1.0-lp160.16.1
As soon as these updates were installed I started getting hangs on the whole server. The Monitoring graphs in the Admin GUI were completely freezing. Whilst the server was still responding to pings I could not SSH into the server at all to check the logs.

After 1 - 2 minutes the server becomes responsive again and all the graphs update (somewhat faster than normal).
I was able to pull the following from journalctl -f. This then loops around every minute or so.

oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=nginx,pid=2850,uid=467 Jun 01 15:47:10 pve-sey-grommunio.the-seymours.co.uk kernel: Out of memory: Killed process 2850 (nginx) total-vm:3922916kB, anon-rss:3873648kB, file-rss:148kB, shmem-rss:148kB, UID:467 pgtables:7664kB oom_score_adj:0 Jun 01 15:49:34 pve-sey-grommunio.the-seymours.co.uk kernel: php-fpm invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0

Not a programmer so need the developers to investigate this. Off to restore snapshot from before the update.

WalterH

Please post the output of the following command: free -h and grommunio-admin user query | wc -l.

Mister2

WalterH Free -h

               total        used        free      shared  buff/cache   available
Mem:           5.8Gi       5.7Gi        94Mi        10Mi        35Mi        94Mi
Swap:             0B          0B          0B

grommunio-admin user query | wc -l

WalterH

x86-64-v4
x86-64-v3
x86-64-v2 (supported, searched)

Server has been running fine since upgraded to OS 16. Only issue I have had in recent weeks is this Mac Mail and Outlook issue.

WalterH

And the output of the following command: ld.so --help | tail -3 ?

WalterH

Mister2 Give the server 2 or 4 GB more RAM, this may fix this issue.

Mister2

@WalterH Also getting very high User and System CPU and obviously, from the error, very high RAM usage. Typically system runs on Admin Graphs at 2.5-3GB GB RAM used. This server is used as part of a homelab just for system and family email (3 users accounts) so not normally stressed and resources are generally OK for good performance.

Mister2

WalterH Do so. After 4 mins the out of memory errors have started again:

oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=nginx,pid=1380,uid=467
Jun 01 16:45:08 pve-sey-grommunio.the-seymours.co.uk kernel: Out of memory: Killed process 1380 (nginx) total-vm:5654360kB, anon-rss:5606256kB, file-rss:72kB, shmem-rss:136kB, UID:467 pgtables:11040kB oom_score_adj:0

Proxmox is reporting 1.75GB of 10GB RAM in use.

Mister2

WalterH Do so. After 4 mins the out of memory errors have started again:

Proxmox is reporting 1.75GB of 10GB RAM in use.

Mister2

WalterH Also after reboot to add the extra RAM to the VM this is the RAM usage which is in line with the normal values I see after a server reboot

free -h
               total        used        free      shared  buff/cache   available
Mem:           9.7Gi       1.8Gi       7.8Gi        13Mi       374Mi       7.9Gi
Swap:             0B          0B          0B

Apart from a short burst where the server usage peaked at 7GB it has now returned to using only 1.8GB. So it looks like something in today's update was running a script or process that required a lot of RAM to run.

Thanks for your tips and advice I shall keep an eye on the resources being used.

WalterH

Mister2 Out of memory: Killed process 1380 (nginx)

but the problem is in the Web server (nginx) process not in the gomox process.

Run this command: tail -f /var/log/nginx/*.log and when the OOM Killer kills nginx, what you see in the logs?

Btw, grommunio is only tested and supported with 4 CPUs, you have only 2 CPUs.

Mister2

WalterH
Will do when the next occurrence happens, all seems quiet on the Admin GUI graphs at the moment. It seems strange that a small upgrade can have such a major effect on the server. I would have expected such a change when I upgraded to OS 16.0, but no additional resources were required at that time. I shall play around with the RAM resources over the coming days to see what the sweet spot is for my usage case and the Grommunio servers requirements.

I fully understand that Grommunio may be tested/supported to run on 4 CPU's but in practice it performs well in a homelab with 1 or 2 and my vCPU settings have been tuned to the needs of my system to give good performance. A recent hardware upgrade will allow me to allocate more vCPU if performance becomes an issue - so thanks for the advice.

souverain

Hi, on my end, even though I've installed all the updates since our last conversation, I'm still experiencing this bug. Do you have any updates, please?

Thanks in advance!

souverain

Ah!
I don't know about you, but since the latest updates, I feel like the bug is gone:

gromox-3.7.256.me40a504-lp156.46.1.x86_64
gromox-debugsource-3.7.256.me40a504-lp156.46.1.x86_64
mapi-header-php-2.1.22.762be6f-lp156.10.1.noarch
gromox-debuginfo-3.7.256.me40a504-lp156.46.1.x86_64
grommunio-web-3.18.7.gec5085ffd-lp156.481.1.noarch

On the other hand, I have no idea which package fixed it...

Mister2

I built a new server today and upgraded it to OpenSuse 16.0. All was well until I copied my Grommunio-Files config.php file to it and the Out Of Memory errors started again. Removing the files stopped the erors but GF do not work now.

I believe that one of the updates released on 1/5/26 is the culprit - most likely the 'crypto-policies 20250714.cd6043a-160000.2.1' update.

I have now reverted to a snapshot taken before the updates and every thing is running as it was before the updates. So I will watch the forum and test updates occasionally until the issue is resolved (hopefully).

The MacOS and Outlook issues are obviously still unresolved as my server is now running on updates from 29/05/26.

FvA

Any update on this? I wrote a short script to restart gromox-http when a curl to 'http://127.0.0.1:10080/EWS/Exchange.asmx' doesn't give a 405 in return. Not ideal, but it keeps Outlook alive for now.

Update, as per souverain's post, upgraded to:
grommunio-admin-api 1.19.11.mc7e4cbd lp156.5.1
grommunio-admin-common 50.10f1afb lp156.51.1
grommunio-admin-web 5.0.0.32.a076725 lp156.172.1
grommunio-antispam 4.0.1 lp156.5.1
grommunio-common 29.9 lp156.19.1
grommunio-cui 1.1.3.b0091e0 lp156.7.1
grommunio-dav 1.6.24.84942ef lp156.1.1
grommunio-dbconf 1.1.2.4bbae9a lp156.15.1
grommunio-error-pages 1.0.10.bb2df37 lp156.26.1
grommunio-imapsync 2.314 lp156.1.1
grommunio-index 1.5.0.gb0a01b9 lp156.2.1
grommunio-release 2025.1.2 lp156.8.1
grommunio-setup 1.1.16.46f2af3 lp156.73.1
grommunio-sync 2.4.8.483c521 lp156.23.1
grommunio-web 3.18.7.gec5085ffd lp156.481.3
gromox 3.7.256.me40a504 lp156.48.1
gromox-debuginfo 3.7.256.me40a504 lp156.48.1
gromox-debugsource 3.7.256.me40a504 lp156.48.1

Will see if the issue is resolved now.

mark1

FvA
I am interested in the workaround script. Would you be willing to share the script with us?

FvA

Sure, fairly straightforward test and restart.

I run this every 2 minutes in a cron job:
grommunio:~ # cat /etc/cron.d/gromox-http-watchdog
*/2 * * * * root /usr/local/sbin/gromox-http-watchdog.sh >/dev/null 2>&1

The script:
`
#!/bin/bash
set -euo pipefail

URL='http://127.0.0.1:10080/EWS/Exchange.asmx'
LOCK='/run/gromox-http-watch.lock'
LOGFILE='/var/log/gromox-http-watchdog.log'

mkdir -p /var/log
touch "$LOGFILE"
chmod 0644 "$LOGFILE"

exec >>"$LOGFILE" 2>&1

log() {
printf '[%s] %s\n' "$(date -Is)" "$*"
}

probe() {
curl -sS -o /dev/null -w "HTTP=%{http_code} TIME=%{time_total}\n" --max-time 10 "$URL" 2>/dev/null || true
}

exec 9>"$LOCK"
flock -n 9 || exit 0

result="$(probe)"

if [[ "$result" == HTTP=405* ]]; then
exit 0
fi

log "probe failed: $result; restarting gromox-http"
systemctl restart gromox-http.service

for _ in $(seq 1 30); do
sleep 5
result="$(probe)"
if [[ "$result" == HTTP=405* ]]; then
log "service recovered: $result"
exit 0
fi
done

log "service did not recover after restart"
exit 1
`

mark1

Thank you very much for the quick reply.
I will test it on my server.

souverain

mark1 please can you give us your feedback after ?

mark1

souverain
After the upgrade to OS 16.0:
gromox 3.7.256.me40a504-lp160.48.1 (all current versions)
grommunio-web 3.18.7.gec5085ffd-lp160.481.3
mapi-header-php 2.1.22.762be6f-lp160.10.1,
no further errors have occurred so far.
After one day of operation, the gromox-http-watchdog.log is still empty.

mark1

Yes, gladly.
The script is already running. I'll report back as soon as there's anything in the log.

« Previous Page Next Page »