Out of Memory error after installing today's updates

Mister2

Earlier today I installed today's updates hoping this would fix the issue I am having with Mac Mail and Outlook hanging since around 25/5/26 see tickets @souverain EWS hangs while gromox-http stays active and @Berndp Outlookprobleme nach Update 25.05.2026 (sorry don't seem to be able to link to these correctly from my browser at the moment)

The updates installed today (running OpenSuse 16.0 appliance with all published updates) are:

crypto-policies 20250714.cd6043a-160000.2.1 gromox 3.7.252.me4b061f-lp160.44.1 gromox-debuginfo 3.7.252.me4b061f-lp160.44.1 gromox-debugsource 3.7.252.me4b061f-lp160.44.1 libstorage-ng-ruby 4.5.314-160000.1.1 libstorage-ng1 4.5.314-160000.1.1 openSUSE-build-key 1.0-lp160.16.1
As soon as these updates were installed I started getting hangs on the whole server. The Monitoring graphs in the Admin GUI were completely freezing. Whilst the server was still responding to pings I could not SSH into the server at all to check the logs.

After 1 - 2 minutes the server becomes responsive again and all the graphs update (somewhat faster than normal).
I was able to pull the following from journalctl -f. This then loops around every minute or so.

oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=nginx,pid=2850,uid=467 Jun 01 15:47:10 pve-sey-grommunio.the-seymours.co.uk kernel: Out of memory: Killed process 2850 (nginx) total-vm:3922916kB, anon-rss:3873648kB, file-rss:148kB, shmem-rss:148kB, UID:467 pgtables:7664kB oom_score_adj:0 Jun 01 15:49:34 pve-sey-grommunio.the-seymours.co.uk kernel: php-fpm invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0

Not a programmer so need the developers to investigate this. Off to restore snapshot from before the update.

WalterH

Please post the output of the following commands:
free -h and
grommunio-admin user query | wc -l and
ld.so --help | tail -3 ?

Mister2

After building up a new server today and upgrading it to OpenSuse 16.0 (a lengthy process as we are still waiting for the OpenSuse 16.0 Appliance ISO), all was well until I copied my Grommunio-Files config.php file to it. I immediately got the same results, persistent Out of memory kills, as when I installed the updates in the first post in this thread. Removing the config.php for the file created during installed brought stability back to the server albeit without Grommunio-Files.

I believe that one of the updates in the first post in this thread is the culprit - possibly crypto-policies 20250714.cd6043a-160000.2.1. I noticed that another thread has been raised today reporting a very similar issue with GF - see 'Grommunio files nur admin Anmeldung geht noch'

I will monitor the updates and await a fix for this issue from the developers.

Marco

Confirming the same OOM behaviour on openSUSE Leap 15.6 and notably WITHOUT grommunio-files installed.

I run the official grommunio Community OVA (6 GB RAM, no swap, Leap 15.6) and ran grommunio-update update community. It moved me from gromox 3.6.63...lp156.42.1 to 3.7.256.me40a504-lp156.48.1 (plus the usual grommunio-web / admin / antispam / python updates - everything stayed lp156).

After the reboot the server became unusable exactly as described in this thread:

pingable, but SSH would connect and authenticate yet never spawn a shell; the VM console was unresponsive too
locks up for a while, briefly recovers, then locks again, looping
I had to hard-reset the VM several times

Caught live, top showed the cause:

PID  USER   ...  RES     %CPU  %MEM  COMMAND
61   root        0       100.0 0.0   kswapd0   (state D)
1848 nginx       4.472g  ...   77.2  nginx

So nginx alone held around 4.4 GB resident (77% of RAM), MemFree was ~ 95 MB, swap is 0, and kswapd0 was pinned at 100% with ₆₇% iowait - the box was thrashing itself to death. This looks like the same nginx/php-fpm OOM runaway reported above.

The key difference from @MMister2's case: I do not have grommunio-files installed at all - rpm -q grommunio-files returns "package grommunio-files is not installed." So in my case grommunio-files cannot be the trigger. That suggests the OOM has a more general cause than GF - possibly the EWS/Outlook hang regression mentioned at the top of this thread, since nginx reverse-proxies those backends and would buffer hung/looping responses in memory.

I've reverted to a pre-update snapshot and I'm stable on the old version, where baseline memory use is only ₁.5 GB of 6 GB - so this is clearly a runaway, not a genuine increase in requirements.

Questions for the devs:

Given this reproduces without grommunio-files, is the underlying cause being looked at more broadly than GF (e.g. EWS/Outlook proxying)?
Is there any recommended interim mitigation for systems that don't use grommunio-files?

Happy to re-test on a snapshot and capture free -h and the oom-kill journal lines if it helps. Thanks!

Mister2

@Marco Update on this issue. When I raised this ticket it seemed that no one else was experiencing the OOM issue I had. I am aware that many of the users on the forum do not use Grommunio-Files and I assumed since I could no longer login to my Grommunio-Files, the issue had affected GF as well.

Subsequently, I found out that GF is not affected just the core Grommunio email as I was able to get GF working again following some reboots and checking of the config.php file. I had just done a clean install of Grommunio using the Appliance ISO from 25/4/25 and then upgraded it to OS 16.0. During my post upgrade configuration I had copied my config.php over rather than merging it into the new servers base config.php.

I have reverted to a snapshot taken before the updates released on 1/6/26 (see original post) as the OOM does not affect this build. I have been monitoring the forum to see if an update has been released that will clear up the issue.

So concur GF is not affected/cause and that the issue is with the main core Grommunio product.

I also have a OS 15.6 appliance that also suffers the same issue, but is running post 1/6/26 updates. Whilst the OOM occur the server manages to recover without hanging the VM - perhaps because it only has 1 MacOS client where my other appliance (OS 16.0 one) has 2 MacOS Mail clients and the OOM memory leak/runaway is duplicated making the OOM's occur more often.

Marco

@MMister2 Thanks for the update — good to have grommunio-files ruled out from the source.

To add to your "scales with client count" point: my setup (details in my earlier post) runs multiple iPhones on constant ActiveSync/EAS push and multiple EWS mail clients (a total of 5 mailboxes). Since you saw heavier OOMs at 2 EWS clients than at 1, my running several EWS clients would put me well up that curve - which fits my server hard-hanging and needing resets rather than self-recovering like your single-client box.

One question for the devs, given I run both EAS and EWS at volume: is the leak specific to EWS, or does ActiveSync/EAS load contribute as well? That distinction would help those of us weighing whether temporarily redirecting certain client types is a usable interim mitigation.

Duese6

I can confirm the OOM issue as well and have some reproducible observations.

I have been running several 3.7.x versions (including 3.7.224, 3.7.240, 3.7.242). With those, I mainly saw issues with Outlook (Autodiscover problems, disconnects, and hanging gromox-http processes).

After upgrading to 3.7.256, the overall behavior improved significantly. However, I am now seeing a reproducible OOM issue, specifically in connection with Apple Mail clients.

From my perspective, this points much more towards a regression or memory leak within gromox itself (likely somewhere in the HTTP / ActiveSync path), rather than something like crypto-policies.

The “files” theory also doesn’t really match a typical OOM scenario, unless it indirectly triggers abnormal memory behavior. What I am seeing looks much more like a leak under a specific client workload.

There is also a related discussion about unusually high EWS traffic with Apple Mail after recent updates, which might point in the same direction:
https://community.grommunio.com/d/2721-huge-ews-traffic-with-apple-mail-after-recent-updates/16

For reference: I have downgraded back to 3.7.130 and the system has been stable since, while the rest of the system is fully up to date.

WalterH

Hi, we see this issue with Apple MAC devices and MAC mail. Version: 3.7.328. Grommunio support is currently on the systems and identified the issue. Apple Mail creates a huge amount of requests and this requests overloads nginx and gromox-http. The only possibility is that the developers limit the number of requests per Apple device. We'll see.

WalterH

Development crated a new gromox version that limits the Apple Mail requests: 3.7.333 , can be found in devel. download.
Download it with wget ... and install it with rpm -Uvh ....
Important, restart all gromox and nginx services:
systemctl restart gromox-http gromox-zcore gromox-midb gromox-imap gromox-pop3 nginx php-fpm

Duese6

@WalterH

I tested the suggested dev build (3.7.333):

The server no longer runs directly into an OOM, but memory usage still increases significantly with just a single Apple Mail client.

In my case, it goes from _6–7 GB up to 20+ GB within a short time and is reproducible. (less than 2 min.)

After downgrading back to 3.7.130, memory usage returns to normal and stays stable.

So it looks like the issue is improved in newer builds, but not fully resolved yet.

grommunio:~ # ps aux --sort=-%mem | head -n 10
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
nginx 24528 10.3 42.4 13959320 13921028 ? S 21:52 1:18 nginx: worker process
gromox 24982 19.5 0.8 1002600 294816 ? Ssl 21:52 2:26 /usr/libexec/gromox/http

CPU is also a bit high I think. For one single client.

Duese6

As an additional observation:

It’s not only the memory usage that increases — the client behavior is also affected.

With Apple Mail, mailbox changes (new mails, folder moves, rule-based deliveries) are noticeably delayed. Updates are often only visible after manually switching folders.

Combined with the increased memory usage and the high EWS traffic, this looks like the server is handling client synchronization inefficiently under this scenario

mwilliams

Thanks for the feedback - we're going to provide an updated build ASAP (within 24 hours).