Appliance initial install package refreshing extremely slow

FelixTECH

Hello,
I am using the VMware Appliance and I am attempting to install Grommunio for the first time. The step of refreshing the packages goes until 89%/21% very fast and smooth in basically seconds. From this point on the speed drops down to less than 10 KiB/sec, after a longer time even to 2 Bytes per second and basically never finishes (at least not in a full day and overnight). It just goes into a denial mode where it doesn't care anymore and refuses to download from a working server on a gigabit internet connection.
The issue can be reliably reproduced every time in all datacenters in different countries on entirely different hardware and even on different hypervisors when installing from the ISO.
Is this normal to take more than a day? If no, where is the issue and why is it not working out of the box? If yes, why is this not fixed?

Felix

WalterH

FelixTECH Install the pciutils package and show the output of lspci.

zypper in pciutils
lspci

crpb

I mostly use vsphere and don't have such issues. Are there any proxies/firewalls in front of your vms that might do some shenanigans?

Maybe the http:// (no ssl) urls get throttled?

PS: I usually use the OVA myself but that wouldn't do any difference.
Maybe check if open-vm-tools are installed but that wouldn't be the problem source in my experience.
sed -i 's/http:/https:/g' /etc/zypp/repos.d/* might do the trick if it's the firewall that doesn't like non-https downloads. I know of such "defaults" on fortigate firewalls.

FelixTECH

crpb The open-vm-tools come by default and they work (and show data in vSphere) right after the first boot.
It is definitely not a firewall issue. There is nothing configured that remotely could trigger such behaviour. Also if it would find something that it dislikes it would notify security. I have tested this across multiple datacenters which use different firewalls and different ISPs.
I have never seen such behaviour before on any operating system and we have 4 kinds of operating systems which have basically nothing in common.

crpb

FelixTECH

What i'm just thinking of is...
Don't do that GUI shit with YAST.

Go to the cli and you could just use grommunio-update upgrade or just the core zypper ref ; zypper up /or zypper dup

crpb

Ok, but it just sounds really odd. I mean yes, zypper is somehow a slowpoke in some regards to apt but i had no such issues with a bunch of vsphere/esxi and a few proxmox and hyper-v based systems.
And i guess your storage isn't a slowpoke aswell if you have access to multiple datacenters :-).

I could try the latest iso here just for kicks but i don't think it will bite me.

Do you have any special storage policies or encryption activated?
And did you try an core openSuse just to see if there would be any diference?

FelixTECH

We use high quality SAS-SSDs for everything except backups. Using SSDs with less chip quality has made issues a few years ago. Turns out saving money on storage is a really bad idea 😉
So storage speed isn't a problem. There are Gigabytes of cache before the SSDs as well.
There are no restrictions to storage and no encryption. We just lock the doors very well.

I have tried it with Hyper-V server with SAS-HDDs, a residential internet connection and the ISO as well. That was even slower and I aborted it at some point.

Nobody of us here has any prior experience with SUSE at all. I'll do some tests with the core OS and GUI/CLI to see if it makes any difference. I'll report back once done.

crpb

FelixTECH Nobody of us here has any prior experience with SUSE at all.

Same here before Grommunio.
As long as you know your way through SystemD it won't be too much of a hassle.

FelixTECH

WalterH

localhost:~ # lspci
00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
00:11.0 PCI bridge: VMware PCI bridge (rev 02)
00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.7 PCI bridge: VMware PCI Express Root Port (rev 01)
02:00.0 USB controller: VMware USB1.1 UHCI Controller
02:01.0 USB controller: VMware USB2 EHCI Controller
03:00.0 Serial Attached SCSI controller: VMware PVSCSI SCSI Controller (rev 02)
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)

crpb

This looks like the same as here..

grom-test-4:/usr/share/grommunio-admin-web # lspci -nnk
00:00.0 Host bridge [0600]: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge [8086:7190] (rev 01)
        Subsystem: VMware Virtual Machine Chipset [15ad:1976]
        Kernel driver in use: agpgart-intel
00:01.0 PCI bridge [0604]: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge [8086:7191] (rev 01)
00:07.0 ISA bridge [0601]: Intel Corporation 82371AB/EB/MB PIIX4 ISA [8086:7110] (rev 08)
        Subsystem: VMware Virtual Machine Chipset [15ad:1976]
00:07.1 IDE interface [0101]: Intel Corporation 82371AB/EB/MB PIIX4 IDE [8086:7111] (rev 01)
        Subsystem: VMware Virtual Machine Chipset [15ad:1976]
        Kernel driver in use: ata_piix
        Kernel modules: ata_piix, pata_acpi, ata_generic
00:07.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 08)
        Subsystem: VMware Virtual Machine Chipset [15ad:1976]
        Kernel modules: i2c_piix4
00:07.7 System peripheral [0880]: VMware Virtual Machine Communication Interface [15ad:0740] (rev 10)
        Subsystem: VMware Virtual Machine Communication Interface [15ad:0740]
        Kernel driver in use: vmw_vmci
        Kernel modules: vmw_vmci
00:0f.0 VGA compatible controller [0300]: VMware SVGA II Adapter [15ad:0405]
        Subsystem: VMware SVGA II Adapter [15ad:0405]
        Kernel driver in use: vmwgfx
        Kernel modules: vmwgfx
00:11.0 PCI bridge [0604]: VMware PCI bridge [15ad:0790] (rev 02)
00:15.0 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:15.1 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:15.2 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:15.3 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:15.4 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:15.5 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:15.6 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:15.7 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:16.0 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:16.1 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:16.2 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:16.3 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:16.4 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:16.5 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:16.6 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:16.7 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:17.0 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:17.1 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:17.2 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:17.3 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:17.4 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:17.5 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:17.6 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:17.7 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:18.0 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:18.1 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:18.2 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:18.3 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:18.4 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:18.5 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:18.6 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
00:18.7 PCI bridge [0604]: VMware PCI Express Root Port [15ad:07a0] (rev 01)
        Kernel driver in use: pcieport
02:00.0 USB controller [0c03]: VMware USB1.1 UHCI Controller [15ad:0774]
        Subsystem: VMware Device [15ad:1976]
        Kernel driver in use: uhci_hcd
        Kernel modules: uhci_hcd
02:01.0 USB controller [0c03]: VMware USB2 EHCI Controller [15ad:0770]
        Subsystem: VMware USB2 EHCI Controller [15ad:0770]
        Kernel driver in use: ehci-pci
        Kernel modules: ehci_pci
03:00.0 Serial Attached SCSI controller [0107]: VMware PVSCSI SCSI Controller [15ad:07c0] (rev 02)
        Subsystem: VMware PVSCSI SCSI Controller [15ad:07c0]
        Kernel driver in use: vmw_pvscsi
        Kernel modules: vmw_pvscsi
0b:00.0 Ethernet controller [0200]: VMware VMXNET3 Ethernet Controller [15ad:07b0] (rev 01)
        Subsystem: VMware VMXNET3 Ethernet Controller [15ad:07b0]
        Kernel driver in use: vmxnet3
        Kernel modules: vmxnet3

WalterH

try ifconfig, are any errors on eth0 like dropped, overrun or framing errors? A small amount of errors is OK.

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.14  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::be24:11ff:fec3:169d  prefixlen 64  scopeid 0x20<link>
        ether bc:24:11:c3:16:9d  txqueuelen 1000  (Ethernet)
        RX packets 10906712  bytes 1624390618 (1.5 GiB)
        RX errors 0  dropped 104925  overruns 0  frame 0
        TX packets 12891622  bytes 131526228661 (122.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

FelixTECH

There are no errors on this.

localhost:~ # ip -stats link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:ab:1b:66 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast
      86022840   91613      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
      34584903   61316      0       0       0       0
    altname enp11s0
    altname ens192

Also if there are any errors that exceed some 0.something% value then I get a monitoring message about it. So I would have noticed if there would have been issues.

WalterH

Try this command, how long it needs to execute and also post the output:

dd if=/dev/zero of=/tmp/whwh bs=4096 count=250000 oflag=dsync

The command creates an 1GB file /tmp/whwh to test the disk speed.
When done remove the file with: rm /tmp/whwh.
Post runtime and output of this command.

FelixTECH

This is in our down datacenters:

250000+0 records in
250000+0 records out
1024000000 bytes (1,0 GB, 977 MiB) copied, 94,2069 s, 10,9 MB/s

This is at a potential customer:

250000+0 records in
250000+0 records out
1024000000 bytes (1.0 GB, 977 MiB) copied, 397.489 s, 2.6 MB/s

It is not a disk speed issue as all other operating systems in many years had never such issues.
This pain is specefic to Grommunio and does not occur on regular openSUSE. When updating openSUSE using YaST2 it is as fast as I would expect from Linux.

WalterH

FelixTECH 2.6 MB/s

looks like an slow mechanical disc or very bad SSDs.

The zypper package manager ist the original SuSE package manager, not modified by grommunio.

FelixTECH

WalterH The 2.6 MB/s is indeed a mechanical HDD. The customer is "happy" with it and doesn't want to upgrade. But normally I do stuff in our own datacenters. I just tested it at that customer's server to see if a different server brand, different Hypervisor, different ISP, different firewall makes any difference.

For further testing I only use the fast servers.
I tested zypper install before doing anything else in the Grommunio Console UI and it took 1 min to download and another 4 min to install all updates.
Then I tested (on a fresh system) the Grommunio Console UI. It took 1 min to "Select software repositories" and then the same as before (1 min download, 4 min install) when applying updates.
On a third test I did the language selection first which took 1 min 15 sec to complete, then the "Select software repositories" took a few seconds and installing like before.
And on a last test I tried first "Select software repositories" and then change the language. Then changing the language took 10 seconds.

So from those tests and the previous experience I can conclude that there must have been an issue on the repo servers that is now gone. And I couldn't replicate this with native openSUSE as that uses different repo servers. So I'm now saying case closed.

WalterH

FelixTECH I did the language selection first which took

Do this mean, you switch the server language to something other then English? If yes, this is a not supportet configuration, it may work (yes may work) but no warranties.

WalterH

Can you add this parameter and test again?/etc/resolv.conf:options single-request-reopen