Migrating Kopano to grommunio with kopano2grommunio.sh is very slow

TGM · 2024-12-17T18:00:23+00:00

Dear all, at this moment I try to migrate the whole Kopano installation (4,6TB attachments, 610GB MySQL database) the migration is very very slow. As exampl...

crpb

Yeah, sadly not but from what you are saying now how it improved i don't think you need that at all.
Just stop the incoming firewallrules or whatever that deliver to the kopano host (postfix) and migrate everything in one go.
Do that at least two times just for tests and then you can just block an late afternoon where the mail system is off for the rest of the day.
And with gromox-mbop you can now also easily cleanup the mailboxes after each test.
I used to run a test import, then ran grommunio-setup again to cleanup everything and did again 😛.
And when you are done testing you can already setup all clients with a freshly cleaned system so the day after the migration you are basically done with the heavy lifting.

TGM

crpb

thanks for the reply. That would be a nice idea. But... At now I copied round about 280GB in the last 20h hours. 280GB of round about 4 TeraByte of data. Maybe is it an option to parallelize the migration?

crpb

Ahh ok, than i didn't realize how much it is 😛.

I would define Groups of people you migrate step by step to the new system.

That would require some tinkering with postfix for sure on both sides and probably on the relayhost(as i guess there is such a thing with those numbers) aswell.

Can't tell you on the spot how to do that but if you are not really certain than maybe ask the support@grommunio.com as i think they most likely did something like that already and would know.

EDIT:

TGM about 20 minutes I migrated 2,5GB and round about 73.000 emails.

ahh ok, now i see it... well, thats due to the number of mails not the size in my experience.

WalterH

TGM Is there any possibility to apply the changed mails (new, moved, delete, ...)

No, but you do not have to. In all my migrations we follow these steps:

we set up the mailboxes in Grommunio
set up Outlook for all users
we switch the MX record to grommunio, so that all new mails arrive in the grommunio system and the users start using Outlook
start the migration, this will fill the users mailboxes.

With this procedure you only need to do a one-time migration.

BTW, the script wants an SSHFS connection, not an NFS connection. You have found the reason why we prefer sshfs.

TGM

WalterH

Just to be sure, that I understand it correctly. I install grommunio and deliver all new emails to grommunio. Then I say the users "Hey, use grommunio. Yes, your old emails are away for now but they came back. Just wait some days....". I start the migration and after the migration is finished all emails are back again?

That would be perfect. Because then I only have to build up a priority list who really needs the email after the weekend and who can wait 2 or 3 days more 😉

Best, Tobias

WalterH

TGM Yes this is the way how we do it.
You write wait some days ... - how many mailboxes do you have and how big are these mailboxes?
200 mailboxes can be migrated in less than 20 hours.

segro

yes, thats the idea for the k2g migration!

crpb

For that i would definitely run at least two test imports just to see if anything might come up which could be fixed before a long weekend. Better safe than sorry 🙂.

TGM

crpb

definitely. That´s the plan. At this moment I am still running the k2g migration script. And after it is finished I will check the log files in with meticulous precision. Nothing for this year, I think. But that´s okay. 😉

@segro perfect, thanks to you for the confirmation.

TGM

WalterH

we are talking about currently 308 mailboxes and we are using after migration 1,2 TeraByte data on the grommunio machine. The migration started at 18.12.2024 09:40:47 and finished at 22.12.2024 04:40:48.

crpb

@WalterH those times can vary extremly

head kopano2grommunio.log; tail kopano2grommunio.log

02.08.2024 14:38:01
02.08.2024 14:38:01 FAIL Error: data file: /root/import/k2g_list.txt not found, aborting.
02.08.2024 14:38:23
02.08.2024 14:38:23
02.08.2024 14:38:23
02.08.2024 14:38:23  ===========================================================================
02.08.2024 14:38:23
02.08.2024 14:38:23
02.08.2024 14:38:23 INFO Kopano 2 grommunio migration start
02.08.2024 14:38:23 WARN Mount attachment directory: /mnt/kopano/
03.08.2024 03:47:35 WARN 0 mailboxes creation failed, 0 migrations failed.
03.08.2024 03:47:35
03.08.2024 03:47:35 INFO Type 'X' to interrupt migration ...
03.08.2024 03:47:35
03.08.2024 03:47:35 INFO Migration done, clean up
03.08.2024 03:47:35 WARN Unmount attachment directory: /mnt/kopano/
03.08.2024 03:47:39
03.08.2024 03:47:39 INFO Total of 70 mailboxes processed
03.08.2024 03:47:39 INFO 70 mailboxes migrated
03.08.2024 03:47:39 INFO Kopano 2 grommunio migration done.

19.12.2024 15:12:43
19.12.2024 15:12:43
19.12.2024 15:12:43
19.12.2024 15:12:43  ===========================================================================
19.12.2024 15:12:43
19.12.2024 15:12:43
19.12.2024 15:12:43 INFO Kopano 2 grommunio migration start
19.12.2024 15:12:43 WARN Mount attachment directory: /mnt/kopano/
19.12.2024 15:12:43 WARN Enter Public Server Key Pass Phrase in next Line
19.12.2024 15:12:46
19.12.2024 15:29:43 WARN 0 mailboxes creation failed, 0 migrations failed.
19.12.2024 15:29:43
19.12.2024 15:29:43 INFO Type 'X' to interrupt migration ...
19.12.2024 15:29:43
19.12.2024 15:29:43 INFO Migration done, clean up
19.12.2024 15:29:43 WARN Unmount attachment directory: /mnt/kopano/
19.12.2024 15:29:44
19.12.2024 15:29:44 INFO Total of 10 mailboxes processed
19.12.2024 15:29:44 INFO 10 mailboxes migrated
19.12.2024 15:29:44 INFO Kopano 2 grommunio migration done.

mwilliams

TGM

The only meaningful way to speed up the process is to parallelize the process, which is possible by splitting the main migration script into multiple individual ones. Given enough hardware resources and parallelization with 6, we've been able to successfully migrate 2.1TB in less than 8 hours. Keep an eye on the mysql of Kopano though, it often has shown to be not optimized well for reading at all, such as the lack of appropriate transaction isolation, for example.

crpb

With that information i would definitely suggest looking into each kopano-mailbox for the "size" aswell as the "count" of items (as with everything on accessing, a file-transfer with 1x500gb is usually faster than 50000x10mb) because otherwise you could end up with N-jobs being finished in 5% time of that one or two other jobs.

WalterH

About one year ago I did a migration with 290 mailboxes and 1,8 GB data, this was done in two days. We used a fast system.