Softaculous

Topic : Way Faster VPS backups

Page 1 of 3

Posted By: domainscoza on November 8, 2018, 9:20 am

I've already logged a ticket for this, but if anyone is interested there is a much faster way to do VPS backups by switching out gzip for lzop compression.

Here is a the what the current virtualizor backup does:

Code
# lvcreate -L12G -s -n backup-v1001 /dev/kvmvg/vsv1001-dbbiylg3awl9iyka-ynfcfcgknopy4drs

# time /bin/dd if=/dev/kvmvg/backup-v1001 bs=24M iflag=direct | /bin/gzip -c > /backup/backup_v1001.img.gz

512+0 records in

512+0 records out

12884901888 bytes (13 GB) copied, 509.413 s, 25.3 MB/s

real    8m29.488s

user    8m15.028s

sys     0m18.228s

When switching the compression method to lzop:

Code

#  time /bin/dd if=/dev/kvmvg/backup-v1001 bs=24M iflag=direct | /bin/lzop -2 > /backup/backup_v1001.img.lzo

512+0 records in

512+0 records out

12884901888 bytes (13 GB) copied, 34.5307 s, 373 MB/s

real    0m39.789s

user    0m17.098s

sys      0m17.185s

Using gzip : 25.3MB/s and 8min 29 seconds for the backup to complete.
Using lzop : 373MB/s and 38 seconds for the backup to complete.

That is a 1300% increase in backup speed and in reduction in backup time.

Note:
1) During these tests, gzip caused higher server load than lzop (lzop was marginally better).
2) The lzop backup was slightly larger than the gzip (300MB on a 12GB disk), but storage is cheap.
3) The /backup directory is an NFS share from our NAS server, so this also writing is across the network.

We have requested Virtualizor to add this as a backup option, it would greatly help with backing / restoring VM's where hours turn into minutes.

Posted By: wolke on November 8, 2018, 10:41 am | Post: 1

impressive. Will check this out.

Posted By: kaushal on November 28, 2018, 7:45 am | Post: 2

Hi,

Sorry for delay in reply, You can use pigz utility. Which has similar performance like lzop. pigz will be automatically detect by virtualizor if installed on the server.

Let us know for further information.

-----------------------
-----------------------
Regards,
Kaushal Vora
Virtualizor Team.
http://virtualizor.com/

Posted By: wolke on November 28, 2018, 11:58 am | Post: 3

Would´nt it be a good idea to install this together with virtualizor automatically?

Posted By: domainscoza on November 28, 2018, 12:45 pm | Post: 4

pigz is good and is as fast as lzop, however creates massive load on the hypervisor since it's using it's default compression (level 6) - This should be adjusted to level 2 to reduce the CPU load, or allow us to configure this value in the virtualizor configuration.

By massive load, I mean it took our AMD Epyc HV's from a load of 5.00 to over 28.00 during the backup process.

lzop however still has the lowest CPU impact, it only increased the load by 2.00

Posted By: wolke on November 28, 2018, 1:00 pm | Post: 5

hmm I did not had this issue. Just installed pigz and startet the backup process manually. Load average goes from 7.x% to 9.x% on an Intel Xeon E3-1245V2
Backup was more than 50% faster.

Posted By: domainscoza on November 28, 2018, 2:56 pm | Post: 6

Quote From : wolke November 28, 2018, 1:00 pm

hmm I did not had this issue. Just installed pigz and startet the backup process manually. Load average goes from 7.x% to 9.x% on an Intel Xeon E3-1245V2
Backup was more than 50% faster.

This is because pigz fires up threads based on the number of cores, you have 4 core on a E3 processor and we are running 32 cores, so 32 threads for pigs, maybe virtualizor can allow us to configure the number of threads for pigz.

Posted By: wolke on November 28, 2018, 3:00 pm | Post: 7

I would bet you have a too high process priority or IO class of realtime inside your backup template. If you lower that, you should be fine.

Posted By: domainscoza on November 29, 2018, 8:23 am | Post: 8

Quote From : wolke November 28, 2018, 3:00 pm

I would bet you have a too high process priority or IO class of realtime inside your backup template. If you lower that, you should be fine.

You sir are 100% correct. Thanks

Posted By: kaushal on November 29, 2018, 9:48 am | Post: 9

Hi,

Thanks @wolke for the answer.

Also I'm assuming that pigz utility will work for your backup compression related issues.

Let us know if you need any further information.

-----------------------
-----------------------
Regards,
Kaushal Vora
Virtualizor Team.
http://virtualizor.com/

Posted By: happym3 on May 20, 2019, 9:32 pm | Post: 10

nice information shared, good work.

Posted By: happym3 on May 20, 2019, 9:34 pm | Post: 11

Softaculous are doing well for some years and so

Posted By: domainscoza on May 21, 2019, 3:51 am | Post: 12

Just an update on this, using pigz, even with the lowest io class selected was still creating more load that we were happy with. So we overwrote the /usr/bin/pigz binary with a symlink to /usr/bin/lz4

The option flags for pigz and lz4 are identical, so it's a complete drop in replacement.

With compression, the backups are pretty fast (390MB/sec on the lowest IO class) and there is little to no load on the servers when backups are done.

Edited by domainscoza : May 21, 2019, 3:52 am

Posted By: wolke on May 21, 2019, 9:20 am | Post: 13

Thanks for sharing this. I wonder what will happen on OS updates. Did you ever got trouble with that?

Posted By: domainscoza on May 21, 2019, 10:50 am | Post: 14

Quote From : wolke May 21, 2019, 9:20 am

Thanks for sharing this. I wonder what will happen on OS updates. Did you ever got trouble with that?

Absolutely no problems at all, works perfect for backups and restores and to get around updates, you just add pigz to your package managers excludes so it won't update pigz and overwrite the symlink.

Posted By: wolke on July 18, 2019, 8:14 am | Post: 15

replacing the symlink to lzop can lead to master desaster!

We tested the replacement and it seemed to work perfectly but then...
We had several servers running into kernel panic after a kernel update took place and the servers had been rebooted.

The console message was:

Quote

kernel panic not syncing vfs unable to mount root fs on unknown-block"

Here is why:
When unpacking the kernel image, lzop will be used also (because of the changed symlink) but fails with several errors:

Quote

cpio - premature end of archive

Unable to decompress /boot/initramfs-3-10[....] - unknown format

and on cloudlinux server also:

Quote

Error communicating with CLN - Bad request

lzop can´t decompress several archives and thus the kernel update (exactly the initramfs generation) fails.

This happened exactly for kernel 3.10.0-962.3.2.lve1.5.25.12.el7.x86_64 on Centos7/CloudLInux

This really nasty bug has been detected by incredible CloudLinux support staff (in 3rd level support). They removed the /usr/bin/pigz symlink to lzop and initramfs generated smoothly and the server could reboot into the new kernel.
We will stay with pigz binary therefore.

Edited by wolke : July 18, 2019, 8:16 am

Posted By: domainscoza on July 18, 2019, 8:46 am | Post: 16

Hi,

Thanks for the heads up on this.

@Softaculous - please can you allow us to specify the location to lzop so we don't need to do the symlink.

Posted By: wolke on July 18, 2019, 10:11 am | Post: 17

There might be an even better alternative than lzop, specially when it comes to cpu time. I´am not into academic overoptimization but it seems that´s worth testing under reallife conditions.

Source: https://hannuhartikainen.fi/blog/everyday-compression-tool/

Abstract:
tool filesize time CPU time
pigz 35125348 1.46 8.82
lbzip2 20301117 5.33 32.16
pixz 19333820 34.90 211.77
pzstd 39684543 0.45 2.51

Posted By: domainscoza on July 22, 2019, 5:09 am | Post: 18

Quote From : wolke July 18, 2019, 10:11 am

Great find, this is really impressive:

Using Lzop:

Code
[root@node02]# time /bin/dd if=/dev/kvmvg/backup-v1002 bs=24M iflag=direct | /bin/lzop -2 > /backup/backup-v1002-lzop.tar.gz

512+0 records in

512+0 records out

12884901888 bytes (13 GB) copied, 53.2806 s, 242 MB/s

real    0m53.527s

user    0m21.149s

sys     0m28.381s

Using pzstd:

Code
[root@node02]# whereis pzstd

pzstd: /usr/bin/pzstd /usr/share/man/man1/pzstd.1.gz

[root@node02 tmp]# time /bin/dd if=/dev/kvmvg/backup-v1002 bs=24M iflag=direct | /usr/bin/pzstd -2 > /backup/backup-v1002.tar.gz

512+0 records in

512+0 records out

12884901888 bytes (13 GB) copied, 20.8995 s, 617 MB/s

real    0m23.406s

user    0m51.984s

sys     0m25.450s

Whats interesting is that the filesize with lzop is 5.1GB and with pzstd it's 3.8GB

Edited by domainscoza : July 22, 2019, 5:11 am

Posted By: wolke on July 22, 2019, 9:26 am | Post: 19

Great results!
@softacoulus: please add the ability to specify the compressor/decompressor that should be used for backups so we do no longer need to do it at OS level.

Posted By: factoriadigital on December 31, 2019, 8:52 pm | Post: 20

Quote From : wolke July 22, 2019, 9:26 am

Great results!
@softacoulus: please add the ability to specify the compressor/decompressor that should be used for backups so we do no longer need to do it at OS level.

Just in case, quoting @softaculous as there was a typo on the mentioned username. Great info guys, thanks.

How it's working for you so far? did you replace pigz with pzstd? did virtualizor made some update on the features requested on this thread or did you find some workaround for implementing lzop or pzstd?

Regards

Posted By: Automata on January 12, 2020, 6:47 am | Post: 21

Quote From : domainscoza November 8, 2018, 9:20 am

Code
# lvcreate -L12G -s -n backup-v1001 /dev/kvmvg/vsv1001-dbbiylg3awl9iyka-ynfcfcgknopy4drs

# time /bin/dd if=/dev/kvmvg/backup-v1001 bs=24M iflag=direct | /bin/gzip -c > /backup/backup_v1001.img.gz

512+0 records in

512+0 records out

12884901888 bytes (13 GB) copied, 509.413 s, 25.3 MB/s

real    8m29.488s

user    8m15.028s

sys     0m18.228s

When switching the compression method to lzop:

Code

#  time /bin/dd if=/dev/kvmvg/backup-v1001 bs=24M iflag=direct | /bin/lzop -2 > /backup/backup_v1001.img.lzo

512+0 records in

512+0 records out

12884901888 bytes (13 GB) copied, 34.5307 s, 373 MB/s

real    0m39.789s

user    0m17.098s

sys      0m17.185s

Hi "domainscoza",

Thank you for sharing this.

+1 for me !

-----------------------
« Stay hungry, stay foolish. »

Posted By: mkools on January 13, 2020, 2:26 pm | Post: 22

Great topic, will follow this. I would love faster backups and restores!

Posted By: factoriadigital on March 18, 2020, 4:02 pm | Post: 23

For @softaculous or if anyone else knows.

We have installed the pigz application to increase the speed for backups since they are very slow but backups continue to be done with gzip.

Can you tell us what changes we must make to enable the pigz to perform the backups?

Using centos 7, installed pigz using package manager, we even uninstalled, reinstalled pigz and restarted whole virtualizor vps node, same result.

Posted By: wolke on March 26, 2020, 8:11 am | Post: 24

Unfortunately Virtualizor still does not allow to select the compressor binary that it should use, so we need to do it on OS level.
You need to replace the gzip binary in /usr/bin/gzip with a symlink to /usr/bin/pigz
You may also want to add an exclusion to /etc/yum.conf, so that your symlink will not be overwritten with a new gzip binary:

Code
exclude=gzip

Posted By: domainscoza on March 26, 2020, 10:04 am | Post: 25

Quote From : wolke March 26, 2020, 8:11 am

Unfortunately Virtualizor still does not allow to select the compressor binary that it should use, so we need to do it on OS level.

We been using the pzstd and it really works great.

This is such a simple feature for Virtualizor to build in and has the largest quality of life benefit with regard to backups. I simply do not understand why they have not built this in yet.

Posted By: wolke on March 26, 2020, 1:04 pm | Post: 26

Quote

We been using the pzstd and it really works great.

Glad to hear. I´m still scared that this maybe is not 100% compatible with gzip. (Because I never tested...) As pigz is proofed working and performance is way better than gzip, I will stay with pigz for now.

Posted By: jasavps on April 13, 2020, 11:03 pm | Post: 27

Quote From : domainscoza March 26, 2020, 10:04 am

Quote From : wolke March 26, 2020, 8:11 am

Unfortunately Virtualizor still does not allow to select the compressor binary that it should use, so we need to do it on OS level.

Agree with this

-----------------------
jasa install vps | vps managed

Posted By: domainscoza on January 22, 2021, 7:44 am | Post: 28

@chirag @softaculous

When can we expect zstd support for backups in virtualizor?

- Dave

Posted By: ServerAstra on February 5, 2022, 7:40 pm | Post: 29

1 year later still no update on this?

Posted By: chirag on February 10, 2022, 6:11 am | Post: 30

Hi,

We have added zstd and lzop compression for backups in coming patch (3.0.9.3).

Let us know for further information.

Edited by chirag : February 10, 2022, 6:12 am

-----------------------
Virtualizor - VPS Control Panel
Follow us on Facebook

Posted By: wolke on February 10, 2022, 12:53 pm | Post: 31

Big THUMBS UP for that!