Way Faster VPS backups https://www.softaculous.com/board/index.php?tid=14659 <![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p49272
Quote From : domainscoza March 26, 2020, 10:04 am
Quote From : wolke March 26, 2020, 8:11 am
Unfortunately Virtualizor still does not allow to select the compressor binary that it should use, so we need to do it on OS level.



We been using the pzstd and it really works great.

This is such a simple feature for Virtualizor to build in and has the largest quality of life benefit with regard to backups. I simply do not understand why they have not built this in yet.


Agree with this]]>
Mon, 13 Apr 2020 23:03:09 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p49272
<![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48982
Quote
We been using the pzstd and it really works great.


Glad to hear. I´m still scared that this maybe is not 100% compatible with gzip. (Because I never tested...)  As pigz is proofed working and performance is way better than gzip, I will stay with pigz for now.
]]>
Thu, 26 Mar 2020 13:04:28 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48982
<![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48979
Quote From : wolke March 26, 2020, 8:11 am
Unfortunately Virtualizor still does not allow to select the compressor binary that it should use, so we need to do it on OS level.



We been using the pzstd and it really works great.

This is such a simple feature for Virtualizor to build in and has the largest quality of life benefit with regard to backups. I simply do not understand why they have not built this in yet.]]>
Thu, 26 Mar 2020 10:04:35 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48979
<![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48978 You need to replace the gzip binary in /usr/bin/gzip with a symlink to /usr/bin/pigz
You may also want to add an exclusion to /etc/yum.conf, so that your symlink will not be overwritten with a new gzip binary:

Code
exclude=gzip


]]>
Thu, 26 Mar 2020 08:11:41 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48978
<![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48912
We have installed the pigz application to increase the speed for backups since they are very slow but backups continue to be done with gzip.

Can you tell us what changes we must make to enable the pigz to perform the backups?

Using centos 7, installed pigz using package manager, we even uninstalled, reinstalled pigz and restarted whole virtualizor vps node, same result.]]>
Wed, 18 Mar 2020 16:02:59 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48912
<![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48396 ]]> Mon, 13 Jan 2020 14:26:53 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48396 <![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48381
Quote From : domainscoza November 8, 2018, 9:20 am
I've already logged a ticket for this, but if anyone is interested there is a much faster way to do VPS backups by switching out gzip for lzop compression.

Here is a the what the current virtualizor backup does:

Code
# lvcreate -L12G -s -n backup-v1001 /dev/kvmvg/vsv1001-dbbiylg3awl9iyka-ynfcfcgknopy4drs


# time /bin/dd if=/dev/kvmvg/backup-v1001 bs=24M iflag=direct | /bin/gzip -c > /backup/backup_v1001.img.gz

512+0 records in
512+0 records out
12884901888 bytes (13 GB) copied, 509.413 s, 25.3 MB/s

real    8m29.488s
user    8m15.028s
sys    0m18.228s



When switching the compression method to lzop:

Code

#  time /bin/dd if=/dev/kvmvg/backup-v1001 bs=24M iflag=direct | /bin/lzop -2 > /backup/backup_v1001.img.lzo
512+0 records in
512+0 records out
12884901888 bytes (13 GB) copied, 34.5307 s, 373 MB/s

real    0m39.789s
user    0m17.098s
sys      0m17.185s



Using gzip : 25.3MB/s and 8min 29 seconds for the backup to complete.
Using lzop : 373MB/s and 38 seconds for the backup to complete.

That is a 1300% increase in backup speed and in reduction in backup time.

Note:
1) During these tests, gzip caused higher server load than lzop (lzop was marginally better).
2) The lzop backup was slightly larger than the gzip (300MB on a 12GB disk), but storage is cheap.
3) The /backup directory is an NFS share from our NAS server, so this also writing is across the network.


We have requested Virtualizor to add this as a backup option, it would greatly help with backing / restoring VM's where hours turn into minutes.


Hi "domainscoza",

Thank you for sharing this.

+1 for me !]]>
Sun, 12 Jan 2020 06:47:32 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48381
<![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48311
Quote From : wolke July 22, 2019, 9:26 am
Great results!
@softacoulus: please add the ability to specify the compressor/decompressor that should be used for backups so we do no longer need to do it at OS level.


Just in case, quoting @softaculous as there was a typo on the mentioned username. Great info guys, thanks.


How it's working for you so far? did you replace pigz with pzstd? did virtualizor made some update on the features requested on this thread or did you find some workaround for implementing lzop or pzstd?

Regards
]]>
Tue, 31 Dec 2019 20:52:29 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p48311
<![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p46694 @softacoulus: please add the ability to specify the compressor/decompressor that should be used for backups so we do no longer need to do it at OS level.
]]>
Mon, 22 Jul 2019 09:26:25 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p46694
<![CDATA[Way Faster VPS backups]]> https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p46687
Quote From : wolke July 18, 2019, 10:11 am
There might be an even better alternative than lzop, specially when it comes to cpu time. I´am not into academic overoptimization but it seems that´s worth testing under reallife conditions.

Source: https://hannuhartikainen.fi/blog/everyday-compression-tool/

Abstract:
tool    filesize      time  CPU time
pigz      35125348    1.46    8.82
lbzip2  20301117      5.33  32.16
pixz    19333820      34.90  211.77
pzstd  39684543      0.45  2.51



Great find, this is really impressive:

Using Lzop:

Code
[root@node02]# time /bin/dd if=/dev/kvmvg/backup-v1002 bs=24M iflag=direct | /bin/lzop -2 > /backup/backup-v1002-lzop.tar.gz
512+0 records in
512+0 records out
12884901888 bytes (13 GB) copied, 53.2806 s, 242 MB/s

real    0m53.527s
user    0m21.149s
sys    0m28.381s


Using pzstd:

Code
[root@node02]# whereis pzstd
pzstd: /usr/bin/pzstd /usr/share/man/man1/pzstd.1.gz
[root@node02 tmp]# time /bin/dd if=/dev/kvmvg/backup-v1002 bs=24M iflag=direct | /usr/bin/pzstd -2 > /backup/backup-v1002.tar.gz
512+0 records in
512+0 records out
12884901888 bytes (13 GB) copied, 20.8995 s, 617 MB/s

real    0m23.406s
user    0m51.984s
sys    0m25.450s


Whats interesting is that the filesize with lzop is 5.1GB and with pzstd it's 3.8GB]]>
Mon, 22 Jul 2019 05:09:07 GMT https://www.softaculous.com/board/index.php?tid=14659&tpg=2#p46687