Standalone backup agent for Linux servers and workstations on-premises or in the public cloud
zyrex
Influencer
Posts: 17
Liked: 2 times
Joined: Jan 21, 2018 7:55 pm
Contact:

Backup size much bigger than actual data

Post by zyrex » Dec 18, 2018 7:33 pm

Hello
I'm running the free version on a system I'm trying to backup (volume level backup)
Everything starts out normally, but after a slow start, it continues far past the data size on disk.
So I have about 2.8TB on the drive, but after 30 hours the backup is still going, and it has transferred 9 TB to the repository.
(Also says something about the slow speed)

I've also tried out file level backup, tried to backup just folder with even less data, but same thing here. A day later, 3TB of 1TB has been transferred....

I just ran a fsck on it, hoping that will do the trick. (ext4)

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup size much bigger than actual data

Post by PTide » Dec 18, 2018 8:06 pm

Hi,

What you've described resembles a known bug with VAL's BitLooker (the thing that makes sure that empty space does not get into backup).
Would you give some overview of the system, please? Specifically, lsblk -af, df -h, distro, and the kernel version.

Thanks!

zyrex
Influencer
Posts: 17
Liked: 2 times
Joined: Jan 21, 2018 7:55 pm
Contact:

Re: Backup size much bigger than actual data

Post by zyrex » Dec 18, 2018 8:24 pm

Certainly!


OpenMediaVault running on Debian 8.11
Kernel 4.9.0-0.bpo.6-amd64

Code: Select all

NAME   FSTYPE            LABEL             UUID                                 MOUNTPOINT
sdf    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sdd    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sdb    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sr0
sdg    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sde    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sdc    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sda    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sdj
+-sdj5 swap                                f9c6d287-9933-47f7-80de-6c1da94b0d5c
+-sdj1 ext4                                ce95a526-c309-4fc8-8783-74bb66e112aa /
+-sdj2
sdh    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring

Code: Select all

Filesystem      Size  Used Avail Use% Mounted on
udev             10M     0   10M   0% /dev
tmpfs            15G  1.5G   13G  11% /run
/dev/sdj1        27G  2.2G   24G   9% /
tmpfs            36G     0   36G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            36G     0   36G   0% /sys/fs/cgroup
tmpfs            36G     0   36G   0% /tmp
folder2ram       36G  167M   36G   1% /var/log
folder2ram       36G     0   36G   0% /var/tmp
folder2ram       36G  948K   36G   1% /var/lib/openmediavault/rrd
folder2ram       36G  1.4M   36G   1% /var/spool
folder2ram       36G   33M   36G   1% /var/lib/rrdcached
folder2ram       36G  8.0K   36G   1% /var/lib/monit
folder2ram       36G     0   36G   0% /var/lib/php5
folder2ram       36G     0   36G   0% /var/lib/netatalk/CNID
folder2ram       36G  468K   36G   1% /var/cache/samba
/dev/md0         11T  2.6T  8.4T  24% /srv/dev-disk-by-id-md-name-filserver-Lagring

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup size much bigger than actual data

Post by PTide » Dec 18, 2018 9:21 pm

Ok, basically that is a Debian8.11-based NAS with a backported kernel. There is a bunch of disks inside that are combined into RAID with mdadm, and that RAID is the main storage for file-server.
The only thing that comes to my mind and that would explain why the software does not skip the empty space is that something is missing from that kernel backport. VAL has been tested on Debian up to v9.4 and we haven't seen such issue yet. Given that VAL hasn't been tested with backports, and a new version of VAL will be released very soon, I'd suggest to wait a little for the update.

P.S. Are you using flash drive as a system drive? I am asking because of that folder2ram thing.

Thanks!

zyrex
Influencer
Posts: 17
Liked: 2 times
Joined: Jan 21, 2018 7:55 pm
Contact:

Re: Backup size much bigger than actual data

Post by zyrex » Dec 18, 2018 9:39 pm

Aha, yes.. That sounds about right.
I will probably get this one updated to Debian 9 soon, but I will wait for the update in the mean time.

The system is running off a flash drive, so that's right about the folder2ram.
I have another system running the same OS (all versions the same) but without the flashdrive (so normal harddrive for system drive), which does not suffer the same bug.

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup size much bigger than actual data

Post by PTide » Dec 18, 2018 10:34 pm

That's interesting...The type of storage where the actual OS resides shouldn't affect the way the backup process works for the other partition that runs atop of RAID.
Would you kindly let us know after upgrading the box and the agent (once it's released) to tell us if it fixed the issue? Because if it doesn't fix it, then it must have something to do with folder2ram, I guess.

Thanks

Matze208
Lurker
Posts: 2
Liked: never
Joined: Jan 21, 2019 2:21 pm
Contact:

Re: Backup size much bigger than actual data

Post by Matze208 » Jan 21, 2019 2:44 pm

Hello
I hope it's okay if I append additional information to this topic, I have the same problem with the free agent, but running on ubuntu. Here while doing file-based backup:

Code: Select all

Veeam Agent for Linux: veeam.
Version: 2.0.1.665
sysname : Linux
release : 4.4.0-62-generic
version : #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017
machine : x86_64

Duration:           02:14:21         Processed:          8.4 GB (100%)
Processing rate:    1.2 MB/s         Read:               8.4 GB
Bottleneck:         Agent            Transferred:        60.7 GB (0.1x)

NAME                   FSTYPE            LABEL     UUID                                   MOUNTPOINT
sda
├─sda1                 ext2                        0375159e-2d8a-4add-9eaf-8df4256d5c28   /boot
├─sda2
└─sda5                 LVM2_member                 GV2jF5-u0pr-A2od-Nc56-JZYm-4Eak-ZRJX12
  ├─pandora--vg-root   ext4                        0b3e905b-6255-43c8-87d2-c1fc78acef2f   /
  └─pandora--vg-swap_1 swap                        6ae46ed2-4afb-4dae-8477-a53cad635dcc   [SWAP]
sdb                    linux_raid_member pandora:0 d94bc9ee-bd62-2504-9d34-5a148ef56b42
└─md0                  ext4                        dea580e9-b5ec-4b7f-bcd4-32da3c402cf9   /media/share
sdc                    linux_raid_member pandora:0 d94bc9ee-bd62-2504-9d34-5a148ef56b42
└─md0                  ext4                        dea580e9-b5ec-4b7f-bcd4-32da3c402cf9   /media/share
sdd                    linux_raid_member pandora:0 d94bc9ee-bd62-2504-9d34-5a148ef56b42
└─md0                  ext4                        dea580e9-b5ec-4b7f-bcd4-32da3c402cf9   /media/share
sr0
loop0
loop1
loop2
loop3
loop4
loop5
loop6
loop7

udev                          981M       0  981M    0% /dev
tmpfs                         200M     24M  177M   12% /run
/dev/mapper/pandora--vg-root   18G     12G  5,0G   70% /
tmpfs                        1000M    8,0K 1000M    1% /dev/shm
tmpfs                         5,0M       0  5,0M    0% /run/lock
tmpfs                        1000M       0 1000M    0% /sys/fs/cgroup
/dev/sda1                     472M     58M  390M   13% /boot
/dev/md0                      7,3T    1,7T  5,3T   24% /media/share
tmpfs                         200M       0  200M    0% /run/user/1000

Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-62-generic x86_64)
Doing a Backup from the normal vg is running fine. The problem is only on the mdadm raid share.

Thanks!

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup size much bigger than actual data

Post by PTide » Jan 21, 2019 3:13 pm

Hi,

Yes, thank you finding the existing thread on the subject instead of starting a new one : )

Would you clarify something, please - do you actually observe a ~50Gb backup file on the repository?

The thing is, that 'transferred' value might get quite big for file-level jobs due to a need to transfer a lot of filesystem metadata back and forth between the source and the target. That is, it does not always reflect the size of the backup file.

It's a known issue and our team is already revising the existing architecture.

Thanks!

Matze208
Lurker
Posts: 2
Liked: never
Joined: Jan 21, 2019 2:21 pm
Contact:

Re: Backup size much bigger than actual data

Post by Matze208 » Jan 21, 2019 3:40 pm

Thanks for your fast reply

Indeed there is a 62,1 GB .vbk file on the backup target.

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup size much bigger than actual data

Post by PTide » Jan 21, 2019 6:12 pm

Well, although filesystem metadata can really take space larger that the actual data, the figure is still big. I'd suggest to contact support team directly so they can confirm that this is the case.

Thanks

zyrex
Influencer
Posts: 17
Liked: 2 times
Joined: Jan 21, 2018 7:55 pm
Contact:

Re: Backup size much bigger than actual data

Post by zyrex » Jan 28, 2019 5:25 pm

PTide wrote:
Dec 18, 2018 10:34 pm
That's interesting...The type of storage where the actual OS resides shouldn't affect the way the backup process works for the other partition that runs atop of RAID.
Would you kindly let us know after upgrading the box and the agent (once it's released) to tell us if it fixed the issue? Because if it doesn't fix it, then it must have something to do with folder2ram, I guess.

Thanks
Alright, I've upgraded the box to the newest OMV, which uses Debian Stretch, newer Kernel (4.9.0-8-amd64) - no backport and upgraded to the newest agent. The result remains the same, unfortunately.

Code: Select all

Duration:           26:07:04         Processed:          10.9 TB (100%)
Processing rate:    121.8 MB/s       Read:               10.9 TB
Bottleneck:         Network          Transferred:        9.1 TB (1.2x)
So this is the 2.6 TB large volume (used space). The backup-file on the B&R server does consume all 9.1 TB.

What's interesting is that the next, incremental backup behaves as expected.

Code: Select all

Duration:           00:33:41         Processed:          2.6 TB (100%)
Processing rate:    3.9 MB/s         Read:               7.8 GB
Bottleneck:         Source           Transferred:        485.6 MB (16.4x)
This would be more in line with how I would've expected the full backup to look.

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup size much bigger than actual data

Post by PTide » Jan 28, 2019 8:44 pm

Would you provide some logs, please? You can upload them to some filesharing and send me the link in PM.

Thanks!

zyrex
Influencer
Posts: 17
Liked: 2 times
Joined: Jan 21, 2018 7:55 pm
Contact:

Re: Backup size much bigger than actual data

Post by zyrex » Jan 28, 2019 9:43 pm 1 person likes this post

Logs sent!

zyrex
Influencer
Posts: 17
Liked: 2 times
Joined: Jan 21, 2018 7:55 pm
Contact:

Re: Backup size much bigger than actual data

Post by zyrex » Jan 31, 2019 5:32 pm

Any ideas, @PTide ?

PTide
Product Manager
Posts: 4876
Liked: 409 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup size much bigger than actual data

Post by PTide » Jan 31, 2019 5:40 pm 1 person likes this post

QA team is still researching the issue. Let me give you a brief overview:

VAL's ability to skip empty space on partitions applies to EXT4 and XFS filesystems only. Since you run EXT4 on that 10TB partition, it should work fine. Considering that everything works on the other box that is identical except for folder2ram presene, I'd say that it is folder2ram what interferes with the process. I'll update this thread once I get any news.

Thanks

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests