Backup size much bigger than actual data

[zyrex]

Hello
I'm running the free version on a system I'm trying to backup (volume level backup)
Everything starts out normally, but after a slow start, it continues far past the data size on disk.
So I have about 2.8TB on the drive, but after 30 hours the backup is still going, and it has transferred 9 TB to the repository.
(Also says something about the slow speed)

I've also tried out file level backup, tried to backup just folder with even less data, but same thing here. A day later, 3TB of 1TB has been transferred....

I just ran a fsck on it, hoping that will do the trick. (ext4)

[PTide]

Hi,

What you've described resembles a known bug with VAL's BitLooker (the thing that makes sure that empty space does not get into backup).
Would you give some overview of the system, please? Specifically, lsblk -af, df -h, distro, and the kernel version.

Thanks!

[zyrex]

Certainly!


OpenMediaVault running on Debian 8.11
Kernel 4.9.0-0.bpo.6-amd64

Code: Select all

NAME   FSTYPE            LABEL             UUID                                 MOUNTPOINT
sdf    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sdd    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sdb    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sr0
sdg    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sde    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sdc    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sda    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring
sdj
+-sdj5 swap                                f9c6d287-9933-47f7-80de-6c1da94b0d5c
+-sdj1 ext4                                ce95a526-c309-4fc8-8783-74bb66e112aa /
+-sdj2
sdh    linux_raid_member filserver:Lagring fe7f3f6f-d7da-a96d-0904-3a4cc578a453
+-md0  ext4              disk              63304388-ea19-40c3-bbe6-3ca08e1f76d5 /srv/dev-disk-by-id-md-name-filserver-Lagring

Code: Select all

Filesystem      Size  Used Avail Use% Mounted on
udev             10M     0   10M   0% /dev
tmpfs            15G  1.5G   13G  11% /run
/dev/sdj1        27G  2.2G   24G   9% /
tmpfs            36G     0   36G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            36G     0   36G   0% /sys/fs/cgroup
tmpfs            36G     0   36G   0% /tmp
folder2ram       36G  167M   36G   1% /var/log
folder2ram       36G     0   36G   0% /var/tmp
folder2ram       36G  948K   36G   1% /var/lib/openmediavault/rrd
folder2ram       36G  1.4M   36G   1% /var/spool
folder2ram       36G   33M   36G   1% /var/lib/rrdcached
folder2ram       36G  8.0K   36G   1% /var/lib/monit
folder2ram       36G     0   36G   0% /var/lib/php5
folder2ram       36G     0   36G   0% /var/lib/netatalk/CNID
folder2ram       36G  468K   36G   1% /var/cache/samba
/dev/md0         11T  2.6T  8.4T  24% /srv/dev-disk-by-id-md-name-filserver-Lagring

[PTide]

Ok, basically that is a Debian8.11-based NAS with a backported kernel. There is a bunch of disks inside that are combined into RAID with mdadm, and that RAID is the main storage for file-server.
The only thing that comes to my mind and that would explain why the software does not skip the empty space is that something is missing from that kernel backport. VAL has been tested on Debian up to v9.4 and we haven't seen such issue yet. Given that VAL hasn't been tested with backports, and a new version of VAL will be released very soon, I'd suggest to wait a little for the update.

P.S. Are you using flash drive as a system drive? I am asking because of that folder2ram thing.

Thanks!

[zyrex]

Aha, yes.. That sounds about right.
I will probably get this one updated to Debian 9 soon, but I will wait for the update in the mean time.

The system is running off a flash drive, so that's right about the folder2ram.
I have another system running the same OS (all versions the same) but without the flashdrive (so normal harddrive for system drive), which does not suffer the same bug.

[PTide]

That's interesting...The type of storage where the actual OS resides shouldn't affect the way the backup process works for the other partition that runs atop of RAID.
Would you kindly let us know after upgrading the box and the agent (once it's released) to tell us if it fixed the issue? Because if it doesn't fix it, then it must have something to do with folder2ram, I guess.

Thanks

[Matze208]

Hello
I hope it's okay if I append additional information to this topic, I have the same problem with the free agent, but running on ubuntu. Here while doing file-based backup:

Code: Select all

Veeam Agent for Linux: veeam.
Version: 2.0.1.665
sysname : Linux
release : 4.4.0-62-generic
version : #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017
machine : x86_64

Duration:           02:14:21         Processed:          8.4 GB (100%)
Processing rate:    1.2 MB/s         Read:               8.4 GB
Bottleneck:         Agent            Transferred:        60.7 GB (0.1x)

NAME                   FSTYPE            LABEL     UUID                                   MOUNTPOINT
sda
├─sda1                 ext2                        0375159e-2d8a-4add-9eaf-8df4256d5c28   /boot
├─sda2
└─sda5                 LVM2_member                 GV2jF5-u0pr-A2od-Nc56-JZYm-4Eak-ZRJX12
  ├─pandora--vg-root   ext4                        0b3e905b-6255-43c8-87d2-c1fc78acef2f   /
  └─pandora--vg-swap_1 swap                        6ae46ed2-4afb-4dae-8477-a53cad635dcc   [SWAP]
sdb                    linux_raid_member pandora:0 d94bc9ee-bd62-2504-9d34-5a148ef56b42
└─md0                  ext4                        dea580e9-b5ec-4b7f-bcd4-32da3c402cf9   /media/share
sdc                    linux_raid_member pandora:0 d94bc9ee-bd62-2504-9d34-5a148ef56b42
└─md0                  ext4                        dea580e9-b5ec-4b7f-bcd4-32da3c402cf9   /media/share
sdd                    linux_raid_member pandora:0 d94bc9ee-bd62-2504-9d34-5a148ef56b42
└─md0                  ext4                        dea580e9-b5ec-4b7f-bcd4-32da3c402cf9   /media/share
sr0
loop0
loop1
loop2
loop3
loop4
loop5
loop6
loop7

udev                          981M       0  981M    0% /dev
tmpfs                         200M     24M  177M   12% /run
/dev/mapper/pandora--vg-root   18G     12G  5,0G   70% /
tmpfs                        1000M    8,0K 1000M    1% /dev/shm
tmpfs                         5,0M       0  5,0M    0% /run/lock
tmpfs                        1000M       0 1000M    0% /sys/fs/cgroup
/dev/sda1                     472M     58M  390M   13% /boot
/dev/md0                      7,3T    1,7T  5,3T   24% /media/share
tmpfs                         200M       0  200M    0% /run/user/1000

Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-62-generic x86_64)

Doing a Backup from the normal vg is running fine. The problem is only on the mdadm raid share.

Thanks!

[PTide]

Hi,

Yes, thank you finding the existing thread on the subject instead of starting a new one : )

Would you clarify something, please - do you actually observe a ~50Gb backup file on the repository?

The thing is, that 'transferred' value might get quite big for file-level jobs due to a need to transfer a lot of filesystem metadata back and forth between the source and the target. That is, it does not always reflect the size of the backup file.

It's a known issue and our team is already revising the existing architecture.

Thanks!

[Matze208]

Thanks for your fast reply

Indeed there is a 62,1 GB .vbk file on the backup target.

[PTide]

Well, although filesystem metadata can really take space larger that the actual data, the figure is still big. I'd suggest to contact support team directly so they can confirm that this is the case.

Thanks

[zyrex]

That's interesting...The type of storage where the actual OS resides shouldn't affect the way the backup process works for the other partition that runs atop of RAID.
Would you kindly let us know after upgrading the box and the agent (once it's released) to tell us if it fixed the issue? Because if it doesn't fix it, then it must have something to do with folder2ram, I guess.

Thanks

Alright, I've upgraded the box to the newest OMV, which uses Debian Stretch, newer Kernel (4.9.0-8-amd64) - no backport and upgraded to the newest agent. The result remains the same, unfortunately.

Code: Select all

Duration:           26:07:04         Processed:          10.9 TB (100%)
Processing rate:    121.8 MB/s       Read:               10.9 TB
Bottleneck:         Network          Transferred:        9.1 TB (1.2x)

So this is the 2.6 TB large volume (used space). The backup-file on the B&R server does consume all 9.1 TB.

What's interesting is that the next, incremental backup behaves as expected.

Code: Select all

Duration:           00:33:41         Processed:          2.6 TB (100%)
Processing rate:    3.9 MB/s         Read:               7.8 GB
Bottleneck:         Source           Transferred:        485.6 MB (16.4x)

This would be more in line with how I would've expected the full backup to look.

[PTide]

Would you provide some logs, please? You can upload them to some filesharing and send me the link in PM.

Thanks!

[zyrex]

Logs sent!

[zyrex]

Any ideas, @PTide ?

[PTide]

QA team is still researching the issue. Let me give you a brief overview:

VAL's ability to skip empty space on partitions applies to EXT4 and XFS filesystems only. Since you run EXT4 on that 10TB partition, it should work fine. Considering that everything works on the other box that is identical except for folder2ram presene, I'd say that it is folder2ram what interferes with the process. I'll update this thread once I get any news.

Thanks

[zyrex]

Right, so yesterday we swapped out the USB-drive with an SSD, and removed the folder2ram since it wasn't needed anymore.
We then started a new full backup, but unfortunately, today it's 51% done and it has transferred 4.3 TB /processed 5.7 TB.

[PTide]

Hi,

Kindly upload logs when the job is finished/failed.

Thanks!

[zyrex]

Right, so, I've been setting up a new array on the server everything was working on.
It's a 5.4TB volume with about 600GB in use.

I had to move around some files and put a ~600GB file on it temporarily, and deleted it later the same day.
Turns out I seem to encounter the same Bitlooker bug? My first full backup was ~600GB (before my file shuffling), but then the next inkremental was another 600GB (after file shuffling), resulting now in full backups at 1.2TB since then.

I was hoping the next full (synthetic) would rectify this, but it didn't.

[yakamoneye18]

Hey everyone,

having the same issue here, and just wanted to check if there is a solution for this.

I installed a hardware Debian server today, installed Veeam Agent for Linux on it and wanted to test it for the first time (VBR and Windows Agent user for years...). I have a SSD with 500GB inside, at the moment there are only 30GB of data. I canceled the backup job after it ran for about 2 hours, transferred 75GB of data and was at 6% progress.

Are there any news about this problem?

Thanks,

Tobias

[PTide]

Hi,

I canceled the backup job after it ran for about 2 hours, transferred 75GB of data and was at 6% progress.

Those are very generic symptoms that might be caused by many different factors. Would you provide some additional data, please? Bottleneck stats, disks layout, target type, and backup mode (volume/file-level) would be enough for the beginning.

Thanks!

[yakamoneye18]

Hi P.Tide,

here a few more information about the test:

output of df -T -h

Code: Select all

Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs  3.9G     0  3.9G   0% /dev
tmpfs          tmpfs     798M   25M  774M   4% /run
/dev/sdc1      ext4      450G  6.8G  420G   2% /
tmpfs          tmpfs     3.9G     0  3.9G   0% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs          tmpfs     798M     0  798M   0% /run/user/0
tmpfs          tmpfs     3.9G   14M  3.9G   1% /opt/omd/sites/ACH/tmp

Veeam Job stats (where do I get more information about the bottleneck stats?):

Code: Select all

 Summary                              Data

     Duration:           02:25:46         Processed:          178.9 GB (7%)
     Processing rate:    21.3 MB/s        Read:               178.9 GB
     Bottleneck:         Target           Transferred:        94.8 GB (1.9x)

Backup mode is "Entire Machine" (would this be the problem - should I use Volume level?).

The target is a QNAP nas. It is attached to our hardware Veeam server via iSCSI (10G Ethernet over fiber) - in VAL I added the local drive on the hardware server with the admin share as repository. The agent is not connected with our VBR installation.

Thanks,

Tobias

[yakamoneye18]

Well, I guess this is called rubberducking - just by talking to you about it, I solved it. I changed it to volume level, selected the device, and now the backup size is 2.9Gb, that's what I wanted.

I am not a Linux expert, and completely new to VAL, so I do not realy understand why this works now, but at least it does

Thanks for listening!

Regards,

Tobias

[PTide]

*quack-qua...Holup for a second! Although I am glad that you've resolved it, something does not add up, and I would like ask a couple of questions:

1.

I changed it to volume level, selected the device, and now the backup size is 2.9Gb

Which device did you select, may I ask? I am looking at your df -T -h and cannot understand which one - none of your devices has that amount of space occupied.
Also you said that there is only 30 GB of data, but I can't see which device holds it.

2.

Code: Select all

tmpfs          tmpfs     3.9G   14M  3.9G   1% /opt/omd/sites/ACH/tmp

This FS looks odd to me, what's that?

Thanks!

[yakamoneye18]

Hi P.Tide,

I selected sdc1 to backup, and I just tried a file level restore, and the restore point was successfully mounted and contained all data I expected.
What really is strange: our monitoring (check_mk - this server is the check_mk server...) sais that the file system usage is 30GB. Veeam saw 15 to process, and df only sees 7 GB on the disk I backed up.

Code: Select all

Summary                              Data

     Duration:           00:04:55         Processed:          15.7 GB (100%)
     Processing rate:    95.4 MB/s        Read:               15.7 GB
     Bottleneck:         Target           Transferred:        2.9 GB (5.5x)

In fact that is a behaviour I cannot at all explain, but like I said, I only use Linux for small webservers, I am mainly a Windows guy....

about the tmp mountpoints - it looks like they are part of the monitoring software - omd is the OpenMonitoring Distribution. But I could not find anything about why this mountpoint is created at all.

[PTide]

Ok, thank you for that info, and one last question then - was it the first succesful job run (i.e. full backup) or it was incremental backup?

Thanks!

[yakamoneye18]

This was the first successfull full backup VAL created, after 2 failed attempts I manually canceled after a few hours.

[PTide]

That's a good compression ratio then - 5.5x. Thank you for the information, would you kindly open a support case and tell our support team about your experience with "Entire machine" mode? The thing is that it doesn't look like an expected behaviour, therefore I'd like our team to investigate that, and they will need your logs.

Thank you!

[yakamoneye18]

Hello P.Tide,

I opened a case (#03664323). While talking to support, I realized that we have two harddrives in the server (1TB each) that we do not need and that are not used. Once I removed these drives, "Full Machine Backup" was just processing our OS disk just like we wanted to. So the problem were the two disks I forgot about... At first, support wanted to check why Veeam is backing up these disks even though there is no file system found on them. Now they tell me this behavior is by design, the harddrives should be excluded via ini file. Since we do not even use these disk, I just removed them from the server and everything is fine now...

Thanks!

Tobias