Comprehensive data protection for all workloads
Post Reply
peters.
Influencer
Posts: 13
Liked: 1 time
Joined: Jun 04, 2012 8:35 am
Full Name: Peter S
Contact:

Size of linux VM backup

Post by peters. »

I'm running Postgresql DB on Centos 6.4 in ESXi 5.1 with vmtools installed. Filesystem is ext4 inside of LVM. VMFS is v3. System is not under heavy load.
Real-used size of filesystem is 6GB, but VMware sees 27GB of used space.

Each day, backup is created on local filesystem (about 2gig archive) and uploaded to remote machine. This causes probably a lot of changed blocks a day and would explain the big difference of used space seen by Vmware & Veeam and the real used space.
My question is, how to backup only the real-used space -> make smaller & faster backups. Currently the backup size is about 20GB with compression used...
I found threads about sdelete on windows, but couldn't find what to do on linux...

Thanks,
Peter
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Size of linux VM backup

Post by Gostev »

Good question. I'd like to know as well...
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Size of linux VM backup

Post by tsightler » 1 person likes this post

As is typical with Linux (and really Windows too) there are a multitude of ways to zero out currently unused space. Probably the most crude, but perhaps most likely method an old-timer like me might use, is simply good old 'cat' and /dev/zero, our trusty supply of never ending zeros, something like this:

Code: Select all

cat /dev/zero > zero.file
rm zero.file
This is pretty crude, effectively creating a file full of nothing but zeros until the filesystem is full, which will cause cat to exit with a insufficient space error, then immediately execurt the rm command to deletes the file. Unfortunately this will leave a potentially significant window where the filesystem is full since deleting a very large file can take quite some time so a slightly less crude, method might be:

Code: Select all

dd if=/dev/zero of=zero.small.file bs=1024 count=102400
cat /dev/zero > zero.file
rm zero.small.file
rm zero.file
This first creates a small file full of zeros, then the huge file full of zeros, then deletes the small file, which is a very fast operation, which should limit the time the filesystem is completely full, and then finally deletes the very large file.

Of course, this method has the same negatives as the Windows sdelete command, namely that it will effectively turn a thin provisioned VMDK into fully provisioned, and it can take a very long time. So, a more modern admin might turn to a more sophisticated tool, like "zerofree" which is typically available as a package on most modern distros. Zerofree is smarter than dd/cat in that it looks for unallocated sectors and writes them as zeros only if they currently contain non-zero data. The primary disadvantage is that the filesystem has to be unmounted, or at least mounted read-only, to use it, and it's still pretty slow, but usage is dead simple:

Code: Select all

zerofree /dev/path/to/block/device
There are also the secure delete tools, which includes sfill. It's really designed to securely erase the data (the default is something like 48 passes), but can be made to sfill with zeros with something like 'sfill -llz' (I've never used it so this is from memory, you should double check).

Also, the crude methods using dd could easily be enhanced with script to loop and check for free space. Might be a little slower, but could be worth it if your trying to do this on a running system. I could probably hack something together fairly quickly, or at least Google something similar and hack it up. I'm sure I've seen some "poor mans" secure wipe scripts which basically do the same with /dev/urandom instead of /dev/zero.
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Size of linux VM backup

Post by yizhar » 1 person likes this post

Hi.

I think that the answer is quite simple - you just need a separate storage for the purpose of the archive.

Some optional examples:

* Attach additional VMDK drive to the VM, mount it as /tmp or /back (or whatever) and use it for the backup.
Exclude this drive from Veeam backups.

* Or, if you wish to backup the backup file and not exclude it, just make the VMDK from above option with size of about 4gb,
this will force reuse of same blocks on the next backup cycle.

* Use a shared folder on the backup server using either NFS or CIFS or other protocol, and store the backup file there.

* Use guest attach iscsi volume which will not be included in the Veeam backup.

Yizhar
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Size of linux VM backup

Post by yizhar »

Hi.

BTW - My suggestions above will not reduce the size of your VBK full backup files (which will still include all those existing "dirty" blocks).
However - it will definitely reduce size of incremental backups either forward or reverse, so it will allow you to have smaller and faster daily backups.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Size of linux VM backup

Post by foggy »

Yes, what Yizhar has suggested (a separate storage for intermediary backups) sounds very reasonable to me. Easier to implement and manage afterwards.
peters.
Influencer
Posts: 13
Liked: 1 time
Joined: Jun 04, 2012 8:35 am
Full Name: Peter S
Contact:

Re: Size of linux VM backup

Post by peters. »

I finally went for separate vmdk for local backup. The main vmdk for VM is thin provisioned. Would it make sense to convert it to thick provisioned+zeroed?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Size of linux VM backup

Post by veremin »

Since you’re going to use an additional vmdk as a place for storing/deleting backup data, I wouldn’t say that having thick provisioned primary disk will be for some help here. Or may be I’m missing something? Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Size of linux VM backup

Post by foggy »

It will reduce the newly created active full size, however would not affect subsequent backups.
peters.
Influencer
Posts: 13
Liked: 1 time
Joined: Jun 04, 2012 8:35 am
Full Name: Peter S
Contact:

Re: Size of linux VM backup

Post by peters. »

Difference after creating extra partition for backup is major: now the increment backup size is ~200MB (compared to 8GB in past). Thank you for help
ASG
Enthusiast
Posts: 75
Liked: 5 times
Joined: Aug 08, 2018 10:19 am
Contact:

[MERGED] Backuping up Linux VM copies whole VMDK

Post by ASG »

Hi,

I've just switched to Veeam from Backup Exec. One thing I noticed is that our Linux VMs Backups are way larger - because B&R copies the whole VMDK (or nearly, about 90%).

Details:
Linux VM with Ubuntu 16.04
One VMDK, Boot partition(s) and than LVM for different partitions.

df prints about ~6GB used on a 60GB VMDK - B&R Backup (no Agent, vSphere Backup with quiescence enabled) backups about 58GB.

I backup to REFS (64KB) and with synthetic fulls weekly.

I hope I put all details that matter here and hope somebody could give me some advise.

Thanks!
ASG
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Size of linux VM backup

Post by PTide »

Hi,

VBR creates VM image-level backup and does not install agents inside the guests, and therefore does not have any knowledge about the underlying guest OS filesystem. Please check this post for additional info and some workarounds.

Thanks
Post Reply

Who is online

Users browsing this forum: No registered users and 199 guests