Windows vs. Linux Target

[stevil]

Chaps,
I've done some testing over the past day or so with different targets.
I have two esxi hosts connected with cisco GBit switches, all brand new top line IBM kit but using DAS rather than SAN/NAS etc..
Veeam is running on a 2003 VM with 4 CPU 4GB RAM.

My test VM is being backup up using VA mode, best compression, local target.

LINUX target
Now, if I point this backup at a linux share on a VM on the other ESXi server I get about 55MB/sec across the LAN, peaking at about 80Mb/sec. This fully loads all 4CPU's on the ubuntu Vm.
I tried increasing the CPU count to 8 on the ubuntu server and performance did increase slightly to an average of around 65MB/sec peaking at 100.5MB/sec.
With one CPU in the ubuntu server the performance was around 17Mb/sec.
CPU usage on my veeam server never rises about about 40%, and always seems to hover around 25%ish.

Windows target
If the backup is pointed at a physical windows file server (capable of 120MB/sec across the WAN) I get an average of around 13MB/sec, peaking at 25MB/sec. At the same time the Veeam server is using around 75% CPU peaking at 100%.

So, it's clear to me that, for me, backing up to a Linux server is way faster than a windows target.

After that long winded intro, I guess my question is.......why is the CPU load apparently transferred to the Linux server during backup? I'm curious as to the mechanics behind the backup process. If I understand it better it may help with our deployment decisions in future. My other question is, how do I get 'linux performance' using only windows?

Regards
Steve

[Vitaliy S.]

Hello Steve,

The explanation is pretty simple. While using Linux as a destination target for the backup job, we deploy an agent into Linux service console (you need to specify SSH credentials for that). That allows us to distribute the CPU load between the backup server and destination Linux server and improve backup job performance.

The same process cannot be applied to Windows machines, that is why all the load resides on the backup server all the time.

Thanks!

[stevil]

Does this mean that in the case of a remote backup more data would be sent across the WAN for the linux host to compress/dedupe?
I'm just running some tests to try and prove this out, but it'd be faster if you already have the knowledge

[stevil]

I'm answering my own question a little here, but I just ran a backup job twice, once with a windows target, the next time with a Linux target. It does indeed seem that more data is fed to the Linux host for it to crunch through, as you'd expect.

Windows LAN transfer = 3.4GB
Linux LAN transfer = 4.4GB

The problem I have is that I want to use a linux host for a WAN backup to take advantage of the synthetic full agent. It seems in doing this that my incremental backups will consume more bandwidth than an equivalent windows backup. Can I disable this behaviour with linux so that the veeam server does all the processing? I've not tried it using WAN target rather than LAN target, maybe that will do what I need?

Cheers
Sorry for all the questions!

[stevil]

OK, I did a test, using WAN target rather than LAN target has no effect

[Vitaliy S.]

When you deploy a remote agent, the VBK storage rebuild operation is performed locally. Though most of the tasks are performed by Veeam backup server. I'm not quite certain about the figures you've posted, have you merged them with the Network Monitor?

Besides, why don't you want to utilize the entire bandwidth? Linux target should be much quicker even with equal bandwidth for Windows and Linux targets.

[stevil]

Hi Vitaliy,
In answer to your questions:
I understand what you are saying about the linux agent doing the storage rebuild/synthetic full. This is a great feature and the only reason I'm looking to deploy a Linux target at the other end of the WAN link (this is discussed in another thread recently). Once we have V6 I'm hoping this thread will be irrelevant

The bandwidth in question is only 2Mbps WAN link, so we're talking about 200k/sec maximum. In a LAN based scenario I agree that linux is faster, across the WAN I would have to transmit an extra GB of data, which would massively outweigh any additional processing power that the linux target would give me.

The bandwidth measurements I'm taking are for data leaving the Veeam VM at the remote site.

So I guess I'm still left with the question, can I use a Linux target and only utilise the storage rebuild/synthetic full feature, and not its realtime processing of 'raw' uncompressed backup data, leaving that job to my local windows Veeam VM ?

Thanks

[Gostev]

Hi Steve, right now this is not possible (traffic compression level cannot be changed for Linux targets), but we will keep this scenario in mind for the future releases. Thanks.

[tsightler]

I have a question. In your testing scenario you stated "if I point this backup at a linux share on a VM", you do mean simply that you chose a linux target as your backup in Veeam, right? You not sharing out a linux mount point with Samba or something?

The reason I asks is that my testing (and production) has always shown the opposite of what you are saying, sending a backup job to a Linux target has always used far less bandwidth than writing to a Windows share. Admittedly this probably changed with the move to incremental jobs rather than reverse incrementals. With reverse incrementals the extra reading and writing was a huge bottleneck for backups to Windows shares and Linux was a clear win. With forward incrementals this advantage may have been wiped away (but the advantage of local agent for transformation is huge).

Also, you mentioned that WAN target didn't help much, but WAN target won't help much with a full backup, it's win is during incrementals. In our case it's lowered our nightly incrementals from 150-170GB/day to 70-100GB/day. which is huge when you're required to keep 90 days of backups online (~75GB/day savings x 90 days = 6+TB of space savings).

[stevil]

Hi tsightler,
Yes, I just added a linux server to veeam and used one of the folders as a backup target, I didn't use samba etc...
Maybe I'll try samba if you've had a different experience
All my testing was with incrementals. I should test differentials as well I guess, but I've spent a lot of time already this week testing, struggling to fit the more mundane work in as well!
You're absolutely correct with your WAN target comments. I would indeed only see a saving with the first incremental. My testing was always with the inital backup. Thanks for pointing that out. I'm looking forward to the next release where the changed block tracking will hopefully be reduced from 1MB down to 4k

[tsightler]

Oh, I wasn't suggesting that you use Samba, I would expect that to give behavior similar to Windows, but I was just trying to make sure that this wasn't what you were doing because you used the term "linux share" and I don't think of a Linux backup target as a "share". Just wanted to clarify the wording so that I was understanding your testing environment.

With incremental runs I could certainly see that backing up to a Windows share would use less bandwidth, but the lack of an agent to perform synthetic fulls is a killer for our scenario.

[bc07]

Steve I also went through weeks of testing to find the best way to backup our environment with Veeam.
Did you try setting to the compression to NONE and storage to WAN target? That could make a difference.

And also go with Windows 2008 (or maybe Vista) or newer as backup appliance and target because of the better SMB protocol that will be used at Windows file shares.

[Gostev]

Definitely recommend at least Low compression with Linux targets, because the data is already compressed with Low compression before it is sent out over wire from backup server. Thus, setting compression to None essentially means making Linux target to uncompress the received data before storing it in the backup file

[stevil]

I didn't realise that little nugget of information. Thanks Gostev. Is that mentioned in the manual? This is the kind of stuff that makes deploying and testing veeam less stressful and more understandable

[Gostev]

No, I do not think this is mentioned anywhere.

[stevil]

You should do an 'experts' manual with all this stuff in it

[Gostev]

We have it! this very forum!

Anyway, I am hoping to make traffic compression configurable in v6, as per our discussion above.

[dter]

Hi,

as i can understand you are using a windows or linux as a target for your backups job.
Did you take the same tests using a NAS ?
What i mean is that if the target is a NAS which provides a CIFS, will make the whole procedure faster ?

Thanks in advance for your comments

[Vitaliy S.]

Dimitris,

The most efficient way (in v5) is to use Linux targets for your back up jobs, however with our upcoming version 6 we'll have a Windows agent that would make job performance comparable to what we had with Linux targets.

Also I don't believe that you will have a boost in performance while using CIFS shares on NAS devices. What is more important here is that you performance depends on many variables such as backup job type (forward or reversed incremental), NAS model etc.

If you want to know more about other users' NAS experience, just search our forums for existing discussions: http://forums.veeam.com/search.php?st=p ... arget+best

Hope this helps!