How to improve backup speed: VCB performance in SAN mode

[tjestr]

2 Jobs are running concurrently on each backupserver (target). There is no performance difference if only 1 job runs. (The job's performance is nearly the same)
I was not able to do the VCB perfomance test yet. I'll try that as soon as possible.

[duhaas]

So here is what I notcied in my env. I started testing veeam on March 23rd, from that time through april 13th my jobs were averaging 30MB/s, which was acceptable to me. Since than its dropped down to about 8MB/s. The one bigger thing I changed was how I went about the configuration of my service console switch and vkernel switch, it was configured like this:

vSwitch0 ----ONE NIC
Service Console
vSwitch1 -----TWO NICS
Vkernel - ISCSI

Now I configured it like this:

vSwitch0 -----TWO NICS
Service Console - Active on nic1 Standby on nic2
Vkernel - ISCSI - Active on nic2 standy on nic1

Can anyone help me brainstorm on why this might of had a negative impact on thing?

[tjestr]

What are the requirements (specifications, best practise) for a SAN based backup target (backupserver) to gain the optimal performance/backupspeed. Like RAM, CPU, Disks, RAID-mode, I/O...

[tjestr]

I've tested the performance of the target (backupserver) via. the IZone IO Filesystem Benchmark tool.
The results can be reviewed here: http://www.familiedohse.de/data/output.xlsx
Please let me know if the performance is OK to run Veeam with 100+ MB/s.

[Gostev]

Hi Falko, I am not sure how to use the data you have provided (I have no background on storage).

I can tell you that Veeam Backup performance is determined solely by VCB processing speed (which can be determined via simple test I have provided above). VCB is the only limiting factor for Veeam Backup - we can process the data as fast as VCB is able to retrieve it for us (and depending on specific VMDK content, Veeam Backup could be even faster than VCB speed test would show, because data is processed by Veeam Backup on the fly and empty blocks are not saved to the VCB proxy hard disk like during the test, but simply discarded with deduplication engine).

As you see from the first posts in this topic, fast storage throughput does not guarantee that VCB is able to work fast, because something else can prevent VCB from achieving high speed.

[tjestr]

You're right, VCB is the bottleneck for Veeam Backup. A VCB backup takes as long as a Veeam backup.
Any suggestions how to "tune" VCB or the target (backup server)?

[Gostev]

We are definitely seeing target storage being a bottleneck very often, so you may want to do some tuning around it (write back vs. write through cache settings, fragmentation etc). Keep in mind that in most cases, FC2 and especially FC4 connection to fast SAN gives much better throughput than single regular hard drive can handle for writes. If you are seeing 50-60MB/s during the above mentioned VCB test executed against your target backup storage, it could very well be that limiting factor is not VCB or SAN for you, but target storage.

Now, given you have achieved fastest possible VCB speed by applying recommendations from the beginning of this thread, and ensured you have fast target backup storage, you can also do the following to further increase the speed.

1. Defrag + Wipe your VMs. This is what could really make Veeam Backup to work much faster than VCB, because deduplication engine drops empty and repeating blocks, so the amount of data that needs to be written to (slow) backup storage is significantly reduced comparing to running VCB without Veeam Backup. Again, keep in mind that target storage is what typically slows everything down in case of FC2/FC4 HBA and fast SAN.

2. Run a few Veeam Backup jobs in parallel with target backup location being on different storages (separate physical hard drives). This again comes to overcoming bottleneck resulted in target storage speed.

Again, just want to cross-reference this thread here because it is perfect real-life example demonstrating how backup storage performance may affect your overall backup performance.

Hope this helps!

[tjestr]

Am I able to set the "Mountpoint" of a backup job for each backup job? So i would be able to mount/stream backup job to disk1 and mount/stream backup job to disk2. So i would be able to spread the load across 2 RAID-sets. (I think veeam mounts/streams the virtual machine vmdk's to C:\....)

[Gostev]

Yes, you can set "mount point" for every backup job individually. And no, Veeam Backup does not stream VMDKs to C:\ unless you explicitly specify C:\ as your backup destination for the job.

[tjestr]

If starting a Veeam Backup the vmdk files are streamed to C:\Dokumente und Einstellungen\%Username%\Lokale Einstellungen\Temp\ (as you can see in the vcbmounter command Veeam uses)
VCB command: C:\Programme\VMware\VMware Consolidated Backup Framework\vcbMounter.exe -h vi.de.qurius.com -u ****** -p ****** -a moref:vm-816 -r "C:\Dokumente und Einstellungen\%Username%\Lokale Einstellungen\Temp\veeamvcb4adac747-3820-4fc7-8e4a-5d778ce4752f" -t fullvm -m san -M 1 -F 1 -L 3

[Gostev]

I guess we went 'bit out of topic here But this is virtual folder, it does not exist in reality (although you can see it with Windows Explorer, and in VCB command line). All I/O attempts to this folder are intercepted by our file system driver, and the data is redirected directly into Veeam Backup processing engine in-memory. Remember that Veeam Backup does VCB processing on-the-fly, so VM data that is pulled from SAN by VCB never hits the staging area, like with other solutions. This is why we can backup 500GB VM with only a few hundred MB available in VCB staging area (something simply impossible with all other VCB-based solutions, where you have to ensure that staging area has enough disk space to host the VM files). This is also why Veeam Backup is 2-3 times faster than competitive solutions, which essentially need to perform the slowest operation (writing data to slow storage) twice: first to staging area on VCB proxy server hard drive, and then to actual backup storage after processing.

Just one of many rocket science technologies we use in Veeam Backup

[tjestr]

Thanks Gostev for the explanation! I do really appreciate it. I will setup the target (backup server) with 2 RAID-sets à 8 disks in RAID 10 mode, write back enabled (currently we have 1 RAID-set à 16 disks in RAID 6 mode, write back enabled). I will setup each backup job to a dedicated RAID-set.
Hopefully this will improve backup speed. I'll keep you posted.

[BearHuntr]

Gostev,

Sorry if I'm missing something here, but you keep referring to a simple "VCB Performance Test" that we can perform, but I'm not seeing any instructions on how to run that test. One of your earlier posts states to run a command, but what command do we run and where? Feeling kinda stupid here! Thanks!

[Gostev]

No problems, it is described in the last post on the first page of this thread:
http://forum.veeam.com/forums/viewtopic ... 3190#p3190

[JorisK]

Great info!

I have reconfigured my HBA's (disabled 3/4 paths) and disabled multipathsupport for Windows and the first tests looks promising. (tests where during production)

We have three queue's running, totally about 4TB of data on one backup server. I'll report the results tomorrow.

[zed999]

excellent thread chaps keep posting the configs!

There are couple of components on VCB performance in a SAN no one has mentioned - CPU and RAM of VCB Proxy and Veeam Backup server(assuming single box is used).

We have Clariion CX3-10, FC 4Gb disks

The suggusted spec for the backup server:
2x Quad Core E7310 at 1.6GHz CPUs
8 GB DDR3 RAM
4Gb HBA
Backup storage will be external SAS connected enclosure (MD1000) via PERC 6/E with SATAII disks in a RAID 5 configuration

Will VCB take advantage of total cores across both CPUs or is it advisable to go with a lesser core count (or even one physical CPU) but a higher clock speed? Or should I be looking at target storage more closley as the potential bottleneck.

Secondly, we do not plan to run concurrent jobs, so will 8GB of RAM be suffuciant or over kill?

What CPU/RAM configurations do you guys have on your VCB/Veeam backup servers and what speeds to you getting?

Look foward to your responses!

[Gostev]

Here's my recommendations based on what I've learned:

CPU: VCB itself will not take advantage of number of cores or clock speed. Multiple CPUs really help when using Best compression, especially with parallel jobs. If you are using default compression and no parallel jobs, then single CPU is more than enough. Faster clock speed will help with overall processing speed (faster data processing), but most likely insignificantly (data blocks hashing, and default compression are not really CPU-intensive operations).

Memory amount: 8GB is definitely an overkill, 1-2GB should be enough for any OS. That is even in case of concurrent jobs - Veeam Backup processing engine does not use a whole lot of memory.

Memory speed: Faster memory is better for compression, deduplication, on-the-fly VCB data pumping.

Backup storage: Backup storage is almost always being main a bottleneck in case of FC4. In case of DAS (and DAS or local storage are recommended for best backup performance), it could be due to bad cache settings bad for RAID mode used, or due to innate hardware slowness (slow disks or slow controller). In case of NAS, 1Gb network will almost always a bottleneck.

[zed999]

thank you for that Gostev, very helpful

I will be sure to post my stats when we get our VCB soloution in!

[mdornfeld]

For what it's worth, we have our Veeam service running on a 2 socket, 3.0Ghz quad core processors, and we found that using BEST compression (not optimal) goes significantly faster and obviously gives smaller backup files as well.

[Gostev]

Makes sense... Best compression = much less data to write to backup storage (which is main bottleneck in most cases). This is extremely useful note for those with powerful Veeam Backup servers. Thanks Matt!

[zed999]

Matt, thats very interesting; you have quite a beefy set of CPUs on you backup server - i assume VCB proxy is on the same box

In terms of CPU useage, how do all the cores look like when a backup is running? also does the backup use much RAM

ps when you said BEST compression goes significantly faster are you refering to the total time taken for backup?

sorry for million and one questions, we really close to purchasing our new hardware soon

[mdornfeld]

Matt, thats very interesting; you have quite a beefy set of CPUs on you backup server - i assume VCB proxy is on the same box
VCBproxy is what runs Veeam and is what performs our Disk to Disk backups.
We have a separate backup server that writes the Veeam backups to tape (via network connection).

In terms of CPU useage, how do all the cores look like when a backup is running? also does the backup use much RAM
CPU utilization on all cores is up in the 90% range when the backup is occuring. RAM usage is minimal.

ps when you said BEST compression goes significantly faster are you refering to the total time taken for backup?
Yes. For an example backup of a 36GB VM.
A Full backup with "Optimal" takes 19 minutes and is 36GB large
A Full backup with "Best" takes 12 minutes and is 15GB large

Good luck with your hardware purchases.

[Gostev]

This is great data, thanks again Matt.

Just to clarify the results, Optimal compression algorithm is designed for lowest CPU load, because this compression is used by our agents, which are running directly on ESX hosts in "Network" backup mode. Optimal compression can give you compression ratios pretty comparable Best compression ratio only if your VM disks have been optimized with defrag&wipe procedure.

[zed999]

Thats some interesting stats Matt, what I think is key is the 50 odd percent reduction in the backup window on the Best job compared Optimal....hmmm I may have to rethink the dual 1.6 GHz Quads.....just out of curiosity what do your dual 3GHz quads look like running that Optimal backup?

thanks for link to defrag&wipe will look into it over the weekend

[cpage]

Is anyone using Veeam backup with a DataDomain DDR as the backup target? Our jobs are running without compression.

We have an Equallogic iSCSI SAN, VCB proxy, non-multipath gig-E to the source and destination. We're seeing about 20 MBs on a single VM backup.

[Gostev]

DataDomain can be quite slow as Veeam Backup target if you are integrating through CIFS (some CIFS implementation issues on their side). You can search for "DataDomain" using forum search for more information as this has been discussed before.

We are looking to address this from our side by changing the way we write backup files to storage (and do it more in the way DataDomain SAMBA server "likes"). Meanwhile, you can try NFS integration instead, this should increase performance.

[kmccubbin]

I don't know if anyone's addressed this, but it seems like it's not only the target drives (in our case an MD1000 array) that can choke speed. If Veeam and/or VCB is installed on a local drive, even though that drive is not the storage target, and that local drive is not particularly fast, it can hurt pretty badly. Going from 7,000RPM to 15,000 RPM drives, without changing any other components, improved our speeds from 50 - 60 MB/S to 90 - 120 MB/S.

The odd thing I'm noticing now is that those speeds only improve to, maybe, 10 MB/s faster on subsequent/incrimental runs. I'm seeing a lot of folks where the post-initial runs improve dramatically. My speeds seem fine, but I wonder if I'm missing something there?

[zed999]

New backup hardware in:

Veeam 3.1 running in BEST mode
Windows 2008 Server x64 (Dell R710)
Dual X5550 2.66GHz
12GB RAM
MD1000 (12TB SATA 7K RPM) 11+1 R5

945.02 GB VM (600GB of actual data fileserver data) sitting on SATA disks

Start 28/08/2009 18:47:05
Finish 28/08/2009 21:09:29
Performance Rate 113 MB/s

pretty awesome concidering we went from using LAN agent based backups giving us a whopping 6MB/s

[Gostev]

Zaher, this is pretty cool speed for Best compression... looks like Veeam Backup engine definitely loves your 8 cores

Kelly, if full and incremental speed are about the same, it means that bottleneck is the VCB data retrieval from source storage - so drivers, multipathing issues may apply etc. Also I think it would be fair to say that at these speeds VCB proxy server hardware may be affecting already, because of on-the-fly incoming data processing (so using server with faster RAM and bus speed may improve the results).

[withanh]

Any preferences to VM or physical machine for the Veeam backup server?

Pros & Cons would be great!

Thanks!

h