Comprehensive data protection for all workloads
tjestr
Enthusiast
Posts: 44
Liked: never
Joined: Mar 05, 2009 9:33 am
Full Name: Falko Dohse
Location: Hamburg
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by tjestr »

2 Jobs are running concurrently on each backupserver (target). There is no performance difference if only 1 job runs. (The job's performance is nearly the same)
I was not able to do the VCB perfomance test yet. I'll try that as soon as possible.

duhaas
Influencer
Posts: 11
Liked: 1 time
Joined: Apr 16, 2009 1:09 pm
Full Name: Duane Haas
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by duhaas »

So here is what I notcied in my env. I started testing veeam on March 23rd, from that time through april 13th my jobs were averaging 30MB/s, which was acceptable to me. Since than its dropped down to about 8MB/s. The one bigger thing I changed was how I went about the configuration of my service console switch and vkernel switch, it was configured like this:

vSwitch0 ----ONE NIC
Service Console
vSwitch1 -----TWO NICS
Vkernel - ISCSI

Now I configured it like this:

vSwitch0 -----TWO NICS
Service Console - Active on nic1 Standby on nic2
Vkernel - ISCSI - Active on nic2 standy on nic1

Can anyone help me brainstorm on why this might of had a negative impact on thing?

tjestr
Enthusiast
Posts: 44
Liked: never
Joined: Mar 05, 2009 9:33 am
Full Name: Falko Dohse
Location: Hamburg
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by tjestr »

What are the requirements (specifications, best practise) for a SAN based backup target (backupserver) to gain the optimal performance/backupspeed. Like RAM, CPU, Disks, RAID-mode, I/O...

tjestr
Enthusiast
Posts: 44
Liked: never
Joined: Mar 05, 2009 9:33 am
Full Name: Falko Dohse
Location: Hamburg
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by tjestr »

I've tested the performance of the target (backupserver) via. the IZone IO Filesystem Benchmark tool.
The results can be reviewed here: http://www.familiedohse.de/data/output.xlsx
Please let me know if the performance is OK to run Veeam with 100+ MB/s.

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

Hi Falko, I am not sure how to use the data you have provided (I have no background on storage).

I can tell you that Veeam Backup performance is determined solely by VCB processing speed (which can be determined via simple test I have provided above). VCB is the only limiting factor for Veeam Backup - we can process the data as fast as VCB is able to retrieve it for us (and depending on specific VMDK content, Veeam Backup could be even faster than VCB speed test would show, because data is processed by Veeam Backup on the fly and empty blocks are not saved to the VCB proxy hard disk like during the test, but simply discarded with deduplication engine).

As you see from the first posts in this topic, fast storage throughput does not guarantee that VCB is able to work fast, because something else can prevent VCB from achieving high speed.

tjestr
Enthusiast
Posts: 44
Liked: never
Joined: Mar 05, 2009 9:33 am
Full Name: Falko Dohse
Location: Hamburg
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by tjestr »

You're right, VCB is the bottleneck for Veeam Backup. A VCB backup takes as long as a Veeam backup.
Any suggestions how to "tune" VCB or the target (backup server)?

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

We are definitely seeing target storage being a bottleneck very often, so you may want to do some tuning around it (write back vs. write through cache settings, fragmentation etc). Keep in mind that in most cases, FC2 and especially FC4 connection to fast SAN gives much better throughput than single regular hard drive can handle for writes. If you are seeing 50-60MB/s during the above mentioned VCB test executed against your target backup storage, it could very well be that limiting factor is not VCB or SAN for you, but target storage.

Now, given you have achieved fastest possible VCB speed by applying recommendations from the beginning of this thread, and ensured you have fast target backup storage, you can also do the following to further increase the speed.

1. Defrag + Wipe your VMs. This is what could really make Veeam Backup to work much faster than VCB, because deduplication engine drops empty and repeating blocks, so the amount of data that needs to be written to (slow) backup storage is significantly reduced comparing to running VCB without Veeam Backup. Again, keep in mind that target storage is what typically slows everything down in case of FC2/FC4 HBA and fast SAN.

2. Run a few Veeam Backup jobs in parallel with target backup location being on different storages (separate physical hard drives). This again comes to overcoming bottleneck resulted in target storage speed.

Again, just want to cross-reference this thread here because it is perfect real-life example demonstrating how backup storage performance may affect your overall backup performance.

Hope this helps!

tjestr
Enthusiast
Posts: 44
Liked: never
Joined: Mar 05, 2009 9:33 am
Full Name: Falko Dohse
Location: Hamburg
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by tjestr »

Am I able to set the "Mountpoint" of a backup job for each backup job? So i would be able to mount/stream backup job to disk1 and mount/stream backup job to disk2. So i would be able to spread the load across 2 RAID-sets. (I think veeam mounts/streams the virtual machine vmdk's to C:\....)

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

Yes, you can set "mount point" for every backup job individually. And no, Veeam Backup does not stream VMDKs to C:\ unless you explicitly specify C:\ as your backup destination for the job.

tjestr
Enthusiast
Posts: 44
Liked: never
Joined: Mar 05, 2009 9:33 am
Full Name: Falko Dohse
Location: Hamburg
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by tjestr »

If starting a Veeam Backup the vmdk files are streamed to C:\Dokumente und Einstellungen\%Username%\Lokale Einstellungen\Temp\ (as you can see in the vcbmounter command Veeam uses)
VCB command: C:\Programme\VMware\VMware Consolidated Backup Framework\vcbMounter.exe -h vi.de.qurius.com -u ****** -p ****** -a moref:vm-816 -r "C:\Dokumente und Einstellungen\%Username%\Lokale Einstellungen\Temp\veeamvcb4adac747-3820-4fc7-8e4a-5d778ce4752f" -t fullvm -m san -M 1 -F 1 -L 3

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

I guess we went 'bit out of topic here :D But this is virtual folder, it does not exist in reality (although you can see it with Windows Explorer, and in VCB command line). All I/O attempts to this folder are intercepted by our file system driver, and the data is redirected directly into Veeam Backup processing engine in-memory. Remember that Veeam Backup does VCB processing on-the-fly, so VM data that is pulled from SAN by VCB never hits the staging area, like with other solutions. This is why we can backup 500GB VM with only a few hundred MB available in VCB staging area (something simply impossible with all other VCB-based solutions, where you have to ensure that staging area has enough disk space to host the VM files). This is also why Veeam Backup is 2-3 times faster than competitive solutions, which essentially need to perform the slowest operation (writing data to slow storage) twice: first to staging area on VCB proxy server hard drive, and then to actual backup storage after processing.

Just one of many rocket science technologies we use in Veeam Backup :wink:

tjestr
Enthusiast
Posts: 44
Liked: never
Joined: Mar 05, 2009 9:33 am
Full Name: Falko Dohse
Location: Hamburg
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by tjestr »

Thanks Gostev for the explanation! I do really appreciate it. I will setup the target (backup server) with 2 RAID-sets à 8 disks in RAID 10 mode, write back enabled (currently we have 1 RAID-set à 16 disks in RAID 6 mode, write back enabled). I will setup each backup job to a dedicated RAID-set.
Hopefully this will improve backup speed. I'll keep you posted.

BearHuntr
Enthusiast
Posts: 26
Liked: 3 times
Joined: Jan 01, 2006 1:01 am
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by BearHuntr »

Gostev,

Sorry if I'm missing something here, but you keep referring to a simple "VCB Performance Test" that we can perform, but I'm not seeing any instructions on how to run that test. One of your earlier posts states to run a command, but what command do we run and where? Feeling kinda stupid here! :? Thanks!

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

No problems, it is described in the last post on the first page of this thread:
http://forum.veeam.com/forums/viewtopic ... 3190#p3190

JorisK
Expert
Posts: 105
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by JorisK »

Great info!

I have reconfigured my HBA's (disabled 3/4 paths) and disabled multipathsupport for Windows and the first tests looks promising. (tests where during production)

We have three queue's running, totally about 4TB of data on one backup server. I'll report the results tomorrow.

zed999
Influencer
Posts: 21
Liked: 1 time
Joined: May 27, 2009 9:05 pm
Full Name: Zaher Butt
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by zed999 »

excellent thread chaps keep posting the configs!

There are couple of components on VCB performance in a SAN no one has mentioned - CPU and RAM of VCB Proxy and Veeam Backup server(assuming single box is used).

We have Clariion CX3-10, FC 4Gb disks

The suggusted spec for the backup server:
2x Quad Core E7310 at 1.6GHz CPUs
8 GB DDR3 RAM
4Gb HBA
Backup storage will be external SAS connected enclosure (MD1000) via PERC 6/E with SATAII disks in a RAID 5 configuration

Will VCB take advantage of total cores across both CPUs or is it advisable to go with a lesser core count (or even one physical CPU) but a higher clock speed? Or should I be looking at target storage more closley as the potential bottleneck.

Secondly, we do not plan to run concurrent jobs, so will 8GB of RAM be suffuciant or over kill?

What CPU/RAM configurations do you guys have on your VCB/Veeam backup servers and what speeds to you getting?

Look foward to your responses! :D

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

Here's my recommendations based on what I've learned:

CPU: VCB itself will not take advantage of number of cores or clock speed. Multiple CPUs really help when using Best compression, especially with parallel jobs. If you are using default compression and no parallel jobs, then single CPU is more than enough. Faster clock speed will help with overall processing speed (faster data processing), but most likely insignificantly (data blocks hashing, and default compression are not really CPU-intensive operations).

Memory amount: 8GB is definitely an overkill, 1-2GB should be enough for any OS. That is even in case of concurrent jobs - Veeam Backup processing engine does not use a whole lot of memory.

Memory speed: Faster memory is better for compression, deduplication, on-the-fly VCB data pumping.

Backup storage: Backup storage is almost always being main a bottleneck in case of FC4. In case of DAS (and DAS or local storage are recommended for best backup performance), it could be due to bad cache settings bad for RAID mode used, or due to innate hardware slowness (slow disks or slow controller). In case of NAS, 1Gb network will almost always a bottleneck.

zed999
Influencer
Posts: 21
Liked: 1 time
Joined: May 27, 2009 9:05 pm
Full Name: Zaher Butt
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by zed999 »

thank you for that Gostev, very helpful

I will be sure to post my stats when we get our VCB soloution in!

mdornfeld
Expert
Posts: 125
Liked: 2 times
Joined: Mar 23, 2009 4:44 pm
Full Name: Matt
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by mdornfeld »

For what it's worth, we have our Veeam service running on a 2 socket, 3.0Ghz quad core processors, and we found that using BEST compression (not optimal) goes significantly faster and obviously gives smaller backup files as well.

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

Makes sense... Best compression = much less data to write to backup storage (which is main bottleneck in most cases). This is extremely useful note for those with powerful Veeam Backup servers. Thanks Matt!

zed999
Influencer
Posts: 21
Liked: 1 time
Joined: May 27, 2009 9:05 pm
Full Name: Zaher Butt
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by zed999 »

Matt, thats very interesting; you have quite a beefy set of CPUs on you backup server - i assume VCB proxy is on the same box

In terms of CPU useage, how do all the cores look like when a backup is running? also does the backup use much RAM

ps when you said BEST compression goes significantly faster are you refering to the total time taken for backup?

sorry for million and one questions, we really close to purchasing our new hardware soon

mdornfeld
Expert
Posts: 125
Liked: 2 times
Joined: Mar 23, 2009 4:44 pm
Full Name: Matt
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by mdornfeld »

Matt, thats very interesting; you have quite a beefy set of CPUs on you backup server - i assume VCB proxy is on the same box
VCBproxy is what runs Veeam and is what performs our Disk to Disk backups.
We have a separate backup server that writes the Veeam backups to tape (via network connection).

In terms of CPU useage, how do all the cores look like when a backup is running? also does the backup use much RAM
CPU utilization on all cores is up in the 90% range when the backup is occuring. RAM usage is minimal.

ps when you said BEST compression goes significantly faster are you refering to the total time taken for backup?
Yes. For an example backup of a 36GB VM.
A Full backup with "Optimal" takes 19 minutes and is 36GB large
A Full backup with "Best" takes 12 minutes and is 15GB large

Good luck with your hardware purchases.

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

This is great data, thanks again Matt.

Just to clarify the results, Optimal compression algorithm is designed for lowest CPU load, because this compression is used by our agents, which are running directly on ESX hosts in "Network" backup mode. Optimal compression can give you compression ratios pretty comparable Best compression ratio only if your VM disks have been optimized with defrag&wipe procedure.

zed999
Influencer
Posts: 21
Liked: 1 time
Joined: May 27, 2009 9:05 pm
Full Name: Zaher Butt
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by zed999 »

Thats some interesting stats Matt, what I think is key is the 50 odd percent reduction in the backup window on the Best job compared Optimal....hmmm I may have to rethink the dual 1.6 GHz Quads.....just out of curiosity what do your dual 3GHz quads look like running that Optimal backup?

thanks for link to defrag&wipe will look into it over the weekend

cpage
Lurker
Posts: 1
Liked: never
Joined: Aug 11, 2009 3:46 pm
Full Name: Cody Page
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by cpage »

Is anyone using Veeam backup with a DataDomain DDR as the backup target? Our jobs are running without compression.

We have an Equallogic iSCSI SAN, VCB proxy, non-multipath gig-E to the source and destination. We're seeing about 20 MBs on a single VM backup.

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

DataDomain can be quite slow as Veeam Backup target if you are integrating through CIFS (some CIFS implementation issues on their side). You can search for "DataDomain" using forum search for more information as this has been discussed before.

We are looking to address this from our side by changing the way we write backup files to storage (and do it more in the way DataDomain SAMBA server "likes"). Meanwhile, you can try NFS integration instead, this should increase performance.

kmccubbin
Influencer
Posts: 17
Liked: never
Joined: Apr 10, 2009 7:27 pm
Full Name: Kelly McCubbin
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by kmccubbin »

I don't know if anyone's addressed this, but it seems like it's not only the target drives (in our case an MD1000 array) that can choke speed. If Veeam and/or VCB is installed on a local drive, even though that drive is not the storage target, and that local drive is not particularly fast, it can hurt pretty badly. Going from 7,000RPM to 15,000 RPM drives, without changing any other components, improved our speeds from 50 - 60 MB/S to 90 - 120 MB/S.

The odd thing I'm noticing now is that those speeds only improve to, maybe, 10 MB/s faster on subsequent/incrimental runs. I'm seeing a lot of folks where the post-initial runs improve dramatically. My speeds seem fine, but I wonder if I'm missing something there?

zed999
Influencer
Posts: 21
Liked: 1 time
Joined: May 27, 2009 9:05 pm
Full Name: Zaher Butt
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by zed999 »

New backup hardware in:

Veeam 3.1 running in BEST mode
Windows 2008 Server x64 (Dell R710)
Dual X5550 2.66GHz
12GB RAM
MD1000 (12TB SATA 7K RPM) 11+1 R5

945.02 GB VM (600GB of actual data fileserver data) sitting on SATA disks

Start 28/08/2009 18:47:05
Finish 28/08/2009 21:09:29
Performance Rate 113 MB/s

pretty awesome concidering we went from using LAN agent based backups giving us a whopping 6MB/s :lol:

Gostev
SVP, Product Management
Posts: 27452
Liked: 4557 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by Gostev » 1 person likes this post

Zaher, this is pretty cool speed for Best compression... looks like Veeam Backup engine definitely loves your 8 cores :mrgreen:

Kelly, if full and incremental speed are about the same, it means that bottleneck is the VCB data retrieval from source storage - so drivers, multipathing issues may apply etc. Also I think it would be fair to say that at these speeds VCB proxy server hardware may be affecting already, because of on-the-fly incoming data processing (so using server with faster RAM and bus speed may improve the results).

withanh
Expert
Posts: 262
Liked: never
Joined: Jul 21, 2009 3:19 pm
Full Name: Darhl
Location: Pacific Northwest
Contact:

Re: How to improve backup speed: VCB performance in SAN mode

Post by withanh »

Any preferences to VM or physical machine for the Veeam backup server?

Pros & Cons would be great!

Thanks!

h
For every expert there is an equal and opposite expert - Arthur C Clarke's Fourth Law

Post Reply

Who is online

Users browsing this forum: corbitech and 24 guests