Going to try Veeam Backup 4.0 tonight

[Wichard20]

Currently been evaluating backup solutions for our VMware environment. Just installed version 3 yesterday and kicked it off 6pm last night. It is currently 3:03PM as I am writing this and it is at 34%. I am backing up 100 Virtual Servers, 6TB total, 28MB/s is the average (On an individual basis, depending on the size I get 14-30MB/s average, the smaller ones shoot up to 100MB/s average). The VCB/Veeam server is a HP BL460c with a Dual Core 3.0Ghz, 3.5GB memory. In the next hour or so I am going to stop the job and upgrade to 4.0. Plan to run 4 concurrent jobs with 25 VMs a piece. We have two SANs in the same fabric. One is a Hitachi AMS1000 which we have five 1.8TB luns each in a seperate raid group. On the server the drive is spanned into one large drive. This spanned drive is where we store the backups. We have 3 vSphere ESX servers using a Hitachi AMS2100. The fabric is all 4GB fiber. I have not done any SAN or server tuning yet and have been researching what I can do. The HBA is qlogic and I am going to make sure the drivers are recent and the backup server has only 1 path to both SANs so there are no multipathing ssues. If anyone have any suggesstions or recommendations for optimal performance please let me know. For example should I not be running 4 streams, etc. Just to clarify we are currently using VCB SAN mode with Veeam and plan to use the new API with Veeam 4.0. Once I am comfortable with a products performance and plan to purchase I will probably will add atleast 1 more physical server for SAN based backups and maybe a 2nd to have a backup server per ESX host. I will report back tomorrow with my results using Backup 4.0 and the new API.

Thanks,
Rich

[Gostev]

Richard, generally speaking there is no sense to run parallel jobs in case with FC4 SAN... with modern FC4 SAN we usually see 100MB/s processing speed on average in VCB SAN mode, and with such speed, single job will be able to load both cores on your server to 100%, so CPU will be bottleneck and parallel jobs will not give you anything. On the other hand, parallel jobs has a number of drawbacks, for instance parallel snapshot operation on the same LUN (which is not recommended by VMware).

I see that you are getting much lower speed than I would expect, and there could be a number of reasons: outdated HBA drivers, enabled multipathing (most likely, based on your numbers), target storage controller settings/speed (second likely cause) and so on. Anyway, it would be interesting to see the results of your testing for v4 and compare them to the above numbers without making any prior changes.

Thank you!

[Wichard20]

Richard, generally speaking there is no sense to run parallel jobs in case with FC4 SAN... with modern FC4 SAN we usually see 100MB/s processing speed on average in VCB SAN mode, and with such speed, single job will be able to load both cores on your server to 100%, so CPU will be bottleneck and parallel jobs will not give you anything. On the other hand, parallel jobs has a number of drawbacks, for instance parallel snapshot operation on the same LUN (which is not recommended by VMware).

I see that you are getting much lower speed than I would expect, and there could be a number of reasons: outdated HBA drivers, enabled multipathing (most likely, based on your numbers), target storage controller settings/speed (second likely cause) and so on. Anyway, it would be interesting to see the results of your testing for v4 and compare them to the above numbers without making any prior changes.

Thank you!

I appreciate the info. I am going to run v4 without any changes (one job, same settings as before except using the new API in SAN only) and will post tomorrow.

[Wichard20]

So its 9:30am Eastern and I am at 35%. I started about an hour and half later (around 7:30pm) then the 3.1 job I ran couple days ago. I am trying to figure out why my speeds are so inconsistent. My Veeam server has 1 path to both SANs. I have not updated the HBA drivers yet but will once the job is completed. My target storage controller settings look correct.

45.00 GB 41 MB/s
45.00 GB 73 MB/s
45.04 GB 64 MB/s
30.00 GB 48 MB/s
20.00 GB 33 MB/s
20.00 GB 52 MB/s
20.00 GB 30 MB/s
45.02 GB 45 MB/s
45.00 GB 57 MB/s
20.00 GB 43 MB/s
30.00 GB 50 MB/s
95.00 GB 51 MB/s
110.00 GB 37 MB/s
45.04 GB 143 MB/s
45.00 GB 105 MB/s
20.00 GB 48 MB/s
45.02 GB 66 MB/s
165.00 GB 59 MB/s
317.85 GB 74 MB/s
20.00 GB 36 MB/s
120.00 GB 31 MB/s
20.00 GB 53 MB/s
70.02 GB 17 MB/s
20.00 GB 37 MB/s
20.00 GB 59 MB/s
45.03 GB 31 MB/s
67.85 GB 29 MB/s
203.39 GB 40 MB/s
50.02 GB 117 MB/s
45.00 GB 96 MB/s
40.00 GB 39 MB/s
45.00 GB 35 MB/s
45.02 GB 27 MB/s
70.00 GB 18 MB/s

[Gostev]

This looks much faster to me then you had with v3, am I right?

The speed is "inconsistent" because of different amount of white space in each VM, and different VSS freeze and snapshot management operation times (which are all accounted in the VM "processing speed" counter). This is normal.

[Wichard20]

This looks much faster to me then you had with v3, am I right?

The speed is "inconsistent" because of different amount of white space in each VM, and different VSS freeze and snapshot management operation times (which are all accounted in the VM "processing speed" counter). This is normal.

Yes it is much faster. Comparing the two jobs, the v4 job has made more progress then the v3 job that ran for 6-7 hours longer.

-Rich

[Wichard20]

Update: The full completed right under 40 hours. I kicked off another job to see how long it takes for an incremntal. I am hoping under 12 hours. Now if I need to reduce my time window what is the best option for me, having multiple Veeam servers? Going off what was recommend earlier I think I would need to get 2-3 boxes dedicated to Veeam with atleast 4 cores and make sure I am not backing up VMs on the same lun. For example if I have 16 luns, I will have one server backup VMs from 8 and the other backup VMs from the other 8. Sound good?

Edit: I am using vSphere so I am utilizing block tracking

[Vitaliy S.]

Richard,

Yes, you can easily use multiple Veeam servers to reduce the backup window, and manage those servers within the centralized reporting management console - Veeam Backup Enterprise Manager.

[Gostev]

Let's wait for the results but I would say you will be fine with one proxy with the current number of VMs, but if your environment is growing, then it would be best to deploy the second backup server proactively.

[Christian]

Richard,

I am currently testing veeam backup also, in the following environment:

5 Vsphere 4 boxes
75 Virtual Machines
3par storage
5 luns (1 TB each)

Destination 6TB sas storage unit

My concern is the snapshots taken in order to perform the backup will fill my luns. Also, I noticed you had jobs running during the day. Did you have any performance impact?

Curious regarding your results.

Thanks
Christian

[Wichard20]

The first incremental did not finish for a mainteance window so I had to cancel it. The second incremental took 36 hours to complete. I went through the logs and one server took 9 hours alone to complete. Its transfer rate was like 6MB/s. Most servers finished up in 5-10 minutes but there are a couple that average 1-2 hours. I might install agent base backups on them to narrow my window.

I am happy with the product and going spec out some dedicated hardware for it. How many cores are optimal?

Christian,

While running during the day is not ideal for the environment but until I narrow my time window it has to due. I have not noticed any performance issues. The environment is production hosting everything from SharePoint to Oracle depending on the project requirements but I have not heard of any complaints from the clients. Our fiber network is not heavily used and even with the backups we are not even close to 50% utilization. If we had a slow fiber network or an ISCI network there might more concern of performance impacts. My understanding of the software is it only takes one snapshot at a time and then removes it so there is not a huge space requirement.

Here is the report from the last full incremental,

45.00 GB 45.00 GB 43 MB/s 0:17:45  
50.04 GB 50.04 GB 62 MB/s 0:13:52  
20.00 GB 20.00 GB 52 MB/s 0:06:30  
317.85 GB 317.85 GB 335 MB/s 0:16:11  
165.04 GB 165.04 GB 129 MB/s 0:21:46  
167.85 GB 167.85 GB 71 MB/s 0:40:29  
45.03 GB 45.03 GB 24 MB/s 0:32:28  
45.00 GB 45.00 GB 61 MB/s 0:12:29  
20.00 GB 20.00 GB 33 MB/s 0:10:26  
20.00 GB 20.00 GB 35 MB/s 0:09:46  
45.02 GB 45.02 GB 77 MB/s 0:09:55  
20.00 GB 20.00 GB 35 MB/s 0:09:40  
20.00 GB 20.00 GB 42 MB/s 0:08:10  
20.00 GB 20.00 GB 38 MB/s 0:08:53  
20.00 GB 20.00 GB 76 MB/s 0:04:30  
30.00 GB 30.00 GB 94 MB/s 0:05:25  
120.00 GB 120.00 GB 81 MB/s 0:25:20  
170.03 GB 170.03 GB 158 MB/s 0:18:22  
45.04 GB 45.04 GB 114 MB/s 0:06:43  
20.00 GB 20.00 GB 42 MB/s 0:08:03  
110.02 GB 110.02 GB 13 MB/s 2:27:24  
45.00 GB 0.00 KB 0 KB/s 0:01:15  
20.00 GB 20.00 GB 22 MB/s 0:15:24  
45.00 GB 45.00 GB 45 MB/s 0:16:54  
45.00 GB 45.00 GB 32 MB/s 0:23:50  
20.00 GB 20.00 GB 39 MB/s 0:08:42  
20.00 GB 20.00 GB 39 MB/s 0:08:44  
20.00 GB 20.00 GB 34 MB/s 0:09:57  
45.00 GB 45.00 GB 31 MB/s 0:24:24  
45.00 GB 45.00 GB 75 MB/s 0:10:15  
20.00 GB 20.00 GB 35 MB/s 0:09:45  
20.00 GB 20.00 GB 0 KB/s 0:-51:-57  
20.00 GB 20.00 GB 79 MB/s 0:04:18  
20.00 GB 20.00 GB 19 MB/s 0:17:54  
45.00 GB 45.00 GB 37 MB/s 0:20:46  
280.00 GB 280.00 GB 55 MB/s 1:26:22  
45.02 GB 45.02 GB 66 MB/s 0:11:43  
45.00 GB 45.00 GB 81 MB/s 0:09:25  
45.00 GB 45.00 GB 45 MB/s 0:16:59  
20.00 GB 20.00 GB 35 MB/s 0:09:40  
50.02 GB 50.02 GB 112 MB/s 0:07:38  
35.03 GB 35.03 GB 29 MB/s 0:20:56  
20.00 GB 20.00 GB 43 MB/s 0:07:51  
45.00 GB 45.00 GB 101 MB/s 0:07:37  
45.02 GB 45.02 GB 99 MB/s 0:07:44  
20.00 GB 20.00 GB 40 MB/s 0:08:33  
45.04 GB 45.04 GB 44 MB/s 0:17:38  
45.00 GB 45.00 GB 52 MB/s 0:14:50  
20.00 GB 20.00 GB 29 MB/s 0:11:37  
95.00 GB 95.00 GB 177 MB/s 0:09:09  
70.00 GB 70.00 GB 58 MB/s 0:20:34  
70.00 GB 70.00 GB 87 MB/s 0:13:43  
45.00 GB 45.00 GB 68 MB/s 0:11:20  
70.00 GB 70.00 GB 34 MB/s 0:35:23  
20.00 GB 20.00 GB 19 MB/s 0:17:49  
67.86 GB 67.86 GB 222 MB/s 0:05:12  
195.03 GB 195.03 GB 6 MB/s 8:56:52  
50.04 GB 50.04 GB 94 MB/s 0:09:04  
40.00 GB 40.00 GB 111 MB/s 0:06:08  
80.00 GB 80.00 GB 158 MB/s 0:08:38  
111.42 GB 111.42 GB 43 MB/s 0:44:02  
40.54 GB 40.54 GB 228 MB/s 0:03:01  
45.00 GB 45.00 GB 67 MB/s 0:11:29  
20.00 GB 20.00 GB 17 MB/s 0:20:12  
40.00 GB 40.00 GB 85 MB/s 0:08:03  
70.02 GB 70.02 GB 275 MB/s 0:04:20  
45.03 GB 45.03 GB 32 MB/s 0:24:04  
45.02 GB 45.02 GB 71 MB/s 0:10:51  
20.00 GB 20.00 GB 26 MB/s 0:13:19  
20.00 GB 20.00 GB 42 MB/s 0:08:02  
70.02 GB 70.02 GB 29 MB/s 0:41:52  
20.00 GB 20.00 GB 36 MB/s 0:09:28  
45.04 GB 45.04 GB 60 MB/s 0:12:42  
20.00 GB 20.00 GB 33 MB/s 0:10:20  
70.00 GB 70.00 GB 123 MB/s 0:09:42  
45.00 GB 45.00 GB 123 MB/s 0:06:13  
95.00 GB 95.00 GB 106 MB/s 0:15:13  
30.00 GB 30.00 GB 94 MB/s 0:05:26  
120.07 GB 120.07 GB 32 MB/s 1:04:49  
35.04 GB 35.04 GB 46 MB/s 0:12:56  
20.00 GB 20.00 GB 36 MB/s 0:09:21  
80.03 GB 80.03 GB 71 MB/s 0:19:21  
110.00 GB 110.00 GB 119 MB/s 0:15:43  
40.00 GB 40.00 GB 39 MB/s 0:17:30  
20.00 GB 20.00 GB 48 MB/s 0:07:09  
45.00 GB 45.00 GB 27 MB/s 0:28:01  
65.02 GB 65.02 GB 21 MB/s 0:54:04  
45.02 GB 45.02 GB 34 MB/s 0:22:46  
45.00 GB 45.00 GB 43 MB/s 0:17:59  
95.00 GB 95.00 GB 16 MB/s 1:44:30  
45.02 GB 45.02 GB 58 MB/s 0:13:18  
20.00 GB 20.00 GB 28 MB/s 0:12:00  
67.85 GB 67.85 GB 82 MB/s 0:14:12  
45.00 GB 45.00 GB 75 MB/s 0:10:13  
45.02 GB 45.02 GB 47 MB/s 0:16:20  
40.00 GB 40.00 GB 91 MB/s 0:07:29  
20.00 GB 20.00 GB 39 MB/s 0:08:49  
70.02 GB 70.02 GB 102 MB/s 0:11:40  

[tsightler]

While Anton might not like this, with that many VM's I'd strongly suggest running at least a few jobs in parallel. We used to run 8-12 jobs in parallel, but since Veeam 4 we've been able to get to 4 jobs. Basically we run a job per VMFS volume, that way there's no change of simultaneous snapshot removals on the same volume, but not so much wasted time with overhead of VSS/snapshot creating/deletion. If we run a single job with all of our VM's (~50 VM's and over 5TB) it takes almost 12 hours. With 4 jobs everything gets done in less than 4 hours, easily fitting into our backup window.

[Gostev]

Anton does not really mind parallel jobs, as long as they do not hit the same source VMFS volume and the same network target

Richard, I recommend modern 4-8 cores computer, for example this one

But I am quite puzzled with slow performance on a few VMs during your testing, please investigate the session statistics from these VMs to see the mode job was using - may be changed block tracking was not leveraged (pre-HW7 VM)? Actually, results for many VMs above look like there were no changed block tracking leveraged - except for those few clocked at 200-300MB/s.

[Wichard20]

Hello,

All the server showed being backed up by SAN using block tracking except for one which Veeam gave an explanation on how to correct. Is that amount of memory recommended as well, 12GB?

Thanks,
Rich

[Gostev]

Rich, 4GB will be more than enough.

[tsightler]

To me those speeds are pretty slow for a system with block change tracking enabled, unless the systems are seeing significant changes every day. We are currently seeing, on average, around 175MB/sec when all backups are taken into account. While we do have a few VM's that are still slow (our Exchange server is only seeing about 35-40MB/sec, and we have two busy SQL servers that are in the 40-50MB/sec performance), almost every other system is >100MB/sec. This is with an all iSCSI environment so we should be at a disadvantage to your FC setup, also, our Veeam server is much less powerful than yours. Obviously that's not much help to you, but it's just a little strange that your not seeing better speeds.

[Wichard20]

I appreciate all the feedback. These are some of the next steps I am taking to get some better performance. I have a new server on the way 8 cores at 2.66GHz, 4GB memory, 4GB Qlogic fiber card. I am also testing a new target storage solution. Our target storage currently is 1TB sata 7.2k drives with small drive count raid groups, I am going to use some fiber drive storage that are 10k. The drives are also in much higher drive count raid group.