Looking for advice

[devlin7]

Hi,
I am fairly new to Veeam. I initally installed 4.1 to test but upgraded to 5.01 when I went live.

I have two Dell R710 servers sharing an MD3000i attached directly via iSCSI over network. Each server has two connections and there are no switches in between.

I have recently purchased a Dell R510 with 20Tb of storage. Obviously, I could install ESX on it but then it would only present 2Tb of storage to the other ESX servers. I decided to install FREENAS on the 510 and just have Veeam talk to a network share. FREENA is installed on a mirrored 120GB partition and I have 9 2TB drives setup in RAID5 + a hot spare, all up I have a share of about 18-19TB. My backup size is about 4Tb for the 8 servers I have. The backup throughput last night was terrible in fact after 16 hours, only two of the 8 servers had completed.

My Veeam backup job is a single job for all servers mainly to take advantage of de-dup.
I thought my slow backups [averaging 13MB/s] were to do with FREENAS but this morning I tried copying a LARGE file from my Ubuntu desktop and I saw speeds of 80MB/s. However, from the Windows 2008 servers I was seeing 15-20MB/s. So after a bit of research I discovered netsh interface tcp set global autotuning level=restricted + a server reboot dramatically improved file copy speeds [I am seeing 50-60MB/s now]

My Veeam backups however are still incredibly slow. In fact, it is currently sitting at 3MB/s

Now I realise the incrementals are going to much much quicker but I am keen to speed up the initial FULL backup and I suppose the weekly rollup.

I could add the FREENAS box as an NFS share directly to each ESX server but I am not sure it will speed anything up.

Has anyone got any suggestions?

[Vitaliy S.]

Hello Graeme,

Could you please create a test job with a single server (probably the smallest one) and back it up pointing to the local storage of Veeam backup server (not the network share). Do you see any difference in performance rates?

If you mount those volumes to a Linux box (or ESX host) and add this Linux box (providing SSH credentials) to the Veeam backup console that may speed up the job processing, though I would first try backing VM up to a different storage just to compare the rate results.

By the way, where is Veeam Backup server installed and what backup job mode do you use? Besides, have you seen any CPU starving problems on your backup server while the backup job was running?

Thank you!

[devlin7]

Hi,

Thanks for the response. I will try a few test backups later and see what I get. Veeam is installed on a Windows 2008R2 server that is also the Vcentre/vsphere machine. This machine is separate to the actual ESX servers. What I don't understand is how one server can transfer at rates of 73MB/s and others at 15MB/s.

When I backed up our 1.96Tb machine to a cheap DLink 323 storage unit using Shadow Protect it took about 28 hours. The DLInk 323 devices are reknowned for terrible performance.

It took a very long time to backup 4TB but at least I have a backup. I will try a few test backups later and see what I get

Status Success Start time 4/02/2011 2:52:40 p.m.
Total VMs 10 End time 7/02/2011 7:51:42 a.m.
Processed VMs 10 Duration 64:59:02
Successful VMs 10 Total size 4.12 TB
Failed VMs 0 Processed size 4.12 TB
VMs in progress 0 Processing rate 18 MB/s

Success 4/02/2011 2:53:39 p.m. 4/02/2011 6:29:55 p.m. 250.01 GB 250.01 GB 20 MB/s 3:36:16
Success 4/02/2011 6:29:56 p.m. 4/02/2011 7:27:08 p.m. 80.02 GB 80.02 GB 24 MB/s 0:57:11
Success 4/02/2011 7:27:10 p.m. 4/02/2011 8:17:18 p.m. 68.27 GB 68.27 GB 23 MB/s 0:50:08
Success 4/02/2011 8:17:18 p.m. 4/02/2011 9:11:51 p.m. 80.02 GB 80.02 GB 25 MB/s 0:54:32
Success 4/02/2011 9:11:51 p.m. 4/02/2011 10:33:58 p.m. 350.00 GB 350.00 GB 73 MB/s 1:22:07
Success 4/02/2011 10:33:58 p.m. 5/02/2011 2:20:30 a.m. 200.00 GB 200.00 GB 15 MB/s 3:46:31
Success 5/02/2011 2:20:30 a.m. 5/02/2011 6:30:56 a.m. 450.00 GB 450.00 GB 31 MB/s 4:10:26
Success 5/02/2011 6:30:56 a.m. 6/02/2011 9:19:08 p.m. 1.96 TB 1.96 TB 15 MB/s 38:48:11
Success 6/02/2011 9:19:08 p.m. 7/02/2011 6:38:08 a.m. 660.01 GB 660.01 GB 20 MB/s 9:18:59
Success 7/02/2011 6:38:08 a.m. 7/02/2011 7:49:36 a.m. 80.02 GB 80.02 GB 19 MB/s 1:11:28

[devlin7]

Hi,

Not quite sure what you mean by "what backup job mode do you use"?

Rgds

[Vitaliy S.]

Having different processing rate is expected. Should more details be required, please have a look at this topic: Processing rate from 70 to 400 MB/s

Regarding the job mode question...what I am referring to is whether you use SAN/Virtual Appliance or Network backup mode?

[devlin7]

Direct SAN. At present, the freeNAS device is attached to each ESX machine as network file system storage.

Any suggestions to improve a backup time of four days? I am looking forward to seeing the incremental backup speeds tonight.

Would it make any difference if I scrap the storage on both my ESX servers and just add a Linux server directly to Veeam?

[Vitaliy S.]

The performance bottleneck might be related either to the source side or to the target. To figure this out, I would try backing up to the local drive to see if that makes any difference.

Besides you may check out the recommendation we have in our F.A.Q. topic, please have a look:
>>> READ THIS FIRST : [FAQ] FREQUENTLY ASKED QUESTIONS <<<

[devlin7]

Thanks for your feedback.

The Veeam backup is installed on a Windows 2008R2 server with 4 CPUS and 12Gb of RAM. CPU never seems to get above 25%. The backup device is a Quadcore server with 24Gb of RAM. I was going to set this up with ESX so Veeam could talk to it directly but I couldn't present the full 20TB of disk via ESX. In fact, I couldn't use iSCSI because of the 2TB limit.

I have read the FAQ from top to bottom and three times sideways, I didn't find it particularly useful.

[devlin7]

I have been this morning and didn't notice that the backup scheduled for Saturday night, started automatically this morning. The rates are completely different this time around and while I realise this is not throughput, it is the only indication I really have.

Success 7/02/2011 9:47:47 a.m. 7/02/2011 10:18:07 a.m. 250.01 GB 250.01 GB 141 MB/s 0:30:20
Success 7/02/2011 10:18:07 a.m. 7/02/2011 10:26:38 a.m. 80.02 GB 80.02 GB 160 MB/s 0:08:30
Success 7/02/2011 10:26:39 a.m. 7/02/2011 10:42:20 a.m. 68.27 GB 68.27 GB 74 MB/s 0:15:40
Success 7/02/2011 10:42:20 a.m. 7/02/2011 10:51:30 a.m. 80.02 GB 80.02 GB 149 MB/s 0:09:10
Success 7/02/2011 10:51:31 a.m. 7/02/2011 10:55:51 a.m. 350.00 GB 350.00 GB 1 GB/s 0:04:20
Success 7/02/2011 10:55:51 a.m. 7/02/2011 11:08:09 a.m. 200.00 GB 200.00 GB 278 MB/s 0:12:17
Success 7/02/2011 11:08:09 a.m. 7/02/2011 11:10:37 a.m. 450.00 GB 450.00 GB 3 GB/s 0:02:27
Validating guest agent availability for the VM
Unable to use guest agent: VM guest OS state is 'NotRunning'
Success 7/02/2011 11:10:37 a.m. 7/02/2011 11:38:39 a.m. 1.96 TB 1.96 TB 1 GB/s 0:28:01
Success 7/02/2011 11:38:40 a.m. 7/02/2011 11:55:04 a.m. 660.01 GB 660.01 GB 687 MB/s 0:16:23
Success 7/02/2011 11:55:05 a.m. 7/02/2011 12:03:53 p.m. 80.02 GB 80.02 GB 155 MB/s 0:08:47