Backup slow after upgrading fra 4.0 to 4.1.1

[3bbolig]

Hi
I did an upgrade fra ESX 3.5 to Vsphere ESX-4.0.0-update01-208167. In the same move i upgraded veeam from 4.0 to the latest 4.1.1. Im doing backup to a Synology NAS 5 disk RAID box and the speed was normally around 45-60 MB/s, but now im only hitting max 25 MB/s. I read the post about issue like this, but its from 2009 and i guess im past the patch vmware announced about the issue

Anybody seen this with the same releases?? im not sure if its vmware or Veeam that have the problem.

I dont have enough space on my SAN to do the backup there.

Now: on veeam 4.1.1 and Vsphere 4.0. 208167
Total size 926,83 GB
Failed VMs 0 Processed size 926,83 GB
VMs in progress 0 Processing rate 24 MB/s


Before running: Veeam 4.0 and ESX 3.5 update 3
Total size 873,90 GB
Failed VMs 0 Processed size 873,90 GB
VMs in progress 0 Processing rate 55 MB/s

The problem is not so much the speed, but the time its taking now.. 11 hours

UPDATE!! After reading another post ive changed the setting from Network Backup to Vmware vstorage API mode Network. Maybe thats gonna help ???

[Vitaliy S.]

Michael,

Yes, that would help for sure, after upgrading to vSphere also please make sure you've upgraded your VM hardware to version 7, so you could use CBT on your incremental job runs.

As for the current speed while using Network Mode (3rd radiobutton), a patch from from VMware should boost your speeds but not as much as you had on your ESX(i) 3 hosts, so I do encourage you to use vStorage API mode.

By the way if you have a backup server running on the VM, try using Virtual Appliance mode, that should give you better performance rates rather than vStorage API network mode. We recommend 4 vCPU for the backup server in order to have best backup performance.

Thank you!

[Gostev]

This is the patch for network backup performance on ESX4:
Network backup slow on vSphere ESX 4.0

[3bbolig]

Ohh

Ive changed the Hardware version to 7 and also to Virtual Appliance mode on the VM. So maybe tonight it will be a lot faster .. otherwise im gonna install the patch.

thx for a quick response i hope it works



/Michael

[Gostev]

Michael, it will only become faster starting 2nd pass (1st job pass will only enable changed block tracking on all processed VMs). Also, be sure that you have selected a vStorage API mode for your job, because other processing modes are unable to leverage changed block tracking.

[3bbolig]

Hi agian

Well almost everything worked out fine.. im down to about 6-7 hours for my backup at a speed around 40-45 MB/s thats ok for me... but one VM is failing and thats the Veeam backup server. Any ideas on why thats the case`?

I get this error in the report::
Backing up file "[SAN] kasperskymanager.42.dk/kasperskymanager.42.dk-flat.vmdk" BackupDisk failed Client error: Failed to open VDDK disk [[SAN] kasperskymanager.42.dk/kasperskymanager.42.dk.vmdk] ( is read-only mode - [true] ) Failed to open VMDK. Logon attempt with parameters [VC/ESX: [10.0.0.7];Port: 443;Login: [];VMX Spec: [moref=vm-77];Snapshot mor: [snapshot-392];Transports: [hotadd]] failed because of the following errors: Server error: End of file

Total VMs 19 End time 26-03-2010 02:24:19
Processed VMs 19 Duration 6:24:09
Successful VMs 18 Total size 926,83 GB
Failed VMs 1 Processed size 891,83 GB
VMs in progress 0 Processing rate 40 MB/s

So the only one not backing up is the actually backup server:-/


I can see that the fail is login issues, but all the other VM's are backup perfectly??

[Vitaliy S.]

Michael,

Glad that you've made it! As for the error do you experience the same behavior for other VMs that are located on the same datastore? Do you see this issue confirming? Could it be a permission issue? Please review the current thread that has the similar issue: http://www.veeam.com/forums/viewtopic.p ... MDK#p13241
If this is not the case it would be great if you could open a support ticket with us and forward all the job logs which contain more information on the current issue.

Thank you!

[3bbolig]

out of 19 VM's on the same datastore its only the backup server VM itself that fails.. but ill check out that post and see if i can figure something out

[3bbolig]

hmm If i change the backup job from Vmware Vstorage API to normal Network backup it works without error.. so my solution to the issue for now are
1 job with 18 VM's backuped via VMware vStorage API mode Virtual App and
1 job for the last VM runnnig veeam via normal Network Backup, hence it doesnt work through the API

Not really a fine way to do it ;/

[Gostev]

Michael, you cannot have Veeam Backup server backup itself in the Virtual Appliance mode, this is documented in the Release Notes.

[3bbolig]

that sux

So from going to 4.0 to 4.1.1 I say no to the good speed and I say no to doing the job in a single job.. Not really sure im that happy that i upgraded :-/

[Gostev]

So from going to 4.0 to 4.1.1 I say no to the good speed

This is statement entirely incorrect. It looks like you did not read the thread I referenced in my first post. Network backup performance was reduced for you due to your upgrade to ESX4, not due to Veeam Backup upgrade.

and I say no to doing the job in a single job..

Both 4.0 and 4.1.1 cannot backup Veeam Backup VM in Virtual Appliance mode. So again this issue has nothing to deal with Veeam Backup upgrade.

If you apply ESX4 patch mentioned in the above-referenced thread, and change your backup mode to vStorage API in "Network" mode, you will both be able to use single job, and have much faster backup performance than you had on ESX 3.5.

[TrevorBell]

Everyone who uses Veeam in a VM has the same issue so its not just yourself who has this problem. I just basically replicate the server to a different location once a week to keep a copy of the VM for DR purposes ...

I only look after 42 Vm`s and i can installed Veeam from .zip file to fully working in around 15 minutes which is also exceptable in a DR situation, So in my case so i could choose not to backup the VM at all and just download the latest release from Veeam and install

My production Stats

28 of 28 VMs processed (0 failed, 0 warnings)

Total size of VMs to backup: 1.08 TB
Processed size: 1.08 TB
Processing rate: 41 MB/s
Start time: 12/03/2010 03:00:19
End time: 12/03/2010 10:39:33
Duration: 7:39:14

Im well happy with these times...

Trev

[3bbolig]

HI agian

I still having problems getting my veeam backup to run smoothly after my upgrade to 4 series. Im ok with not backing up the veeam Vm itself but now i get this error on my fileserver vm
Creating snapshot CreateSnapshot failed, vmRef "vm-61", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "True" A snapshot operation cannot be performed.

the other vm backups fine. All VM are runnning hardware version 7

Any help on this issue?

[Vitaliy S.]

Hello Michael,

Please try to create snapshot on this VM using vSphere Client with quiesce enabled. Are you successfull with that?

Also try disabling VMware Tools Quiescence option and enable Veeam VSS if you're running Windows OS on your file server and re-run your job.

Thanks!

[3bbolig]

If i try to create a snapshot from Vsphere client i get the error below. Does this have something to do with disk size on 256 GB ? i recently tried to increase the disk size from 192GB to 256 GB

Create virtual machine snapshot
Fil Server Clinton - 10.0.0.8
File <unspecified
filename> is larger than
the maximum size
supported by datastore '
<unspecified datastore>

edit: also im not too keen on changing too much on the job as i now finally have it working on 18 out of 19 VM's. and i use to work with Vmware tools quiescence enabled.

[Vitaliy S.]

Yes, you're correct. It seems like your datastore is formatted with less block size required to make a snapshot.

Here is an existing topic with the similar error message, please have a look:

http://www.veeam.com/forums/viewtopic.p ... arger+than

Thank you!