backup performance and compression level

[Gostev]

Ah, via Software iSCSI Initiator?

[bc07]

No. Normal FC SAN, vmfs storage, I later copied the vbk file (1.95TB) to the backup server in 7hours. I used the SAN for the VBA as target because it is faster than over a file share to the storage on the problematic backup server.
But that does not fix the issue(s) on the backup server because Veeam on the backup server has a problem with all storages as destination including the one where I backed up to from the VBA.

[Gostev]

Oh, so you are backing up inside VMDK file attached to VBA?

[bc07]

Yes with VBA to another vmfs volume (vmdk). At the moment that is the fastest way for me to backup out file server. From the backup server it would takes days to backup the file server.

[bc07]

I did a fresh install on the backup server (where we are having the backup performance issues with Veeam) on a separate disk drive and only installed win2008 x64 SP2 and Veeam and all the current drivers for mainboard and FC-HBA. The issue is still the same: when making full backup of large VM's (tested with single vmdk file) the processing rate and write speed to the target goes down from 100+MB/s to 50MB/s after 25minutes and later down to 10-20MB/s.
It seems it is hardware related not sure if it is AMD, Chipset or BIOS related and the question is if buying Intel instead of AMD guarantees that we don't have the same issue again. Server upgrade would cost around $1300 more if we go with Intel instead of AMD.

[Gostev]

You could ask for moneyback

[bc07]

Oh, Veeam gives money back?

I just found out it is not or not to 100% hardware issue! I'll try to fix it (backup performance issue....) tomorrow.

[bc07]

Status Veeam backup performance issue:
I found out that the two things I’ve noticed (fastest backup speed only with no compress/WAN target and rapidly degrading transfer/process rate at backup of (large) vmdk files) are caused by the same issue.
After more testing and the previous results I can say for sure that the issue (in my case):
- is NOT the source or target storage
- is NOT AMD/Intel CPU or chipset related
- is NOT any kind of connection issue/bottleneck (SAN, Network)

At a system without the issue, there is no performance (processing rate) difference between no compression and doing any kind of compression (except of course CPU/memory are too weak for the higher compression).

[bc07]

Actually there are there things I've noticed. The third one is that when having multiple vmdk's per VM that the processing rate drops significantly after each vmdk's being processed when deduplication is enabled.

[tsightler]

So, please don't just tell us what was NOT the problem, tell us what your problem was, that would be far more valuable.

[Gostev]

I think Enrico likes intrigue

[theflakes]

I'm anxiously awaiting the solution to this as I've seen the same problem with gradual slowdowns for our largest VM when conducting full backups on it. I've tested several scenarios with no resolution found yet. I finally gave up and lived with it as performance on all non-full backups is good and it only affects one VM.

[bc07]

Because Veeam should have noticed that issue if they had a good QA. The problem is reproduceable.
I spent 80+ hours including nights and weekends to troubleshoot the problem. The official Veeam support is terrible they just go over the same things like having a bottleneck somewhere or storage issue. The only helpful tips I got here in the forum.
The last time I found a "bug" in a product after spending weeks of testing I didn't even get a Thank You.

Tom the "fix" would not help you.

[tsightler]

So you're saying you are unwilling to share the fix with the community that at least tried to assist? Sure, maybe the "fix" won't help me, that's OK, but my professional curiosity is still wants to know what you found. I understand being frustrated with support and having to spend so much of your time to resolve an issue that you feel should not exist in the product, but I think it is extremely unfair to engage the community for assistance, admit that the only help you did receive was from them, but then refuse to share your findings with us just to spite Veeam. Many of us here have spent untold hours on this forum answering questions or trying to reproduce issues that other members are seeing just to help them, that's part of what makes the community great.

Still, it is your right to withhold the information, and I respect that, even if I'm disappointed.

[bc07]

I've shared and sharing it with the community but not with Veeam.

[Gostev]

I've shared and sharing it with the community but not with Veeam.

I take it as "Thank You" for all my assistance and sleepless hours thinking what else to suggest in attempt to help you find out the root cause... only a tip of an iceberg here in this topic, since we also exchanged countless private messages discussing troubleshooting steps and their results.

by Gostev » 29 Mar 2011 01:52

Yep, that is 2am on my local time.

[bc07]

Well, you get paid by Veeam. I get only invoices from Veeam.

The tips mentioned in the forum did not point out the actual problem, they were just more helpful than what support suggested. I still had to find out the problem through trial and error ....

[Gostev]

I only get paid by Veeam to work in the office from 9am to 6pm, and during that time I am mostly doing my primary job, not these forums (you can guess that by looking at my regular forum hours). Just like Tom, I do not get paid by Veeam for participating in this community during my personal time, such as 2am at night. You do not get paid for your hobbies, unfortunately. Actually, I wish I did, because I would be millionaire by now... few hours everyday, and even more on weekends at overtime rate really adds up

Posted at 11:20pm on Friday night

[joergr]

I still had to find out the problem through trial and error ....

....but you won´t at all tell it to this forum, right? This is more than strange to me to say the least. I am always very curious about very different problems and solutions. Now that is what a forum is made for.

Joerg

[rkovhaev]

Because Veeam should have noticed that issue if they had a good QA. The problem is reproduceable.
I spent 80+ hours including nights and weekends to troubleshoot the problem. The official Veeam support is terrible they just go over the same things like having a bottleneck somewhere or storage issue.

I am assigned to Enrico's support case. and I'll be reproducing the issue in our environment. Let me put the record straight, you are saying that the issue is with the Veeam software itself, yet you have addressed it solely by making some changes in your environment and using shipping Veeam Backup code - without any updates to our product. Something clearly does not match up in this story...

Additionally, please do not forget that we use proprietary VMware API and are completely dependant on hypervisor arhictecture and behavior, so backup performance issue with high CPU/DISK usage VMs could caused by VMware architecture peculiarities. Try to create a snapshot on VM that runs huge DB and serves hundreds of requests and you get performance issues with VM itself... here is the quote from VMware:

When a disk operation is performed within the guest, the disk I/O is recreated by parsing each snapshot delta file in the chain. This produces additional disk overhead on the host because more than one file must be opened and processed to recreate the I/O data for the guest operating system. For best performance, remove all snapshots in the guest operating system or store performance-sensitive data on an independent virtual disk.

So, maybe during the backup your storage performance drops significantly or ESX/ESXi is not really good at reading data from parent disks and writing to snapshot while reading disk/providing Veeam Backup with data. So far I've been in Virtualization it always gets down to storage limits. That is why support always looks into storage and tries to find a bottleneck.

Nevertheless, we'll try our best to reproduce it and share results with the community.

[Gostev]

Some offtopic posts were removed. No matter how you feel about this, please be polite and stay on topic when posting. Since OP confirmed he has the issue resolved, but is unwilling to share the resolution with community, there is no point of keeping this discussion topic open. Thread is locked.