3PAR storage snapshot issue

[chjones]

I challenge the idea that Veeam is "Enterprise Ready", based on our experience with Storage Snapshots.

We are close to 90-95% virtual and in our two main Datacentres we use Veeam to protect all production Virtual Machines. We also use HP 3PAR Storage so were very excited about the 3PAR Storage Snapshot Integration introduced in Veeam v7. After some initial teething problems (the proxies were unable to see the snapshot volumes, this was fixed by patches from Veeam Support) we were able to use Storage Snapshots on one of our nightly jobs. This worked well for a few months.

We then expanded this about a month ago to use on all backup and replication jobs (some of our backup jobs take 8-10 hours to complete so only requiring VM snapshots to exist for minutes instead of many hours is a huge benefit). However, since enabling snapshots on all jobs we are inundated with warnings from vCenter about "consolidation needed" and frequent backup job failures stating "Error: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server". On subsequent retries the jobs will eventually complete and the consolidation warnings from vSphere are resolved.

We have disabled Storage Integration for now until this is resolved.

I have a support case open regarding this and have been advised that the QA Team has been engaged to look into this issue with the aim of developing a permanent fix. The issue, so I'm told, is the way Veeam makes connections to vCenter. For every VM in a job there is a new connection to vCenter to work out what the VM is, where it is, configuration, etc. Then if you have Storage Snapshots there is yet another connection to vCenter for every disk the VM has to work out which storage the VM is on, is it snapshot capable. So for a job with 50 VMs, each VM has at least 2 disks, you can have well over 100, or even over 200, connections to vCenter.

We have investigated vCenter and resources are not exhausted and ports are available, however the support team are suggesting there are simply too many connections trying to be made to vCenter and this is causing our issues. It seems a very inefficient way to connect to vCenter. Why not establish a connect at the start of the job and use that to gather all the information you require. For smaller VMware deployments there likely won't be any noticeable problems, but when you say "Enterprise" you are talking hundreds of VMs and Veeam just isn't there yet. I hope it will be soon, but not currently.

[foggy]

Chris, thanks for sharing your experience. Could you please post you support case ID number? Thanks.

[chjones]

Hi Alexander,

My support case is 00548259.

I received an updated Veeam.Backup.Core.dll two days ago from the engineer, Mark, I've been working with (btw, I wanted to give props to Mark, he's been a great technician to work with!).

We have run through two nights of backups since using the updated DLL and enabling Storage Snapshots on all Backup and Replication jobs and we haven't had any errors or jobs failing and retrying, and no alerts from vCenter about VM consolidation being required.

This is the explanation of the changes made with this new DLL:

"The updated dll contains a different logic. In short words: instead of opening multiple connections to VCenter, Veeam obtains metadata regarding all VM's from that VC using only one connection. That may slightly increase job duration but helps to avoid the initial problem. The fix should be included in upcoming version 8."

This seems like a much better use of resources and has been a definite improvement for us. We are replacing our Veeam Proxies in the next 6-12 weeks with HP ProLiant BL460c Gen8 Servers that will be have direct 8Gb Fibre Channel to the 3PARs so we're hoping with these new servers and the updated way Veeam connects to vCenter that our backups should be much more reliable and faster.

I was shocked at the speed in which an updated DLL was provided. Impressive!

[veremin]

Hi, Chris, thanks for sharing the found resolution; much appreciated. For those experiencing similar problems, please, contact our support team directly - you will be provided with the updated DLLs, once your issue is confirmed to be the same one.

[Gostev]

In the context of this topic, sound almost like this DLL enables physical server backup
Vladimir, please split this sub-discussion into the new topic, whenever you get a chance.

[GarthH]

Dont forget to get the updated dll's after every patch release, applying any updates overwrites the dll fix

[veremin]

There is no golden rule in this case. Some of the private fixes are incorporated into the next patch, the other (requiring logic changes) into the next release. So, once the patch is release, it's worth checking whether the issue shows up again or not. Thanks.

[chjones]

So ... Patch 4 was released. I logged a support case, 00587435, and requested if there was an updated Veeam.Backup.Core.dll that is compatible with Patch 4. I was advised by the engineer, and I quote:

"This fix was included in latest patch and you can safely install Patch 4."

Despite the fact the fix is not listed in the Patch Release Notes, which is why I contacted support, I took the advice of the engineer. This is what we pay our maintenance contract for, they know the product better than I ever will.

We updated, problem returned. Exactly the same issue.

I have case 00590726 currently opened and have been told that I was incorrectly advised in my previous case and the fix is NOT included in Patch 4.

"I've verified this moment, the fix is not included in the patch 4, my apologies for the misinformation once again. We've requested the fix for build .871 to our R&D department, I'll send it once it's ready.
Thank you for your understanding."

The last update I received 10 hours ago is that the fix has been released for testing and we will be contacted as soon as it's available for download. I appreciate the very quick turnaround to try and get the fix out to us. I can never say Veeam is slow on support turnarounds.

However, I am disappointed with the Veeam Technical Support Team for the misinformation we were provided. We trust the vendor's technical support team to provide accurate and reliable information, particularly around fixes they have released for the product. I understand when you have a tech issue that hasn't been seen before there will be time involved to troubleshoot, but getting a simple yes or no wrong if a fix is included in a publically released patch is a problem.

When Veeam spams our vCenter and hits the connections limit it's almost a Denial Of Service attack on our vCenter. For now we are putting up with the issue as the jobs fail and retry and eventually succeed.

Hopefully the fix will be released soon and it will be permanently resolved in Version 8. My rant is over

[Gostev]

Hi, Chris.

I've forwarded this to Support Management... this will be investigated, but human do make mistakes and miscommunicate, so while you do have full right to rant right here, sometimes this kind of stuff cannot be avoided I guess.

But yes, there is no chance that fix could have been included in Patch #4 in the second half of May, because at that time this patch was already in code freeze and final testing stage. Which is exactly why Vladimir suggested that you double check with support. Which is what you did, actually

Release Notes are usually the most trustworthy sources, as they are approved directly by QC.

Sorry again, and thanks!

[chjones]

Hi Anton,

Thanks for the reply. I fully understand that humans do make mistakes, and I did read the release notes and didn't see this issue mentioned, hence the support call to verify. It is what it is and Veeam B&R has been a brilliant product for us and makes huge strides with every new release (I am really looking excited about WAN acceleration for Replica jobs in v8!!!!).

We received an updated DLL on Friday (unfortunately the backups had already begun by that time) so we applied the update yesterday and had no failures overnight and no consolidation warnings from vCenter which was great. If the jobs run without issue again tonight I'll be closing the support case.

I've always said that I am very impressed with the turnarounds we receive from Veeam Support and the speed in which we received the updated file was fantastic.

Now to work out an easy way to change all of our jobs from Reverse Incremental Jobs to Forward Incrementals and retain the integration with our other backup product (we still backup a lot of data that Veeam cannot protect) and we'll have an almost perfect setup. Reverse jobs runs at 10-20MB/sec and test Forward jobs have run at over 150-250MB/sec. INSANELY FAST! This is the next item on my to-do list

[Gostev]

Sure, but you would have to pay for this speed with the disk space consumption

We are planning to introduce the new backup mode option to primary jobs in v8 (similar to what Backup Copy does), basically this will be the golden middle between disk space consumption and performance. This new backup mode maintains most benefits of reversed incremental backup (same low disk space usage), but requires 2 I/Os per block instead of 3 I/Os per block, so transform should be correspondingly faster. Also, because its forward incremental, backup complete real fast thus reducing the time VM spends on snapshot.

Anyway, glad the original issue is solved for you. We are looking for a better way to fix this then the current hotfix (it is still not very optimal), but I have confirmed that at least the current hotfix implementation will be included in v8.

[veremin]

Now to work out an easy way to change all of our jobs from Reverse Incremental Jobs to Forward Incrementals

It should be possible with the use of PowerShell. If you need help, feel free to create a separate topic in corresponding subforum, and I will help you to write the required script. Thanks.

[emachabert]

Hi,
Encountering the same behaviour here and opened a support case ( # 00643238) to obtain the dll.
They told me Patch4 has the fix, which isn't true as stated above.

Anton, could you please tell the support it is not included in Patch4.

thank you

[m_zolkin]

Hi Eric,

My name is Mike Zolkin and I am in charge of the Veeam Customer Support team, EMEA & APAC. You've been dealing with one of our newcomers. Thank you for the heads up. We'll train him and make sure he won't do such faults in future. Please accept our apologies for any inconvenience caused by this incident.
If you need any assistance, you could always reach out support management by logging to cp.veeam.com and using "Talk to Manager" button.

Thank you,
Mike

[emachabert]

Hi Mike,

Thank you for your feedback too.
We applied the fix and we will enable storage snapshots on monday to avoid any problem this week end.

Even if you have newcomers making little mistakes, you are still aproximately 999% faster than the competition

[hlawatsch]

Hello everybody,

I know , this topic is a little bit older but we had the same Problem on one of our 3par System.
So I had two Questions:
1.) Was a patch released in the meatime which fixes this Problem
2.) If not, can someone provide me this DLL ? That would save my day

BR
Stephan

[foggy]

Stephan, what Veeam B&R version are you at? According to Anton's post above, the fix should have been included in v8. If you're still on v7, please contact technical support to get the fix.

[hlawatsch]

we are at v7 but we have discussed in company recently to go to v8 not knowing that this would solve our snapshot problem

[foggy]

Another reason to upgrade then.