Reflections on Veeam and some hype around it

[Yuki]

We are an MSP and mange several dozens networks across the globe. Mostly manufacturing, research and bio-tech companies. We've been transitioning many to Veeam as we virtulize the environments. However, the more we work with Veeam, the more we seem to see "hype" that differs from reality. We keep hearing how great the product is, but after using it for a while we see that it is good, but not without issues that seem to come up again and again.

Overall, Veeam B&R is a great product and mostly works as advertised. Yet, some of the promises appear to have small print or an equivalent asterisk (not in print, but in actuality). Some of the experience with support has also been surprising and not in a good way.

Let me give you some examples. One of the reasons we started to switch people to Veeam was because of the claims of greater backup performance and completion times. Then we found out that Veeam places much heavier I/O requirements on the storage compared to other software we've used. This is mostly the case with Reverse Incremental or Synthetic full backups. The catch here was that in some cases storage that was more then adequate and showed great performance with other backups, now became the bottleneck. So while the approach to backups was more efficient (VMDK block level changes compressed and deduplicated), the I/O strain has resulted in significant performance issues.

In the process of investigation of these problems, we've contacted both Veeam and hardware vendors. We had representatives form all companies do the investigation portion, running some tests and then taking results back for analysis. From QNAP we had their senior engineer install special performance analysis tools on the storage and Dell used their DPACK to collect information about hosts, VMs and storage sub-systems. QNAP admitted an issue with their current firmware and said they will be working on a fix (it has been at least 6-9 month, no fix yet). Dell on the other hand looked at the storage load on servers and where Veeam claimed storage was the bottleneck, showed that server's storage throughput is not actually peaking and has headroom. With two companies claiming their product is working fine, we've had no luck having them actually figure out who's information is incorrect as both can't be right at the same time. To be frank, i would have liked to see this information as a side-note either in the official manual or at least in some pre-sales white-papers.

On separate occasions when I had to contact support, it was going well until we came to an issue where tech didn't know what was going on or how to fix it. Now, Gostev and others have said several times in the forums that we can always escalate an issue to 2-d tier just by asking. We've tried to do so and were told that there is no need to escalate and that the 1-t tier tech will continue working on it. On one recent occasion tech slowed down in response on a rather strange problem we've had with replication. Then after a silence period that was longer than we would have expected he came back saying that he can only provide limited support due to us running on a trial license (!). We sorted out the license issue (he never asked for license type or company name), but a lot of time has been lost. This was affecting a production environment for one of our more storage dependent clients. We've also seen Veeam support techs try to shift the issue to other vendors prematurely (being refereed to Microsoft support on an issue that happens on Windows, but in relation to Veeam). I've been most impressed with Russian support staff to which the calls roll during certain times and wonder if the ratio of high level support personnel in Russia is higher than in USA?

In the process of performing our own troubleshooting, we've discovered 2 bugs. One had to do with name UID of the veeam server and it's use of proxies (all of a sudden it couldn't use B&R servers proxy to process jobs after Veeam configuration was restored into a new VM). The second one was related B&R not picking up active settings based on the check boxes marked in a job configuration (was trying to use seeding when it wasn't told to do so). I've made a note of that to the tech and mentioned that this part of the Veeam code is likely re-used between versions, so this is likely to surface for some in v7 (if they happen to do some of the things we did), but he appeared to have no interest in either.


We wanted to see what it would take to become a public Veeam backup and replica destination, from licensing perspective (since we already run our own SANs and servers for clients in a datacenter and are adding a second facility). So i went to fill out the page asking for someone to contact us and provide information. I have not received a response after multiple requests.


As many here, i receive and read Gostev's news letter and once my own post here in the forums was "Best post of the week" (were, ironically, i was advocating for Veeam not to give out free licenses and to continue paying good wages for support engineers and programmers). Anyway, so I read all of these e-mails and often see not just good tips - but rather critical information on restoration of systems like domain controllers. That by default, in many common environments, some extra prep work is needed in order for veeam to restore systems correctly. I would like to see a page where Veeam would list most recent KB articles based on some of the information provided by forum members and bloggers so it becomes a part of the official knowledge base. It seems that manual and KBs


I still love Veeam for ease of use and no-nonsense deployment and configuration, but it feels as though a lot of information reported to support or in forums is not making it into manual and KB in order to keep up the appears of a perfect product. I do want to see those footnotes in deployment manual referring to special steps needed to restore some common systems or things to be watching out for. If the official guide doesn't mention a critical part/step/config that is only floating somewhere in the forums or is mentioned in Gostev's letter, well..... - it may not help someone during a production restore after following the official documentation.

Lastly, i think it would be great to see veeam assign client managers to MSPs so we get some extra attention. After all - we do a lot of heavy-lifting and often provide the first tier of support to clients before there is any talk of contacting Veeam. And we do it just because Veeam is a decent product, not because we get commissions (we don't. we don't resell any products). Yet, we make recommendations to our clients to buy hardware/software worth of hundreds of thousands each year. Client that had some major issues with Veeam replicating and backing up their 16TB file server is spending $250,000 on additional storage. Yet, we feel like that its worth paying for higher levels of support since the experience so far has been mixed. Instead of telling clients (who certainly have the money) to go all in on top level Enterprise versions with Premium support. We would have hard time justifying these recommendations after reporting our experience, especially when we are bombarded with sales calls from Nakivo and others competitors.


I really want Veeam to keep on improving and become an exceptional product/company even more so than you are right now. I believe in the work you guys are doing and i'm glad you are listening to customers a lot more than some of your competitors.

[Yuki]

Re-reading the post now and seeing all the typos I've made. Should have written this earlier int he day when the mind was still functioning.

Cheers

[Gostev]

Hello Yuki, thanks for the feedback. One thing you need to remember to do if you are not happy with the quality or speed of technical support provided, or require escalation to higher tiers, is to contact our support management directly. To do that, go to the customer support portal and click Talk to a Manager.

Obviously, not all tier 1 support folks are gurus, some of them could be new hires. They are working according to their work description, and they will try to troubleshoot the issue before escalating the support case further. Otherwise, I assume, they might get in trouble internally for escalating basic/known issue.

I think we should also consider to provide a shortcut to tier 2 support for our long-term Cloud Provider Partners, who by default are very experienced with the product.

[Fiskepudding]

Excellent post Yuki.

You touch many aspects here. I agree on most. Actually I am quite active in the DC restore thread.
Everything does not “just work”!

About the "hype", I have never seen a company NOT hype their own products, pointing out their good sides, and keeping quiet on their "less good"/bad sides.

Support can always get better. However, we are quite happy with Veeam support.
I base that on our experience with previous backup vendors like Symantec and Acronis.
Never felt that Veeam “blamed” some 3. party software or hardware. With Acronis the solution to a problem was always: upgrade, reinstall, or “it must be the storage”!...
they would not go further with support unless we did a reinstall and upgrade, even if were at latest build or just 1 build behind, even if no fixes where related to the problem.
Back to Veeam

I have also reported some bugs to WEB-support. And they seemed to take it serious; they recreated the issues and assured it will be fixed in newer version. But if they ignore important feedback regarding bugs, that is bad news and hopefully not according to guidelines.

And I agree with I/O load, our 8 bay QNAP struggle to keep up, so after moving to Veeam this does no heavy jobs any more. On the other hand QNAP has its general issues with performance, but they are affordable. Good for copy job when it arrives I guess.

In the end Veeam works very well for us. But, there is always room for improvement, and I hope that your post somehow reaches the right persons at Veeam.
I see Anton already replied, so I hope your points are noted.