Veeam 9.0 Copyjob -> Catalyst: A perf disaster!

[froopd]

In Drammen Kommune (one of Norway's largest cities) we are using Veeam for backup and HPE StoreOnce as backup media.
Refer to Veeam Case #01827400.

When Veeam 9.0 came, we change from CIFS to Catalyst to avoid the previous know performance problems related to StoreOnce CIFS against Veeam.
The backup/restore jobs have been working perfectly since we changed. However for Security we had two StoreOnce boxes in two different locations With StoreOnce CIFS replication between and when changing to Catalyst we have to do this replication through the backup Application (Veeam 9) using Backup Copy Jobs.

These backup copy jobs have been a complete performance and stability disaster, and our conclusion now is that we almost cannot use the Veeam product to get Catalyst data replicated from on StoreONce Box to another.

Anyone else having the same problems/issues? If not and you are running Veeam With Catalyst against two StoreOnce systems: How have you been able to make this work properly? I have looked through all documentation, but cannot really find a good way of getting the Catalyst Data over from StoreOnce#1 to StoreOnce#2 using Veeam. I have contacted HPE, but they cannot giv me a good answer how to do it, and I have not had any solution yet from Veean either, so anybody out there who have set up this without any issues, please tell me how.

Main problem seem to be that Veeam in the copy job is not doing a "Box to Box" transfer: They are inflating the data on SO#1 moving it back to the Proxy and then sending it to SO #2. This operation puts a lot of extra stress on SO #1 and SO #2 and the network in between. In addition it looks as if this process also generated significant read activity on SO#2 which jams the network even more.

I would like to og back to CIFS for this. However that will take me back to the disastrous performance in the primary backup/restore jobs, so that is not an option.

Nother option is to switch backup software to a sofware supporting Catalyst and Catalyst replication (NetBackup, Backup Exec, Data Protector), but I kind of like Veeam, so I would prefer not to change backup software.

Any ideas how to solve this will be very much appreciated.

Regards
Frode

[foggy]

Hello Frode, could you please give more details on your issue and setup? What is reported as the bottleneck for the backup copy jobs? Do you have per-VM backup chains enabled on the repositories? What servers are specified as gateways for the repositories? Have you reviewed the configuration guide?

In this case data indeed needs to be rehydrated and sent over to the gateway server prior to be written to the target store. However, overall performance of the v9 backup copy job writing to the Catalyst Store should be better than the one of the v8 job writing to the CIFS share and all its synthetic activity is performed locally, without data rehydration. There also will be improvements in this area in the upcoming 9.5.

[jmmarton]

I think what he's getting at is that we don't support Catalyst to Catalyst replication, so when copying from one SO to another SO we have to rehydrate the data first. I've run into something similar with customers who have Data Domain and asked about DD Boost replication support when copying from one DD to another.

Joe

[foggy]

I understand that, however would like to know whether the setup is configured optimally.

[Arnfinn]

Hi.
Referring to Veeam call: Case # 01816162.

I was running Veeam 8 against a StoreOnce using CIFS with StoreOnce replication to another Remote StoreOnce system over a low bandwidth WAN link 100 mbit.

This was working without any problems in Veeam 8 and my bandwidth was more than sufficient to have the boxes in sync.

However there were known performance issues related to using CIFS on StoreOnce with granualr operations (like takling up file Explorer). These problems were eliminated by moving to Catalyst in version 9, so we decided to move to Catalyst to increase performance of our granualr operations.

Catalyst has no internal replication, so this meant we had to set up "replication" in Veeam.- This was done using BCJs. We implemented this a couple of months ago and since this time my Veeam backup solution has been a nightmare to administer:

The good news: The backup jobs themselves using Catalyst works perfectly and restore performance and granualr restore performance is super.

BUT the really bad news: The BCJs are completely unstable, they are using a lot more bandwidth than the CIFS solution when replicating over WAN, and they seem to not only do read on the source StoreOnce, but they also try to read data back to the Veeam proxy from the target StoreOnce over Wan when copying a backup image from source to target.

Is there anybody out there who has been able to make this work for more than minimal data amounts? If so: How did you set up the solution?
I have tried several configuration changes in Veeam (incluig recommendations from Veeam), but nothing seems to stabilize this solution.
I have also contacted HPE: The answer from HPE is that for copying data between two StoreOnce systems, especially over a LB link, it is not recommended to do it via a proxy server like Veeam does, but to do a box to box replication like NetBackup OST and Data Protector does. which makes sense. However it seems that the only option in Veeam is to inflate data on one side transfer it back to the Veeam proxy, partially dedupe it on the Veeam proxy and then send it over to the target side over WAN to be stored. This is working very slowly compared to the old CIFS replication, and for some reason it also causes read activity on the target StoreOnce where I assume datathen are transferred back to the veeam Proxy over my WAN-link which makes the performance even worse. Information I have from Veeam is that there is no plans in Veeam to support Cataløyst replication which would solve this problem. As a result the BCJ are reporting errors, hanging, aborting and my Veeam logs every day has turned into a nightmare to check. So currently my offsite backups are good for nothing after upgrading to Veeam 9 and using Catalyst against StoreOnce.

I am now seriously considering moving back to CIFS to get my offsite copies back in shape. However this will reintroduce the granualar performance problems related to using Veeam against StoreOnce CIFS and it will also force me back to run fulls instead of Synthetic fulls.

If anybody out there has done this migration successfully, a feedback on how, will be very much appreciated, since going back to CIFS is something I rather would not like to do.

Br
Arnfinn

[foggy]

Hello Arnfinn, if I'm getting right, it is the same setup as in the original post of this thread, right? Anyway, the question I'd ask are the same.

[vdellachiesa]

We run two SO with Veeam 9 and catalyst. For the first couple of months we had nothing but problems. Our experience with the backup copy jobs were poor. We were reciving cofc 1404 error messages every night. With a 3rd party,Veeam and a private hotfix things finally settled down. We know have two bcj that ship about 140 vms across a 10gb pipe. 100 in one direction and 40 in the other direction. I don't know incremental number we push but across the pipe though.

[rhannevig]

Hi

A simple workarround is to run two different jobs, depending on the job time and the job window that you prefer. Just simply run one job to one storeonce, and one to the other.

[SyNtAxx]

We gave up on Veeam Catalyst 'integration' and converted everything back to CIFS. We have two StoreOnce 6500s with copious amounts of stored data on it replicating at the hardware level. To say that forcing a customer to go from hardware based replication to software based replication at the added expense of more administrative overhead is an enterprise solution is in my opinion silly. I'm not sure how much my opinion counts for, but there is very serious chance we will be seeking a replacement product, or even going back to previous product as a result if there is not intent on Veeams' part to fully support product integrations such as Catalyst. I do like the Veeam product, generally speaking. There are a few things I dislike and have had to find solutions to get around.

[DagMoen]

I hope Veeam and HP can manage to work together to find a solution for this asap.
I need to do a similar setup and need a solution to get veeam backupdata on one StoreOnce replicated to another.

Rgds

Dag

[CraigL2112]

Wanted to chime in here on this issue and offer my take:

We, too, are considering dumping the Catalyst "integration" entirely, even given the quite-impressive performance increase of backups and returning to CIFS given the issues with having to use BCJ to replicate to our off-site StoreOnce. We've hit several issues, and the performance is quite a bit worse than when we let the StoreOnce appliances replicate.

SyNtAxx and I discussed this topic in a thread around the time Version 9 came out -- NOT having true Catalyst Copy support is a huge step backwards. If Netbackup, Backup Exec and Data Protector can all do it, why can't Veeam?

The official response, back then, was something along the lines of "we don't want to replicate corrupt data" and a sob story about how device to device replication could not be trusted -- or something like that. IIRC, several other StoreOnce users hoisted up the 'BS' flag.

Veeam is clearly a top-shelf product, with some great features. Support is almost always outstanding and it has never let us down when it comes to restoring. However, it is very clear that the decision to not support Catalyst Copy has caused issues. Hopefully this is rectified in a future release........

-Craig

[jazzoberoi]

+1 to what Craig said.

We've decided to not deploy the Veeam-HP 'Catalyst' integration till the 'Catalyst Copy' feature has been deployed.

[Gostev]

IIRC, several other StoreOnce users hoisted up the 'BS' flag.

If you think this is BS, then you are more than welcome to spend a few weeks at any Veeam support office and see yet another customer got hit by storage-based replication issues live I love it how you question my statements solely from your point of view, without having access to the same "big data" I have (support stats for almost 200'000 companies over 8 years)...

And for the record, I don't actually recall "several other users hoisting up this flag" anyway. In my recollection of all previous discussions on this subject, most commonly the statement was rather "I understand there's a data corruption risk - but just let me take, because it's my choice". This does not sound like a good solution for an enterprise backup though!

I do have some solid ideas and am intend to work with HP to hopefully come up with the reliable solution based on Catalyst replication (granted we see good business results working together with the current integration, of course). But so far they have been very willing to modify their code to suit Veeam-specific needs (which is what made the current integration possible in the first place), so nothing is impossible!

[Regnor]

Can't we use the built in replication of the StoreOnce and have a Veeam Proxy running at the target side to do scheduled corruption checks?
@Gostev: Does it really happen that often that storage-based replication makes problems? In the case of StoreOnce HPE claims to be reliable and uses checksums etc for replication; OK they can't say anything different

[foggy]

Can't we use the built in replication of the StoreOnce and have a Veeam Proxy running at the target side to do scheduled corruption checks?

Backup health checks cannot be done outside Veeam B&R jobs.

@Gostev: Does it really happen that often that storage-based replication makes problems? In the case of StoreOnce HPE claims to be reliable and uses checksums etc for replication; OK they can't say anything different

It is right the fact that StoreOnce replication is "reliable and uses checksums etc for replication" results in it accurately syncing all the storage corruptions occurred on source. It's not about reliability of StoreOnce replication, the main concern here is that any storage-based replication is not content aware and will replicate bad data just as well as it replicates good data.

[brentdowling]

We had the exact same experience. We patiently waited for v9 to support catalyst to resolve CIFS performance issues only to find that now the issue is getting replication completed in a timely manner instead of the actual backup. So now instead of technical calls about backup and recovery throughput, we are still making support calls on replication performance. It seems that this could have been easily solved by just handing the copy task off to the appliance. It is possible to spoof the veeam system by pointing the veeam server at the remote copy of the catalyst data. I've been looking for a way to just use the catalyst copy API externally to at least get a copy off site. Seems like it would not be that difficult to do.

[Regnor]

It is right the fact that StoreOnce replication is "reliable and uses checksums etc for replication" results in it accurately syncing all the storage corruptions occurred on source. It's not about reliability of StoreOnce replication, the main concern here is that any storage-based replication is not content aware and will replicate bad data just as well as it replicates good data.

Then we shouldn't write any bad data to disk
But you're write bad data or blocks could be possible, and for the reason that I don't trust any single device, the storage based replication should also not be trusted...
Regarding health checks; wouldn't Backup Validator do the job?

[foggy]

Yes, you can use Veeam Backup Validator, which basically recalculates checksums for all data in the backup chain and compares the results to the previously stored values.