No Catalyst Copy support?

[SyNtAxx]

I don't know, my opinion on having catalyst 'integration' and no catalyst copy is leaving the job half done, what ever the reason. Sorry if that stings.

-Nick

[chjones]

I agree that the StoreOnce Catalyst integration isn't there yet, there is still more to be done.

We use HP Data Protector (DP) for over 50 Catalyst Object Copies every day. We have many remote sites that all backup to local Catalyst Stores via DP and then an object copy is automatically run by DP to replicate the data to StoreOnce's at our two main datacentres so we have multiple disk copies of data available. I was hoping I would be able to use this same setup for Veeam v9 in the following way:

1. Veeam backs up to local StoreOnce Catalyst Store
2. Use Powershell to initiate a DP object copy job to use Catalyst Replication to replicate the data to a StoreOnce at another site

The data Veeam writes to the Catalyst Store is not able to be imported into DP so we cannot perform any actions on it after the backup completes. Even if we used Veeam's Backup Copy jobs to create another copy of the data on Catalyst Store at the other datacentre we would still have the problem of integrating the data into DP.

DP is the product we use to write data to LTO6 Tape and you can't have two products controlling the same tape libraries so we need DP to control the tapes. All we need is Veeam's Catalyst data to be able to be imported into other products that utilise Catalyst Stores (DP and NetBackup) and we'd have a good place to start.

[SyNtAxx]

Yeah, I tested Catalyst integration and v9 backup copy. Its not doable for an enterprise with thousands of vms, especially since it copies one single vm at a time. Fail. The best I can do is use file share level access for the StoreOnce units with per vm chains to keep file sizes in check and allow the hw to do the work un-hydrated as intended.

-Nick

[Gostev]

Nick, do you mind if we review your logs to determine where the bottleneck is? Thanks.

[SyNtAxx]

Sure, have a look.

Background info for the test environment:

Fresh install of Veeam v9 using native SQL Express (as provided in the veeam installer)

Backup server: Fresh Window 2012 R2, 12 core @ 2.5GHz and 24GB RAM

Proxy: Windows 2012 R2, 10 core @ 2.5GHz and 12GB RAM

Source Storage: 3par v400 @ 944 drives

Backup Targets: 2 @ HPE StoreOnce 6500s connected to LAN via multiple 10gE and FC 8gbps

Replication link between data centre(s): 2Gbps dedicated to StoreOnce Traffic.

Both StoreOnce Units added as Catalyst integrated devices and mounted locally to the proxy/gateway.

Let me know where you want the logs uploaded to.

-Nick

[foggy]

The best would be to open a case and share the ID here. Support guys will provide you with the location for logs upload. Thanks!

[Gostev]

Yes, I just need a case ID with debug logs. I will hand it directly to QA team responsible for scalability testing.

[SyNtAxx]

I guess my questions are the following:

1) Under what premise am I opening the case on ? (The app doesn't support catalyst copy)

2) Backup copy is a single VM operation, am I missing something ?

3) When I call, what do I tell the support engineer? (Gostov told me to call? )


Thanks,

Nick

[nefes]

1) Backup Copy job is working slow, when source and target are both Catalyst repositories.
2) Backup Copy job without WAN acceleration does use parallel processing in v9 (new feature).
3) You can tell them that R&D asked you to provide logs, and give them a link to this topic.

[SyNtAxx]

1) Backup Copy job is working slow, when source and target are both Catalyst repositories.
2) Backup Copy job without WAN acceleration does use parallel processing in v9 (new feature).
3) You can tell them that R&D asked you to provide logs, and give them a link to this topic.

This is from page 271 of the veeeam 9 user manual:

Note:
"Backup copy jobs do not support parallel processing. Multiple VMs in the job are copied one by one, subsequently. Data for VM disks are also copied subsequently, not in parallel."

I'm going to test again and see if parallel processing works as you indicate.

-Nick

[foggy]

We'll update the manual, online help is already updated, as far as I can see.

[millardjk]

1) Backup Copy job is working slow, when source and target are both Catalyst repositories.
2) Backup Copy job without WAN acceleration does use parallel processing in v9 (new feature).
3) You can tell them that R&D asked you to provide logs, and give them a link to this topic.

It's that #2 that's killing me. While my client is using a pair of 2900s instead of 6500s, the 6 VMs he's trying to copyjob are 13TB in size. Writing to the SMB share and letting StoreOnce do parallel copy with proprietary WAN acceleration is giving us better performance--especially when you factor in copy interval timeouts--than Catalyst-CopyJob-WAN Accelerator provided.

I would really like to let the client have the option to "trust Veeam or trust HP" rather than having to chose between Catalyst & SMB.

[pbu]

Hi,
I build a test setup with 2x StoreOnce VSA 1TB with Catalyst Stores
StoreOnce version 3.13.1-1546.2
Veeam B&R 9 can see these CatalystStores and write to them, in the StoreOnce you can see the data coming in on the StoreOnce Catalyst Write Activity in the StoreOnce GUI

When a Copyjob starts you see activity again on the Replication Data Job Outbound (on the primary StoreOnce VSA) and Replication Data Job Inbound (on the secondary StoreOnce VSA)
What I expected to see is when Veeam starts a Copy Job it should make use of the Catalyst Copy Job features and the transport activity should showup as StoreOce Catalyst Copy Job Outbound Throughput and StoreOce Catalyst Copy Job Inbound Throughput.

I think when the data is replicated via the Data Job channel it gets re-hydradrated, transported and then dedupped and compressed again on the secondary StoreOnce VSA.

The idea was to make use of the Catalyst Copy Job features with the orchestration of Veeam B&R 9, but that seems not to be the case now...

[foggy]

Peter, your observations are totally correct. Catalyst Copy is not supported in this version of Veeam B&R.

[pbu]

So what is the big benefit we get with Catalyst Support of Veeam B&R 9??
We can use the Catalyst Store instead of CIFS or NFS shares for a primary Repository

But Catalyst Stores can not Replicate like CIFS or NFS Stores

[SyNtAxx]

So what is the big benefit we get with Catalyst Support of Veeam B&R 9??
We can use the Catalyst Store instead of CIFS or NFS shares for a primary Repository

But Catalyst Stores can not Replicate like CIFS or NFS Stores

Exactly, what is the benefit....not much except a nice talking point.

What i've done is set up backup to SAN location --> copy to SO 6500 (CIFS Mode) --> Hw Replicate --> remote SO 6500.

-Nick

[millardjk]

Theoretically, the Catalyst I/O performance is higher than using SMB (would everyone just *quit* calling it CIFS!?!), especially when used in conjunction with per-VM file chains.

[SyNtAxx]

It might very well be, I am using the per vm chains now which is fantastic btw! I am seeing about 800MB/sec when sending data to the SMB Store Once Shares as viewed from the target SO. I am running a single giant proxy server with 20gbps LACP. when I have multiple copy jobs running I can get about 80% of the pipe full, so I can probably stand to adjust the number of repo streams a bit higher.

I do believe in the product and the value it brings, despite some of my complaints.

[tsightler]

Weekly synthetic fulls can be a big benefit with Catalyst, no more requirement to run active fulls to the StoreOnce all the time can be huge and you can use BCJ with per-VM chains. Based on your setup you can use the same server to talk Catalyst to the local StoreOnce to read the data, and talk Catalyst to the remote StoreOnce to write the data, so you're still technically talking Catalyst on both sides. Do this with per-VM and you get parallel BCJ with local Catalyst and Catalyst across the wire. Still not as efficient as Catalyst Copy, but can be quite workable in many situations.

[Gostev]

Yes, I would say the biggest benefit of Catalyst is instant synthetic full backup creation and transformation (instead of physically moving data within StoreOnce, we just set or update references to the exiting data blocks). Results in 10 or more times faster processing for those operations, and solves the biggest support issue StoreOnce users have experienced.

Additionally, when you do Catalyst over WAN to a low bandwidth mode Catalyst store, you will also save bandwidth due to advanced source-side dedupe by Catalyst.

And Jim is correct, generally better performance and reliability over SMB protocol (I don't know if it is still the case, but StoreOnce used to leverage a terribly outdated SMB server version).

[millardjk]

Based on your setup you can use the same server to talk Catalyst to the local StoreOnce to read the data, and talk Catalyst to the remote StoreOnce to write the data, so you're still technically talking Catalyst on both sides. Do this with per-VM and you get parallel BCJ with local Catalyst and Catalyst across the wire. Still not as efficient as Catalyst Copy, but can be quite workable in many situations.

Interesting architecture (Catalyst->LAN->Proxy->WAN->Catalyst) to "keep it talking Catalyst", but what if you have another proxy in play because you're also using the second StoreOnce for local backups (Catalyst->LAN->ProxyA->WAN->ProxyB->LAN->Catalyst)? Does Veeam recognize that the ultimate source & target are both StoreOnce Catalyst Stores and still "keep it talking Catalyst"? If so, does it still make sense to use WAN Accel for connections <100Mbps (when permitted by license & infrastructure)?

In my design, we have a StoreOnce in each of two locations for local backups, and we're doing copy jobs from each local to the respective remote StoreOnce. Small jobs have been working well, but bigger ones are...troublesome, especially when we also try to use WAN Acceleration to beef up the efficiency of the 20Mbps WAN.

[haslund]

Yeah, I tested Catalyst integration and v9 backup copy. Its not doable for an enterprise with thousands of vms, especially since it copies one single vm at a time. Fail. The best I can do is use file share level access for the StoreOnce units with per vm chains to keep file sizes in check and allow the hw to do the work un-hydrated as intended.

Hi Nick,

Since v9 you have the option of using Parallel Processing in Backup Copy Jobs to avoid copying a single VM at a time.

• Backup copy parallel processing. With v9, backup copy jobs will now process multiple VMs in
parallel, just like primary backup jobs. This improves the backup copy and retention processing
performance due to removing “dead time” between each VM, and will further speed up processing
when per-VM backup file chains are enabled on the target backup repository

See https://www.veeam.com/veeam_backup_9_0_ ... _en_wn.pdf for all the other new awesome new features and enhancements.

[foggy]

Interesting architecture (Catalyst->LAN->Proxy->WAN->Catalyst) to "keep it talking Catalyst", but what if you have another proxy in play because you're also using the second StoreOnce for local backups (Catalyst->LAN->ProxyA->WAN->ProxyB->LAN->Catalyst)?

Gateway server is specified per repository, so you could add the same store as a separate repository for remote office and assign ProxyA to be the gateway for it.

[millardjk]

Since v9 you have the option of using Parallel Processing in Backup Copy Jobs to avoid copying a single VM at a time.
See https://www.veeam.com/veeam_backup_9_0_ ... _en_wn.pdf for all the other new awesome new features and enhancements.

Awesome stuff; unfortunately the training materials for VMCE9 explicitly states that BCJs don't support parallel processing (module 4: deployment. Enabling parallel processing).

It may be supported in truth, but if you get asked about it on the exam, I hope the "correct" answer isn't based on the training material.

[Gostev]

Review of Catalyst benefits (in German, but mostly graphs and numbers)
https://mycloudrevolution.wordpress.com ... -catalyst/

[Markus.K1985]

Thanks for sharing Gostev!

[Gostev]

Nice, the author himself Thanks to YOU!

[Markus.K1985]

Also a must Read for Catalyst Users:

Veeam with HPE StoreOnce Catalyst Configuration Guide
https://www.veeam.com/wp-hpe-storeonce- ... guide.html

[johna8]

We have 3540 setup for catalyst and Veeam at each local site.
Writes are fine, however backup copy jobs seem to run on average around 14 MB/sec.
We are having to throttle the network - however from other experiences can you achieve a decent read MB/sec?

Our issue is our rate of change, given we are trickling the data via off-site - Veeam just can't get up due to the slow reads.
I might have to look at perhaps StoreOnce replication over Veeam backup copies.

The other alternative is using faster storage for the landing zone to allow faster read/writes for a shorter period - rather than having the StoreOnce and backup jobs continuously trying to read the data.

From the German article it seems that the reads were quite high for the copy job?

[foggy]

Just to confirm, do you see source reported as the bottleneck for the copy job? What StoreOnce firmware version do you have?