No Catalyst Copy support?

[CraigL2112]

Hello everyone -

I'm very pleased to see Catalyst support added to v9 and am eager to kick the tires and see how much better the performance is, especially over FC. Am curious since there doesn't appear to be Catalyst Copy support (for directing StoreOnce to replicate to another StoreOnce appliance), what the play is to replicate backups that live on a Catalyst store? The only thing I can think of would be do maybe do a Backup Copy to a NAS share on the second appliance? Obviously, this would probably require rehydration on the source side, then RE-dedupe on the second appliance.......

Just thinking out loud here. Anyone have any ideas?

-Craig

[nefes]

Please review this post by Gostev, which describes why we don't support Catalyst Copy.

[CraigL2112]

Doesn't this somewhat defeat the purpose of Catalyst, though? IE: Only replicating new blocks to the target appliance?

[Gostev]

This particular purpose, it does defeat. But Catalyst does many other things which we all do support.

[marco.horstmann]

Only replicating new blocks to the target appliance?

Hi Craig,
Backup Copy will copy only new backuped blocks too. Ok, its maybe a little bit more then a Copy directly
between StorOnce with their own protocol, but its more secure reliable in my opinion. If you have corruption
inside your first StorOnce you get it replicated to the second system. With Backup Copy you have additional
healthchecks to make sure that our backupfiles wasn't corrupted by e.g. a media error or spontaneous bit state change.

Regards
Marco

[CraigL2112]

Guys -

Thanks a ton for your responses. I appreciate it.

I'll be honest -- this feels like somewhat of a step backwards for us StoreOnce users. As it stands now, StoreOnce fields the replication -- with this change, that load is moved downstream to Veeam. I can only assume you guys have internally measured the performance impact of this....is there any data you can share?

Fellow StoreOnce users -- what do you think about there not being actual Catalyst Copy support?

-Craig

[Gostev]

Craig, what change are you talking about? You can still setup and use StoreOnce replication if you like, just as you did before? It's just that we do not recommend this approach in conjunction with data protection, and thus don't want to encourage it by adding built-in management. Thanks!

[CraigL2112]

Gostev -

I can only assume you mean if we are backing up to a CIFS share. I'm talking about if I do a Catalyst backup and NOT being able to do a Catalyst Copy (like we do in Netbackup) to get the data moved to our other StoreOnce which lives offsite.

-Craig

[SyNtAxx]

I'm interested in this scenario as well. I have a StoreOnce 6500 in two locations. One is my primary Datacentre and the second is the DR location. Running Backup copy jobs for 1000s of vms is not optimal in my opinion.

-Nick

[CraigL2112]

I agree, Nick. In your case especially, you paid big bucks for those 6500s and they should be doing your heavy lifting (replication!) for you -- not your backup software. Catalyst Copy is one of the selling points of StoreOnce in general, so this decision to not support it is somewhat puzzling.

I'd be curious to hear what HP/IBRIX engineers think about the rationale of "you might replicate corrupt data" as the reason for not supporting it. I wonder why Symantec doesn't make the same claim.....

-Craig

[Gostev]

Actually we did discuss this with HP many times on many levels, as obviously they were pushing us hard to implement this as well. They understand and agree with our position. On our last meeting at VeeamON, we brainstormed some ideas together on how we can build robust solution based on Catalyst replication, and will consider them for the next major release.

I wonder why Symantec doesn't make the same claim.

Because they sell backup, not restore. We sell availability, and actually care for your ability to restore when needed.

[CraigL2112]

Don't get me wrong.. I'm a huge fan (and paying customer) of Veeam..we literally have a 100% success rate with restores. Can't beat that!

...but we also have a 100% restoration success rate of many, many TB of data we have backed up with Netbackup. It's the whole garbage-in-garbage-out scenario -- if you do a crappy job installing and configuring ANY product, your end result (in this case, your ability to restore) may not be all that stellar.

If you haven't, you really may want to consider polling customers about the decision to not support Catalyst Copy -- I'm suggesting this with my 'customer' hat on. I think you'll find most who see that there is Catalyst support (and understand what it means) will assume that Copy is part of it and be highly disappointed when they find out it's not.

The best analogy for this I can think of for this... is like returning a kickoff in football, running 70 yards in the open field then purposely tackling yourself. It makes my head hurt to think about.

Again, this is with my 'customer' hat on.

With all of this being said, I'm excited to do some tests with v9 to see how much faster our jobs run over FiberChannel, and how long replication takes with the Copy job..................

-Craig

[Gostev]

As a matter of fact, we did poll customers - implicitly, through those support cases on failing recoveries.

I can tell you quite honestly that we ourselves did not realize this issue existing 5 years ago, until we got our first few thousands of customers. But now with nearly 200'000 customers, the absolute number of occurrences when storage-based replication failed customers data protection strategy big times became scary. And more often than not, it costs people their jobs. Honestly, I much my users "highly disappointed" with lack of some features in Veeam, than losing their job a few months later.

It's unfortunate that you could not attend VeeamON, as I gave a number of real-world examples on a couple of related breakout sessions. It was an eye opener for many, from those conversations I overheard leaving the room.

Anyway, you don't need to convince me in benefits of storage-based replication either - we will gladly leverage it as soon as we find a way to build a reliable solution based on one.

[SyNtAxx]

Ok, so can somebody explain to me how a large enterprise can move his data from one location to the other if they cant reliably count on built in replication from a unit such as a HP StoreOnce 6500 (big $) ? Backup Copy jobs on 1000s of vms isnt going to cut it. Im all ears.

-Nick

[nefes]

Have you tried backup copy job in v9 between Catalyst stores already? With all improvements, made in v9, it should take reasonable time.

[SyNtAxx]

Have you tried backup copy job in v9 between Catalyst stores already? With all improvements, made in v9, it should take reasonable time.

I haven't even installed the product yet. However, from past experience with Backup Copy jobs it involves creating more policies and creates more administrative effort and potentially more server horsepower (in my case not an issue) to allow for these jobs to remain running in the background. You haven't answered my question, or is your answer the only 'reliable' method is Veeam backup copies?

I've been looking for the information Gostev mentioned was presented at VeeamOn, is there availability to this data or a recording of the event ?

-Nick

[nefes]

Since I'm not in Product Management team, I can't comment on any feature that will or will not be in any future version.
For v9 I am not aware of any other reliable way to copy backups from Catalyst to Catalyst except Backup Copy job.

[Gostev]

Unfortunately, VeeamON sessions are available for attendees only.

Also, I just wanted to clarify the issue, as it is being misinterpreted elsewhere. Storage-based replication by itself is extremely reliable and efficient engine, which is why we ARE looking for ways to leverage it in our solution. I have not seen it corrupt "good" backups while replicating those, or anything like that.

What makes ANY storage-based replication a bad choice for data protection TODAY are the following major issues, which are not specific to HP StoreOnce:

1. It is not content-aware, and as such it will replicate bad (corrupted) data just as well as it replicates good data. So, if your source backup files are corrupted, they will still be replicated without raising any flags, leaving you with no ability to restore from either on-site or off-site repository in the time of need. This is not the case with Backup Copy jobs, which validates data as it copies it over.

2. It does not make you meet the 3-2-1 backup rule, that every data protection strategy must be designed around, in the "2" part. Specifically, there are no 2 different medias with this solution: 2 identical storage device in sync with each other cannot be consider separate medias by any means, as backups remain in the same fault. For example, if backups (or data inside of them) is deleted on the source storage (accidentally, or due to a bug in software) - then this deletion will be propagated to the target storage, leaving you with no good backups anywhere. Again, this is not the case with Backup Copy jobs. Along the same lines, Cryptolocker also became a big problem lately, with encrypted source backups being extremely reliably and efficiently replicated to the target storage

There are also seems to be storage-specific replication issues with certain vendors, and I gave some recent examples during my VeeamON sessions. But I rather not post them here, as I know this will cause endless email threads with each vendor I will mention. And in any case, eventually they all come down to the above point 1 being the main issue.

Bottom line - thanks to our extremely large user base, we have been observing the above two issues consistently causing many of our users to actually lose their data, which resulted in my current position regarding leveraging "dumb" storage-based replication in data protection designs. But again, we do have plans to work closely with backup storage vendors to alleviate these issues by adding some intelligence on top of storage-based replication, which is indeed the ultimate off-siting engine.

[pirx]

We'll have soon 2x StoreOnce 6500, mainly because of the replication feature. So Catalyst Replication is must have for our next backup application. It' wouldn't make sense to buy this device (especially with the need to replicate data) and don't use it. I think there are also a lot of checks in Catalyst replication. Besides the 1500 VMs we've also to backup physical servers, so if we use this feature for them, there is no point to say it's not safe. We are also thinking about using the Oracle RMAN and MSSQL Catalyst Plugins (looking fine so far), also with replication.

I was really excited to hear that Veeam now supports Catalyst, but honestly you should have implemented Replication too and let the user choose what is best for them.

[chjones]

Wow this topic has really had some interest, wasn't expecting this. I also think this has gone down another path of Storage Replication (Catalyst Level) vs Application Replication (Veeam Level). Doesn't Veeam perform integrity checks on the backed up blocks and also has SureBackups, so wouldn't these capabilities negate the issue of replicating corrupt data? (Happy to be shot down on this one if I'm off the mark)

One issue I have with the way Veeam works with Catalyst is that, as I've previously stated, we use HP Data Protector to also backup physical servers to Catalyst Stores as well as initiate the replications. HP have a weird licence structure with StoreOnce. There is a StoreOnce Catalyst licence which is required on the StoreOnce to allow the use of Catalyst Stores. There is also a Catalyst Replication licence, which allows you to setup replication of Catalyst Stores between StoreOnce Devices.

We currently have the Catalyst Licence so we can write to StoreOnce Catalyst Stores with both Veeam and Data Protector.

The odd thing is that for Data Protector or Symantec NetBackup to write to a Catalyst Store all you need is the base Catalyst licence. You can then trigger a Catalyst Replication from within the backup product which will replicate the data to another Catalyst Store. Even though, at least in the Data Protector world, you need to have servers acting as StoreOnce Gateways at both the source and destination these gateways just act as listeners to each StoreOnce and report the replication progress back to the management server (Data Protector). The actual replication is completed solely by the StoreOnce devices, without the need for a Catalyst Replication licence.

However, if you to write data to a Catalyst Store and then have the StoreOnce automatically replicate new data to another Catalyst Store then you need the Catalyst Replication licence. This is not ideal as there is no knowledge or reporting that the replication has been completed, it's internally on the StoreOnce's. I don't really want to have to pay for another licence which for something which, I believe, is not as good as a controlled and reported replication.

I know in my company we would much rather have some level of awareness that a replication of data has occurred and be alerted when it fails.

My initial issue I posted about is that there are now several products that can write data to a StoreOnce Catalyst Store. There is Data Protector, NetBackup, BackupExec (I believe?), Veeam, MSSQL and Oracle, yet none of them can be made aware of data that each other has written. I don't need Data Protector or NetBackup to be aware of the contents of the data the other application has written to the Catalyst Store but I would love to be able to copy that data elsewhere, to another Catalyst Store or to an LTO tape. For example Data Protector doesn't need to know how to read the contents of a VBK file, that's Veeam's job, but would be such an improvement if Veeam data was written to a StoreOnce in a way that the other products could interact with the data and move it. That's my dream.

[pirx]

1. It is not content-aware, and as such it will replicate bad (corrupted) data just as well as it replicates good data. So, if your source backup files are corrupted, they will still be replicated without raising any flags, leaving you with no ability to restore from either on-site or off-site repository in the time of need. This is not the case with Backup Copy jobs, which validates data as it copies it over.

But shouldn't Veeam notice that the source data is corrupt? I mean... the problem doesn't seem to be that the replication is failing and silently changing blocks. It's the fact that the source backup is corrupt. I haven't looked in detail in Veeam's backup copy (I see that you write that it checks the data), but there is also the source repository which is the base for the copy to a second location.

2. It does not make you meet the 3-2-1 backup rule, that every data protection strategy must be designed around, in the "2" part. Specifically, there are no 2 different medias with this solution: 2 identical storage device in sync with each other cannot be consider separate medias by any means, as backups remain in the same fault. For example, if backups (or data inside of them) is deleted on the source storage (accidentally, or due to a bug in software) - then this deletion will be propagated to the target storage, leaving you with no good backups anywhere. Again, this is not the case with Backup Copy jobs. Along the same lines, Cryptolocker also became a big problem lately, with encrypted source backups being extremely reliably and efficiently replicated to the target storage

HP takes a replicated backup job as 2 different media. I'm not quite sure that the statement about the deleted files on one side is true for StoreOnce Catalyst Replication (not sure what the difference between Catalyst Copy and Catalyst Replication is). At the moment we are using Data Protector with Copy Jobs (Replication) and data that is deleted on one side is not automatically deleted on the other side. This is also true for HP's Oracle SO RMAN Plugin that is using Catalyst Copy.

Maybe we are talking about different things. I'm not talking about Replication inside StoreOnce which simply transfers all data from a source to a target store. I'm talking about StoreOnce Replication that is controlled by an application, in our case HP Data Protector or the RMAN Plugin. I'd like to see the same feature in Veeam. At the end it's an question how we can get the data to a second location and StoreOnce Replication looks good enough to us, especially with physical servers, VM's and a large amount of TB each day.

[matsholm]

I agree with both "CraigL2112" and "SyNtAxx" when it comes with the lack of intregration with Storeonce Catalyst Copy. Saying from a software company that hardware replication isn't good enough is just crap. I've been working with Storeonce (and other DD and NetBackup) appliances for so many years with many big customers and I must say that Veeams reply to why not implement storage based replication is very strange. If you're not trust HPE do have a reliable block-replication between Catalyst stores then why do you trust HPE to even be a backup repository? With your approach your even saying that all DR setups in the world that rely on mirroring or other storage based replication is worthless?
How should we handle our big customers that were really looking forward to have a good replication implemented, "nefes" reply sying that backup copy jobs between copy jobs in v9 should be reasonable is not good enough. What is implemented in v9 that would be so much better handling a backup copy job comparing with catalyst copy? Can Veeam do a backup copy from one store to another without rehydrating the data?
/Mats

[foggy]

Saying from a software company that hardware replication isn't good enough is just crap.

We're not saying it is not good enough, we're saying completely different thing:

Also, I just wanted to clarify the issue, as it is being misinterpreted elsewhere. Storage-based replication by itself is extremely reliable and efficient engine, which is why we ARE looking for ways to leverage it in our solution. I have not seen it corrupt "good" backups while replicating those, or anything like that.

What makes ANY storage-based replication a bad choice for data protection TODAY are the following major issues, which are not specific to HP StoreOnce:

1. It is not content-aware, and as such it will replicate bad (corrupted) data just as well as it replicates good data. So, if your source backup files are corrupted, they will still be replicated without raising any flags, leaving you with no ability to restore from either on-site or off-site repository in the time of need. This is not the case with Backup Copy jobs, which validates data as it copies it over.

Can Veeam do a backup copy from one store to another without rehydrating the data?

Currently data needs to be rehydrated and sent over to the gateway server prior to be written to the second store. However, overall performance of the v9 backup copy job writing to the Catalyst Store is much better than the one of the v8 job writing to the CIFS share and all its synthetic activity is performed locally, without data rehydration.

[pirx]

Currently data needs to be rehydrated and sent over to the gateway server prior to be written to the second store. However, overall performance of the v9 backup copy job writing to the Catalyst Store is much better than the one of the v8 job writing to the CIFS share and all its synthetic activity is performed locally, without data rehydration.

And this rehydration is one point we want to avoid in future. Did you compare the performance of Veeam backup copy with SO replication? I fear in larger environment it's just a question if you are able to replicate/copy at all.

[matsholm]

foggy I still don't buy your arguments for not using storage based replication. Primary storage are normally not content aware either and a lot of companies and organizations have their primary storage replicated/mirrored (on block level not cintent) without any problems although your right about replication bad blocks. But StoreOnce and other vendors have a CRC engines working in background to make sure that data that the appliacation writes to storage are exactly the data written to disk (and also CRC for replication), so all data is checked.
To me it sound like you didn't have the time (or will) to implement this as a copy solution, are you planning to have that in future or on other appliance vendors?

[foggy]

It's not about bad blocks and CRC checks, it's mostly about replicating corrupt data, for example, if the source repository was compromised after data was written to the storage (think Cryptolocker or whatever). In this case you're just copying unrecoverable backups bit by bit to the secondary location.

Yes, we do have such plans, as stated above:

Bottom line - thanks to our extremely large user base, we have been observing the above two issues consistently causing many of our users to actually lose their data, which resulted in my current position regarding leveraging "dumb" storage-based replication in data protection designs. But again, we do have plans to work closely with backup storage vendors to alleviate these issues by adding some intelligence on top of storage-based replication, which is indeed the ultimate off-siting engine.

[SyNtAxx]

So, there is no way to replicate an HP Catalyst store outside of a backup application (Veeam,Data protector)? In other words the HP StoreOnce 6500 can not replicate its own native catalyst stores on its own?

-Nick

[Gostev]

I think it will be best to confirm directly with HP. There could be a workaround, or some functionality like that coming soon. For example, I know for sure that Data Domain can replicate DDBoost stores on its own, outside of a backup application.

[chjones]

I've asked that question of HP and the answer was no, there is no way to have data on a Catalyst Store automatically replicated to another StoreOnce. A StoreOnce can have a replication licence applied, but that only works for VTL and NAS storage, not for Catalyst. With Catalyst (if you have a Catalyst Replication Licence) the replication must be initiated by your backup application.

HP weren't very forthcoming on reasons why the is the case, so I can only assume its so that replicated data is known to your backup product (whats the point of replicating that data if your backup product doesn't know it's been replicated to the destination? Importing Catalyst Objects into your backup product can be done, but it's not simple) ... and possibly also so HP retain some market share as their backup product, HP Data Protector, integrates the best with Catalyst and wants to manage the replication. I hope I'm wrong and this control over the replication can be allowed to other applications to control.

[pirx]

I still think that Veeam should be able to use the similar functionality as HP Data Protector with Catalyst protocol. HP DP knows about the replicated data and there is nothing going on behind DP's back. If objects in the source catalyst store are lost, the replicated data is still in the store on the other device. And I think it would also be possible to compare data in both stores to get the similar behaviour that Veeam backup copy has.

Maybe I'm missing the point. I can understand that this was not implemented in v9, but not this whole discussion about how unsafe appliance based replication is. If blocks are simply synced between both devices - yes. But AFAIK with StoreOnce Catalyst protocol, replication is managed by the backup application. So it's only a question what the application is capable of.