Best replication compression ESX 3.5 > ESX 4 enviornment

[tsightler]

Also, some notes on "pull" replication:

Advantages
- Only one, only in case of ESXi targets: keeps VMDK rebuild traffic local to target site (not across WAN); meaning 3x less traffic during incremental sync (read replaced block - write to VRB - write new block to VMDK)

Well, and one more, VERY big one for us:

- One click failover from any loss since Veeam server is co-located with replica's.

[Gostev]

So where's the white-paper? Whitepaper's that no one knows about aren't really worth creating. Saying you don't distribute them on the web site seems silly, how would I know to ask my salesperson about it if I don't know it exist. The "best practice" issue seems to be one of Veeam's weakest points. Is this possibly to encourage customers to use Veeam partners for implementation services? I guess I could see that, but as a "do-it-ourselves" type of IT department, I want all the docs I can get.

The reason is simpler. Actually we have over 10 technical whitepapers like that, but they were not particularly ready for public distribution before (written by our SE, not properly formatted or proofread). We are actually finishing cleaning them up right now, and after this is done, we will be able to make them public.

[Gostev]

Well, and one more, VERY big one for us:
- One click failover from any loss since Veeam server is co-located with replica's.

You are right, I missed the operational benefit.
Updated my list accordingly!

[glennsantacruz]

@tsightler -

Transactionally replicating the SQL database to the remote site is interesting, but seemingly very dependent on understanding how Veeam is handling transaction with regard to the replica state. For instance, does Veeam open a transaction (and keep it open) for the duration of the replica job? If so, your target "transactionally consistent" SQL database will not be aware of any replica state for mid-replica failure. On the other hand, if Veeam is regularly (transactionally) updating the database with replica job state on an ongoing basis, you may have a good point (with one exception - the replication job finishes and your network fails before the database is committed; you then have a "good" replica and a "bad" database). It would seem that you would certainly reduce the risk window with this method, but it still doesn't seem safe. Again, having a clear architectural understanding of how Veeam is doing this (without, of course, revealing any trade-secret items) is important.

@gostev -

Yes, we did discuss this a few weeks ago. However, two major items were missing from that discussion: 1) you mentioned a whitepaper (for which I'm still waiting) and 2) there's no coverage of how replication works at a low level. We did cover the point regarding keeping a valid copy of the SQL database, but we did not get into the mechanics of pull versus push (and that push is apparently not a safe method). Nor did we discuss whether the blocks are written to a vrb file prior to being injected into the VMDK. It was a very good discussion, but it was incomplete.

And one more point to add on the pros/cons of "push" replication (to reiterate Tom's statement): safety. Disadvantage on the "pull" side is lack of safety. Maybe that elusive whitepaper will explain it fully and give us a good path for best-practice replication. Personally, I really want to "push", but Tom's points really have me rethinking that strategy.

[donikatz]

Yes, this elusive draft white paper... is it available to our sales reps?

[Gostev]

@Doni yes, for now you can get this white paper through your sales rep. I have also asked the corresponding SE to update it with couple important points (currently missing) based on initial feedback of users who tried to implement it, before it is made public. Anyhow, it does not really have any additional info comparing to what already covered here (except of step-by-step of setting up native SQL database replication w/screenshots).

@glenn, why do you think that "push is apparently not a safe method" comparing to pull? Technically, there are absolutely no differences. Just a matter of what will be accessed over WAN - source or target, but the code itself is the same, and moreover unaware whether it is "pulling" or "pushing". So no matter what you choose to do, and no matter at what moment the disaster hits, you will always be able to failover to last know good state your Veeam Backup server knows about.

[glennsantacruz]

The reason I don't see "push" as a safe method (again, in the context of a mid-replication failure) is detailed throughout this thread. Basically, if you are writing blocks to a VRB and VMDK, and then experience a failure on the Veeam server that is pushing, you're left with an inconsistent replica. True, subsequent replication passes should correct this problem -- however, that assumes the failure situation is temporary (maybe a network hiccup), and that the Veeam server will reattempt. In the situation that the pushing Veeam server (or datacenter) is destroyed, we are unable to power-on the replica VMs in the surviving datacenter (because they are in an inconsistent state). If we had a copy of the Veeam database that was used by the pushing Veeam server, we could rollback the replica to a "last known good state". The chief problem here (and Tom's point, if I understood correctly), is that an attempt to maintain a copy of the "pushing" database is prone to failure: if we have a good copy of the database prior to the site failure, we'll certainly have all of the "known" rollback points. BUT, that particular copy of the database will be unaware of the "in-flight" replication writes into the VMFS and VRB files, so a rollback point will not be "true".

I think it's probably time to change the conversation direction. Instead of us trying to determine how best to assure "safe" replication (in the event of failure), why not turn it over to Veeam to prove?

So here's my new query: will Veeam please demonstrate the safety of replication jobs in the event of both temporary (network blip) and long-term (site outage) failures?

You've already pointed out that this thread (and others) likely are already deeper than the whitepaper that you'd referred to, so I'm asking Veeam to please seriously review this topic and provide a new up-to-date whitepaper on the subject. In fact, I'll be very happy to write this up, provided that Veeam will review and make corrections.

[tsightler]

Right, as part of my DR design, I have to assume the complete loss of the primary datacenter and/or source site. I don't get the luxury of assuming that my source site was destroyed but somehow the Veeam server survived the destruction, or that the loss of the datacenter was predicatable.

When we tested "push" replication we intentionally failed the replication by "powering off" the Veeam server, thus leaving an incomplete replication on the remote destination. Without the original Veeam server we could not figure out any way to recover from this situation. We tried to use a backup copy of the Veeam database from before the "interrupted" replication cycle, but Veeam seemed to simply ignore the partial VRB file and try to power up the tainted VMDK file. We opted for the "pull" replication because in this scenario Veeam seemed to handle the failed replication cycle properly.

Are you saying that Veeam is supposed to be able to recover from a failed, in progress replica even if the primary Veeam server is also lost in the process?

[Gostev]

Yes, but of course as I explained you need to replicate Veeam Backup configuration database to standby Veeam Backup server in the DR site. This is the key requirement. If you do not replicate config database to standby Veeam Backup server, then you are exposed even with pull scenario - in case if something happens with your Veeam Backup server in DR site, you will only be able to power on latest state - while failover to earlier time will not be possible.

[glennsantacruz]

Yes, but of course as I explained you need to replicate Veeam Backup configuration database to standby Veeam Backup server in the DR site. This is the key requirement. If you do not replicate config database to standby Veeam Backup server, then you are exposed even with pull scenario - in case if something happens with your Veeam Backup server in DR site, you will only be able to power on latest state - while failover to earlier time will not be possible.

I really hate to be so slow on understanding all of this, but I'm still not convinced this works properly. I'm unclear as to what question you're answering with "Yes, but..." ; is it Tom's question about mid-replication failure recoverability?

When you say, "replicate Veeam backup configuration database to standby..." are you describing a *real-time* transaction-based replication of this SQL Server? Or are you saying that a post-replication-job SQL backup (and subsequent copy to standby site) is sufficient?

[Gostev]

Yes, I was answering Tom's question about mid-replication failure recoverability. And as for SQL replication, I am talking about built-in SQL Server replication, not post-replication-job SQL backup.

[glennsantacruz]

So as a summary, then, will you confirm that this setup is completely safe (from any mid-replication failure):

1. Production datacenter running Veeam with local SQL server
2. DR datacenter running Veeam with local SQL server
3. Both Veeam servers "push" replicas to the corresponding remote datacenter
4. Both Veeam servers are using in-built SQL Server native transactional replication to a target database located in the remote datacenter

If this is the case, then Tom can gain the network efficiency of "push", and we can all rest assured that we will not have any issues with replicas failing at any point in this scenario. Is this correct?

[Gostev]

Absolutely correct, thanks for summarizing Glenn.

[glennsantacruz]

Now I think I'm just crazy for extending this conversation even further.

Given that we can accomplish "safety" of push replicas by implementing real-time SQL replication, we now must cross the bridge of SQL changes... we could:

1) Research/implement transactional replication for SQL Express ( so as not to introduce additional licensing burdens on our Veeam environment ). However, SQL Express has limitations with respect to transactional replication.
2) Use "full blown" SQL installations in both sites, with "full blown" transactional replication ( and "full blown" Microsoft licensing costs... )
3) "Cross-point" the Veeam servers. What I mean is to point the "prod" datacenter Veeam to a SQLExpress instance in the DR datacenter, and vice-versa. This would (naturally) introduce latency on all the Veeam jobs, with respect to database updates, but we'd be facing that very same latency with real-time replication (wouldn't we?). At face value, this seems just crazy, but the more I consider it, the more I'm liking it. I'm especially considering Tom's case here - this could give the "best case" of push replication without the nasty problem of site failure (or other mid-replication issues...)

Thoughts?

[Gostev]

Glenn, I believe the whitepaper I mentioned is talking about option 1 from your list. I don't think there are any SQL Express limitations that affect this, since our SE has actually tested the concept himself before writing the whitepaper. Have you get a chance yet to obtain the WP through your sales rep and review?

I agree option 2 is an overkill.

Option 3 is interesting! The amount of data Veeam Backup stores in the database is pretty minimal (job configuration and sessions history), so this may work very well.

Also, just so you know, ability to perform "Import Replica" (similar to importing backups) is on our short list of replication improvements (not in v5, but high priority). This should allow to remove the requirement to sync configuration databases for replica recovery purposes.

[donikatz]

"Also, just so you know, ability to perform "Import Replica" (similar to importing backups) is on our short list of replication improvements (not in v5, but high priority). This should allow to remove the requirement to sync configuration databases for replica recovery purposes."

Nice!

[glennsantacruz]

There is no whitepaper from my perspective. Nobody knows about it (other than your posts), nobody has seen it (other than your SE), nobody has been given a peek at it -- it's vapor as far as I'm concerned. Our sales rep didn't even know how to spell Veeam - you think they're going to have access to a mystical whitepaper? I doubt it.

From what I've gathered, SQL Express 2005 limits you to "Subscriber" only with regard to replication (see http://www.microsoft.com/Sqlserver/2005 ... ntegration ). That's a pretty major limit, and would force you to write your own replication code/schedule it to run/etc. -- and that's the whole crux of this problem. Your point is that we must have *real-time* replication on this Veeam database, and that intermittent synchronization will not provide the proper level of safety. Surely a home-grown replication job on that database would not suffice... unless that whitepaper has more magic than expected.

I'm rolling forward with testing option 3.

[Gostev]

There is no whitepaper from my perspective. Nobody knows about it (other than your posts), nobody has seen it (other than your SE), nobody has been given a peek at it -- it's vapor as far as I'm concerned. Our sales rep didn't even know how to spell Veeam - you think they're going to have access to a mystical whitepaper? I doubt it.

Wow... no comments...
You should seriously consider changing your "Veam" reseller

[donikatz]

Glenn-- I've asked my rep for it, haven't heard back yet. In the past he's been able to hook me up with unpublished white papers, so I'll let you know if I get my hands on it.

[glennsantacruz]

I'd very much appreciate that - you can PM me if you want to avoid the "whitepaper police" (apparently those whitepapers are hush-hush...)

[Gostev]

Resellers don't even need to ask Veeam reps - all documentation is readily available Veeam Partner portal - both public and technical/internal like this one. We have been distributing internal materials like that past 2 years, and there were zero problems... I guess the issue is specific to some new resellers.

Anyhow, as I've said I will be pushing to make these technical whitepapers public once they are updated and properly formatted. This is almost done.

I have also asked for access to Partner Portal so I can download and send it to you guys if you cannot get is from your resellers in a timely manner.

[Gostev]

apparently those whitepapers are hush-hush

Only those which are not yet ready for prime time are hush-hush... other are available for public download today.
http://www.veeam.com/whitepapers.html

[Bunce]

I've been reading this with interest and thought I'd throw in a question..

Given the importance of a 'valid' VBR server at destination that knows about the current state of all rollbacks, and the difficulties/risks in achieving this in reference to Tom's use-case of a mid-replication failure, I'm wondering the following..

Is it possible to embed metadata into the vbr files which would allow a simple utility installed at the destination site to re-index all files to re-populate the SQL database?

This was (actually IS for us -> we haven't decided to switch to VBR yet) one of the advantages of esXpress (as I'm sure Glen knows) in that you could simply re-index a backup target, and within a few seconds/minutes have available a list of all replica's that you can restore from.

Dependencies on SQL-Replication and therefore license & cost implications maybe possible for companies with existing infrastructure but they are likely to have a higher budget and therefore already in the SAN-Replication market and likely not have a need for host based replication products anyway..

While we like really like some of the features of VBR, we keep coming across setbacks such as this, which while at the time appeared minor features of a product, are actually quite important (lack of ability to exclude local disks per VM, flexible retention schedules/purging, configurable concurrent backups/throttling, native backup to NFS targets without dependency on hosts, etc).

Guess that comes hand-in-hand with different products. Each have pros and cons

Andrew

[glennsantacruz]

This was (actually IS for us -> we haven't decided to switch to VBR yet) one of the advantages of esXpress (as I'm sure Glen knows) in that you could simply re-index a backup target, and within a few seconds/minutes have available a list of all replica's that you can restore from.

You nailed it in more ways than one. We are coming from the esxpress camp, and our initial impressions of Veeam were quite good. Only after committing ( purchasing ) the Veeam product are we discovering nuances in relation to safety of replicas & backups. I'm not bashing the Veeam product but just pointing out that it takes awhile to really uncover the low-level bits that ultimately make a very significant impact on overall Enterprise backup/replication. esxpress definitely had its issues (otherwise we wouldn't have moved camps), and our "doe-eyed" first impressions of Veeam in comparison to esxpress are starting to become more realistic... The whole dependency on a local SQL database was one of the "warnings" from the esxpress camp, but we're just now beginning to see these differences. It's a shame that neither vendor gets "real" in their forums (probably for fear of the other vendors "jumping on" that post to point out flaws...). I think that's part of the same reason we've not seen any definitive "here's how it works" explanations in this thread (I've asked a few times for a true engineering explanation of the replication process, but have been pointed to a vapor whitepaper), so we're left to our own "discovery" methods to ensure we're properly protecting our environments. I suppose the philosophy is, "to each customer his own", meaning that each customer is responsible for the labor involved in proving their DR plan with this product. Admittedly, there are many different environments and goals in DR, but I also would expect there to be some "basic" explanations that apply to 80% (or more) clients -- namely, remote datacenter replication (and safety of that replication -- think of the Seinfeld episode where you "take the reservation but don't *keep* the reservation").

While we like really like some of the features of VBR, we keep coming across setbacks such as this, which while at the time appeared minor features of a product, are actually quite important (lack of ability to exclude local disks per VM, flexible retention schedules/purging, configurable concurrent backups/throttling, native backup to NFS targets without dependency on hosts, etc).

Admittedly, one of the reasons we switched to Veeam was the support we saw in these very forums. Based on your comments, it looks like you are also an existing (or former) esxpress user. From our perspective, these forums are a night/day difference from esxpress - I'm not bashing, again, but I must say that there was a very consistent message of, "we're working on that", or "it's on our roadmap", or "submit a support request and we'll work with you directly - and not share our results with the community...". In fact, I'm still waiting for an answer on why their dedup appliance provides the same statistics if you just rename a vmdk... You'd expect that a vmdk rename would result in 100% dedup (provided the VM were not powered-on). I've yet to test these scenarios with Veeam, but you can probably guess that I'll be vocal on these forums about unexpected results...

Pros/cons, yes - that's the nature of the beast. Here's hopes that Veeam is more pro than con when compared to competition. Frankly this thread has reduced our overall impression of Veeam (considering the difficulty of getting a straight answer to how the product works -- which you'll see that we still don't have, even after asking a few times...)

To Veeam - I hope you're seeing this as constructive criticism instead of "complaining". This is a basic principal of physics: "without friction, you have no forward momentum." I'm giving you friction in hopes that we can help move your product forward....

[tsightler]

1) Research/implement transactional replication for SQL Express ( so as not to introduce additional licensing burdens on our Veeam environment ). However, SQL Express has limitations with respect to transactional replication.
2) Use "full blown" SQL installations in both sites, with "full blown" transactional replication ( and "full blown" Microsoft licensing costs... )
3) "Cross-point" the Veeam servers. What I mean is to point the "prod" datacenter Veeam to a SQLExpress instance in the DR datacenter, and vice-versa. This would (naturally) introduce latency on all the Veeam jobs, with respect to database updates, but we'd be facing that very same latency with real-time replication (wouldn't we?). At face value, this seems just crazy, but the more I consider it, the more I'm liking it. I'm especially considering Tom's case here - this could give the "best case" of push replication without the nasty problem of site failure (or other mid-replication issues...)

Thoughts?

So I have thoughts, just been too busy to get them out lately.

We were serious leaning toward using option "2" because we already have dedicated MSSQL servers, although we don't currently use them for Veeam, but we could. Unfortunately, those SQL servers are currently being replicated with Veeam, so it adds an interesting twist. Would I simply start replicating all of our SQL servers via transactional replication and quit using Veeam for those? Maybe. It does add a layer of complication we're not to sure about, it's yet another service which we'd need to be alerted when it's broken and another service to check on daily. It also gets complicated as you scale. We have four sites replicating (remote sites replicating to main datacenter, main datacenter replicating to secondary datacenter) and may soon add sites 4-6. Now I need 6 SQL servers, each transactionally replicating to it's partner site. That's 6 places for things to go wrong. Very iffy.

So, after thinking about all of those things, we had also dreamed up a few other options:

1. Basically, our first idea was exactly what you're thinking for "Option 3". Put the SQL server on the "target" side of the Veeam "push" server. This seems reasonable, our remote sites could all use a single SQL server in the main datacenter, while the Veeam server in the main datacenter would use a SQL server in the failover datacenter. It would be easy enough to proceduralize the failover to the secondary Veeam servers in the even of a disaster.

2. Our second thought was to keep doing basically what you're currently doing, sending a copy of the database after each replication job but, for added protection, take a storage system or file system level snapshot as well. That way, if a new Veeam replication cycle starts, and a crash happens during the cycle, you can just revert your replication targets to the point of the last replication cycle. This is also pretty easy to proceduralize if your environment can easy support snapshots.

3. Third option...just say the heck with it and try to wait a few months for Veeam 5 with SureBackup. Why does this help? Well, Veeam VBK files are significantly more robust than replica's. They can be imported into another Veeam server, and an "incomplete" VBK file (one that was interrupted in progress) can still be recovered. Actually, if you have a large VBK file with 10 servers, and 5 were backed up before the crash, those servers will be available to their latest recovery point, while the others can be rolled to their previous point. No need for the "original" SQL database. Since you can simply run your VM straight from the VBK files, there's very little need for replication anyway.

[glennsantacruz]

I agree about your assessment regarding transaction replication. As the environment becomes more complex, the transactional replication diagrams become burdensome and prone to failure.

Your option 1 is what we're currently testing, and we likely will run in this mode until Veeam 5 is released. So far, so good (as long as we're aware of what could go wrong and how to recover).

Option 2 gets tricky if your Veeam server is itself virtualized -- you'd have to ensure that all cached writes are flushed (and Veeam doesn't really care much to snapshot/vss freeze itself) if it's using a local SQL instance. External instances, though, would lend well to this approach.

Your option 3 is also appealing. I wonder about the time involved to import these VBK files into a Veeam server and how much that impacts RTO. I would anticipate that "readily available" replicas would far outweigh the VBK import approach, but it will be nonetheless worthwhile to test. This would certainly eliminate a nasty headache. I just hope the months between now and Veeam 5 are relatively disaster-free....

[tsightler]

Option 2 gets tricky if your Veeam server is itself virtualized -- you'd have to ensure that all cached writes are flushed (and Veeam doesn't really care much to snapshot/vss freeze itself) if it's using a local SQL instance. External instances, though, would lend well to this approach.

Your option 3 is also appealing. I wonder about the time involved to import these VBK files into a Veeam server and how much that impacts RTO. I would anticipate that "readily available" replicas would far outweigh the VBK import approach, but it will be nonetheless worthwhile to test. This would certainly eliminate a nasty headache. I just hope the months between now and Veeam 5 are relatively disaster-free....

I think you might have missed what I was saying regarding option 2, not that I can blame you, after I read it I couldn't make much sense of it either, I'll try again.

Prior to this discussion you were running a replica, backing up the SQL database, and shipping it to the remote site. I was simply saying to do the same thing, but, after shipping the backup copy of SQL database to the remote site, take a snapshot of the remote replica's. This way, if a failure occurs during the replication and you need to failover, you simply revert your snapshots of your remote replica's which are now clean, and you don't even need Veeam to fire them up unless you want an older rollback. If you do, for some reason, need an older rollback you can restore the backup copy of the SQL server, which is in sync with the snapshot, and your good to go. It doesn't matter if the Veeam server is virtualized or not, you don't even need the Veeam server at the remote site if you only want to revert to the previous complete replica.

Importing VBK files in our environment is quite fast. We have quite a number of VBK's that are ~1TB and it takes about 5 minutes to import one with 30 VBR files, even less if I cull off the majority of the VBR files, which I'd be likely to do in a disaster. We'd probably carefully size the VBK files that were targeted for "failover/DR" to keep them small and thus easy to import/repair quickly. We'd lose the de-dupe, but still gain the compression over a replica. I think it would be fine for our RTO requirements.

I'm curious actually, would you be willing to share your RTO target? I ask because I don't really think Veeam is an ideal product for very low RTO's, say <15 minutes. We use it for RTO's that are 30-60 minutes, but use other technologies for our low RTO targets like our ERP environment.