Physical Exchange Backups - DAG

[r-low]

Hi,

We have a customer who has 16 physical Exchange Servers. They have 2 physical locations and each location contains 8 physical Exchange Servers. There are multiple DAGs configured for these servers and we are working to ensure that we have the most efficient backup configurations in place.

Currently the Veeam Agent for Windows (VAW) can't tell which Exchange Server is holding the main role for each DAG. That functionality doesn't exist, to my knowledge. As a result, we have to backup all of the hosts to ensure that the data is protected. However, the deduplication rates for these jobs aren't as good as the backup system that Veeam is replacing. The old backup solution was DAG aware and their full backups of all of their Exchange Servers was approximately 22TB in size. They are using Veeam to backup this environment now and the full backup is approximately 49TB in size.

The backups are being saved to a CIFS share on an HP StoreOnce 5500. We have to use the CIFS option as opposed to backing up to a Catalyst repository because the VAW doesn't include the API for Catalyst configurations.

My main questions are, what is the best way to back up these servers? Should the servers be backed up one at a time in individual jobs, or grouped by protection groups? We currently have them configured in protection groups based on their physical location. We could do 16 jobs, each with a separate server object in each job but would that get us a better dedup rate on the storage repository?

Is there a timeline for when the VAW will be able to write directly to a Catalyst repository? Is there a plan for the VAW to have some central intelligence built-in so that it can properly support a physical Exchange DAG Configuration? I look forward to hearing back from someone in the community soon.

Thanks,

R-Low

[Rick.Vanover]

Hey Ryan - I can tell you that the DAG logic you are seeking is planned in the next update to Veeam Agent for Windows. If I had to give you a date I'd say it's coming within the next 60-120 days to be generally available.

[Dima P.]

DAG logic you are seeking is planned in the next update to Veeam Agent for Windows

That would require Veeam B&R server as a central management unit for all Veeam agents. Backup job should be a failover cluster job type.

Currently the Veeam Agent for Windows (VAW) can't tell which Exchange Server is holding the main role for each DAG.

Any chance you have a single node where all DBs are stored in a passive state?

Is there a timeline for when the VAW will be able to write directly to a Catalyst repository?

Wont be a part of next major release but it's now in the high priority list. Are you going to use Catalyst added to Veeam B&R as repository with Windows server gateway? Thanks!

[r-low]

Hi,

In this particular case the Veeam Agents are currently being managed by a B&R Server. We can work to reconfigure the backup jobs once the VAW is able to support and interpret the DAG configurations.

I have done a follow-up with the customer and I'm waiting for feedback as to the configuration of each of the Exchange Server hosts. If there is a particular host that is running in passive mode then I'm assuming that the implication is that it can be excluded from the backup job, correct?

The Catalyst support would be of significant value for this particular customer. My understanding is that the DAG support should be available with Update 4 but the Catalyst support wouldn't be available for quite some time after that, correct? In the current configuration we are presenting the Catalyst repository through a Windows gateway server, yes. It will be used to backup the virtual machines in the environment but unfortunately that won't be an option for the physical servers at this time.

If you could have the Catalyst support available for the physical agents by the end of next week, it would be much appreciated. Thanks.

[DMermod]

Has this DAG logic been implemented yet? If not, is there some way (beta?) that we can implement it.

We have an Exchange 2016 migration (from 2010) we are implementing in January. We are moving it from virtual to physical and have the need for this feature.I'm trying to avoid looking at another backup product for this. We utilize Veeam for pretty much everything and this missing feature is holding us back from continuing to do so.

If at all possible, please let me know.

Thank you!

[Dima P.]

Dennis,

Has this DAG logic been implemented yet?

It's going to be included in upcoming Update 4.

If not, is there some way (beta?) that we can implement it.

Should be possible to request a demo via your local sales team if I am not mistaken. Cheers!

[DMermod]

Will this solution work with an IP-less DAG or will it require an IP for the windows agent on physical Exchange 2016?

I'm preparing our environment and will need to know this as it looks like I will have to rebuild the DAG completely if I get it wrong.

Also, is the update any closer to being released?

Thank you,

- Denis

[Dima P.]

Denis,

IP-less DAGs are going to be supported by Agent Management functionality in upcoming Veeam B&R 9.5 Update 4, but not by standalone agent deployment. Thanks!

[ianbutton1]

Me too!
Our scenario - Exchange 2010 DAG (6TB in 22 mailbox databases) on three physical mailbox servers; backup storage is StoreOnce 4700 with Catalyst. Old backup (not Veeam) - Full weekly in c.36 hours (OK), daily diffs in under 2 hrs (each db in a separate job for concurrent threads, also speeds up retry if a single database backup fails).
Now testing VAW 2.2: backing up Passive DAG member to CIFS share on StoreOnce, Full backup of half the databases (all in one job) took 9 hrs (very good) at c.80MB/sec; but first incremental is now crawling at 10MB/sec (extremely bad). Throughput graph always shows incorrect results (e.g. During Full, a peak every 3 mins of up to 904MB/sec, zero in between!). Concurrent jobs (1 database per job) doesn't work as each job locks the source and they have to run in series, taking much longer)
It would be really useful to have Catalyst store as repository option for VAW (removes dedupe load from source) and concurrent threads allowing retries of only failed databases. This might speed things up. Until these appear, I fear that Veeam will not be usable for our Exchange backups.
On the other hand, I might have missed something . . . ??

[ianbutton1]

PS - The 10MB/sec incremental has processed 3% in 2 hours - i.e. nearly 3 days for 100%. That's why it's not really usable! No other StoreOnce traffic is competing for bandwidth. I suspect it's the dedupe in the source that's slowing it down (pull data from StoreOnce CIFS, compare, write back to StoreOnce - CIFS read & write graphs are identical).

[Dima P.]

Hello Ian.

Now testing VAW 2.2: backing up Passive DAG member to CIFS share on StoreOnce

Please stay away from testing DAG backup with this version. Official DAG support is going to be provided in with the Veeam B&R Update 4 (you can request RTM version in via our support team).

The 10MB/sec incremental has processed 3% in 2 hours - i.e. nearly 3 days for 100%. That's why it's not really usable! No other StoreOnce traffic is competing for bandwidth. I suspect it's the dedupe in the source that's slowing it down (pull data from StoreOnce CIFS, compare, write back to StoreOnce - CIFS read & write graphs are identical

Issue indeed might be related to the deduplication storage but that's only an assumption. Please open a case with our support team and let them investigate the debug logs. Thanks!

[DMermod]

Dima,

I have installed the Update 4 RTM and VAW 3.0 and am unable to process my IP-less DAG.

Is there something I am missing?

I have tried using the failover cluste option and it fails immediately.

The server option works but backs up all of the databases regardless of the active/passive status.

Thanks,

- Denis R Mermod

[Dima P.]

Hello Denis,

For IP-less DAG you need to create backup job managed by backup server with the type server (not failover cluster).

The server option works but backs up all of the databases regardless of the active/passive status.

All databases are going to be processed, database copy is not going to be excluded (as we need to grantee the application consistency) .

[DMermod]

Dima,

Well that's horrible news and I wish that was clarified earlier.

So if I move my DAG to IP based are you saying that the passive copy will be backed up only?

If so, I have exactly two days (today included) to do this before our Exchange migration....

Hoping for a quick response so I can get started.

As a follow up, will I still need to chain backup jobs for the IP-less DAG solution so they are processed at separate times or can they all be in the same job?

- Denis R Mermod

[DMermod]

When I got into work this morning I moved my test bed to an IP based DAG.

I was able to use the failover cluster option, but I still received a backup of every database.

It seems as though both options copy both active and passive databases regardless.

Please correct me if I'm wrong.

- Denis R Mermod

[Dima P.]

will I still need to chain backup jobs for the IP-less DAG solution so they are processed at separate times or can they all be in the same job?

You can use a single backup job for that. Job chaining is not required.

It seems as though both options copy both active and passive databases regardless.

That's correct.

[ianbutton1]

I have the same issue - DAG backup using VBR 9.5 Update 4 is processing all copies of every database. This seems unnecessary and takes a lot of extra time when we have 3 servers each with about 5TB of mailbox databases - 53 hours so far for the initial Full! That's 10TB of unnecessary data being pushed around our network - pretty inefficient. Backup of just the passive member should be sufficient. It will be interesting to see how the incrementals perform over the next few days.

[doum]

Hi,

we have an Exchange 2016 DAG, on two physical servers (2012 R2).
We was waiting for 9.5 U4 to migrate the backup of exchange to veeam because I had read that 9.5 U4 will be exchange DAG aware...but it seems I'm wrong ?

Our database are currently half active on one node, and half active on another node.
with our current backup software, we can backup only passive DB.

Is it possible currenlty with veeam 9.5 U4 to backup exchange node and backup only passive database on each node ?
Backup all DB two times is a important time and more important space loose

I can't find any clear procedure to how configure an exchange DAG with Veeam and what's new with 9.5 U4

thanks for your help

[doum]

Just to fully understand

why did you say that it would be included in 9.5u4 and finally say that even in 9.5u4 Veeam will backup the active and passive databases?

Finally, what is changing in 9.5u4 regarding Exchange DAG ?
I may return the internet, but I don't see any difference with before.

[andyman3000]

R-low if it’s any help I am backing up a single 6 node physical DAG. 3 in DR 3 in Prod. I do not point to the dag via protection group or build a job off of that as it would enumerate all 6 nodes in two locations pointing to one backup repository. Correct me if I’m mistaken but dag is more of a geo cluster not failover cluster.

I have one job for 3 in prod to the prod repo with app processing enabled. And it seems to clear the exchange logs as needed, as the nodes are active in prod.

Second job of 3 in DR to a DR repo.

Each server is backed up and I believe is having redundant copies of the databases.

Second note I am very sorry you have to backup to Store Once CIFS. We bought the Agent licensing thinking Catalyst would work and to our surprise it did not. We were competing for resources for VMware Catalyst backups and it put me through hell for months. I’m happy to report I’ve moved Agent jobs to HPE Apollo windows based repos with DAS.

But aside I did not get this bad of performance in incrementals. I am curious what is your storage settings set at for your StoreOnce CIFS repos as well what is your compression set to on the job. My job compression was set to Dedupe friendly. Repo was set to per backup files, local target, I’ll check the other setting when I get back maybe they will help you.

-Andy

[ianbutton1]

Andy - if this is what works best, perhaps Veeam should change the instructions for IP-based DAG backup in their documentation https://helpcenter.veeam.com/docs/backu ... l?ver=95u4
(Doum - this is the new stuff about DAG backups)
But I followed these instructions and all servers were backed up! 53 hours for a Full (3 servers, 4.5 TB each=14TB), 8 hours for an incremental (support case id 03360737).

I quite agree about the lack of Catalyst support for Agents - it's a dreadful hole in Veeam's coverage, given that the agents should hand them the market for backing up physical Windows & Solaris machines. And I can't even do a scheduled file copy job to a StoreOnce CIFS repo, never mind to a Catalyst store.

-Ian

[doum]

For me it's not a correct management of a DAG.

Having to backup each database two times (or more if you have more that 2 copies) is absolutly inefficient
Even our old, crappy backup system was able to backup only passive database since exchange 2010...

[Dima P.]

Hi folks,

Let me sum up your last questions.

I had read that 9.5 U4 will be exchange DAG aware...but it seems I'm wrong ?

Yes Veeam B&R 9.5 U4 is Exchange aware. You can create managed by backup server job with the type failover cluster which was not possible before

Is it possible currenlty with veeam 9.5 U4 to backup exchange node and backup only passive database on each node ?

Unfortunately that's not possible in current version. Backup is created on the image level without additional exclusions.

Finally, what is changing in 9.5u4 regarding Exchange DAG ?

Before Update 4 the only option for DAG was to create a single backup job for every node and use job chaining to protect entire DAG.

I quite agree about the lack of Catalyst support for Agents - it's a dreadful hole in Veeam's coverage, given that the agents should hand them the market for backing up physical Windows & Solaris machines. And I can't even do a scheduled file copy job to a StoreOnce CIFS repo, never mind to a Catalyst store.

Catalyst support is planned and we are working on it.

Having to backup each database two times (or more if you have more that 2 copies) is absolutly inefficient

I agree that processing can be improved and will add all your votes to this feature request.

[doum]

Ah ok

thank you for these clarifications.
I didn't understand that the difference was the native support of the DAG via the cluster instead of having to create 2 jobs.

I'm currently doing the first full backup of our exchange DAG.
20 hours to backup the first server. It's gonna take 20 hours to backup the second one too.

let's hope that the following incrementals are faster: D

Thank you for the votes on the feature. By the way, do you have a uservoice or something similar?

[Dima P.]

Benoit,

Thanks for the update! We track user requests or votes from the forums and then update our internal tracking system.

[mark49808]

Adding my vote to improve this logic. I dont think i can move off a competitors product until you have better support for backing up 1 copy only of passive databases. Its just too much storage otherwise. Unless you have some other creative solutions?

[Dima P.]

mark49808,

Noted, thank you! Speaking of workarounds: you could exclude entire DAG node from protection group but that would work only if this node contains only database copies. Cheers!

[mark49808]

If i have a single node with only passive copies, can i back up from that?

If not, will veeam dedupe help me at all here? or am i really looking at 4x the data?

[doum]

It's what I finally make.

I move all my database "out" of one server to have only passive copy, and I backup only this server
not perfect but until Veeam has a real support of active/passive copy in exchange DAG, it's better.

Veeam dedup was helpless.
currently our backup is something around 7To, it was 12To when I backed up the two exchange server.

[mark49808]

Thank you doum. Does this properly commit transaction logs, etc on all servers? if so that seems like a viable workaround until this behavior is improved.