Replication job with multiple servers (Application-awareness question)

[Simon_LBC]

For the moment I created two separated replication job, but this is annoying as sometime a job is overlapping the other for some reason, so I would like to setup a single replication job between my production and backup server. Both are Hyper-V of course (2012 R2) and I am using latest Veeam Backup & Replication 9.5 (standard version).

In the replication job I have 8 servers (VMs) where 2x are AD Domain Controllers, who require "Application Awareness" for the replication (and for the backup job) and the 6x others servers are Linux Servers and Windows Server (basic File Server) who don't gain anything from "Application Awareness" I think. Except maybe that I have one of the Windows Machine that have a small running SQL-Express server on it, does it requires Application-awareness in this case too?

I am also currently a bit confused between the Application Awareness in the "Job Setting" tab (who apparently relate to disc destination) and in the job itself (Guest Processing tab). Did one setting impacts on each other? Moreover, I would like to know if I can manage Application-awareness as "per VM" to activate it only on the 2x AD-DC's and not on the 6x other servers? I think that maybe I can do something about that in the "Applications..." buttom from the "Guest Processing" tab maybe, but I am not skilled enough to be sure what all settings are intended for.

Could you please help me creating a great replication job that will also leads to a backup job for the same 8x VMs servers. At some point I think that the settings for the backup job "Application Awareness" would be very similar, this is why I didn't talk about the backup job that much at this point

Thanks a lot folks! I know it's many questions in a single thread, but in fact the point is Application-Awareness you see

[PTide]

Hi,

I am also currently a bit confused between the Application Awareness in the "Job Setting" tab (who apparently relate to disc destination)

I am not sure about which AAIP settings are you talking about, would you be more specific please?

In the replication job I have 8 servers (VMs) where 2x are AD Domain Controllers, who require "Application Awareness" for the replication (and for the backup job) and the 6x others servers are Linux Servers and Windows Server (basic File Server) who don't gain anything from "Application Awareness" I think.

Windows server still do. "Application Awareness" is not for applications only, but also for the filesystem. Without AAIP there will be no integrity preservation of opened files inside Windows VMs.

Except maybe that I have one of the Windows Machine that have a small running SQL-Express server on it, does it requires Application-awareness in this case too?

Unless you are ok with having crash-consistent state of the machine in the backup, I'd suggest to turn AAIP on for that server too.

Moreover, I would like to know if I can manage Application-awareness as "per VM" to activate it only on the 2x AD-DC's and not on the 6x other servers?

Yes, it can be configured per-VM.

Could you please help me creating a great replication job that will also leads to a backup job for the same 8x VMs servers. At some point I think that the settings for the backup job "Application Awareness" would be very similar, this is why I didn't talk about the backup job that much at this point

So, you want to have BOTH a replication job and a backup job for the same set of VMs?


Thanks

[Simon_LBC]

I am not sure about which AAIP settings are you talking about, would you be more specific please?

On the "Job Settings" menu, if I click on "Advanced..." button then in the "Hyper-V" tab, I found a option check for "Enable Hyper-V Guest Conscience". Then in the "Guest Processing" menu of the job, I also have a setting called "Enable Application-aware processing". Which is basically two very similar technical terms. Do I need to enable both settings to use them?

Windows server still do. "Application Awareness" is not for applications only, but also for the filesystem. Without AAIP there will be no integrity preservation of opened files inside Windows VMs.

Unless you are ok with having crash-consistent state of the machine in the backup, I'd suggest to turn AAIP on for that server too.

Oh I see, so you recommend to turn on "Application-awareness" on ALL Windows VMs, no matter what's software running on the VMs (SQL, AD Controler, etc)? And what about the Linux VMs? Does turning on "Application-awareness" on all server's VM impact on replication or backup process? Is it making backup slower? Does it impact on live servers who are in use?

Yes, it can be configured per-VM.

So this settings can be found somewhere in the "Applications..." button? Where I see a list of my VM...?

So, you want to have BOTH a replication job and a backup job for the same set of VMs?

Correct! I have a replication job between my two local servers/storage for a full failover/hardware redundancy, performed every few hours (with a few checkpoint retention) then I also have a backup jobs for the sames VMs/datas to an off-site location (using a Veeam Cloud Connect provider) for disaster recovery purpose and also for data retention/incremental (versioning, sort of... in the case someone trash data accidentally 5 days ago... you know what I mean). By the way, I am planning very soon to have a second replication job (or a second destination?) to an off-site server in a colocation center for "instant" disaster recovery.

[PTide]

If you enable both, then AAIP will be preferred over quiescence whenever possible. You have Linux and Windows in the same job, with both options enabled AAIP will be used for Windows VMs (unless you've specifically disabled AAIP for certain VMs) and quiescence will be used for Linux VMs.

Does turning on "Application-awareness" on all server's VM impact on replication or backup process? Is it making backup slower? Does it impact on live servers who are in use?

AAIP does give any significant impact on backup/replica speed. AAIP for Windows VMs relies on VSS services. To coordinate proper VSS and indexing activities, Veeam deploys a small executable component inside a VM. It is installed only during VSS quiescence procedure and removed immediately after the processing is finished, producing very low impact on VM performance and stability. Regarding Linux VMs that have Linux Integration Services installed - there were reports that some freezes, or crashes might occur on certain systems/kernels. However, in most cases those problems can be resolved with updating Integration Services or kernel.

Where I see a list of my VM...?

That's correct. Select the "Enable application-aware processing" check box, click "Applications", in the displayed list select the VM and click "Edit". To define custom settings for a VM added as a part of the VM container, you must include the VM in the list as a standalone object. To do this, click "Add" and choose a VM whose settings you want to customize. Then select the VM in the list and define the necessary settings.

I have a replication job between my two local servers/storage for a full failover/hardware redundancy, performed every few hours (with a few checkpoint retention) then I also have a backup jobs<...>

Your design is fine as long as the replica job and the backup job do not overlap. Since those are the same servers in both jobs, AAIP settings can be identical for them. You can also configure a remote replica from backup, or even make it vice-versa - backup from replica.

Thanks!

[Simon_LBC]

Very clean and clear instructions! Thank you. Just to conclude, this is how I modified my replication and my backup job :

I put back my 2x domain controllers VMs into the primary (and single now) replication job with the other VMs. I BOTH enabled the "Enable Hyper-V Guest Conscience" in the job setting (left the "Take crash consistent..." option checked (by default)), then I also enabled the "Enable application-aware processing" in the Guest Processing section. Since I understood this feature is not supported by Linux (no VSS), I go into "Applications" and then "Disable AAIP" for the VMs that are Linux based and left default (mandatory VSS) for all the Window's VM.

Then, I did pretty much the same for the backup job...

Thank you again!

[Simon_LBC]

AAIP does give any significant impact on backup/replica speed. AAIP for Windows VMs relies on VSS services. To coordinate proper VSS and indexing activities, Veeam deploys a small executable component inside a VM. It is installed only during VSS quiescence procedure and removed immediately after the processing is finished, producing very low impact on VM performance and stability. Regarding Linux VMs that have Linux Integration Services installed - there were reports that some freezes, or crashes might occur on certain systems/kernels. However, in most cases those problems can be resolved with updating Integration Services or kernel.

Dear PTide,

After 48 hours running my modified replication job with the new settings for AAIP, I noticed that this is actually increasing A LOT the replication time for the job, more precisely for my file server VM's where I have some very large discs to replicate (12 TB disc). Without AAIP, my entire replication job (8 VMs, including this large VM) was taken around 20 minutes to complete, now the same job increased to 1h20 minutes (+1 hour) with the AAIP enabled. When looking at the logs, I see that the reason why it increased by 1 hour, is when Veeam replicate the large discs of this VM. The other VMs are not taking really more time, but this specific VM increased replication time per 1 hour.

It's not that much of a problem, but I now have to change my replication approach, since previously, without AAIP, I was replicating my servers every 2 hours during business hours, but obviously with a 1h20 replication time, I now have to cut my replication cycles per the half and I now replicate every 4 hours.

Is is and expected behavior or not?

Thanks!

[PTide]

Would you share the bottleneck stats, please?

Thanks

[Simon_LBC]

Yes, with great pleasure :

Load: Source 89% > Proxy 7% > Network 19% > Target 8%

FYI, my Veeam server is a VM and is located on the "target" (replica) server and the source is actually located on the "production" server as well. I don't know if it's the best scenario, but I had more ressources on the "replica" server, so I decided to install Veeam server on this server (in a Windows Server 2012 R2 VM).

[PTide]

Hmmm, my first guess that it might VSS snapshot process what is taking so long. I would suggest to contact support team so they can take a look at the logs, as such increase in processing time does not seem normal to me.

P.S. Please also post your case ID if you decide to proceed to the support team.

Thanks!

[Simon_LBC]

I'll surely open a service case since I have an active support contract (of course) and if I can somehow "optimize" my replication and backup tasks, I am surely in! I'll open a case by Monday morning since I am leaving for the weekend now and I'll let you know.

By the way, as per the last 48h, I also have a two or three occurrences where the VSS task fail on some VMs (not all) and then the job need to run for a "retry" for those VMs. But it goes fine by the second try.

[Simon_LBC]

Hmmm, my first guess that it might VSS snapshot process what is taking so long. I would suggest to contact support team so they can take a look at the logs, as such increase in processing time does not seem normal to me.

P.S. Please also post your case ID if you decide to proceed to the support team.

Thanks!

Done! CASE ID is 03329677

Thank you.

[Simon_LBC]

Hi PTide... at this point my ticket brings no solution yet and maybe you can help me out figuring why my replication job now takes so long. Apparently the bottleneck is the source (the production server), but why's that? I suspect that maybe I have some issues with my current Veeam -vs- servers layout? Let me clarify how my IT infrastructure is installed :

- I have two identical servers (HP Proliant), a production one and a replication one (everything is offline on the replica)
- The production server have a high-performance SAN storage directly attached (fiber channel) and replication server have a Entreprise-NAS storage directly connected with iSCSI.
- My Veeam server is a virtual server (W2012R2) and is live on the replication server with plenty of ressources allocated. I decided to put Veeam server on replication server since it's the only live VM on this Hypervisor and it sounds like a good plan to me (is it or not?)

Now I am brainstorming; maybe I can set a "Backup Proxy" on the replication server (physical host) for Off-Host proxying? Since Off-Host proxy is not supported on a virtual server (such as the Veeam server)? Or maybe the key would be to turn the entire Veeam server into a physical machine somehow?

This is pretty much that.

[PTide]

Hi Simon,

<...>at this point my ticket brings no solution yet

Is the ticket closed already, or they are still investigating?

Regarding the layout:

I decided to put Veeam server on replication server since it's the only live VM on this Hypervisor and it sounds like a good plan to me (is it or not?)

It is a good plan - if the production goes down you can control failover process from VBR. On the other hand:

If application-aware image processing is enabled for the job, Veeam Backup & Replication connects to VM guest OSes, deploys runtime processes on VM guest OSes and performs in-guest processing tasks.

That is, it might be the runtime process deployment what takes so long, and this is something what the support team should be able to confirm/disprove.

<...>maybe I can set a "Backup Proxy" on the replication server (physical host) for Off-Host proxying?<...>

You can, but you shouldn't really do that - off-host proxy should reside closer to a data source.

I would also check if manual VSS snapshot creation in the large VM takes long.

Thanks!

[Simon_LBC]

Is the ticket closed already, or they are still investigating?

No it's not close and it's still under investigating. I immediately provided the 10 last days of Veeam logs when opening the ticket, but the technician didn't mention if he found anything. Then he ask me to wait a few more cycles (days) to see if there's change in the situation (??) but since the problem is going on for 7 days now, I don't see the point. This is what I answered to him on our last communication. To be perfectly honest I am totally confused and I was expecting a bit more of a solution after he inspected the logs...

[PTide]

Well, provided that:

1. The problem is visible and reproducible by switching AAIP for that particular VM on/off

2. It's clearly not the first full sync of the replica but just an incremental run

I can't see how waiting might change anything. That is, feel free to escalate the case.

Thanks!

[Simon_LBC]

Hi PTide,

I think that I just found the reason (not the fix) for why my replication is taking so much time... I don't understand how it's related to AAIP (but apparently is it), but I found that replication job is hanging on the "Merging chekpoint" step for the "File Server" VM (the VM that is 13 TB large). Apparently the snapshot process of this VM only take around 20 minutes, but the "MERGING" step is taking at least 45+ minutes itself, what brings me to a total of 1h+ replication delay for this VM!

What misleading me on that, it's because when I look at the basic HTML log inside Veeam GUI, there's no separate informations for creating the VSS shapshot and merging old snapshot delay. But in my understanding the VSS snapshot creation of 20 minutes is pretty much the expected time. But the "MERGE" process for 45+ minutes sound like very long to me.

Do you have any suggestion for reducing the time for the "MERGING snapshot" step of the replication? Did the merging time is in direct relation with how many snapshots I keep in my setting? For the moment I keep 6 checkpoints because I replicate 3 times a day and I want to keep 48h checkpoint in the case of a cryptoware attack or something but maybe it's "overkill" and maybe I can reduce it to 3??? or did the amount of checkpoint doesn't impact on merging processs?

Now I just can't understand why the Veeam technician doesn't immediately point this information when he first inspecting my full server log?

[Simon_LBC]

Hi again PTide, well, I am currently looking at my replication's job this morning and this time the VSS Snapshot creation part is currently for 50+ minutes and only complete to 47% as for the "File Server" VM... so yesterday I was under the impression that the slow delay is only for the "Merging Snapshot" portion of the job, but apparently this delay is variable and the VSS Snapshot creation portion is also sometime VERY long... DAMN! I I thought that I found a lead to the problem, but no...

Any my ticket to the Veeam support is still hanging... I transmitted the last logs very early by yesterday morning , and still no return.

[PTide]

Hi,

There is not much what I can do via this forum at this point, and that's why I asked you to escalate the case "Talk to Manager" button if you're not satisfied with the level of support provided.

Thanks!

[Simon_LBC]

Thanks PTide and I totally understand. I finally got an answer from the Veeam technician a few minutes ago and here's the answer "after checking your logs, the delay of 1h30m for replicating your VM "File Server" appears to be a normal and expected delay and this delay has been consistent during the past 7 days".

What the ***??? He totally dropped the fact that the job increased from 20 minutes to 1h30m (or more) when turning on AAIP.

Yes, I think it's definitely the time for the "talk to your manager" red button!!! (I don't know if I should cry or rage)

[PTide]

I don't know if I should cry or rage.

Niether will makes things better, so I'd suggest to stay polite, yet strict.

Thanks

[Simon_LBC]

Niether will makes things better, so I'd suggest to stay polite, yet strict.

Obviously. No need to worry, I'm a professional