Veeam 6.5 ... Almost there, Proxy problems ...

[lars@norstat.no]

First of all i would like to say that Veeam 6.5 was a huge improvement over the last version and i'm very happy that you implemented the function we asked for in what you call intelligent load balancing.

This is working great and i'm processing my Exchange server at about 6 GB/s now which is somewhat acceptable. The bottle neck is now the target SAN and that is where i want to be ....

But now that the replication issue is done away with and i can continuously replicate about 60 vm's i turned my attention to backup and a problem came up. First i will explain the setup.

I have the main Veeam Server and 4 Proxies, all of them running 8 cores and 32 GB of RAM and they handle all the replication using HOT ADD mode. Source and Target proxy are set to Automatic.

I also have a Backup server connected to the SAN via Fibre Channel and this server have it's own disks internal to save Backup jobs. Now the problem is that when i activate the Backup server it will take on replication jobs too and fetch data via SAN and then transferring via network to one of the Proxies that will HOT ADD the data down to target. Even though i have switched of the Failback to network option. (The backup server to not have access to the target SAN)

Now, i do not want the Backup server to be involved in replicating and vice versa so i set the backup job to use only the backup server, but on the replication proxies i still have to set automatic because if i choose the servers i want to use it reverts to the old way of doing it where one proxy server reads the data and the other one writes it and everything is transferred on the LAN between them.

Setting proxy manually for each job is not a good way of doing it anyway because what if i add a proxy later, then i would have to edit 60 jobs .... No, what i would like is the ability to set a proxy to only handle replication jobs, backup jobs or both. Both would be the default of course and the setting does not have to be easy to find if this confuses people, but in a setup like mine I then have the ability to control the flow of data perfectly and i could finally buy our Veeam Licenses

[dellock6]

Hi Lars,
you can simply remove the proxy role from your backup server, and let it act only as a central console. Simply go to the "backup infrastructure" area, select the proxy menu and remove proxy installed in the veeam server itself. So you would always use the other proxies for every job.
About your design, 8 cores and 32 gb ram is an overkill for proxies, remember you need 2 vcpu per cuncurrent job, and I've seen in my tests you need no more than 4 gb of ram to execute two cuncurrent job per proxy. 32 Gb of unneeded ram is a problem since VMware had an overhead in memory consumption caused by all that ram it has to support.
Also, sure you are talking about 6 "GB" per second? How can it be?

Luca.

[Vitaliy S.]

Hi Lars,

Since direct SAN mode is the most efficient way to retrieve VM data, your proxy server deployed on the backup server receives the highest priority. Currently, the workaround suggested by Luca seems to be the only possible way to make sure your replication jobs are processed be HotAdd proxy servers. You may also want to configure your default backup proxy server to work in the network mode only, should help.

Anyway, thank you for your feature request and for your positive feedback on 6.5 release.

Thanks!

[lars@norstat.no]

I really miss the possibility to draw something here even with ASCII but the editor collapses all spaces so that does not work. There is always a lot of confusion but i will try to explain.

First 6GB/s is the Processing rate, NOT the transfer rate of raw data.

I have 60 jobs running concurrently and divided by 5 servers that is 12 jobs each so i do not run 2 jobs on each server although i guess that was only an example.

There are 5 servers proccessing replication jobs Veeam Server itself + 4 dedicated virtual Proxies.

I have one Pysical Backup server running backup jobs or i want that server to run only backup jobs. Removing proxy function from the backup server will not help because then it can't run any jobs ?1?

As long as i disable the backup proxy everything is fine and the Veeam server + 4 proxies handle the replication, but as soon as the Backup server is activated it starts to handle replication jobs aswell and i want it to just sit and wait for a backup job to come by.

Also, SAN mode is the most efficient way to collect data. But it is not the most efficient way to replicate if that server has to transfer the data via the network to another proxy that in turn have to write the data to disk. So the whole logic behind the priority is flawed.

[tsightler]

As long as i disable the backup proxy everything is fine and the Veeam server + 4 proxies handle the replication, but as soon as the Backup server is activated it starts to handle replication jobs aswell and i want it to just sit and wait for a backup job to come by.

This is the part where I get confused. You say you have a "backup server" so you make it sound like this physical backup server is a completely separate Veeam server, but if that was the case it wouldn't even be aware of the replication jobs, only it's own jobs.

I think what you mean here is that you have a "physical backup proxy server", which is still associated with the same Veeam server that is also handling the replication. Is that correct?

Also, SAN mode is the most efficient way to collect data. But it is not the most efficient way to replicate if that server has to transfer the data via the network to another proxy that in turn have to write the data to disk. So the whole logic behind the priority is flawed.

It's not really "flawed", there's simply not enough information available to make a better decision. The proxy selection algorithm has no real information regarding the connectivity between two sites, we can't really just assume that the network will be slow. What if it's local replication within a datacenter over 10GbE? In that case, using SAN mode to retrieve the data would provide the best performance.

Still, I think I understand your issue. You want your SAN mode proxy to be used for backups only, because your replications are already limited by the target so using a "slot" on the direct SAN proxy for replication is a waste of resources. Unfortunately, you can only do this by manually selecting the proxies at this point in time. As far as your concern about adding a proxy in the future and not wanting to edit the jobs, it might be possible to address this with Powershell.

Honestly though, if I never wanted to mix my replication and backup jobs/proxies, I'd probably just install two separate Veeam servers, each with their own resources and monitor them with EM.

[lars@norstat.no]

This is the part where I get confused. You say you have a "backup server" so you make it sound like this physical backup server is a completely separate Veeam server, but if that was the case it wouldn't even be aware of the replication jobs, only it's own jobs.

I think what you mean here is that you have a "physical backup proxy server", which is still associated with the same Veeam server that is also handling the replication. Is that correct?

Yes

It's not really "flawed", there's simply not enough information available to make a better decision. The proxy selection algorithm has no real information regarding the connectivity between two sites, we can't really just assume that the network will be slow. What if it's local replication within a datacenter over 10GbE? In that case, using SAN mode to retrieve the data would provide the best performance.

Exactly, there is not enough information and that's what humans are for, but if you don't give me the possibility to choose, then i can't correct the situation.

Still, I think I understand your issue. You want your SAN mode proxy to be used for backups only, because your replications are already limited by the target so using a "slot" on the direct SAN proxy for replication is a waste of resources. Unfortunately, you can only do this by manually selecting the proxies at this point in time. As far as your concern about adding a proxy in the future and not wanting to edit the jobs, it might be possible to address this with Powershell.

Nope, i can't manually choose proxies because then the intelligent load balancing won't work, if not that would at least be a work around. I think that ILB not working when choosing proxies manually should be classified as a bug ...

Honestly though, if I never wanted to mix my replication and backup jobs/proxies, I'd probably just install two separate Veeam servers, each with their own resources and monitor them with EM.

Of course i could do that and there probably wouldn't be a licensing issue with that, but the problem is that then the Backup server wouldn't be aware of the replication going on and be able to stop them for taking backup and that in turn would lead to some snapshot and CBT issues ....

[tsightler]

Nope, i can't manually choose proxies because then the intelligent load balancing won't work, if not that would at least be a work around. I think that ILB not working when choosing proxies manually should be classified as a bug ...

How does this break the load balancing? Based on what you're saying you want specific proxies used only for replication jobs, and specific ones used only for backup jobs. You can easily pick a subset of proxies for a job and load balancing still occurs between this subset of proxies.

Of course i could do that and there probably wouldn't be a licensing issue with that, but the problem is that then the Backup server wouldn't be aware of the replication going on and be able to stop them for taking backup and that in turn would lead to some snapshot and CBT issues ....

Actually, the second server should still be aware as we won't take a second snapshot if an existing Veeam server already has a snapshot on the VM, and even so, there are no conflicts with CBT.

[lars@norstat.no]

How does this break the load balancing? Based on what you're saying you want specific proxies used only for replication jobs, and specific ones used only for backup jobs. You can easily pick a subset of proxies for a job and load balancing still occurs between this subset of proxies.

So the new 6.5 version has something called intelligent load balancing that we and surly others requested. This allows a replication job to stay on one proxy server. In other words the proxy mounts both source disk and target disk to itself and transfer data between them. Much faster if you have only 4 GB network and 8 GB fibre channel.

I have 5 servers as you know and they are all in one location. This means that for one server to read the data, transfer it via the network to another proxy who then writes the data is just ... well stupid. And it works great too. Now if i choose to set the proxies manually for all the replication jobs then some jobs will stay on one server, but most will choose one source proxy and then another target proxy. (This is completely random) in other words the old and inefficient way of doing it.

Actually, the second server should still be aware as we won't take a second snapshot if an existing Veeam server already has a snapshot on the VM, and even so, there are no conflicts with CBT.

Ok, so with the servers on the same Veeam server it works as follows:

At the time a backup job is scheduled to start it ask's if there are replication jobs running and if there are then it will wait for 15 hours or something like that until the server is available if not the backup will time out (not an issue). When it becomes available it will pause the replication of that server or rather the replication won't start because there is a backup job running against that server. When the backup server finishes the replication continues.

Now how will that work with two Veeam servers ? How often will the backup server poll to check if the server is available for backup and won't the replication just start again before the backup has a chance to start ? Remember that i'm running continuous replication 24/7.

[tsightler]

I have 5 servers as you know and they are all in one location. This means that for one server to read the data, transfer it via the network to another proxy who then writes the data is just ... well stupid. And it works great too. Now if i choose to set the proxies manually for all the replication jobs then some jobs will stay on one server, but most will choose one source proxy and then another target proxy. (This is completely random) in other words the old and inefficient way of doing it.

I would agree that this is a bug and you should open a support case. The logic for selecting the same source and target proxy for replication should not change just because you selected to lock replication to a subset of proxies. I personally have not seen this behavior in my own testing. In my environment my replications are all local and the system always picks the same proxy for source and target if they both have access to the same data source

Now how will that work with two Veeam servers ? How often will the backup server poll to check if the server is available for backup and won't the replication just start again before the backup has a chance to start ? Remember that i'm running continuous replication 24/7.

With two servers if a replication is running on one server, the backup will run as well, each will take and manage it's own snapshot. They may overlap for a short period of time, but this shouldn't be too much of an issue if your are running continuous replication as those cycles are likely to be quite short. The only issue would be if you are using application aware processing with both the replication job and the backup job as there could be some conflicts, but I'm guessing if you are using continuous replication you are not using AAP on the replication jobs.

[lars@norstat.no]

With two servers if a replication is running on one server, the backup will run as well, each will take and manage it's own snapshot. They may overlap for a short period of time, but this shouldn't be too much of an issue if your are running continuous replication as those cycles are likely to be quite short. The only issue would be if you are using application aware processing with both the replication job and the backup job as there could be some conflicts, but I'm guessing if you are using continuous replication you are not using AAP on the replication jobs.

But didn't you just say ....

Actually, the second server should still be aware as we won't take a second snapshot if an existing Veeam server already has a snapshot on the VM, and even so, there are no conflicts with CBT.

And i'm running AAP on replication jobs to ensure that replicas are consistent, or should i ? Have i misunderstood this function ?

[tsightler]

Sorry, that was a little bit of a misspeak. With two separate Veeam servers the job second server will still take a second snapshot, but there won't be any conflicts between the two snapshots or with CBT as it's designed to handle this case.

AAP on the other hand is not aware of multiple processes so there might be a conflict if the sessions happen to overlap and you might get job warnings or even failures, based on how the job is configured, but even without AAP your replicas would be "crash consistent". My experience is that most customers that are running near-continuous replication aren't using AAP because they do not have an infrastructure that can perform the freeze/unfreeze cycle consistently without causing impact to transactional workloads like SQL/Exchange. If you're not having any such problems with it then there's no reason not to use it as it sounds like your environment has plenty of fast I/O available. Most environments that I run into are I/O starved, even though many don't even know it.

Historically most CDP replication solutions have not offered options for consistent replication, although a few products do offer the concept of consistency points, i.e. points in time where the CDP source system is brought to consistency on a schedule. This point is then marked on the replica target as a consistency point, but you can still revert the replica to points that are not consistent.

In the end, I agree with your assessment that using a single Veeam server would be ideal for your use case, and that there is a small bug in the selection algorithm when manually selecting proxies (I've yet to be able to reproduce that in my test lab though). I encourage you to open a support case and provide logs for this issue in the hoping of getting that fixed. I was just trying to offer some possible options to try in the interim. I think using two servers would be a valid option with some consideration around the AAP. There might be other options based on how your jobs are configured (for example, a Powershell script that pauses the replication job long enough to run a backup job on the other server).

[lars@norstat.no]

So is there no registry hack or config file that i can change to make the backup server only touch backup jobs or disable network mode all together so that the backup server can't retrieve data with direct SAN and transfer to another proxy ?

[Vitaliy S.]

No, there is no such registry key, at least I haven't heard about it.

[tsightler]

And honestly, the only likely way to find out for sure is to open a support case and let our support teams review the behavior you are seeing.