Backup and Replication to DR Site Design Proposal

[CaptainFred]

Hi,

I am starting a new project to backup and replicate our vSphere environment using B&R v7. I wondered if I could run my proposed setup by you all to get any feedback on improvements/alterations I could make or any issues that I may have and need to address. We don't use B&R in production yet but use another product for vSphere to tape backups. Sorry for the length of this but thought it best to give you all the details up front and thanks in advance for any help provided.


Current/Production Environment:
-------------------------------

- vSphere 5.0 Enterprise (upgrading to 5.1 soon) with 6 ESXi hosts on primary site
- vCenter on Windows 2008 R2 Server VM using SQL for DB
- Dell iSCSI 10GBE SAN storage, around 30TB capacity
- Approx 125 Windows VMs some thin, some thick, B&R tells me a total of 18.5TB (will be increasing when we virtualise more soon)
- Windows 2003/2008 R2 mixed mode domain (not replicating any primary site DCs)
- Applications include 3rd Party, File Data, Exchange 2010 DAG (2 VMs) not using RDMs, SharePoint, Ecommerce (IIS), SQL 2005/2008, etc
- We also host many websites, in a multiple DMZ environment protected by firewalls (separated by vmnic's/vswitches in the vSphere environment)
- Servers all use DHCP reservations, IP range 172.18.x.x, approx 20 server VLANs
- Physical Windows 2008 R2 Backup Server with VMFS Datastores directly presented to it for SAN mode backups
- 100Mbit Internet Link
- Primary Site Backup Storage - Dell iSCSI 10GBE SAN - 23 x 3TB SATA drives in RAID 6 (1 x Hot Spare) around 57TB usable. Connected to Backup Server as NTFS volume


DR Site:
--------

- vSphere 5.1 Essentials Plus with 1 ESXi host
- vCenter on Windows 2008 R2 (or 2012) Server VM using SQL for DB
- Deploy Veeam Backup Proxy on vCenter or other VM
- 50Mbit Internet Link (will be connected to primary site by site to site VPN using Cisco firewalls)
- DR Storage - Dell iSCSI 10GBE SAN - 23 x 3TB SATA drives in RAID 6 (1 x Hot Spare) around 57TB usable
- Will be creating a DR Active Directory Site with a VM DC connected to the production domain that will also run DHCP, IP range 172.20.x.x


Business Requirements:
----------------------

1. There is no definite RPO or RTO, but at the moment we do nightly tape backups so this will be a major improvement on that.

2. At least twice a year some members of the business want to visit the DR site (where there are PCs/desks etc they can use) and test all applications and services without affecting the production replication. After the test any changes they have made can be lost (it's only a test). Unfortunately the automated tests that B&R can do are not enough for internal audit, they want users to actually test applications and services at the DR site.


My Proposal:
------------

1. Backup to Primary Site Backup Storage (NTFS) every 4 hours approx, stored as Veeam backup files.

2. Replicate from Primary Site Backup Storage (as seed) to DR Storage (VMFS Datastore) every 4 hours approx (after previous job I guess). I would prefer this to be replication I think so the servers actually exist at the DR site as VMs (VMDKs). I suppose this a debatable point because I can't use WAN acceleration if I use replication at present.

3. Re-create the Primary Site Network VLANs, DHCP scopes etc, but with 172.20.x.x range instead of 172.18.x.x. Configure DR site DHCP server identically to Primary Site but with different IP range to avoid conflicts etc.

4. Use SureReplica for business requirement 2, for when users come to test, which would be separated from production replicated VMs. One problem with this is giving users access to the SureReplica VMs through the proxy and also I would need a DHCP server in that Virtual Lab otherwise none of these VMs will get IPs.


Questions/Discussion:
---------------------

I have already setup the primary site backup storage, created a full backup and some incremental backups (all ok so far) and also on the same site connected the DR storage to a test ESXi host and setup a replication job to replicate from the primary site backup storage (as seed) to the DR storage. What I didn't expect is that it is using the seed backup repository but also then connecting to the production VMs to get the latest snapshot and also replicating that, but not updating the repository with that latest backup. I'm not certain I'm doing this the right way so really hoping someone is going to tell me whether I am or whether I should be doing it slightly differently? For info, that first full backup took that 18.5TB to around 6.8TB and took 22 hours.

The replication is seems slow, maybe it's because it's the first full run, not sure? I've only connected the DR storage to my test ESXi host by one 1GBit connection (and the test ESXi host is connected to the LAN by just one 1Gbit connection), I thought that around 100MB/s transfer would be ok and of course I didn't expect it to be snapshoting the production VMs as part of the replication. I also thought that when it's in the DR site it won't be getting anywhere near 100MB/s but actually it's currently running at about 40-50MB/s, still way in excess of the ~5MB/s bandwidth I will have to the DR site when the storage is installed and everything configured on the DR site. Is this what you would expect? Anyway to stop it taking the snapshot from the production VM when replicating?

I suppose there is many ways I could do this all which may not be right or wrong. I just thought that maybe I should divide up the storage at DR into VMFS and NTFS and use the WAN accelerated backup to the NTFS storage and then replicate the VMs from that into actual VMs for testing/failover etc? Not how I would prefer I think, especially as it will use more storage all the time and I won't be able to show management all our replica VMs sitting there on the DR vSphere environment just waiting to be used.

Would point 4 of my proposal above work?

Any thoughts?

Thanks for reading all of this and if I have missed anything or if anything isn't clear let me know.

Simon

[Vitaliy S.]

Hello Simon,

Unfortunately the automated tests that B&R can do are not enough for internal audit, they want users to actually test applications and services at the DR site.

If users can automate application verification via scrips, then you can use these scripts as a part of the SureBackup job as well.

What I didn't expect is that it is using the seed backup repository but also then connecting to the production VMs to get the latest snapshot and also replicating that, but not updating the repository with that latest backup. I'm not certain I'm doing this the right way so really hoping someone is going to tell me whether I am or whether I should be doing it slightly differently? For info, that first full backup took that 18.5TB to around 6.8TB and took 22 hours.

But why it should update the backup repository you were using as a seed with the latest VM data? Seeding is just a one time operation that basically restores the VM from the backup and then maps production VM to the restored one to update it with the recent VM changes. This operation should be used when you do not want to transfer the entire VM over the WAN link. Once initial job pass is done, you do not need this seed anymore.

The replication is seems slow, maybe it's because it's the first full run, not sure? I've only connected the DR storage to my test ESXi host by one 1GBit connection (and the test ESXi host is connected to the LAN by just one 1Gbit connection), I thought that around 100MB/s transfer would be ok and of course I didn't expect it to be snapshoting the production VMs as part of the replication. I also thought that when it's in the DR site it won't be getting anywhere near 100MB/s but actually it's currently running at about 40-50MB/s, still way in excess of the ~5MB/s bandwidth I will have to the DR site when the storage is installed and everything configured on the DR site. Is this what you would expect? Anyway to stop it taking the snapshot from the production VM when replicating?

After you do replica mapping/seeding, Veeam replication jobs has to calculate all the VM digests to detect what blocks have changed after seeding. Can you please tell me where you store replica metadata? Is it on the source site repository? BTW, can you confirm that you're using proxy servers on both sites?

...and of course I didn't expect it to be snapshoting the production VMs as part of the replication.... Anyway to stop it taking the snapshot from the production VM when replicating?

Not sure I understand the question... If you want to backup/replicate a VM you need to create a snapshot first and then start transferring VM data.

I just thought that maybe I should divide up the storage at DR into VMFS and NTFS and use the WAN accelerated backup to the NTFS storage and then replicate the VMs from that into actual VMs for testing/failover etc? Not how I would prefer I think, especially as it will use more storage all the time and I won't be able to show management all our replica VMs sitting there on the DR vSphere environment just waiting to be used.

Would point 4 of my proposal above work?

Well...certainly using backup copy jobs with WAN acceleration enabled to transfer your backups to the offsite location is a recommended way, but I don't completely get what do you mean by "replicate the VMs from that into actual VMs for testing/failover etc". Do you want to run a backup restore job from these files? If this is the case, then using replication jobs is definitely a preferable way to have "ready to run" and verified copies of production VMs on the DR site.

Hope this helps!

[CaptainFred]

Thanks for your reply.

Unfortunately the automated tests that B&R can do are not enough for internal audit, they want users to actually test applications and services at the DR site.

If users can automate application verification via scrips, then you can use these scripts as a part of the SureBackup job as well.

Yes I know but that’s what they want to do. They also have to test things like phones and business processes which would be slightly different because there is less people who would be utilised in the DR situation. So would my plan of using the Virtual Lab (SureReplica) work (remember I need to run the test and keep production replication ongoing)?

What I didn't expect is that it is using the seed backup repository but also then connecting to the production VMs to get the latest snapshot and also replicating that, but not updating the repository with that latest backup. I'm not certain I'm doing this the right way so really hoping someone is going to tell me whether I am or whether I should be doing it slightly differently? For info, that first full backup took that 18.5TB to around 6.8TB and took 22 hours.

But why it should update the backup repository you were using as a seed with the latest VM data? Seeding is just a one time operation that basically restores the VM from the backup and then maps production VM to the restored one to update it with the recent VM changes. This operation should be used when you do not want to transfer the entire VM over the WAN link. Once initial job pass is done, you do not need this seed anymore.

I see. I think I have misunderstood the point of seeding. I thought the point was so that only one backup is required to be run from the production storage, so it’s not impacted by 2 jobs.

Can I ask how you would recommend I deploy in my situation? Am I actually going to need to have 2 jobs running every 4 hours (one backup job one replication job)? Will they conflict if they happen to get to the same VM at the same point in time?

The replication is seems slow, maybe it's because it's the first full run, not sure? I've only connected the DR storage to my test ESXi host by one 1GBit connection (and the test ESXi host is connected to the LAN by just one 1Gbit connection), I thought that around 100MB/s transfer would be ok and of course I didn't expect it to be snapshoting the production VMs as part of the replication. I also thought that when it's in the DR site it won't be getting anywhere near 100MB/s but actually it's currently running at about 40-50MB/s, still way in excess of the ~5MB/s bandwidth I will have to the DR site when the storage is installed and everything configured on the DR site. Is this what you would expect? Anyway to stop it taking the snapshot from the production VM when replicating?

After you do replica mapping/seeding, Veeam replication jobs has to calculate all the VM digests to detect what blocks have changed after seeding. Can you please tell me where you store replica metadata? Is it on the source site repository? BTW, can you confirm that you're using proxy servers on both sites?

Yes the “Repository for replica metadata” is in the source site backup repository. Is this correct?

I haven’t deployed the DR proxy server yet because I’m preparing the replica VMs on the DR storage locally before I physically take it to the DR site. There is a 1Gbit connection between all the components at the moment, it should easily be able to utilise 100% of a single 1Gbit connection.

...and of course I didn't expect it to be snapshoting the production VMs as part of the replication.... Anyway to stop it taking the snapshot from the production VM when replicating?

Not sure I understand the question... If you want to backup/replicate a VM you need to create a snapshot first and then start transferring VM data.

This is kind of related to my misunderstanding of seeding, I think, see above.

I just thought that maybe I should divide up the storage at DR into VMFS and NTFS and use the WAN accelerated backup to the NTFS storage and then replicate the VMs from that into actual VMs for testing/failover etc? Not how I would prefer I think, especially as it will use more storage all the time and I won't be able to show management all our replica VMs sitting there on the DR vSphere environment just waiting to be used.
Would point 4 of my proposal above work?

Well...certainly using backup copy jobs with WAN acceleration enabled to transfer your backups to the offsite location is a recommended way, but I don't completely get what do you mean by "replicate the VMs from that into actual VMs for testing/failover etc". Do you want to run a backup restore job from these files? If this is the case, then using replication jobs is definitely a preferable way to have "ready to run" and verified copies of production VMs on the DR site.

Ok I am correct in saying that this is the recommended approach?:

Primary Site contains Production Storage and Production running VMs
Use Backup Job – to backup VMs to Backup Repository (NTFS volume) located on Primary Site
Use Replication Job – to Replicate VMs to ESXi/vSphere (VMFS datastore) located on DR Site


Extra questions:

A. Can the Production and DR Site Servers that are running B&R and/or Proxies be physical or virtual or a combination? Eg. Physical on Production Site, Virtual on DR Site?

B. Can the Production and DR Site Servers that are running B&R and/or Proxies be running on different operating systems?

C. When I select a VM and the choose the option to “Restore guest files (windows)” I can restore files and if it’s an Exchange server I can restore email out of databases, but if it’s SQL server the option to restore database(s) doesn’t appear, do I have to do something different if the server contains SQL? Note I have not selected “Enable application aware image processing” or “Enable guest file system indexing” because it says it’s optional - do I need to enable one of these for SQL?

Thanks

[Vitaliy S.]

So would my plan of using the Virtual Lab (SureReplica) work (remember I need to run the test and keep production replication ongoing)?

Yes, that would definitely work.

Can I ask how you would recommend I deploy in my situation? Am I actually going to need to have 2 jobs running every 4 hours (one backup job one replication job)? Will they conflict if they happen to get to the same VM at the same point in time?

No issues with two jobs processing the same VM. Please read this topic for further information > Double backup

Yes the “Repository for replica metadata” is in the source site backup repository. Is this correct?

Yes.

I haven’t deployed the DR proxy server yet because I’m preparing the replica VMs on the DR storage locally before I physically take it to the DR site. There is a 1Gbit connection between all the components at the moment, it should easily be able to utilise 100% of a single 1Gbit connection.

Depends on how your network is already utilized, but yes, it should take the entire bandwidth. If not, then please look through this thread to make your jobs use the entire bandwidth > VB replication not using the entire WAN bandwidth

When you move your replicas offiste, make sure you deploy the secondary proxy server. This is essential when you're replication over the WAN link.

Ok I am correct in saying that this is the recommended approach?:

Primary Site contains Production Storage and Production running VMs
Use Backup Job – to backup VMs to Backup Repository (NTFS volume) located on Primary Site
Use Replication Job – to Replicate VMs to ESXi/vSphere (VMFS datastore) located on DR Site

Yes, looks good to me.

Extra questions:

A. Can the Production and DR Site Servers that are running B&R and/or Proxies be physical or virtual or a combination? Eg. Physical on Production Site, Virtual on DR Site?

B. Can the Production and DR Site Servers that are running B&R and/or Proxies be running on different operating systems?

C. When I select a VM and the choose the option to “Restore guest files (windows)” I can restore files and if it’s an Exchange server I can restore email out of databases, but if it’s SQL server the option to restore database(s) doesn’t appear, do I have to do something different if the server contains SQL? Note I have not selected “Enable application aware image processing” or “Enable guest file system indexing” because it says it’s optional - do I need to enable one of these for SQL?

A. Yes, no limitations with that, but I would prefer to have virtual proxy server to leverage hotadd capability during restore and backup process.
B. Yes.
C. To restore SQL Server items, you should be using SQL AIR wizard > VMware : [FAQ] FREQUENTLY ASKED QUESTIONS

Also I would strongly recommend using application-aware image processing for all Windows VMs, see this topic for more details > >>> READ FIRST : [FAQ] FREQUENTLY ASKED QUESTIONS <<<

Thanks!

[CaptainFred]

Yes, that would definitely work.

Ok so how do I deal with access to the Virtual Lab for the DR test users without causing conflicts bearing in mind that all the servers are DHCP and that DHCP runs on the DC which I'm not replicating and not only that the servers in the Virtual Lab will need a domain to connect to so somehow I will need a DC in there as well? Sounds very complicated - I don't see how it will work. Am I explaining this properly?

Can I ask how you would recommend I deploy in my situation? Am I actually going to need to have 2 jobs running every 4 hours (one backup job one replication job)? Will they conflict if they happen to get to the same VM at the same point in time?

No issues with two jobs processing the same VM. Please read this topic for further information > Double backup

According to that thread, they would conflict if the replication job happened to get to the same VM as the backup job was working with at the time? Specifying different times won't necessarily help because both jobs will be working through a list of over 100 VMs. This is why I thought you could backup to local backup repository and then replicate from that offsite so that only ONE job is running on the production storage and ONE backup per VM not two.

Extra questions:
A. Can the Production and DR Site Servers that are running B&R and/or Proxies be physical or virtual or a combination? Eg. Physical on Production Site, Virtual on DR Site?
B. Can the Production and DR Site Servers that are running B&R and/or Proxies be running on different operating systems?
C. When I select a VM and the choose the option to “Restore guest files (windows)” I can restore files and if it’s an Exchange server I can restore email out of databases, but if it’s SQL server the option to restore database(s) doesn’t appear, do I have to do something different if the server contains SQL? Note I have not selected “Enable application aware image processing” or “Enable guest file system indexing” because it says it’s optional - do I need to enable one of these for SQL?

A. Yes, no limitations with that, but I would prefer to have virtual proxy server to leverage hotadd capability during restore and backup process.
B. Yes.
C. To restore SQL Server items, you should be using SQL AIR wizard > VMware : [FAQ] FREQUENTLY ASKED QUESTIONS

Also I would strongly recommend using application-aware image processing for all Windows VMs, see this topic for more details > >>> READ FIRST : [FAQ] FREQUENTLY ASKED QUESTIONS <<<

Thanks!

A. Not sure I understand - better to have virtual proxy server on the production site or DR site? If my physical backup server on the production site can perform SAN mode backups surely this is better than hotadd? Or is it not?


New Question:

D. When I select a datastore or cluster to backup, it builds the list of VMs to backup and goes through them in what seems to be a random order. How can I get it to go through them alphabetically (or even another order I specify)?

Thanks for your help

[Vitaliy S.]

Ok so how do I deal with access to the Virtual Lab for the DR test users without causing conflicts bearing in mind that all the servers are DHCP and that DHCP runs on the DC which I'm not replicating and not only that the servers in the Virtual Lab will need a domain to connect to so somehow I will need a DC in there as well? Sounds very complicated - I don't see how it will work. Am I explaining this properly?

If you're going to verify an Exchange server, then of course you will need a DC and a DNS VMs in the application group up and running prior starting the Exchange verification job. Basically, you need to have all VMs replicated that verified application depends on.

Here is more info on the network routing, please look it through > VMware : [FAQ] FREQUENTLY ASKED QUESTIONS

Also if you take a look at our User Guide (page 268) you will see how network is configured for virtual labs.

According to that thread, they would conflict if the replication job happened to get to the same VM as the backup job was working with at the time? Specifying different times won't necessarily help because both jobs will be working through a list of over 100 VMs. This is why I thought you could backup to local backup repository and then replicate from that offsite so that only ONE job is running on the production storage and ONE backup per VM not two.

Yes, there will be a conflict, but no jobs will fail. The VM will stand in a queue in the second job. Once first job is complete, your second job will start processing the selected VM, so there is no need to bother with job scheduling.

A. Not sure I understand - better to have virtual proxy server on the production site or DR site? If my physical backup server on the production site can perform SAN mode backups surely this is better than hotadd? Or is it not?

If you have a proxy server acting in the SAN mode on the main site, then of course this is preferable. I was referring to the DR site, where hotadd proxy can be used to write VM data to the datastore directly.

D. When I select a datastore or cluster to backup, it builds the list of VMs to backup and goes through them in what seems to be a random order. How can I get it to go through them alphabetically (or even another order I specify)?

Do you want them to be processed or displayed in the alphabetical order? Btw are you adding VMs explicitly or in the container? The recommended way to add VMs to the job is to use VM folders/datastores/hosts etc. In this case all new VMs added to these folders (to your VI) will be picked up by the backup job automatically.

[CaptainFred]

If you're going to verify an Exchange server, then of course you will need a DC and a DNS VMs in the application group up and running prior starting the Exchange verification job. Basically, you need to have all VMs replicated that verified application depends on.

Here is more info on the network routing, please look it through > VMware : [FAQ] FREQUENTLY ASKED QUESTIONS

Also if you take a look at our User Guide (page 268) you will see how network is configured for virtual labs.

Ah ok that throws a spanner in my plans! I thought (possibly incorrectly) that it can cause problems snapshoting a domain controller and it’s not supported by MS. Also lots of issues can occur like USN rollback etc - http://support.microsoft.com/kb/875495

I have been reading the manual and FAQs but tbh I find this area quite complicated.

Yes, there will be a conflict, but no jobs will fail. The VM will stand in a queue in the second job. Once first job is complete, your second job will start processing the selected VM, so there is no need to bother with job scheduling.

Ok so it won’t error? It will simply wait until the VM is available?

If you have a proxy server acting in the SAN mode on the main site, then of course this is preferable. I was referring to the DR site, where hotadd proxy can be used to write VM data to the datastore directly.

Yes, I’m planning this:

Production Site – Physical Backup Server and Replication Source Proxy – Windows Server 2008 R2 SP1 - SAN mode connectivity (direct access to vSphere datastores)
DR Site – Virtual Backup Server and Replication Destination Proxy – Windows Server 2012

1. Is this the best design?

2. I could give the DR Site VM direct access to the SAN vSphere datastores or wouldn’t this work in the same way as it does for a physical server or would it make no difference in terms of performance compared to hotadd?

3. Referring to the “hotadd proxy” and the explanation on page 28 of the manual – does this method actually attach the VMDK of a VM to the Backup VM so it can copy the data straight out of the VMDK? Even though it’s still attached to the real VM?

4. Even so, the data still has to be transferred across the network so is this why it’s not as good as a physical server with direct access to the SAN/vSphere datastores?

Do you want them to be processed or displayed in the alphabetical order? Btw are you adding VMs explicitly or in the container? The recommended way to add VMs to the job is to use VM folders/datastores/hosts etc. In this case all new VMs added to these folders (to your VI) will be picked up by the backup job automatically.

I would like them to be processed in alphabetical order. I have added the cluster. Same thing happens when I select folders/datastores/etc. Is this expected or a bug?


F. I know it’s only just been released but I thought I might as well ask – can I use Windows Server 2012 R2 as the VM on the DR site?


Thanks again for your help, much appreciated.

[Vitaliy S.]

Ah ok that throws a spanner in my plans! I thought (possibly incorrectly) that it can cause problems snapshoting a domain controller and it’s not supported by MS.

There are no problems with snapshotting DC VMs, just make sure you use application-aware image processing option. Here is an existing topic for additional details > AD Controller System State Issue

Ok so it won’t error? It will simply wait until the VM is available

Correct.

Production Site – Physical Backup Server and Replication Source Proxy – Windows Server 2008 R2 SP1 - SAN mode connectivity (direct access to vSphere datastores)
DR Site – Virtual Backup Server and Replication Destination Proxy – Windows Server 2012

1. Is this the best design?

2. I could give the DR Site VM direct access to the SAN vSphere datastores or wouldn’t this work in the same way as it does for a physical server or would it make no difference in terms of performance compared to hotadd?

3. Referring to the “hotadd proxy” and the explanation on page 28 of the manual – does this method actually attach the VMDK of a VM to the Backup VM so it can copy the data straight out of the VMDK? Even though it’s still attached to the real VM?

4. Even so, the data still has to be transferred across the network so is this why it’s not as good as a physical server with direct access to the SAN/vSphere datastores?

1. Yes, you may want to additionally deploy virtual proxy servers on the main site for the redundancy reasons.
2. You can not use SAN mode to write/restore VM data to the datastore, SAN mode can only be used to retrieve VMs, so there is no sense in configuring direct SAN mode on your DR site.
3. Correct. The reason why you can attach virtual disk to another VM is a VM snapshot, it "unlocks" virtual disks, so Veeam job can back these disks up.
4. Replication jobs transfer VM data over the network, that's correct and that is why you need to have proxy servers deployed on both sites to minimize the amount of traffic sent over the WAN link.

I would like them to be processed in alphabetical order. I have added the cluster. Same thing happens when I select folders/datastores/etc. Is this expected or a bug?

Yes, this is expected, VMs are processed in the order returned by vCenter Server. Can you please elaborate on why do you want to process VMs in alphabetical order?

F. I know it’s only just been released but I thought I might as well ask – can I use Windows Server 2012 R2 ?

Please see this topic for the answer > Hyper-V 2012 R2 support

[CaptainFred]

Ok when you say no problems snapshotting DC VMs, does this include Windows 2003 / 2008 / 2008 R2 DCs? We have a mixed mode 2003/2008 R2 domain at the present because we still have some 2003 DCs (we will be removing them eventually, hopefully soon ish).

Is it recommended then to use the application-aware image processing option but not so necessary to use the guest file system indexing option? I don't really understand the point of the indexing option if it's optional and it indexes the VM anyway. I'm guessing that enabling the application-aware image processing option will slow the backups down a bit because of the extra processing?

I suppose there is no definite reason why I would like to process VMs in alphabetical order (or a specified order) but it seems more logical to me. Is there no way around this then? Not sure why vCenter returns them in such an odd order, it's not even the order they were created... at least I don't think it is!

Thanks again for your help, sorry for the delay in getting back to this thread.

[foggy]

Ok when you say no problems snapshotting DC VMs, does this include Windows 2003 / 2008 / 2008 R2 DCs?

Yes, all the mentioned versions are supported. There is another huge existing topic regarding domain controller recovery, worth considering for further reading.

Is it recommended then to use the application-aware image processing option but not so necessary to use the guest file system indexing option? I don't really understand the point of the indexing option if it's optional and it indexes the VM anyway.

Guest OS indexing is required if you need to search for guest OS files in the selected VM backup at a specific restore point using Veeam Backup Enterprise Manager (1-click FLR functionality).

I'm guessing that enabling the application-aware image processing option will slow the backups down a bit because of the extra processing?

There is some processing overhead required to perfrom proper application quiescence, however this will allow for automatic DC recovery without any manual intervention or additional AD administration tasks at all (as described in the topic referred above).

[CaptainFred]

I've been working through this further and come across a few other issues/questions:

1) Performing Storage vMotion of a VM resets CBT so it backs up the whole VM again. Is this true? Seems like a big problem because we occasionally move VMs about to optimise storage and performance, some of which are several TBs so then it would have to try and send several TBs across the WAN link? This is bad enough if it was just the backup but sending that amount of data even with compression and dedupe across our 50Mbit WAN link is not going to happen! Maybe I'm missing something here. I realise this is more a VMware issue.

2) When I setup my replication job and use a backup as a replica seed is this meant to be just a one off when first setting it up to help not copy as much data across the slow WAN link or a permanent going forward solution?

3) Regarding the design, I saw this: http://helpcenter.veeam.com/backup/70/v ... eding.html recently so should I actually be having 2 backup respositories? like this...

Use Backup Job – to backup VMs to Primary Site Backup Repository (NTFS volume)
Use Backup Copy Job (using WAN acceleration) - to copy VMs to DR Site Backup Repository (NTFS Volume)
Use Replication Job – to Replicate VMs to DR Site ESXi/vSphere (VMFS Datastore) using DR Site Backup Repository as Seed

Seems like a lot of backups/copies of the same data, I'd have at least 2 copies of the VMs at DR, one in a VBK and one as the VMDK itself.

Thanks

[veremin]

1. Yep, Storage Migration results in CBT reset.

2. Replica seeindg is more about one-time operation that helps to reduce traffic sent trough the bandwidth during the initial replication cycle.

3. Such scenario guarantees required backup data redundancy. Though, it's up to you whether to follow these best practices or not.

Thanks.

[CaptainFred]

1. Ah so if I moved a 2TB VM between datastores at the production site it would then have to replicate the whole 2TB (depending on compression and dedupe of course)? No way around this/workaround? Surely this must be causing people problems, especially on even slower WAN links.

2. Ok but in my design above would this actually mean better performance and mean I could replicate more data because I'd be able to use WAN acceleration rather than just replication and proxies at each end?

3. Is this more of a recommendation, ie. not essential? I suppose this means I could have more rollback points in the VBK and less snapshot rollback points on the DR VMs themselves?

Thanks for the help

[veremin]

1. Nope, the reset CBT means that whole VM image will be read by VB&R; however, only changes will be transferred.

2. Not sure whether I completely understand you, but the backup copy jobs and replication ones are completely different and are used for different purposes. Also, the replica seeding procedure described above applies only to replication jobs and not to backup copy jobs. Likewise, WAN Accelerators are only utilized by backup copy jobs.

3. Yep, it’s recommended way to guarantee backup data redundancy. Though, as mentioned above, it’s up to you to decide whether to follow it or not.

Thanks.

[CaptainFred]

Ok thanks.

Regarding the backup report email that is sent after the job is complete, am I right in saying that the “read” amount is the amount of changed data in the VM and the “transferred” amount is the amount it’s actually transferred to backup AFTER compression and dedupe?

I've just attempted a replication job of a VM from vSphere 5.0.0. I've selected my source and destination proxies but got the message "hotadd isn't supported for this type of disk". The destination proxy is installed on Windows Server 2012, could this be because technically hotadd isn't supported in vSphere 5.0.0 on Windows Server 2012? I can raise a support case if necessary about this but thought it might be simple issue if it's just the OS support. If I turn off the failback to NBD mode I get this error (couldn't find anything in the KB or forum for this and it doesn't have any snapshots):

Error: Client error: An existing connection was forcibly closed by the remote host Exception from server: Failed to open VDDK disk [[Datastore 1] SERVER1_R/SERVER1.vmdk] ( is read-only mode - [false] ) Failed to open VMDK. Logon attempt with parameters [VC/ESX: [192.168.1.1];Port: 902;Login: [****];VMX Spec: [moref=79];Snapshot mor: [79-snapshot-1];Transports: [hotadd];Read Only: [false]] failed because of the following errors: Failed to process [srcReplicateVddkDiskContent] command.

[foggy]

Regarding the backup report email that is sent after the job is complete, am I right in saying that the “read” amount is the amount of changed data in the VM and the “transferred” amount is the amount it’s actually transferred to backup AFTER compression and dedupe?

There is a good topic describing all the counters used in the job stats and report, worth reviewing.

I've just attempted a replication job of a VM from vSphere 5.0.0. I've selected my source and destination proxies but got the message "hotadd isn't supported for this type of disk". The destination proxy is installed on Windows Server 2012, could this be because technically hotadd isn't supported in vSphere 5.0.0 on Windows Server 2012?

Hotadd is supported on Windows 2012, however there are several limitations of using hotadd mode outlined in the following KB article: http://www.veeam.com/KB1054, please review whether some of these applies to your case.

[CaptainFred]

Yes, I've read a few threads like that one and just wanted to double check what I'd read is correct. Reading that thread I believe I am?

I've also read that KB article before but I wondered whether it needs an update because

vSphere 5.0 and later releases have a block size of 1 MB only.

as it states here: http://kb.vmware.com/selfservice/micros ... Id=1003565 - I'm not so sure that it means that I can't use hotadd in vSphere 5.0 or later with disks 256GB or larger.

None of the other points apply. I can't find anyone mentioning my particular error which is worrying. Is it because I am replicating from vSphere 5.0.0 to vSphere 5.1.0? Works if I don't use the proxies. I'm also doing vCenter to host because I haven't finished installing the DR vCenter yet, from what I've read that should still work?

[foggy]

Yes, I've read a few threads like that one and just wanted to double check what I'd read is correct. Reading that thread I believe I am?

You're correct about transferred. As for the read data, then it is the amount of data that has been actually read from source VM disk (blocks reported by CBT minus overlapping snapshot blocks, zero-data blocks, blocks of swap files, etc., so basically less than the amount of changes).

None of the other points apply. I can't find anyone mentioning my particular error which is worrying. Is it because I am replicating from vSphere 5.0.0 to vSphere 5.1.0? Works if I don't use the proxies. I'm also doing vCenter to host because I haven't finished installing the DR vCenter yet, from what I've read that should still work?

Replication to the higher version is supported, as well as replication to the standalone host, so I suggest contacting support with this.

[CaptainFred]

So if it says it read 60GB for example, it's actually read more because thats the figure after ignoring overlapping blocks, zero data blocks etc?

Well it seems the problem with hotadd was partly that my backup username has a dash in it, I've created a new username and it's working for most VMs but not all, so will raise a support case now about it.

Is this still valid: http://www.slideshare.net/VeeamSoftware ... and-tricks it mentions a lot of registry keys but not where to create them and there is a couple of Veeam keys in the registry?

Does the DownloadStreamsNumber (DWORD) improve TCP streams for backup or replication or both?

I ran a create synthetic full the other day, I had a full backup which was about 7TB, and 9 incremental backups of around 300-400GB each. It took 30 hours to create it! Would this be expected? I didn't have "transform previous full backup chains into rollbacks" enabled but would like to!

The backup server is physical and high spec, 12 x 2Ghz cores, 32GB RAM, backup storage is iSCSI 10GBE 24 x 3TB SATA drives in RAID 6. The new full backup file it resulted in is 10TB which surprised me a bit too. Will it automatically delete the 7TB full backup file depending on the rentetion/restore points?

Thanks

[foggy]

So if it says it read 60GB for example, it's actually read more because thats the figure after ignoring overlapping blocks, zero data blocks etc?

No, it's the actual amount of data read (no need to read blocks that do not need to be copied).

Is this still valid: http://www.slideshare.net/VeeamSoftware ... and-tricks it mentions a lot of registry keys but not where to create them and there is a couple of Veeam keys in the registry?

Most of the registry keys should be created under the standard HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication hive. Not sure which keys are you referring to exactly.

Does the DownloadStreamsNumber (DWORD) improve TCP streams for backup or replication or both?

Both.

I ran a create synthetic full the other day, I had a full backup which was about 7TB, and 9 incremental backups of around 300-400GB each. It took 30 hours to create it! Would this be expected? I didn't have "transform previous full backup chains into rollbacks" enabled but would like to!

With such a large full I would say that this is expected (synthetic backup activity depends mostly on the target storage performance).

The backup server is physical and high spec, 12 x 2Ghz cores, 32GB RAM, backup storage is iSCSI 10GBE 24 x 3TB SATA drives in RAID 6. The new full backup file it resulted in is 10TB which surprised me a bit too. Will it automatically delete the 7TB full backup file depending on the rentetion/restore points?

Previous full will be removed according to the specified retention settings.

[CaptainFred]

I've just been watching the webinar "SureReplica -- put your replicas to work" and about half way through the question regarding what happens to replication when a SureReplica job is running. It started breaking up so I'm not 100% sure but I think they said that the SureReplica job stops when the replication job runs, so this means I wouldn't be able to use SureReplica for DR site testing by actual users because they would be half way through testing and production replication would start and stop all the SureReplica VMs! Is this what would happen?

This is a business requirement as I explained in my first post in this thread.

If, instead I used backup copy to copy my backup to DR, could I use a SureBackup job to bring up the VMs for testing and while in the background the backup copy job from production was still running, without affecting the SureBackup VMs running at the DR site?

Please can you clarify this as it's very important I get this right.
Thanks

[Vitaliy S.]

Yes, SureBackup lab will shut down when either backup or replication job starts. Please find more information on this in our Help Center.

Please be aware that backup copy jobs cannot be used as a source for SureBackup jobs, since the main use case is to verify your backups first and only then send them to the offsite location for archival purposes. And I'm not sure how this would help, as backup files will be modified by the backup copy job anyway, so it will lead to the same behavior that was described above.

I would suggest disabling replication jobs for the time you need to verify all applications manually, and then enable them back.

Hope this helps!

[CaptainFred]

Oh. So bearing my business requirement in mind, what would you suggest? Could I use a backup copy to copy them to DR then a replication job to copy them again to DR using the backup copy as seed thus meaning I have 2 copies at DR, one for testing with, one to keep up to date from production? (Plus I would have best of both worlds, in backup format and VMDK format).

Or if I replicate the VMs to DR, stop replication and power them on manually for the testing, then power them off after testing has been finished then start replication again will replication continue like before and overwrite any changes made during the testing? ie. bringing them back in sync to production state?

[Vitaliy S.]

The first case would still require stopping either a backup copy job or a replication job before running any tests, but it is, indeed, recommended to have 1 backup onsite, stand-by replicas and archival backup copies in the DR location.

Anyway, getting back to your business requirement, I would suggest one of the following scenarios:

1. Verify your onsite backups (don't forget to temporary disable backup jobs), while running replication jobs. This will protect you from any DR situation that might happen during verification process.

2. Verify your VM replicas, by running all the tests manually. The changes that you will make will discarded and VM replicas will again be in-sync with your production VMs. See this topic for additional details > VMware : [FAQ] FREQUENTLY ASKED QUESTIONS

[CaptainFred]

Sorry I don't understand. How will either of these options meet my business requirement?

Are you saying that my statement below is correct?

Or if I replicate the VMs to DR, stop replication and power them on manually for the testing, then power them off after testing has been finished then start replication again will replication continue like before and overwrite any changes made during the testing? ie. bringing them back in sync to production state?

It won't have to replicate the whole VM to get it back in sync will it?

[Vitaliy S.]

If you need to perform extended testing and these tests overlap with regular job schedule, then you will need to disable your jobs until all your tests are complete. My first advice will give you both protection and time for testing (since you will be testing separate backup files/VM replicas).

It won't have to replicate the whole VM to get it back in sync will it?

Correct, only VM changes will be transmitted, not the entire VM.

[CaptainFred]

This won't meet the business requirement because while testing is occurring I won't have replication to DR running so in the unlikely event of a disaster during a test the replica VMs will be out of date.

To confirm then, any changes that are made to a replica VM will be overwritten/discarded the next time replication runs?

[Vitaliy S.]

This won't meet the business requirement because while testing is occurring I won't have replication to DR running so in the unlikely event of a disaster during a test the replica VMs will be out of date.

Running SureBackup verification and regular backup/replication jobs targeted to the verified VMs at the same time is not possible. There should be a pause between these activities.

To confirm then, any changes that are made to a replica VM will be overwritten/discarded the next time replication runs?

Yes, see our sticky F.A.Q. for further details:

Q: I want to test replica failover. Can I simply power on any replica restore point with vSphere Client?
A: Yes. Because each restore point is a snapshot, its content will remain intact, because all disk changes inflicted by the running VM will go into the new snapshot file that host automatically creates to protect the selected snapshot's state (standard VMware snapshot tree functionality). For the same reason, Veeam replication job will not be affected either. Testing replicas with v6 is really a breeze - thanks to the new way of storing replica restore points.

[CaptainFred]

I now have my main replication job running (on site to start with) of 59 production VMs of various sites, totaling about 14.6TB. I've selected my local backup repository as the repository for replica metadata.

1) Is the "repository for replica metadata" for storing replica data before it's sent to the replica destination?
2) My repository for replica metadata has 223 files, 147 foldes and is 8.10GB at present. Should I be backing this up to tape or offsite occasionally?
3) if the repository for replica metadata was lost for whatever reason, would it just re-create it? Would this affect the amount of data sent at the next replication job run?
4) Will the repository for replica metadata keep growing or does it size manage itself? shrink/cleanup files after the job has completed etc?

Thanks

[Vitaliy S.]

1) Is the "repository for replica metadata" for storing replica data before it's sent to the replica destination?

Not really, this repository doesn't store any replicated VM data, it is used for storing VM replicas descriptive information.

2) My repository for replica metadata has 223 files, 147 foldes and is 8.10GB at present. Should I be backing this up to tape or offsite occasionally?

No, you do not need that. If you happen to lose this repository, VM replicas metadata will be rebuild on the next job run.

3) if the repository for replica metadata was lost for whatever reason, would it just re-create it? Would this affect the amount of data sent at the next replication job run?

Yes, it will be re-created. The amount of traffic sent will not be large, though digest recalculation (metadata rebuild) operation will take some time.

4) Will the repository for replica metadata keep growing or does it size manage itself? shrink/cleanup files after the job has completed etc?

It depends on the retention policy you choose. Please see this topic for additional info > Repository for Replica Metadata 20GB