requirements for off-site and connection

[seapro220]

Hello all.
I've been trying to determine the following info, and know that there are real actual users out there with experience, that could potentially help me with the pitfalls associated with off-site/DR replication. So here goes -

I have vmware with 2 hosts and 3 TB of space. My 6.5 VBR backups are ~1.3TB and are configured with weekly and incremental backups. I have a remote site, but not connected in any way at this time. I can create a VPN between to do the replication between, but don't know if the additional overhead associated with have a 'tunnel' up will cause a lot of latency across the wire. My circuits are as follows - 10x10 at the main site and 20/1 at the remote site - so theriodically my bottleneck is at the 'main' site since it's restricted to 10Megs up and during the day that size will have to be reduced due to production work. The goal is to set up a remote/DR site in which I can replicate my data for DR. I've been reading some and it seems that most people will replcate the data local, then take that 'local' device to the remote site and re-establish that connection..due to the large amount of data needed to be 'seeded' the first time. I have several other questions and hope someone has already experienced these questions and can help.
1. i believe i need a remote proxy. If i purchase a relative decent new pc with windows 7 and atleast 4 gigs of storage, will this surfice?
2. i'll be using a Synology NAS box for the remote storage and configuring it with Raid-5.
3. how do i determine how much space i need? Do i take my 'actual space' - of the servers and use that as a real number...and then add my replication/incremental-backup data to it?
4. do i just need to worry about what is in my actual .vbk files?
5. if i dont use a vpn, but rather configure a rule on my 'remote' site firewall to nat/allow all traffic from my 'main' site and direct it to the proxy - will this work, or do i need a vpn to route the traffic over?

so many questions...so many forums ...

thanks -

[Vitaliy S.]

Hi Mark,

1. If you want to store your backups offsite, then yes, you need to create a repository on the DR site. If you want to replicate VMs, then you need to create a backup proxy to accept data coming from the source proxy. I guess you mean 4 gigs of RAM, not storage, right?
3. If you're going use replication jobs, then you will need enough space to provision full copies of our source VMs + additional space for restore points (snapshots). If you're referring to backup files, then as far as I've understood, you're already running backup jobs locally, right? If yes, then you will need to the same amount of space on the DR site.
4. VBK files are full backups of your VMs, VRB and VIB files are incremental restore points, so all files should be protected.
5. VPN connection is definitely preferred. There is an existing topic with the same question, should be useful > Replication over NAT (Internet)

Thanks!

[Yuki]

Synology may not have enough disk throughput to handle the creation of the reverse incremental or synthetic fulls in a timely manner.

We are seeing a similar issue with 12 bay QNAP in RAID5 - reverse incremental are very slow due to the need to read/write data when creating full daily. As a result, your backups may run into the next day when your next backup or replication are scheduled to run. I recommend either going RAID10 or using something more robust altogether.

In either case - before you commit to this setup - use veeam in trial mode to test your configuration and see if it is working for you.

1) remote proxy is recommended but not 100% needed if you have a VPN or point to point link.
2) See above.
3) For FULL - calculate the current total actual data usage. For incremental, its the whatever the data change rate between your backups. So if you do daily and 10 gigs of files change in a day - that's what you should count. Now, there will be compression applied to all backup files, but because some data (video, images, already compressed) is not very compressible - it's hard to make predictions. you may see as much as 70% reduction in size or as little as 10%.
4) ? you need to worry about all files...
5) you want a VPN or a dedicated non-public link such as point-to-point L2 or MPLS. VPN will cause overhead and will eat up a portion of your available throughput, but it is the right way to do this.

[veremin]

Remote proxy is recommended but not 100% needed if you have a VPN or point to point link.

I want to mention that having two proxy scheme for remote replication is indeed a best practice.

Otherwise, there will be no compression, throttling rules, and so on, since all of them in case of remote replication are implemented between two proxies. And it is always preferable to have compression in the aforesaid scenario, so that only compressed traffic is transferred between the sites via WAN.

Hope this helps.
Thanks.

[seapro220]

Thanks for all in responding. As I suspected and begin to review more info, it does become complicated with so many different scenarios..
1. proxy local and storage local -- doing that now.
2. proxy local and storage remote - not feasible due to the limitations of time and WAN links.
3. proxy local AND remote - with replication between. If I seed the 'remote' then potentially the additional proxy server local, connected to the remote proxy server might be sufficient to replicate the vrb files considering my WAN links and time constraints.
4. proxy local, store local and try to do some type of replication of the data. The daily vrb file changes shouldn't be an issue, but would require a 'manual-to-sata- copy of the vbk files weekly. This doesn't sound like fun.

I know that there are multiple programs, like robocopy and others..but what's the 'real' world doing - which is practical???

my full vbk files are about 1.3 TB and daily vrb files are approx 400megs. using a quick lookup on calculations, i should be able to replicate 400 megs in approx 15 minutes - at the outside.

Does this sound correct to all ?

mark

[seapro220]

I forgot to add the following -

i have a 10 Meg up at the Main site, and a 20 Meg site remotely.

[Vitaliy S.]

2. proxy local and storage remote - not feasible due to the limitations of time and WAN links.

Why is it not feasible? You can copy already created backup files on the main site to the DR site, and then map backup job to these files. In this case only incremental backups will be created. Reversed incremental mode seems like a good choice to me, as you will not need to run full backups on weekly, 2 weeks cycle etc.

If I seed the 'remote' then potentially the additional proxy server local, connected to the remote proxy server might be sufficient to replicate the vrb files considering my WAN links and time constraints.

Proxy servers do not replicate backup files, proxy servers are used to retrieve source VMs data and replicate existing VMs as they are on your production servers.

I know that there are multiple programs, like robocopy and others..but what's the 'real' world doing - which is practical???

Please use forum search to see examples from our existing customers > http://forums.veeam.com/search.php?st=0 ... ckup+rsync

Thanks!

[seapro220]

Thanks for the info, I'm currently in a 'hold' status as I haven't received my additional 1512+ unit. I expect to get it on tuesday or wednesday, and will put it in place. Here's the thoughts -

1. build the unit, and seed it with the 'full vbk' files.
2. take the unit offsite and initiate a nightly backup process to replicate the daily changes.
3. i'll monitor that for a bit and see what happens.

concerns or thoughts ..

1. i've been told, by veeam, that in order to set up "replication" that the only way to do this is to have a proxy server, which is at the remote site, configured as the proxy server to do this. I understand this is needed for a 'replication' job or process, but it would require a (a) vmware server at remote site, and (b) almost twice the amount of data as I'd have essentially a 'full' replicate of data being stored, and replicated - off site. Does this sound right, and is anybody using this process? I know with vmware i can accomplish the same processes in a similiar process. I could trim down the # of servers that I want to have in 'hot stand by' by reducing the requirements at the remote site (server side and data storage) and the cost of the licenses might be cheaper... but just don't know.
On the other hand, I've been asked to provide a method of testing a "DR" solution - and this might be the method of validating this process, and getting the 'toys' @ the home side (err - remote site).

My goal initially is to "replicate" (and i know i used this term incorrectly initially) my backups offsite and now it seems that I've quickly (potentially) morphed this project into something different. Doesn't this sound like ALL IT requests??

[Vitaliy S.]

1. i've been told, by veeam, that in order to set up "replication" that the only way to do this is to have a proxy server, which is at the remote site, configured as the proxy server to do this. I understand this is needed for a 'replication' job or process, but it would require a (a) vmware server at remote site, and (b) almost twice the amount of data as I'd have essentially a 'full' replicate of data being stored, and replicated - off site. Does this sound right, and is anybody using this process?

Well...you can have a single proxy on the source site, but in this case you will not be able to use traffic compression between your two sites, which doesn't sound like a good idea assuming you're replicating over the WAN link. There is no need to use a dedicated server for the backup proxy, if you have any existing VM on the remote site, just install Veeam proxy role on it and use it in replication jobs.

[seapro220]

Hello all.

I first must a apologize for any confusion earlier in talking about replication vs remote-copy or synchronizing as I might have been confusing the two. Initially I've been looking at 'backing up via replication', but am now concentrating on copying (replicating) my data externally so that I have a 'copy' if something goes wrong at the main site
As I mentioned earlier, I was in 'hold' mode due to the equipment shipment. I have now configured the remote 'nas' box and 'seeded' it with the initial run - which consisted of "all" the data from my primary site (vbk and vib) files.

I haven't had a chance to either use/play with RoboCopy or any of the other 'copy' programs yet as I stumbled upon another tool - RichCopy - and am trying it out to see how well it works. The reasoning to try and use the RichCopy program is because I can send multiple 'copies' across the wire at the same time - and it's a GUI for me to monitor (initial curiosity and self-validation). The initial process has worked, but not as great as I'd thought as my timing seemed to run longer that I'd expected - and I'd originally selected 'all' the folders for 'replication/copy' and not just the .vib file (selected vbk files accidentally). I restarted the process and it completed. I'm sure that there is a method of using RoboCopy (or some other tool) which can 'push' more that 1 file at a time, so I need to look into those other products and do some 'speed' testing.

I've also created a remote share, created another repository (at the remote site) and taken a current, but not currently being used VM (30 gigs) and am testing it as a remote-backup. My 'pipe' is pretty full currently as production is under way - but I'm curious to see how it'll preform 'across-the-wire'. The initial test won't really result in any 'hard' info as my internet-pipe is active now, and the initial backup is a 'full' (never written anything there via this job before) ..but never the less - I'm curious.

More as I learn..

[veremin]

I'm sure that there is a method of using RoboCopy (or some other tool) which can 'push' more that 1 file at a time, so I need to look into those other products and do some 'speed' testing.

Windows 7 and Windows Server 2008 R2 include an updated version of Robocopy that includes a multi-threaded copy feature. Usage of this option in most cases results in faster or significantly faster file copy operations.

In general, this feature enables multiple files to be copied simultaneously. By default, it is set to 8, however, you can specify any number from 1 to 128.

Code: Select all

robocopy D:\Test C:\Temp\folder /S /MT:10

For instance, in the above-mentioned example, Robocopy will copy the content of the D:\Test folder, including subfolders, with 10 threads. Furthermore, please be aware that the progress monitors don’t appear to be accurate in multi-threaded mode. It starts copying 10 files at once, but only shows you the progress of 1 of them.

More information regarding Robocopy parameters can be found here.

Hope this helps.
Thanks.

[seapro220]

Ok - thanks.

A few follow up inquires. I've make a simple batch program, which at this time is 'supposed' to do the following - copy, list what's being copied to a log file, and exclude .vbk files. on 2 of the batch commands i'm copying 'all' from local to remote, and the 3rd option is using the exception. Here's the batch program - please take a look and let me know what I'm missing - or suggest other corrections.

c:
robocopy s: t:\avebackups /m /s /e /l /z /ts /fp /mt:10 /log+:c:\dailyinc.txt --- copy all from local to remote\directory ---
robocopy s:\vmbackups t:\vmbackups /m /s /e /z /l /ts /fp /mt:10 /log+:c:\dailyinc.txt /xf *.vbk /s --- copy all from local\dir to remote\dir and exclude .vbk files ---
robocopy r: t: /l /m /z /ts /fp /mt:10 /log+:c:\dailyinc.txt --- copy all from local to remote ---

A couple of follow up questions -

1. does robocopy copy only the 'newer' files? Meaning, if i have 3 files local, and generate 2 additional files between batch runs - will robocopy see that there are 3 files already there, and only copy the 'newer' or 2 files?
2. what switches can i use to keep the directories in sync? As Veeam changes/removes files during it's backup process (incremental and truncating backups) I'll keep X-# of files local. If I dont utilize some type of switch command, then I could potentially have 10 (good) files locally, but have 11 or more files at the remote site - and have to compare/delete the 'old' files at the remote site.

[veremin]

Syntax looks good for me.

As for your questions:

1. AFAIK, this’s exactly how Robocopy's supposed to work.
2. Additional /mir switch should be used in this situation. This mode keeps trees in sync by optionally deleting files out of the destination that are no longer present in the source.

Please, also be aware that robocopy method isn’t considered to the best practice in case of you using reversed incremental mode, since, from robocopy perspective, each job run produces completely new .vbk along with a bunch of corresponding .vrbs, thus, these newly-created files will be copied each time your batch runs.

So, it’s recommended to utilize such functionality only if your backup mode is set to forward incremental.

Hope this helps.
Thanks.

[seapro220]

Thanks again - Status update ..

No, my backups are not configured as Reverse Incremental. I got caught in that loop before..

2 jobs still running. I'm guessing because I originally didn't have all the data in all the directories. Will monitor it today and see what happens.

I don't like the fact that I can't 'see' what's going on, unless i occasionally 'read' the log file to see what files are/have been copied so far.
I had seen the /mir switch, but hadn't initiated it yet. It seems like the 'answer' to keeping all files 'in-sync' by providing a mirror copy. Perhaps whenever the jobs are finished, I'll change my batch jobs to reflect this.
No matter whether I initiate the job w/o any other switches - i still can't get the /MT:xx to work. I keep getting a "invalid parameter /MT:10" error message. I have to find a way around this as I have to many files to copy and can't have the 'whole' thing caught on some large files and not be able to 'push' more than 8 files @ a time.

thanks again ..

[veremin]

I’m wondering what version of Windows you’re using, since not until Windows 7/Windows 2008 R2 has the multithread functionality been introduced in Robocopy.

Thanks.

[seapro220]

Sorry, I was off-line for a couple of days while I sorted other things out.
I have switched to a windows 2008 server that is running the 'newer' robocopy program. All seems well, but I only be able to utilize/copy/mir 1 file at a time. I have specified the /MT:12 which should indicate the program to copy 12 files at a time-as opposed to the default of 8 - but it's not working. I know this because I have purposively not specified a 'log' file and using the 'on-screen' display to watch what is actually happening (12 copies/pushes/sync at a time). Any thoughts or suggestions?

[veremin]

Your assumption about multithread functionality not working properly is based on what you’re seeing in the “on-screen” display, isn’t?

But, as I’ve said previously:

Furthermore, please be aware that the progress monitors don’t appear to be accurate in multi-threaded mode. It starts copying 10 files at once, but only shows you the progress of 1 of them.

Or am I missing something obvious? If so, don’t hesitate to let me know about it.

Hope this helps.
Thanks.

[seapro220]

Hmm.
I remember that statement mentioned earlier, but upon reviewing the logs (maybe not too closely earlier) it only seemed to be copying 1 at a time. I'll review the logs again and see if I'm wrong. I'd expected, within the log file, to see 'all' the files being copied at a time and notice some type of progression (%) of what was being done during this process. like ..
file 1 copying...
file 2 copying...
the same X10 - showing all 10 as 'active' at this time.

On another job, with the same parameters the job 'completed' and has been 'waiting' for the changes. I remember that you could set the timeline to check, on one of the switches but wonder if it is any type of resource 'hog' on the system if it checks ever 5 minutes, or 20 minutes. I'd think not (probably a quick check at the source side for x# of changes - sees nothing and goes back to sleep) but just wanted to see what others are doing. I know it depends on the # of change files, the timing of how quickly you want the jobs 'offsite' etc ...
Any suggestions on how I can 'trim' the log file to perhaps reflect only the files that are copied - whenever the job runs? my first initial 'log' file was pretty much inclusive (it's also set to append instead of create) but still...is was 32M in size.

[veremin]

In the way I see it, neither log nor progress monitor reflects correctly multi-thread mode. So, if you’re still a little bit anxious about it, the only option you have is just testing robocopy-utility with and without specified parameter and seeing whether there is tangible difference between them.

For the purpose of complete explicitness, it might be worth testing it with a big amount of files of considerable size and multithread option specified to something like 32, for instance.

There are also existing tests regarding it conducted by enthusiasts. Results can be found in this article or in this presentation.

Hope this helps.
Thanks.

[seapro220]

Ok. So I've been running this for a few days and for some reason - still seem to have some trouble 'reading' what's going on. For instance, i initiated the job around 6:18 pm. this is what the log looks like .. Where in here am I being told that I'm having "12" files copied at a time ?? It would appear to me that, at most - I'm getting 2 files copied @ once - instead of "up to" 12 files.

-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows
-------------------------------------------------------------------------------

Started : Wed Feb 06 18:33:03 2013

Source : s:\vmbackups\
Dest : t:\vmbackups\

Files : *.*

Exc Files : *.vbk

Options : *.* /TS /FP /S /E /COPY:DAT /PURGE /MIR /Z /M /MON:5 /MT:12 /R:1000000 /W:30

------------------------------------------------------------------------------

1 s:\vmbackups\
12 s:\vmbackups\AD Boxes\
14 s:\vmbackups\All Production Servers\
13 s:\vmbackups\exchange2010\
8 s:\vmbackups\pgp - linux server\
*EXTRA File 63.9 m 2013/02/03 01:36:34 t:\vmbackups\pgp - linux server\pgp - linux server2013-01-30T020058.vrb
36 s:\vmbackups\reporting-task-for-ssrs-report-Daily-San-Disk-performance-info_13\
19 s:\vmbackups\virtual center server\
Newer 58729 2013/02/06 00:21:26 s:\vmbackups\AD Boxes\AD Boxes.vbm
100%
New File 797.8 m 2013/02/06 00:21:15 s:\vmbackups\AD Boxes\AD Boxes2013-02-05T190051.vib 0% 1% 2% ... 7%

[seapro220]

Forgot to add. In this specific job, ALL files are large, ranging from 2Gb to 18Gb in size each.

[veremin]

As I’ve said previously, it seems that none of the variants, be it log or progress monitor, is noticeable for correct reflection of multi-thread mode.

So, you’ll have to either take as a matter of course that it’s supposed to work this way, or conduct your own independent test in order to verify whether everything is ok.

Probably, in order to get more subtle answer regarding this tool you should ask its direct developers (Microsoft).

In addition, there is a tool called RichCopy, which is a free computer utility program developed by Ken Tamaru of Microsoft Corporation to copy file directories. It provides a complete graphical user interface (GUI), whereas Robocopy is a command-line utility. In the way I see it, it should provide you with some visual representation of performing tasks.

Hope this helps.
Thanks.

[yizhar]

HI.

I suggest that you reconsider a different approach:

1. At the main site - continue to have regular local backups.

2. At the remote site - install and configure a veeam repository on a windows computer.
Create another job to backup to the remote site.

Some main differences between that approach vs trying to copy backup files:

* The repository (running on remote site) understands the internals of the backup files, and can do things like reverse incrementals or synthetic full by itself, without the need to copy the large bkf files over wan again and again.

* The repository is maintaining the backup files and responsible for their validity.
vs file copy utilities (robocopy/rsync etc) which might corrupt the file if the transfer isn't completed for any reason - for example during the transfer you don't have a reliable backup at remote site.

So - I suggest that you first try this approach instead of robocopy.

However one disadvantage of remote backup -
The backup window will need to be longer, and the production vm will be in snapshot state until its backup is completed,
so you should try to avoid running disk write intensive (such as sql/exchange maintanance) during that time.

Yizhar

[seapro220]

So - I've been working/testing this for a couple of weeks and here's what I've found -

vbk transfer is only possible via 'snail mail' (copy to external drive and transfer to remote site). I knew this from the start - so not surprised.
robocopy seems to be the 'best' method so far and is keeping the files/directories "in-sync".
vrb files are transferring, for the most part, during off-hour timelines. Sometimes it runs into the day, but with QOS in place - the users aren't complaining yet.

As I have several Veeam jobs (non-Microsoft, veeam server, sql, exchange, etc) I get failures during the weekly transform process due to the robocopy job hasn't completed yet. The 'retry' options are working, but now I have 2 vbk files, instead of 1. If there isn't any issue with this (I presume Veeam is smart enough to know what failed, and restart w/o issues) then I'm not too vulnerable. My issue currently is with 1 large job (for me anyway) that backs up 18 servers at a time, and writes 1 large vbk file (~720 Gbs). The vrb files for this 'incremental/forward' job is averaging around 13Gb and transferring - even at full throttle - is time consuming.
I probably need to break this job up to smaller sizes - just not sure because if I have 3 10Gb files to transfer, as opposed to 1x30Gb file - the time remains the same.

I'd thought of creating/using the suggestion above, by Yizhar, but don’t know if it'll work for me given my limited internet connection (10x10 at main site, 20x1 at remote) and timeline available. I presume that the reference to disk-write-intensive processes is referring to the initial 'snap' process and not during the 'whole' back up timeline. ???

[yizhar]

HI.

1. Regarding the direct backup over wan (instead of local backup and then sync), the best way to know the answer to "how long would it take", is to try on a sample VM.
I am backuping up several VMs (about 800gb total VM size before dedup/compress) to a remote site via 10mbps line, and it takes about 20 hours for initial backup but few hours for following CBT runs.

2. I suggest that you look at the new product "Veeam Cloud Backup" which is now available, and see if this can be a better solution to sync the files from local main site to remote shared folder - instead of robocopy.
Few links:
http://www.veeam.com/cloud-backup-vmware-hyper-v.html
http://rickrbyrne.wordpress.com/2013/02 ... p-edition/
http://forums.veeam.com/viewtopic.php?f=2&t=15499

Yizhar

[seapro220]

Hi.
Just an update so far -
I have successfully broken up the large job into 2 smaller ones and changed my sql job from a daily to a weekly one. In regards to using Cloud storage for my backups at this time, it doesn't seem to make sense to me as I need to have a copy of my data local - as well as one which is stored 'in the cloud' somewhere. I might not be using a specific 'named' cloud storage, but am essentially accomplishing the same process. I'm in search now for the components to build myself a 'lab' at my remote site and once it is completed, I'll probably look into setting up a veeam server over there and letting the 2 servers proxy and manage the data between them. Having a 'lab' off-site will also allow me the abilities of building/testing/off-loading some non-intensive (monitoring and management servers) at my remote site as well as reducing the work load of my VBR server. For me, having the data local (local restores of either full, or incremental file changes) - as well as working copies off-site is the best method of preserving the integrity of our data and keeping in-line with the 'best practices' of Veeam's product support.

[yizhar]

HI.

When I mentioned "veeam cloud edition" above, it was related to your private backup over wan to your own location.
The cloud edition has support for simple file share, and you can try and see if it works better then a robocopy script for syncing backups from primary to your own DR site.
I haven't tested it myself.

Yizhar