Seeding v7 backup copy

[krismason]

Hi v.Eremin

Thanks for the reply... I think your misunderstanding - its hard to explain without showing you

The KB states that only a VBK can exist to seed the first run of the backup copy - so there cannot be any incrementals - if the first seed didnt complete (which it didnt only 1TB of 10TB did) then it will only copy whats changed for whats been seeded, the issue was the my backup copy interval kicked in before the full seed completed (due to poor scheduling my mistake) and the first run taking a long time to complete as the digests need to sync for WAN acceleration (it then began to run a regular unseeded backup copy over my WAN as the next interval had been started which was evident in transfer speed and amount of data being transferred)

My point is to try and help others not encounter the issue perhaps the KB should be updated? I have now changed my backup copy interval to 100 DAYS so that it can run for as long as it needs for the first time (which is extreme) but after it has been seeded I can change to a more practical/realistic schedule - does this make sense?

[veremin]

There might be indeed sort of confusion between us.

AFAIK, it should be possible to map a backup copy job to a chain with increments. The only requirement is that this chain should be created by a backup copy, not backup job.

As to your second question, I think your case is similar to the situation described in the adjacent thread. Interval expired and broke digest synchronization. Thus, global cache became the primary source of deduplication. That in its turn led to increased amount of transferred data.

Thanks.

[krismason]

Hi Vladimir,

Yes it is indeed possible to map a copy job with increments (but not successfully seed it) e.g. when you have large data set (say a 4TB VM) over a 20MB Link. As per the KB1856 mentioned on how to seed there is no mention of what/how to time this seed.

The way I have been successful for my VM's that are sub 500GB over my current comms infrastructure was to seed the backup copy(VM by VM - 1 at a time) (disabled the job once succesfull), relocate the copy to remote site and create new repository, change source, set WAN accelerator for source and target and run the job.

I have successfully backup copied a bunch of VM's using this method however they all complete within 24 hours of me starting the copy at the remote site and when I have started the copy say I start it at 2:00PM I set the backup copy interval to Copy every 100 day at 1:59PM (so that the job has 23:59 minutes to complete) considering the digests take some time on the first run and link speed issues this covers most of my VM's except the bid ones. This also gave me flexibility that if one job failed for 1 VM I only had issues with that seed (I didnt affect a vbm/vbk seed for my whole enviroment)

The issue arises on the setting on the copy job "Copy Every (days, hours, minutes) at x:xx am/pm". It seems that the Days/Hours/Minutes is completly ignored and the job runs everyday (when enabled) at the time interval specified you would assume if you set it to run every 100 days at 1:59PM that it would run once every 100 days however it seems to run once every day - can you please explain this to me?

This has inedvertanly caused my issue which is that one running the first job takes more than 23:59 to run.. hope this makes sense?

Thanks

Kris

[veremin]

The issue arises on the setting on the copy job "Copy Every (days, hours, minutes) at x:xx am/pm". It seems that the Days/Hours/Minutes is completly ignored and the job runs everyday (when enabled) at the time interval specified you would assume if you set it to run every 100 days at 1:59PM that it would run once every 100 days however it seems to run once every day - can you please explain this to me?

Backup copy job has continuous schedule, which means it always runs. As to copy interval, it identifies how often a backup copy job should look for a new restore point in source location and how often it should synthetically create a new restore point in target location. Thanks.

[foggy]

...if you set it to run every 100 days at 1:59PM that it would run once every 100 days however it seems to run once every day - can you please explain this to me?

If you feel that the interval setting is completely ignored (backup copy job attempts to create restore points more frequently), feel free to contact technical support so they could take a look at what is happening.

[jmentz]

Hi All

I have also been struggling with this and have spent days trying to get this back on track and will be calling support on Monday. Here is my scenario:

1) Created a new local Backup Copy to external media to serve as a seed for offsite repository via WAN Accelerator.
2) Disabled daily backup job from executing at the local site to ensure no changes to local backup files occur before Offsite Copy job is running correctly.
3) Transported the offsite seed files (VBM and single VBK file) to remote site and copied on to offsite repository.
4) Rescanned the offsite repository to synchronise the VBM / VBK - successful.
5) Changed the local Backup Copy job to point to remote Offsite repository (via WAN Accelerator) and mapped the backup - successful.
6) Ran a "Sync Now" on the Backup Copy job and it started successfully and was doing "digest calculations" which was taking a very long time to run as it is a 700GB file. (This was early evening.)
7) At 12am I received an email alert saying: "Copy interval has expired. Initial copy did not complete, and will resume on the next copy interval." It seems this happened because my backup copy interval was set for every day at 12am.
8) I tried kicking off the process again the next morning by doing a Sync Now, but this time it created a new 16500kb vib file on the remote repository and ignored the VBK.
9) Recognising this as an issue, I stopped the job and I deleted the new VIB file and overwrote the altered VBM file with the original one from the USB disk I transported to the remote site.
10) I went through the same process of scanning the repository and mapping the file to the backup copy jobs.
11) I started the job again but it did not work properly and reviewing logs it is failing because it is missing the new VIB file I have deleted!! Getting the following error: "Client error: File not found. Agent failed to process method {Stg.OpenReadWrite}."
12) Reviewed the logs and noticed the following: "Client error: File does not exist. File: [E:\Backups\Offsite Copy\Offsite Copy2014-10-11T151204.vib]".

I have tried all I can to try and recover from this but having no luck. If I replace the VBM file with the original one, why does Veeam still look for the VIB file?? Will it help if I delete the Offsite Copy job and create a new one pointed to the original VBM file?

I would hate to have to start the whole process again as it has taken days just to get to this point already and this is only the first site out of 17!! You can see why I am quite despondent! :cry:

Any advice on how to avoid this situation from happening when seeding offsite repositories and how to recover from this situation if it does happen without having to start all over again??

Many thanks
John

[tsightler]

I have tried all I can to try and recover from this but having no luck. If I replace the VBM file with the original one, why does Veeam still look for the VIB file??

You need to remove the backups from the database by going to Backups...Disks and then right-clicking the backup and choosing "Remove from backups". This will cause the Veeam server to "forget" about the backup files that it thinks are there (including the VIB file), then you can rescan the repo and start the process over.

Will it help if I delete the Offsite Copy job and create a new one pointed to the original VBM file?
Should not be required

I would hate to have to start the whole process again as it has taken days just to get to this point already and this is only the first site out of 17!! You can see why I am quite despondent!
No reason to start over as long as you have the original VBK/VBM.

Any advice on how to avoid this situation from happening when seeding offsite repositories and how to recover from this situation if it does happen without having to start all over again??
Make sure that for the very first pass, after creating the seed, that you set the interval on the backup copy job to something very long, say 30 days or something, so that this initial process doesn't get interrupted. After that, be sure to set the interval to what should be correct for your environment, typically 24 hours and in sync with the start of your backup jobs.

Reach out to me with a PM if you get stuck or if support doesn't help you. BTW, one thing that was confusing to me is that when the first interval was interrupted and the next started and created a VIB you seemed to think that was a problem, but that's fully expected. With no error why did you feel this was a problem?

[jmentz]

Hi Tom

Thank you so much for your advice on this. Since writing this post last night I did delete the Offsite Copy job and kicked off a new one after pointing to the original VBK/VBM in the offsite repository. It ran through the digest calculation and then eventually started transferring data. So far it has transferred about 35GB to a new VIB file. I will remember about the "Remove from backups" option if I encounter this issue again in the future. I was incorrectly under the impression the backup chain information was exclusively kept in the VBK file.

Just to clarify my issue with the VIB being created - I was (perhaps again mistakenly) under the impression that no additional VIB should be created as nothing has changed at the source. I performed the local backup copy, stopped the local backup job, raced the USB disk over to the offsite destination and started the job. Even now I am somewhat puzzled why I am looking at a brand new VIB file (35GB and growing) being created in the offsite repository when nothing has changed - there is no new data in the local backup files as there has been no new local backups performed since I did the backup copy job.

I will definitely take your advice about changing the interval to something long the next time I end up seeding a job like this!

Thanks again
John

[jmentz]

Just another quick follow up to above - I think I must have a fundamental misunderstanding as to how the process works in the back end. Will discuss with support on Monday but if any insight to the following will be appreciated from an educative perspective!

The following processes have run and are currently running on this offsite copy job:

Code: Select all

11/10/2014 4:39:25 PM :: Hard disk 2 (500.0 GB) - DURATION 12:19:57
11/10/2014 4:39:25 PM :: Calculating disk 1 digests (100% done) - DURATION 8:52:07
12/10/2014 1:47:50 AM :: Changed extents are not available.
12/10/2014 1:47:50 AM :: Digests are missing on source, loading them from target
12/10/2014 4:59:25 AM :: 0.0 KB transferred over network, 0.0 KB obtained from WAN Accelerator cache
12/10/2014 4:59:50 AM :: Error: The remote procedure call failed and did not execute
RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [192.168.200.20].
12/10/2014 4:59:51 AM :: Busy: Source 90% > Source WAN 9% > Network 0% > Target WAN 0% > Target 0%
12/10/2014 4:59:51 AM :: Primary bottleneck: Source
12/10/2014 4:59:51 AM :: Network traffic verification detected no corrupted blocks
12/10/2014 5:00:16 AM :: Processing will be retried (retry 1 out of 5)
12/10/2014 5:00:18 AM :: Copying restore point 9/10/2014 9:09:47 PM from backup repository Local Repository
12/10/2014 5:00:18 AM :: Queued for processing at 12/10/2014 5:00:18 AM
12/10/2014 5:00:20 AM :: Required backup infrastructure resources have been assigned
12/10/2014 5:00:28 AM :: VM size: 640.1 GB
12/10/2014 5:01:15 AM :: Hard disk 1 (120.0 GB)
12/10/2014 5:00:38 AM :: Using source WAN Accelerator Local
12/10/2014 5:00:38 AM :: Using target WAN Accelerator 192.168.200.20
12/10/2014 5:00:45 AM :: Saving SERVER.vmx
12/10/2014 5:00:45 AM :: Saving SERVER.vmxf
12/10/2014 5:00:47 AM :: Saving SERVER.nvram
12/10/2014 5:01:15 AM :: Hard disk 1 (120.0 GB) 76GB read at 5 MB/s - DURATION 4:28:43 (so far...)

When I look at the destination repository I can see the new VIB now almost approaching 50GB. I am already in trouble for exceeding the data usage limits on this link so I am praying the offsite copy process is not currently trying to reconstruct new VIB files by transferring all the data from the source repository again! If it starts to transfer the 500GB disk I will have to stop the process and hand in my resignation. What is the point of doing a seed if there is still a requirement to shift such massive amounts of data??

[tsightler]

I don't think it is recognizing your seed because the initial seeding process was interrupted and it has now moved on to copying into an increment. This will happen if it never finished the first pass of creating digests. Also, please be sure you have the latest patches installed (patch 4) as there were major fixes around some of this. It's really hard to tell just from some screen dump info however but the fact that it's creating such a large VIB is iffy.

However, when you say "nothing has changed" do you mean that the source is not continuing to make backups? I mean, what will happen when you use a seed is that it will recognize that as the base, but the next run will create an "increment" which is all the changes that have continued to be built since that base was created so if it takes 3 days to move the seed data, then there is three days worth of data that must be "caught up".

It sounds like you created the initial "seed", then moved the seed to the remote site. This was all good, however, once at the remote site you were using WAN accelerator, which wasn't part of the initial copy. That means it has to create hashes for the data so that the source side knows what data is in the remote copy. Since this was interrupted it's making bad assumptions that data is missing and thus sending too much data. Normally I would expect a new VIB but it would only be the changed data. I believe the failure of creating the hashes is the problem. You probably needed to reinitialize the WAN accelerators when you "started over".

It really is complex to understand everything that can go wrong in a forum post, but it's obvious with a Webex, so hopefully support can get you there quickly.

[jmentz]

Thanks for your insights Tom, much appreciated! Just one quick question - when you say reinitialize the WAN accelerators, do you mean remove and then add again?

[foggy]

I believe Tom meant clearing the accelerators' cache (right-click WAN accelerator > Clear cache). If you simply re-install WAN accelerator with the same settings, it will re-use the existing cache, which is not what you would need in this case.

[jmentz]

Unfortunately I think I am fighting a losing battle with the WAN Accelerator / Backup Copy seeding. I am actually in the process of trying to implement this at a number of sites and encountering similar issues everywhere. Maybe somebody would have some advice...

I am now working on a site which has a fairly slow ADSL connection. As with the other sites I made a backup copy to external USB and transported it to the Target Site to seed the offsite repository. The VBK file is approx 2TB in size. I then mapped the backup copy job successfully to the offsite repository and started a Sync Now. After running for close to 48 hours it only made about 2% progress. For some reason the backup copy interval also ignored my setting of 10 days and kicked in last night at 12am and seemingly started the whole process again. At the rate it was going, the initial sync will only be finishing in a couple of weeks. Being close to tears, I drove to the source site disconnected the Veeam backup machine and drove it over to the target data centre and connected locally to the network in an effort to get this initial sync completed at LAN speed and then take it back to the source site.

I constructed a DMZ using the onsite firewall at the target site to allow the backup machine to retain it's original IP address by still being able to connect to the Target WAN accelerator and target repository. I successfully verified connectivity. So far so good. I kicked off the Backup Copy job but now it got stuck with message Building VM List!! I had a look at the logs and noticed that it appears to be talking back to the ESX servers located in the source site!!! See below:

Code: Select all

[19.10.2014 11:38:09] <01> Info     [Soap] Trying to download xml from 'https://192.168.1.2:443/client/clients.xml'.
[19.10.2014 11:38:30] <01> Error    Unable to connect to the remote server (System.Net.WebException)
[19.10.2014 11:38:30] <01> Error       at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
[19.10.2014 11:38:30] <01> Error       at System.Net.WebClient.DownloadString(Uri address)
[19.10.2014 11:38:30] <01> Error       at Veeam.Backup.ViSoap.CClientConfig.DownloadXmlWithRetries(WebClient webClient, String address)
[19.10.2014 11:38:30] <01> Error    A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 192.168.1.2:443 (System.Net.Sockets.SocketException)
[19.10.2014 11:38:30] <01> Error       at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
[19.10.2014 11:38:30] <01> Error       at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
[19.10.2014 11:38:30] <01> Info     [Soap] Trying to download xml from 'https://192.168.1.2:443/client/clients.xml'.
[19.10.2014 11:38:51] <01> Error    Unable to connect to the remote server (System.Net.WebException)

Why cant Veeam build it's VM list based on the VBM and VBK files that are present in the local repository? Why does it need to speak to the ESX servers??

I then re-IP'd the Veeam backup PC and put it on the same subnet as the Target Repository and re-established the IPSec tunnel to the source site so that the ESX servers can be reached. However, as soon as I changed the IP address of the Veeam backup PC I was no longer able to get in to the console! I am getting a message saying that the "Backup and Replication Service is not running. Please start the service and click Retry". As soon as I change the IP back to the original IP address I can get in to the console but of course not route to the source site via the IPSec tunnel.

So in conclusion, this is an absolute nightmare. Offsite seeding doesn't seem to really help when you have a large volume of data. The WAN acceleration doesn't seem to help if you have a slow internet connection. It is ironic that these two things which is supposed to overcome the limitation of slow WAN links are having exactly the opposite effect in my case.

I am sure there is a big chance that I am perhaps just stupid and don't really understand what I am doing. But it will be great if somebody could advise me what the best way is to implement and seed an offsite Backup Copy routing with up to 2TB of data over slow (1MB/s) WAN links.

I guess for now I have no choice but to take the backup PC back to the source site, putting the backup jobs on hold for a few weeks and hope no disaster strikes, start a new sync and just wait and wait and wait...

[jmentz]

FYI, this is what the logs indicate after the backup PC was re-IPd:

Code: Select all

[19.10.2014 12:44:04] <01> Info     ------- Started new session of Veeam Shell -------
[19.10.2014 12:44:04] <01> Info     Multi session mode: True
[19.10.2014 12:44:04] <01> Info     Terminal Services session ID: 2
[19.10.2014 12:44:04] <01> Info     Remoting options: []
[19.10.2014 12:44:04] <01> Info     Registering TCP client channel [bstcp]
[19.10.2014 12:44:47] <01> Info     Networking error: [Exception has been thrown by the target of an invocation.. A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 192.168.1.13:9392]
[19.10.2014 12:45:29] <01> Info     Show retry\cancel messagebox with owner window. Invoke requered: [True]

Restarting the Veeam services did not seem to help.

[jmentz]

Sorry for spamming this thread - just want to share with you guys some of the challenges I am facing with getting this all to work and tap in to your experience and get some advice.

In addition to the site I have talked about I thought I would check in to one of the other sites which I have seeded yesterday and noticed the following progress on just the first ONE of the VMs - 7 others are included in the backup copy job:

19/10/2014 8:37:16 AM :: Hard disk 2 (600.0 GB) 20.0GB read at 824 KB/s :: Duration 7:04:36

I am really starting to question the value of seeding and using the WAN accelerator as it seems it actually takes just as long - if not longer - to perform the first backup copy job! At this rate this site will also take weeks to sync up.

I have scoured the forums and one through the manual countless times (believe me I have had lots of time while waiting!) and do not believe I have made any fundamental mistakes when performing the seeding process. So my questions to you guys are as follows:

What is the best way to seed an offsite repository that is connected via a low bandwidth connection?
Realistically, should I allow weeks for the first sync to complete?
During this time, what is the effect of daily backups running on the ongoing backup copy job? Does it invalidate the data that has already been sent or is it simply a matter of catching up one day in distant future?
What is the point of the WAN accelerator if it doesn't actually speed up backup copy jobs?
Is it better to rather just do the first sync the data via direct connection (not using WAN accelerator) and once in sync enable the WAN accelerator?

Your advice will be much appreciated! I have another 10 sites to do and at the rate I am going I am feeling rather depressed...

[mc1903]

I am just about to start down the path of 'offsite backup seeding' and after reading the forums I am completely confused!

I have had Veeam B&R 7.x running for the past 10 months backing up to a single repository on a Windows NAS - this was very easy to setup and works 100%.

I now need to move to the next stage and place a copy of my backups in an offsite location. I have approx 20Tb of backup data in my current repository.

Is there an 'end-to-end walk through' document available that I can review?

I now have a second Windows NAS temporarily installed at the primary location, but what is my next step?

Also, assuming I get a copy (Seed) of the data onto the second NAS, what happens to the backup jobs whilst it is offline being moved to the offsite location? Do the jobs fail or do you edit the jobs to not make a 2nd copy?

I hope the seeding process is better addressed in version 8 - perhaps a wizard would help reduce the confusion.

Martin

[foggy]

Why cant Veeam build it's VM list based on the VBM and VBK files that are present in the local repository? Why does it need to speak to the ESX servers??

For example, if VMs are added into the job via container, we need to expand it each time the job runs in order to identify the exact list of VMs to copy (not to miss a newly added ones). Also add possible disk exclusions here.

John, could you please tell us what kind of primary and secondary storage do you have and what are the bottleneck stats for the backup copy jobs? What did support say regarding your issues?

[foggy]

Is there an 'end-to-end walk through' document available that I can review?

Martin, basically, the seeding process is pretty simple and is described in a couple of sentences in the second post of this tread. Since you can only map backup copy job to the single full backup only, you need to perform the steps outlined in this post.

[mc1903]

Martin, basically, the seeding process is pretty simple and is described in a couple of sentences in the second post of this tread. Since you can only map backup copy job to the single full backup only, you need to perform the steps outlined in this post.

Thanks Foggy,

I must be as thick as a brick as I have read that post already, but I just do not 'get' it.!!

I think I need to see it done and then it will click.

Martin

[foggy]

You can always test the procedure on some small VM locally before going and seeding all backups offsite.

[veremin]

I must be as thick as a brick as I have read that post already, but I just do not 'get' it.!!

Not sure whether it helps or not, but we have a KB article regarding backup copy seeding. May be, you will find it more comprehensible. Thanks.