Initial "replication" over removable storage

[hblb]

Hi,

Currently we are using Veeam Backup to replicate a 40GB machine to a remote DR site down a slow WAN link – the replication statistics are indicating that the initial replication will take around 3 days.

Other than speeding up our WAN link is there any way to avoid the significant length of time required to create initial replicas across slow links – i.e. can we physically ship VM images and then start replication?


Thanks

[Gostev]

Hello hblb, support for initial replication over physical storage is one of the features we are considering to add to the product soon.

[bwestover]

I'd like to second that request. We have a few VMs that are 100s of gigabytes in size and sending the full VM over the WAN at any time is just not going to be feasible.

Could this be integrated into the backup/restore procedure?

For example you create .vbk file of your VM, then you can physically transport that file to the DR site. Then during the backup, it would be cool to have a 'restore as replica' function that restores the VM on the target host, and registers it in Veeam as a replica of the source VM.

[Gostev]

Sounds good - thanks for this.

[andygee]

I'd like to third this request. The replication is the main reason we are considering this product, unfortunately we are having issues getting the replication to go across the wan and complete.

If I register the server as a host name in the dns onsite and replicate initially locally and then just update the dns will veeam know any difference?

Andy

[Gostev]

Andy, you are genious!

This might work because source and target server are stored using DNS names (and not VirtualCenter refIDs).
So Veeam Backup should not notice the difference.

Unfortunately, I cannot officially recommend this workaround yet (before our QC get a chance to test this), but probability that it will work is pretty high.

[bwestover]

On the subject of workarounds for the 'initial replica' problem, I too have come up with something. I doubt this can be 'officially' recommended either as its a modification to the product, but it seems to be working.

Procedure:
I created a job to replicate a VM where the source and target were actually the same host.

Then I moved the replica VM, using physical media, over to another host at the DR site.

Then I modified the database record for that job to point it at the new host.

Then I started the replication again, and now its going to the new host.

It wasn't very hard, the database seems very simple. The job details were held in a single table, and I only had to modify two columns to update the target host and the target storage location.

Still, I wouldn't recommend this in any production environments as this could be risky to the stability of the product. I'm performing all of this on a test server.

It is working, I think, the only thing is the performance is much lower than I would have hoped. I have a fairly slow WAN link, however since only the changed data blocks should be replicated, it should still complete in a reasonable amount of time.

I suspect due to my unorthodox method it may be performing an initial replica instead of an incremental.

Is there a way to confirm that it is running an incremental replication versus an initial replica?

Thanks for your help. I appreciate all the attention you give to feature requests and suggestions.

[Gostev]

That should work as well!
Developers will be able to identify whether it is incremental run from the log files - if you want us to investigate, please send them to our support.

[bwestover]

Ahh!! the logs. I was having a hard time finding them until you said 'support' Help->Support Information BINGO!!

I looked in there. It wasn't too hard to see that it was creating an initial snapshot rather than replicating changes.

I messed up the target directory

I straightened it out and now that method is working beautifully.

Thanks for your help! My eval is going well and we will likely be buying licenses soon!

[eric]

Ahh!! the logs. I was having a hard time finding them until you said 'support' Help->Support Information BINGO!!

I looked in there. It wasn't too hard to see that it was creating an initial snapshot rather than replicating changes.

I messed up the target directory

I straightened it out and now that method is working beautifully.

Thanks for your help! My eval is going well and we will likely be buying licenses soon!

I have a few questios about this as we would like to test it as well. 1) Did you have to install SQL2005 tools to get into the tables? 2) Do you find that the incremental replicas are still sending all of the backup data? It appears that it does when I watch the status of incremental replication now.

TYIA,

Eric

[bwestover]

Yes, I used SQL Server Management Studio to modify the database. The express version is free from MS website.

After I got the directories correct on the target, it now copies only the changed blocks when I replicate to my new target.

So my operation was to replicate the VM from the source back to the same source. (source and target are same host). I did this to create the appropriate database entries in Veeam Then I:
1)make backup of replica VM.
2)Transport backup manually to new target.
3)restore VM on new target
4)modify database to alter job to point at new target
5)run replication again

At that point it runs and copies only the differences since the last time it ran.

This is an unsupported method, but for my testing and until such a feature is included in VeeaM backup, it works for me.

[eric]

1)make backup of replica VM.
2)Transport backup manually to new target.
3)restore VM on new target
4)modify database to alter job to point at new target
5)run replication again

For step 3) Do you add the replica to the server Inventory? Or does Veeam do that?

Eric

[bwestover]

Not sure if Veeam would do it or not. Im not sure if it matters.

I think i DID register it myself and made sure to use the same <vmname>_replica convention i had used on the original host.

[eric]

What table and columns are you modifying in the db? When I open dbo.BJobs the target_host_id has an entry but not target_dir and target_file...

Can you provide more info...

[bwestover]

I dont feel comfortable posting instructions on how to modify the back end database.
This is totally unsupported and could cause serious problems.

In my test environment, it was worth testing, but I dont want a casual reader to see the instructions and corrupt their database because of me.

If you'd like to email me

exanthus-at-gmail dotcom

I'll send you the script I used.

[eric]

I dont feel comfortable posting instructions on how to modify the back end database.
This is totally unsupported and could cause serious problems.

In my test environment, it was worth testing, but I dont want a casual reader to see the instructions and corrupt their database because of me.

If you'd like to email me

exanthus-at-gmail dotcom

I'll send you the script I used.

No problem. This is only my test environment as well...

[Gostev]

For step 3) Do you add the replica to the server Inventory? Or does Veeam do that?

Eric

Yes, Veeam Backup does this for you.

[eric]

After reviewing the simple procedure and db changes that need to be made to make local replication>physical transport>then remote replication possible it seems that this is something that would be easy for Veeam to program. You simply need to be able to modify the job to point to a new host/data store inside Veeam. Although Veeam could probably add some additionaly functionality/error checking in there as well.

Eric

[Gostev]

Yes, from the coding perspective this feature is fairly easy to add.

[AlexanderDonov]

Originally posted by bwestover


1)make backup of replica VM.
2)Transport backup manually to new target.
3)restore VM on new target
4)modify database to alter job to point at new target
5)run replication again

-- Having trouble with #1 , I created a backup job to send over to other esx host. My target vm is the vm_replica. That way I could probably restore as I understand, but the job keeps failing. See the log. :


[11/11/2008 1:37:21 PM]Info Task result: Backup failed\nClient error: Cannot read data from the socket. Requested data size: .\nEnd of file\n{\nCannot read data from the socket. Requested data size: .\nEnd of file\nData manager: Exception in the read loop.\n}\nCannot backup file in the service console mode. File: [/vmfs/volumes/47fa637e-3b890a40-c0b9-00145e7a8220/mhlengs4/mhlengs4-D-flat.vmdk]. VBK: [2ac8aa0d-0048-4661-9b84-f3f6a95815ab (928)\mhlengs4-D-flat.vmdk]. RBK: [].\nClient failed to process the command. Command: [backup].\n\nServer error: File cannot be delivered by the management channel. Source path: [/vmfs/volumes/47fa637e-3b890a40-c0b9-00145e7a8220/mhlengs4/mhlengs4-D-flat.vmdk].\nbad open: Device or resource busy\n
[11/11/2008 1:37:21 PM]Info Disposing client from thread 1
[11/11/2008 1:37:21 PM]Info (brsvmwesx01.crbard.com) Service: closed
[11/11/2008 1:37:21 PM]Info Disposing client from thread 1
[11/11/2008 1:37:21 PM]Info Task session {2199a160-0aa0-4b72-b4ca-e0a84d81e2c4} has been completed, status: Failed, 107,378,013,277 of 143,772,095,488 bytes, 10 of 35 objects, details: "Backing file "/vmfs/volumes/47fa637e-3b890a40-c0b9-00145e7a8220/mhlengs4/mhlengs4-D-flat.vmdk"\nBackup failed\nClient error: Cannot read data from the socket. Requested data size: .\nEnd of file\n{\nCannot read data from the socket. Requested data size: .\nEnd of file\nData manager: Exception in the read loop.\n}\nCannot backup file in the service console mode. File: [/vmfs/volumes/47fa637e-3b890a40-c0b9-00145e7a8220/mhlengs4/mhlengs4-D-flat.vmdk]. VBK: [2ac8aa0d-0048-4661-9b84-f3f6a95815ab (928)\mhlengs4-D-flat.vmdk]. RBK: [].\nClient failed to process the command. Command: [backup].\n\nServer error: File cannot be delivered by the management channel. Source path: [/vmfs/volumes/47fa637e-3b890a40-c0b9-00145e7a8220/mhlengs4/mhlengs4-D-flat.vmdk].\nbad open: Device or resource busy\n\n"
[11/11/2008 1:37:21 PM]Error Backup failed at AgentProvider.BackupClient.Backup(String filenameToBackup, String backupFilename, String folderInsideBackup, String filenameInsideBackup, String rollbackFileName)
[11/11/2008 1:37:21 PM]Error at VeeamManager.CMethodScBackup.BackupFile(CBackupFile file)
[11/11/2008 1:37:21 PM]Error at VeeamManager.CBackupJob.BackupFiles(CDBSessionInfo taskSess, CVirtualMachine vm)
[11/11/2008 1:37:21 PM]Error at VeeamManager.CBackupJob.BackupVM(CVmTask vmTask, CDBBackup currentBackup, CDBBackup previousBackup)
[11/11/2008 1:37:21 PM]Error at VeeamManager.CBackupJob.Backup()
[11/11/2008 1:37:21 PM]Error Client error: Cannot read data from the socket. Requested data size: .
[11/11/2008 1:37:21 PM]Error End of file\n{\nCannot read data from the socket. Requested data size: .
[11/11/2008 1:37:21 PM]Error End of file\nData manager: Exception in the read loop.\n}\nCannot backup file in the service console mode. File: [/vmfs/volumes/47fa637e-3b890a40-c0b9-00145e7a8220/mhlengs4/mhlengs4-D-flat.vmdk]. VBK: [2ac8aa0d-0048-4661-9b84-f3f6a95815ab (928)\mhlengs4-D-flat.vmdk]. RBK: [].\nClient failed to process the command. Command: [backup].\n
[11/11/2008 1:37:21 PM]Error Server error: File cannot be delivered by the management channel. Source path: [/vmfs/volumes/47fa637e-3b890a40-c0b9-00145e7a8220/mhlengs4/mhlengs4-D-flat.vmdk].
[11/11/2008 1:37:21 PM]Error bad open: Device or resource busy\n
[11/11/2008 1:37:21 PM]Info Job operation: Checking retention policy
[11/11/2008 1:37:21 PM]Info Job operation: Running post job command
[11/11/2008 1:37:21 PM]Info Job session {a6bd8b43-0f7d-49df-a435-8f2da1a34898} has been completed, status: Failed, 107,378,013,277 of 143,772,095,488 bytes, 1 of 1 tasks, 0 successful, 1 failed, details: ""
[11/11/2008 1:37:21 PM]Info Job has been stopped
[11/11/2008 1:37:21 PM]Info VeeamBackup Manager has been stopped

[bwestover]

It sounds like your doing the first step twice. Let me see if I can clarify.

#1 Create and run a replicate job for your source VM. Use the source host as the target host (makes a copy "vmname_replica" on the same host as the original source VM)

#2 Physically backup the offline replica copy, that step 1 just created. (you can do this however you want. I simply gzipped the VM files and then copied them to an external drive. Transported them to my intended target, copied them back and unzipped)

#3 With replica VM now in place on new target host, modify the DB to alter the job to point at the new target.

#4 Run the job (now modified) again, it should do a differential backup of your source VM onto your target host.

To be clear, you only create a single Veeam job.

[bwestover]

GZIP is a native linux tool, and its included with ESX. You will need console access to your server to perform the compression. Alternatively, if your more comfortable with windows, you could use Veeam FastSCP to copy the VM to your local workstation, then compress and transport using whatever method your comfortable with

[AlexanderDonov]

Thanks that did clear up what I was doing. I never used gzip. Do I have to install it on ESX host or on Windows in order to compress the vm_replica?

[renpen]

bwestover,

Hi there,

Could you give me the exact steps required to alter the target in the database.

Thanks

[Gostev]

Just wanted to update everyone that replica seeding functionality is coming as a part of our 4.0 release (scheduled for end of Q3 2009).

[bwestover]

Gostev, thats great news. I can't wait for the new version. We're up to 10 sockets licensed now (just added 2 more) and its going great.

renpen,
I have not re-worked the database statements for the new version 3. I don't expect there have been significant changes, but I don't think its a good idea to modify your database without testing.

If you want me to send you alter statements anyway, email me at "exanthus at gmail dot com"

[bwestover]

I have been unable to make the method I had used previously work with version 3.0 or 3.1. Something must have changed in the way Veeam tracks sessions, or identifies the server object or volume object. After I apply my alter statement to the table, the job no longer functions.

I will no longer be able to send the alter statements via email, as they don't work and actually corrupt the job in question.

We will just have to be patient and wait (eagerly) for the new version which will include this feature.