Backup copy job coping wrong (random) disk

[Marten_med_e]

Support case ID# 00949961

This is something I discovered about two weeks ago, I was unsure if it was my bad or a bug, so until I was sure I didn't want to post to this forum, but as Veeam support was able to reproduce the issue last Friday (26/6) I decided to start a thread.

This concerns Backup copy jobs where the following conditions are true, for me anyway, I don't know if the support has found other/more/different conditions. This is going to be a long post, will try to keep it simple.

1. Source server, one Windows Virtual server with six disks, as seen from VM config, one scsi card, six disks, ESXi 5.5.0, 2456374, I will use virtual machine stofs02_Fileserver_ny hard disk 5 SCSI (0:4) only as example in this post.


From VMware PowerCLI, Get-Vm xx |Get-Harddisk


Note that three disks has the same name, so I have the backup jobs “locked” down to using the same proxy, stoba004 witch uses only Network as transport mode, as to KB 2042 http://www.veeam.com/kb2042 (not same error but similar? )


I have done a ordinary Backup job, selected disks only and I will use virtual machine stofs02_Fileserver_ny hard disk 5 SCSI (0:4) only as example in this post.




Backed up data verified with file restore, the correct data/disk is backed up.




So far no problems or errors, Veeam is working as expected. This backup had been running for week before I started to configure the backup copy job.
But I did a ordinary Backup copy job, chose objects to process, from backups, “1-Backup Job STOFS02_KFU”, this picture is from today, with a workaround applied as a try from Veeam support tech.




After I clicked "Add" the object is added to the list and the total size is calculated,


The outcome of a recalculation isn't allways constant, sometimes all disks in the server, sometimes the whole server and sometimes any random disk from the server,


This is from the Job progress window, take an extra look at what disk is being proccesed, not the one in the Backup job for sure and look for all different sizes, nothing makes sense.


And look closely now, the disk on disk, that sounded funny, but this is the backup files from the repository, a new disk, this one 1.1 TB


And this was confirmed with file restores, this is from harddisk 4 SCSI (0:3)


This is the correct disk, here from the "primary" Backup job,


This is a short video I did to show that if I make a restore from Monday (Måndag) the files in the backup is from disk 6 SCSI (0:5) and from Wednesday (Onsdag) they are from disk 4 SCSI (0:3), both disks is wrong disks, the status for the backup job was "Successful"
https://dl.dropboxusercontent.com/u/590 ... 0error.mp4

All five jobs with this server experiences this phenomena, some jobs have mixed four disks in the same backup file, I have had backup copy job where the disks in the local backup copy repository was different from the one in the cloud repository. To be sure what disk is copied I had to make a restore as the progress windows and statistics windows can show another disk than the selected disk and the copied disk.

I have 42 Backup copy jobs configured and those five are the only one with this error.

Cheers

[Marten_med_e]

As a follow up, I have received confirmation from two sources that the provided workaround is working, I have myself applied it to two of the Backup copy jobs for test, these jobs have now been running three days with the new configuration in place but as both jobs right now is occupied one with merging and the other is waiting for resources (another cloud backup is running a health check) I have not confirmed it my self.

Anyway, this is the provided workaround. Use it on your own risk, I haven't been able to confirm this my self, have seen the backup copy job logs before and after and it looks right.

If you experiences this error contact Veeam support and let them confirm if this workaround is right for your problem.

1. Create the regular Backup job. Do not check the "Configure secondary destinations for this job".

2. Create an empty Backup Copy job, do not add any "Objects to process", I only changed the repository, leave the "Enable the job when I click finish" check box, checked.

3 Go back to the Backup job made in step one, check the "Configure secondary destinations for this job", click next.


4. Select the "empty" backup job created in step two, click your way trough the wizard and finish.


This is said working, the numbers, VM size, Total size and the "Job progress, Processed data" sizes don't add up, but that is just cosmetic errors.


But once again, If you experiences this error contact Veeam support and let them confirm if this workaround is right/working with your problem.

Cheers

[Shestakov]

Hello Mårten,

As was described in the related topic, if you have several backup jobs of the VM, with different types of exclusions and set the source of backup copy job "from infrastructure" or "from backups"(as I found out in your support case you did), the backup copy job copies the latest restore point.
Do you have several backup jobs making backups of the server? If yes, please choose "From Jobs".

It`s hard to imagine another way how excluded disks can be backed up, since backup requires snapshot, while backup copy job doesn`t take snapshots, but simply copies restore points of chosen VMs. You can review logs to check if the snapshot is taken during backup copy job(I believe it`s not).

Thanks!

[Marten_med_e]

Thanks for your explanation, but it really don't add up in my mind.

As was described in the related topic, if you have several backup jobs of the VM, with different types of exclusions and set the source of backup copy job "from infrastructure"....the backup copy job copies the latest restore point

I get this, backup copy job copies the latest restore point of the chosen object, and copies the data from backup repository instead of the production environment. In my set up all disks (data) belonging to that server is included, as expected. But even here I have stumbled across something unexpected, where all datastores belonging to a server is included even if I select only one of the servers datastore and with datastores for this server those are different LUNs in the san, each has only one wmdk file in that LUN or if I include the server and then exclude the LUNs I don't want included in the backup.




And from the run of the backup copy job, I cancelled this as it isn't working as I expected.

As was described in the related topic, if you have several backup jobs of the VM, with different types of exclusions and set the source of backup copy job "from backups" the backup copy job copies the latest restore point.

If I do a backup job with one server and only one of several disks included from that server, the .vbk file from that backup job only include data from that disk, this I have confirmed with file restores. In my world this .vbk file is the "from backups", it is not from infrastructure, it is not a backup job (configuration) it just is the backup, or am I thinking up side down here?

If I now configure a backup copy job and chooses "Objects to process:" + "Add" + "From backups..." and choose the backup job that only includes data from the only disk that is included in that backup job, Veeam chooses the last restore point from the server (as from infrastructure?) to process in the backup copy job, but that restore point is from a disk not included in the backup job, this makes no sens to me.
This should create a copy of the backup job and include the same data as the backup job, otherwise, for me, this isn't a backup copy job of a backup.

Cheers

[Shestakov]

If I do a backup job with one server and only one of several disks included from that server, the .vbk file from that backup job only include data from that disk, this I have confirmed with file restores. In my world this .vbk file is the "from backups", it is not from infrastructure, it is not a backup job (configuration) it just is the backup, or am I thinking up side down here?

By current design, if you choose "From jobs..." the job will copy restore points created only by the chosen job, while both "From backups" and "From infrastructure" copy all found restore points of the VM. It`s made in a sake of convenience of VMs search.

I see where your concern is coming from. But "From backups" also can`t mean "from some exact backup file", since you need backup copy job to copy new backups constantly.
If you have an idea how to make these names / logic more explicit, please share, we will definitely take your opinion into consideration.

By the way, if you take a look at the "Action" list of the backup copy job, you can see the name of the copied restore point.
Thanks!

[Marten_med_e]

By current design, if you choose "From jobs..." the job will copy restore points created only by the chosen job, while both "From backups" and "From infrastructure" copy all found restore points of the VM. It`s made in a sake of convenience of VMs search.

But in my setup, I got only one disk included in the backup copy job, not all, but the documentation dosen't say that all restore points will be included. Just that it will search for restore points of selected VM(s) in all backups created.
From the beginning I thought this disk was randomly selected but it turned out to be the disk with the latest snapshot created that ended up in the backup, but the backup copy job could have the wrong disk included and not always the same, why don't I get all disks included as ITWWT does in his thread? Sorry, but I can't really accept that it is supposed to work this way.

And how are we (me anyway) going to understand that this is as expected from the text in the documentation?

Cheers

[Shestakov]

But in my setup, I got only one disk included in the backup copy job...

There are no exclusion settings in the backup copy job itself. And as was mentioned, backup copy job does not create backups. It just copies existing backups offsite.
You just need to choose the backup job with excluded disks as the source of the backup copy job.

I see what caused the confusion. But we tried to explain the behavior in the documentation:

From Backup. You can select VMs from available backups. When a backup copy job runs, Veeam Backup & Replication will search for restore points of selected VM(s) in all backups created on the Veeam backup server. You can limit the search scope by selecting only specific repositories for the backup copy job.
From Jobs. You can select VMs from available backup jobs. When a backup copy job runs, Veeam Backup & Replication will search for restore points of selected VM(s) in backups created for selected jobs.

However, I agree that the documentation should be explicit to the maximum and should not cause confusions. We will think how to improve it. Thanks!

[Marten_med_e]

There are no exclusion settings in the backup copy job itself.

I meant I got only one disk included in the backup job... not backup copy job. But I really would like that was a possibility.

And as was mentioned, backup copy job does not create backups. It just copies existing backups offsite.
You just need to choose the backup job with excluded disks as the source of the backup copy job

I have reconfigured my jobs, so now they are using From Jobs and it works as desired.

One part of me wants to continue fight this, because I think this is unacceptable but as Veeam think this is expected behavior and only points to the documentation, I do understand what you try to explain happens when we choose From backups.., but my backup copy jobs doesn't behave like that. Nor as you describe,

backup copy job does not create backups. It just copies existing backups offsite.

If my setup had copied all disks from the server I would have accepted this as in this thread http://forums.veeam.com/veeam-backup-re ... 28971.html.
But in my case it just copies the disk with the newest snapshot and as this can vary depending on the backup resources available, it can be any disk of five in the server but it is never the sixth disk. But I do not know if it's worth it, I made my point and has showed that what is really in the vbk can be something different than expected, without any warning or "heads up". If you remove a server from a job we get a warning about it but if the wrong disk is processed, nothing.


Cheers

[Shestakov]

One part of me wants to continue fight this, because I think this is unacceptable but as Veeam think this is expected behavior and only points to the documentation, I do understand what you try to explain happens when we choose From backups.., but my backup copy jobs doesn't behave like that. Nor as you describe

I`ve contacted your support engineer and asked her to review the logs again to find out if the behavior really differs from the design. Since backup copy job can not backup or exclude disks, the only mistake it can make is to choose wrong restore point to copy.
Once we have updates, I`ll post them here.

If my setup had copied all disks from the server I would have accepted this as in this thread http://forums.veeam.com/veeam-backup-re ... 28971.html.

Looks like the cases in both threads are similar, you and another customer chose VMs "From Backup" and VBR copies the newest VM restore point instead of the one you were waiting for. Glad it`s solved by switching to "From Jobs..."

Once again, sorry if for the inconvenience and thanks for the feedback!

[Shestakov]

We`ve made some additional testing and the issue was not reproduced.

Previously you stated that the backup copy job chose "wrong" disks to copy. Could you please list backup jobs having the VM selected to back up?
And which VM disks are chosen in each job as well as the jobs` schedule.

As was mentioned above VM disks could not be excluded or backed up randomly. The most likely the backup copy job chose restore points to copy against expectations.
Thanks!