Backup from a restored machine very slow

[ppcfox]

Veeam Backup from a VM is normal.
When this machine is restored with Veeam then the next backup is very slow, this depends on machines
with very much unused space. It seems that this unused space after restore is not recognized as such.

sorry for my worse english.

[Gostev]

Hi, I am not sure I fully understand what you are saying, and the whole scenario. If this will be easier for you, feel free contact our office in Germany, ask to talk to a Systems Engineer, and explain him the issue. He will then forward this on to me (they know me well ). Contact details can be found on our web-site. Danke!

[Gostev]

I think I understand what you are saying now. However, the speed should remain the same after restore, as we backup entire image (along with white blocks). We also restore the entire image in the same form, so restored image is bit-identical (so any white space which existed before will still be there). I am assuming you are running the latest product release.

[ppcfox]

Hi Anton,
yes we are running the latest software 5.0.2.
We recognized the problem since we had changed from ESX to ESXi (4.1.0,260247).

Greetings from germany
Jörg

[Vitaliy S.]

Jörg,

There shouldn't be any difference if you backup VMs from ESX or ESXi host accordingly. Could you please clarify if you see performance degradation on the incremental pass or full run? Have you changed anything else? Backup mode? Target datastore?

Thanks.

[tsightler]

When Veeam performs "agentless" restores (which I think are the only possible restores with ESXi) the it appears to loose the ability to skip "empty blocks". To verify this behavior I performed this simple test:

1. Created a new 40GB VM, completely empty (no OS)
2. Ran a test full backup, took 1:08sec, reported 599MB/sec, very little I/O reported from storage
3. Delete test VM
4. Restore from Veeam using ESX agent mode
5. Ran a test full backup, took 2:40 sec. reported 252MB/sec, very little I/O reported from storage
6. Restore from Veeam using ESX agentless mode
7. Ran a test full backup, took 10:47 sec. reported 63MB/sec, storage array reports 10 minutes of sustained 70MB/sec transfer

Basically, if you perform a restore of a VM that has lots of empty space using agentless mode, all of that empty space becomes "used". I think you can work around this by forcing the restore to thin and then using VMware tools to convert the restored VMDK to thick, but I haven't done that in a while. We still run full ESX so we just make sure our restores are done with the agent.

[Gostev]

Looks like full pass CBT failure at 7, which made us scan through all blocks. Usually does not have to deal with restores... full pass CBT fails all the time on random VMs for no good reason, including freshly created VMs. More info > CBT on first (full) backup, how?

[tsightler]

Perhaps, but in my environment this is 100% reproducible after an agentless restore (discovered this a very long time ago), thus the responses to the OP that this shouldn't happen are incorrect, right?

If it's truly the "CBT on first full backup" issue should it correct itself after an incremental for the next full backup?

[Gostev]

No, it will never start working unless you do certain sequence of actions involving manually disabling/enabling CBT and powering VM off/on many times (in certain order). This is just an idea though, need to see log files to say if initial CBT worked or not in this case.

[tsightler]

OK, I'll send you my logs offline.

[Gostev]

Thanks!

[tsightler]

I'm pretty confident this issue is NOT related to CBT at all. I think the restore of a VM using Veeam via agentless always creates a VMDK that appears to VMware as being "fully" used, thus requiring a read of the entire VMDK during a full backup pass. Why do I think this? Here are my reasons:

1. As far as I know, CBT is not required for "unused block elimination", Veeam can skip unused blocks on VMDK's on ESX 3.x hosts, which don't even have this feature
2. I cannot make a "thin" disk from a VMDK that was restored by Veeam as think via the agentless method. If I attempt to do this I end up with a "thin" disk that is just as big as the "thick" disk because VMware thinks the disk uses the entire space.
3. These issues do not occur when using "agent based" restore mode on ESX

[Gostev]

You are right, I just came back from devs and opened the topic to write the same thing. It is a matter of how thick disk is created, in agentless mode thick disk is uploaded as file. Where as in agent-based mode, and in agentless mode in case of thin disks, disk is first created as disk, and is then "stuffed" with the content, including zeroed blocks. But for VMware, all block now appear to have content, so initial CBT will return all blocks including those containing zeroes. Thanks.

[tsightler]

Well, actually, I was wrong on point #2. If I use vmkfstools to clone the "thick" disk that's full of zero's to a thin disk, I will actually end up with a mostly empty "thin" disk. It appears that vmkfstools is smart enough to recognize zero'd blocks and not copy them to the "thin" disk.

The easiest manual workaround seems to be to simply restore the VM as "forced thin" and, if you want "thick" disk, simply convert to "thick" after the restore via command line. Veeam might want to consider using this restore method as the "default" method for restoring files via agentless mode, you could restore as thin and convert to thick after the restore was complete. This also has the added advantage of typically being much faster (thin restores seem much faster than thick restores for agentless mode).

Of course, I've somewhat hijacked this thread, so I don't even know for sure if this is the issue the OP is referring to, but it's something I've noticed and thought it might be what he was referring too. I guess we should wait to hear from him.

[ppcfox]

Tom, thank you very much for clarification.
I tested the restore with "thin". After restore i changed the format with storage vMotion (quicker than console input).
It works!

Greetings
Joerg

[tsightler]

You can also use the datastore browsers "inflate" option, but the VM has to be powered off, which is the great thing about using storage vMotion to do it. Glad to know that solved your problem, well, at least explained the problem.