-
- Novice
- Posts: 4
- Liked: never
- Joined: Feb 10, 2011 6:31 am
- Full Name: Admin
Backup from a restored machine very slow
Veeam Backup from a VM is normal.
When this machine is restored with Veeam then the next backup is very slow, this depends on machines
with very much unused space. It seems that this unused space after restore is not recognized as such.
sorry for my worse english.
When this machine is restored with Veeam then the next backup is very slow, this depends on machines
with very much unused space. It seems that this unused space after restore is not recognized as such.
sorry for my worse english.
-
- Chief Product Officer
- Posts: 31754
- Liked: 7259 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup from a restored machine very slow
Hi, I am not sure I fully understand what you are saying, and the whole scenario. If this will be easier for you, feel free contact our office in Germany, ask to talk to a Systems Engineer, and explain him the issue. He will then forward this on to me (they know me well ). Contact details can be found on our web-site. Danke!
-
- Chief Product Officer
- Posts: 31754
- Liked: 7259 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup from a restored machine very slow
I think I understand what you are saying now. However, the speed should remain the same after restore, as we backup entire image (along with white blocks). We also restore the entire image in the same form, so restored image is bit-identical (so any white space which existed before will still be there). I am assuming you are running the latest product release.
-
- Novice
- Posts: 4
- Liked: never
- Joined: Feb 10, 2011 6:31 am
- Full Name: Admin
Re: Backup from a restored machine very slow
Hi Anton,
yes we are running the latest software 5.0.2.
We recognized the problem since we had changed from ESX to ESXi (4.1.0,260247).
Greetings from germany
Jörg
yes we are running the latest software 5.0.2.
We recognized the problem since we had changed from ESX to ESXi (4.1.0,260247).
Greetings from germany
Jörg
-
- VP, Product Management
- Posts: 27347
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Backup from a restored machine very slow
Jörg,
There shouldn't be any difference if you backup VMs from ESX or ESXi host accordingly. Could you please clarify if you see performance degradation on the incremental pass or full run? Have you changed anything else? Backup mode? Target datastore?
Thanks.
There shouldn't be any difference if you backup VMs from ESX or ESXi host accordingly. Could you please clarify if you see performance degradation on the incremental pass or full run? Have you changed anything else? Backup mode? Target datastore?
Thanks.
-
- VP, Product Management
- Posts: 6032
- Liked: 2859 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Backup from a restored machine very slow
When Veeam performs "agentless" restores (which I think are the only possible restores with ESXi) the it appears to loose the ability to skip "empty blocks". To verify this behavior I performed this simple test:
1. Created a new 40GB VM, completely empty (no OS)
2. Ran a test full backup, took 1:08sec, reported 599MB/sec, very little I/O reported from storage
3. Delete test VM
4. Restore from Veeam using ESX agent mode
5. Ran a test full backup, took 2:40 sec. reported 252MB/sec, very little I/O reported from storage
6. Restore from Veeam using ESX agentless mode
7. Ran a test full backup, took 10:47 sec. reported 63MB/sec, storage array reports 10 minutes of sustained 70MB/sec transfer
Basically, if you perform a restore of a VM that has lots of empty space using agentless mode, all of that empty space becomes "used". I think you can work around this by forcing the restore to thin and then using VMware tools to convert the restored VMDK to thick, but I haven't done that in a while. We still run full ESX so we just make sure our restores are done with the agent.
1. Created a new 40GB VM, completely empty (no OS)
2. Ran a test full backup, took 1:08sec, reported 599MB/sec, very little I/O reported from storage
3. Delete test VM
4. Restore from Veeam using ESX agent mode
5. Ran a test full backup, took 2:40 sec. reported 252MB/sec, very little I/O reported from storage
6. Restore from Veeam using ESX agentless mode
7. Ran a test full backup, took 10:47 sec. reported 63MB/sec, storage array reports 10 minutes of sustained 70MB/sec transfer
Basically, if you perform a restore of a VM that has lots of empty space using agentless mode, all of that empty space becomes "used". I think you can work around this by forcing the restore to thin and then using VMware tools to convert the restored VMDK to thick, but I haven't done that in a while. We still run full ESX so we just make sure our restores are done with the agent.
-
- Chief Product Officer
- Posts: 31754
- Liked: 7259 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup from a restored machine very slow
Looks like full pass CBT failure at 7, which made us scan through all blocks. Usually does not have to deal with restores... full pass CBT fails all the time on random VMs for no good reason, including freshly created VMs. More info > CBT on first (full) backup, how?
-
- VP, Product Management
- Posts: 6032
- Liked: 2859 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Backup from a restored machine very slow
Perhaps, but in my environment this is 100% reproducible after an agentless restore (discovered this a very long time ago), thus the responses to the OP that this shouldn't happen are incorrect, right?
If it's truly the "CBT on first full backup" issue should it correct itself after an incremental for the next full backup?
If it's truly the "CBT on first full backup" issue should it correct itself after an incremental for the next full backup?
-
- Chief Product Officer
- Posts: 31754
- Liked: 7259 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup from a restored machine very slow
No, it will never start working unless you do certain sequence of actions involving manually disabling/enabling CBT and powering VM off/on many times (in certain order). This is just an idea though, need to see log files to say if initial CBT worked or not in this case.
-
- VP, Product Management
- Posts: 6032
- Liked: 2859 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Backup from a restored machine very slow
OK, I'll send you my logs offline.
-
- Chief Product Officer
- Posts: 31754
- Liked: 7259 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
-
- VP, Product Management
- Posts: 6032
- Liked: 2859 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Backup from a restored machine very slow
I'm pretty confident this issue is NOT related to CBT at all. I think the restore of a VM using Veeam via agentless always creates a VMDK that appears to VMware as being "fully" used, thus requiring a read of the entire VMDK during a full backup pass. Why do I think this? Here are my reasons:
1. As far as I know, CBT is not required for "unused block elimination", Veeam can skip unused blocks on VMDK's on ESX 3.x hosts, which don't even have this feature
2. I cannot make a "thin" disk from a VMDK that was restored by Veeam as think via the agentless method. If I attempt to do this I end up with a "thin" disk that is just as big as the "thick" disk because VMware thinks the disk uses the entire space.
3. These issues do not occur when using "agent based" restore mode on ESX
1. As far as I know, CBT is not required for "unused block elimination", Veeam can skip unused blocks on VMDK's on ESX 3.x hosts, which don't even have this feature
2. I cannot make a "thin" disk from a VMDK that was restored by Veeam as think via the agentless method. If I attempt to do this I end up with a "thin" disk that is just as big as the "thick" disk because VMware thinks the disk uses the entire space.
3. These issues do not occur when using "agent based" restore mode on ESX
-
- Chief Product Officer
- Posts: 31754
- Liked: 7259 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Backup from a restored machine very slow
You are right, I just came back from devs and opened the topic to write the same thing. It is a matter of how thick disk is created, in agentless mode thick disk is uploaded as file. Where as in agent-based mode, and in agentless mode in case of thin disks, disk is first created as disk, and is then "stuffed" with the content, including zeroed blocks. But for VMware, all block now appear to have content, so initial CBT will return all blocks including those containing zeroes. Thanks.
-
- VP, Product Management
- Posts: 6032
- Liked: 2859 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Backup from a restored machine very slow
Well, actually, I was wrong on point #2. If I use vmkfstools to clone the "thick" disk that's full of zero's to a thin disk, I will actually end up with a mostly empty "thin" disk. It appears that vmkfstools is smart enough to recognize zero'd blocks and not copy them to the "thin" disk.
The easiest manual workaround seems to be to simply restore the VM as "forced thin" and, if you want "thick" disk, simply convert to "thick" after the restore via command line. Veeam might want to consider using this restore method as the "default" method for restoring files via agentless mode, you could restore as thin and convert to thick after the restore was complete. This also has the added advantage of typically being much faster (thin restores seem much faster than thick restores for agentless mode).
Of course, I've somewhat hijacked this thread, so I don't even know for sure if this is the issue the OP is referring to, but it's something I've noticed and thought it might be what he was referring too. I guess we should wait to hear from him.
The easiest manual workaround seems to be to simply restore the VM as "forced thin" and, if you want "thick" disk, simply convert to "thick" after the restore via command line. Veeam might want to consider using this restore method as the "default" method for restoring files via agentless mode, you could restore as thin and convert to thick after the restore was complete. This also has the added advantage of typically being much faster (thin restores seem much faster than thick restores for agentless mode).
Of course, I've somewhat hijacked this thread, so I don't even know for sure if this is the issue the OP is referring to, but it's something I've noticed and thought it might be what he was referring too. I guess we should wait to hear from him.
-
- Novice
- Posts: 4
- Liked: never
- Joined: Feb 10, 2011 6:31 am
- Full Name: Admin
Re: Backup from a restored machine very slow
Tom, thank you very much for clarification.
I tested the restore with "thin". After restore i changed the format with storage vMotion (quicker than console input).
It works!
Greetings
Joerg
I tested the restore with "thin". After restore i changed the format with storage vMotion (quicker than console input).
It works!
Greetings
Joerg
-
- VP, Product Management
- Posts: 6032
- Liked: 2859 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Backup from a restored machine very slow
You can also use the datastore browsers "inflate" option, but the VM has to be powered off, which is the great thing about using storage vMotion to do it. Glad to know that solved your problem, well, at least explained the problem.
Who is online
Users browsing this forum: Bing [Bot] and 88 guests