Backing up large vCloud Environment - problem over time

[Dan Feeley]

Greetings,

We are new to Veeam and are excited to see it supports vCloud, which is what we are evaluating. We are using vCloud in a large software testing lab deployment with 500+ vApp configurations within a given org in vCloud. This is a highly leveraged "fast and thin" provisioned environment.

Our target backup in Veeam is vcloud->Org->Org VDC. This will backup all vApps under the given VDC since it is not possible for us to itemize or identify 500+ vApps that are constantly changing.

We are performing SAN to SAN backups. It works really well the first few days. Our initial full backup ran only 17 hours. However, over time (about two weeks) the entire process has slowed down significantly. Its not the actual backup of vApps that is the performance problem, or the SAN, or anything infrastructure. Our throughput stats remain the same. What seems to be happening is Veeam is keeping track of many, many vApps that have now been deleted under the Org VDC and its lagging the system out.

So now the initial "building vapp list" which took only 20 minutes on the very first full backup is now taking 4.5 hours. What I think is happening is Veeam is looking for approximately 120 vApps that were their prior, but now have been deleted. It clearly states "vapp-name is no longer accessable. Please make sure this was intentional." And I also think the lagging between switching vapps during backup as the backup now takes 40 hours and not 17. Again, the actual throughput stats of a given backup is fantastic, but the time it takes to switch between vApps has grown and grown.

Is there a setting or anything that can be done to tell Veeam to ignore a vApp that is now missing, when it fact it is not missing but intentionally removed? Remember, we cannot itemize our vApp list in the initial backup definition and we must choose a higher level target object (ie: Org VDC).

Any help would be appreciated.

Thanks.

[Vitaliy S.]

Hello Dan,

As far as I know there is no such setting or registry key, but can you please open a support case and send us job log for further investigation? If the problem is indeed in those vApps, then we can potentially include a hotfix for this problem in the next update of Veeam B&R.

Thank you.

[Dan Feeley]

Greetings,

I've opened up a case (case #00446068). I've uploaded my job logs from the weekend full which shows a 4.5 hour vApp calculation time. My first full backup log has a 47 minute vApp calculation time. This extended 4 hour calculation also occurs on dial incremental backups too.

As a means to troubleshoot the problem with support we have removed the target "org VDC", refreshed and re-added the same "org VDC". The assumption here is that it will be reduced back to a quick calculation time. So I should have more results to provide within a few hours.

Again the time lag seemed to be not only in the initial calculation of the entire job but also switching between individual vApps.

I'll post the results here.

Thanks.

[Dan Feeley]

So we tried to remove and "refresh" (using a little refresh button) the target "org VDC" to see if that would shake off the fact that its holding onto old vApps. It did not. It still took about 4.5 hours to calculate the entire vApp list. You could actually watch the vApp list build one vApp at a time, with about a 20-30 second delay between each vApp name to appear in the log window.

Just for another test I cloned the job. The entire org VDC vApp list of over 500 vApps was built in 5 minutes on the cloned job! vApps were added about 5-8 vApps per second to the job list.

So is there a way to link a cloned job to the original source backup files? I got the impression that it was treated as a completely unique job, and even with my feeble attempt at renaming the cloned job to the same as the source, it incremented my storage directory by adding a "_1" to the folder name. If there is a way to link the clone back to the original files then maybe it can pick up where it left off with its backup history, but in this clean state? I ask because I got the sense that the clone was running a full backup and not its scheduled incremental for today. Again, I am new to Veeam B&R to this point, so forgive me if I am a bit lost in its operations.

I'll report this in my case too.

Thanks.

[tsightler]

Cloning a job creates a new backup job that, if left alone, will create a new full backup, however, you should be able to clone the job, delete the old job, then use the "Map backup" feature to map the new job to the old backup file (this is on the "Storage" section of the job properties, you should see a "Map Backup" link under the Repository selection pull-down).

[veremin]

Since running cloned job is virtually an equivalent of running a new full backup, I wonder whether you tried to run a new active full backup of original job and see whether the issue shows up or not. Thanks.

[Dan Feeley]

After speaking with support we are going to do a webex and come up with some ideas, but it appears the problem maybe related to the VM retention policy, or just the sheer size of our vCloud environment and the history that rapidly develops with the retention period x VM retention policy.

Cloning the job and remapping back doesn't help. only creating a net-new point in time backup seems to fix it, but then I really am making changes quite often.

Honestly if you could add to your product the option to backup "catalogs" vs "my cloud" workspaces in vCloud that would be a tremendous improvement. As it stands right now I can only backup at the "org VDC" level because I cannot target the vApp names in any meaningful way. My structure looks similar to this:

vCloud Director
--> vCloudDirector (cell - 1 object)
-->--> Lab (target organization - 1 object)
-->-->--> Lab VDC ( VDC for the lab org - 1 object)
-->-->-->--> ALL VAPPS UNDER LAB VDC (500+ objects changing all the time)

So instead of displaying all the vApps as a flat structure you could add
-->-->-->-->-->Catalogs (25 catalogs in my example)
-->-->-->-->-->--> Catalog name (1 object)
-->-->-->-->-->-->--> ALL VAPPS UNDER GIVEN CATALOG ( 20-30 vApps in my example)

You could also treat the "my cloud" vApps as its own catagory/tree view, so the vApps in peoples running "my cloud" spaces are not treated the same as the catalogs.

This would probably allow me to break down the problem to something more manageable.

[dellock6]

Hi Dan, dealing with retention of deleted VMs in our vCloud environment was an issue for us too, even before v7 when we where mapping jobs to underlying resource pools in vCenter. One quick solution is to lower the default parameter of 14 days to something really low, like 1 or 2 days, unless there is a need in the SLA to keep also deleted VMs for longer times.
Other than that, I'm interested too in this topic, keep us updated!

Luca.

[veremin]

Though, it' not recommended to set 1 as number of “deleted VM retention period”, since under certain circumstances, during retries, for instance, it might produce certain issues. So, 2 would be good enough. Thanks.