Snapshot and ESXi host config backup suggestions

[seadave]

Been using Veeam for about a year now. While it has definitely narrowed our backup window and significantly reduced the amount of data that traverses our production and backup segments, we have had some stability issues linked to Veeam's bull in a china shop approach to snapshots out of the box and how it processes disk consolidation. I thought I would provide some suggestions that might make it into v10 and IMHO improve the product.

First off, the easiest option I think would be to include an option to do periodic ESXi host config backups from Veeam. Right now I use a script to do this available from this site:

http://www.shogan.co.uk/vmware/another- ... ed-to-1-3/

It uses PowerCLI and pops up a nice little GUI that will create a TAR/GZ file of the host's settings (IP, VLANs, Networking, Storage, etc). It has on multiple occasions saved me hours of reconfig times when rebuilding a host for various reasons (the latest was a failed USB device that the host was booting off of). Anyway, I think allowing Veeam to run the esxcli config export command and storing it in the Veeam repository would save a lot of people lots of time and make for a more complete DR system.

The other issues I have had is with snapshots. The first issue is that out of the box, the local and virtual backup proxies that Veeam uses defaults to 12 concurrent tasks. This equates to the max number of snapshots that can be run at once (only took me a month of working with tech support to figure that one out, but I digress). Trying to do that many snaps at once killed our Equallogic based SAN. Just like trying to copy multiple files off a slow disk, it also massively slowed down the process. We scaled this back to 4 concurrent tasks and all is; mostly, now well. I think the default should be reduced to prevent initial deployment issues and some type of bold notice in the manual/release notes that this is a major tuning factor and should be reviewed (If it is and I've missed this, I'll hang my head in shame, but this wasn't made obvious to me during our deployment).

Finally, I've been bitten several times with excessively large in duration and size snapshots (after searching the forums, I do not appear to be alone when it comes to this). I'm not sure if this is possible, but it would be great if Veeam had some sort of limit to cancel a Backup/SNAP if it consumed more than a percentage of remaining free space. Maybe this isn't possible due to how the VMWare API works, but it seems like in combination with Veeam One, the job could be canceled if remaining free space was less than say 5%. We run a lot of 1.5TB to 2TB VMs on our network. The only time our Exchange Server experienced an unplanned outage was due to this. We had exported a large number of emails during an archive system deployment and the next time Veeam ran, it created a massive snapshot and filled up the LUN. We monitor closely and caught it before damage was done and we were able to grow the LUN, but it had at least 20% of free space before this happened so we weren't expecting any problems. I realize this is one of the shortcomings of VM based compared to native SAN snapshots, but some of us don't have the option to leverage those. Anyway, any software application that will cause a fault like this because it isn't monitoring available resources can be made better by doing so in my opinion.

Looking forward to v9!

[seadave]

Another suggestion. I like to chain my jobs. Job A starts at 7, when that finishes, job B starts, etc. It would be great if you could sort the jobs based on that instead of Name, Type, Status, Last Result, etc. I would expect sorting via "Next Run" would do this but it appears to use a strange criteria:

List non scheduled jobs first, then jobs assigned a specific time, then remaining chained jobs based on their After "Job Name" alphabetical listing. Would be better if it listed them in chaining order. Would make it easier to adjust.

[foggy]

The other issues I have had is with snapshots. The first issue is that out of the box, the local and virtual backup proxies that Veeam uses defaults to 12 concurrent tasks. This equates to the max number of snapshots that can be run at once (only took me a month of working with tech support to figure that one out, but I digress). Trying to do that many snaps at once killed our Equallogic based SAN. Just like trying to copy multiple files off a slow disk, it also massively slowed down the process. We scaled this back to 4 concurrent tasks and all is; mostly, now well. I think the default should be reduced to prevent initial deployment issues and some type of bold notice in the manual/release notes that this is a major tuning factor and should be reviewed (If it is and I've missed this, I'll hang my head in shame, but this wasn't made obvious to me during our deployment).

Looks like this is what storage latency control was designed for.

Another suggestion. I like to chain my jobs.

We typically do not recommend chaining, due to some undesired consequences it incurs.

[seadave]

I've tried storage latency control with limited to no improvement. Perhaps I'm not leveraging correctly but doesn't appear to address the issue fully. I'll give it another look. Your post may explain another issue we have experienced (non-consistent Synthetic Fulls being created for Tape jobs). So in our case we have 10 jobs. Should we schedule each one to start within a minute of each other so they simply queue up and then wait for resources? Seems like you won't have a consistent processing of jobs that way. Some will start before others depending on how long the job takes on a particular day. I do see how this would narrow the backup window though. I will give this a shot tonight.

Is the explanation as to why you wouldn't want to chain as explained in the article you linked to in the manual? If not it should be.

[seadave]

Looks like this is what storage latency control was designed for.

I've tried to turn this on at 20%. A little nervous that I can't be more granular but looks like that is only available to Ent Plus customers.

Perhaps you could explain the relationship between "Enable parallel processing" in Global Options and "Max Concurrent Tasks" in the Veeam Proxy.

Will Veeam still attempt multiple concurrent tasks with "Enable parallel processing" disabled, or does disabling that effectively make "Max Concurrent Tasks" equal to 1?

Thanks

[foggy]

Should we schedule each one to start within a minute of each other so they simply queue up and then wait for resources?

Right, this is how we recommend scheduling jobs letting our automatic resource scheduler do the job.

Seems like you won't have a consistent processing of jobs that way. Some will start before others depending on how long the job takes on a particular day.

Not sure I follow this. Jobs will start according to their schedule, priority is given to the tasks of the job that has started earlier.

Is the explanation as to why you wouldn't want to chain as explained in the article you linked to in the manual? If not it should be.

Seems it's not there, I will forward your request to the documentation team. Thanks.

I've tried to turn this on at 20%. A little nervous that I can't be more granular but looks like that is only available to Ent Plus customers.

Right, custom thresholds are supported in Enterprise Plus only.

Perhaps you could explain the relationship between "Enable parallel processing" in Global Options and "Max Concurrent Tasks" in the Veeam Proxy.

Will Veeam still attempt multiple concurrent tasks with "Enable parallel processing" disabled, or does disabling that effectively make "Max Concurrent Tasks" equal to 1?

Parallel processing enables parallel processing of VMs and disks within a single job. With parallel processing disabled, all VMs (and disks) in the job will be processed sequentially, however a single proxy still can process tasks from different jobs in parallel. The amount of concurrent tasks per proxy is effective in any case.

[seadave]

Let see if I understand this.

So let's say you have two jobs. One with 4 VMs and one with 2 VMs. The number of concurrent tasks in your proxy is set to 4. The first job is scheduled to start at 18:00 and the second at 18:01.

With parallel processing enabled, the first four VMs in the first job are processed and the second job is queued until one of the first for VMs in the first job is completed processing (with the assumption that will take longer than one minute).

With parallel processing disabled, the first VM of four in the first job is processed at 18:00, and the first VM of two in the second job is processed at 18:01.

Is that correct?

[foggy]

Correct (with the assumption that all VMs in your example have single disk).

[seadave]

@foggy

Based on the link you sent (which is good, thanks), I might suggest that:

"Task limitation settings define the number of tasks that the backup infrastructure component can perform concurrently, or at the same moment. The maximum number of concurrent tasks that a backup proxy or repository can perform depends on the number of CPU cores available on this backup proxy or backup repository. It is strongly recommended that you assign task limitation settings using the following rule: 1 task = 1 CPU core. For example, if a backup proxy has 4 CPU cores, it is recommended that you limit the number of concurrent tasks for this backup proxy to 4."

is changed to say:

"Task limitation settings define the number of tasks that the backup infrastructure component can perform concurrently, or at the same moment. The maximum number of concurrent tasks that a backup proxy or repository can perform depends on the number of CPU cores available on this backup proxy or backup repository and your storage throughput. It is strongly recommended that you assign task limitation settings using the following rule: 1 task = 1 CPU core. For example, if a backup proxy has 4 CPU cores, it is recommended that you limit the number of concurrent tasks for this backup proxy to 4. Even if this is the case, you should still closely monitor your storage throughput because if your SAN is not able to keep up with the number of tasks you assign, it will be the limiting factor and not the number of cores you assign to your proxy. Test and benchmark to determine the optimal setting."

[foggy]

Thanks for the suggestion. I will pass this to documentation team as well, however repository tasks is a bit different story, described in adjacent chapter.

[Shogan]

Been using Veeam for about a year now. While it has definitely narrowed our backup window and significantly reduced the amount of data that traverses our production and backup segments, we have had some stability issues linked to Veeam's bull in a china shop approach to snapshots out of the box and how it processes disk consolidation. I thought I would provide some suggestions that might make it into v10 and IMHO improve the product.

First off, the easiest option I think would be to include an option to do periodic ESXi host config backups from Veeam. Right now I use a script to do this available from this site:

http://www.shogan.co.uk/vmware/another- ... ed-to-1-3/

It uses PowerCLI and pops up a nice little GUI that will create a TAR/GZ file of the host's settings (IP, VLANs, Networking, Storage, etc). It has on multiple occasions saved me hours of reconfig times when rebuilding a host for various reasons (the latest was a failed USB device that the host was booting off of). Anyway, I think allowing Veeam to run the esxcli config export command and storing it in the Veeam repository would save a lot of people lots of time and make for a more complete DR system.

Hi seadave,

I was reading my morning e-mail and read the Veeam newsletter and noticed "ESXi Host Config Backup" and it caught my attention leading me to this thread. Great to see the tool is still being used and of some use too! I've pretty much stopped developing it further (although it fairly simple PowerCLI) so I'm glad to see it still works.

Thanks for the mention