Veeam 7 Upgrade - BackupExec Now Slow

[Berniebgf]

Hi All,

Recently upgraded a customer to Veeam 7 from 6.5, they use backupexec 2010 to backup Veeam files to tape .

We upgraded to Veeam 7 making no changes to jobs in Veeam or BackupExec. (only enabled parallel processing)

Now some of the backups to tape (2 out of 3 jobs) are taking twice as long, we are trying to work through the issue and have tried the following.

1. Check for overlapping disk / tape jobs
2. Checked for IO load during backup to tape jobs
3. implemented registry change to Veeam for "tape drive discovery interval" (think that's it)

We can see from the BackupExec logs that the job throughput has halved since upgrading to Veeam 7.

This customer is using the Vendor drivers for the library and not the Symantec Drivers, so Veeam 7 is picking up the tape drive as well.

My next step was to update the devices drivers to Symantec drivers and reboot the system to clear the library out of Veeam 7 (hopefully).

Anyone seen this before? any advice?

thanks

Bernie.

[Vitaliy S.]

Hi Bernard,

If you're not planning to use Veeam to offload your backups to tape, then changing tape drivers might do the trick, though I haven't seen this behavior reported before...

BTW, does BE has any information on what process has slowed down? Is it an actual write process or there other steps (for example, tape preparation) that has affected overall performance?

Thanks!

[veremin]

Hi, Bernie,

Implementing the said registry key, as well as, installing Symantec proprietary drivers should completely exclude Veeam from any tape activities. So, if after these steps you still experience degraded performance, then, Symantec support team will be the best place to raise this issue.

Thanks.

[Berniebgf]

Thanks for the responses,

Just looking in more detail at the information I have from the customer, I am thinking it may be a Disk IO workload issue.. (But still need to clean up the drivers).

There are three jobs that run...

1. 7:30pm -> 1am --> ~5GB Min
2. 11pm -> 12pm (Next day!) --> ~2.5GB min
3. 3am -> 9am --> 2.5GB Min.

So the first job runs at full speed but the next two drop by half (before Veeam 7 they were all 5GB min) I see there is tape job overlap which is something to look at.
Previous these jobs all ran fine (@5GBmin) even with some overlap on the tape jobs..

I wonder if Veeam 7 has some changes that are causing jobs to run at this later time.... Will check out the Job history and see what is going on at these later times and see if there is any other IO workload going on.

Will report back.

Bernie

[Berniebgf]

To answer you question Vitaliy:

It is not the verify that is taking time, it is the data write workload. I have no more info than that at this stage.

thanks

Bernie.

[veremin]

As mentioned, with the said tricks implemented, Veeam should not be involved in tape activities. So, it shouldn’t cause any effect on Symantec jobs performance at all.

Anyway, once you collect a bit more information from customer, we will be able to analyze it more thoroughly. Thanks.

[Berniebgf]

Hi all,

Just a quick update, this is definitely an IO workload issue driven by Veeam's "parallel processing" and its ability to push allot of concurrent IO.
The customer has staggered Veeam backups during the day and night with schedules to not overlap the disk jobs with multiple tape backup jobs in BackupExec.

The additional IO workload due to Parallel Processing on the source Veeam repository is now effecting tape backups which obviously feed from same disk system (backing up Veeam files to tape) where previously it did not.

Schedules are being modified to "optimize" the IO workload balance between Veeam backup and BackupExec.

Bernie

[Berniebgf]

Hi all,

Update on this issue, turns out the issue was not related to the IO workload by Parallel processing. Turning Parallel processing off did not resolve the issue.

The issue (as mentioned previously) was the fact that the library element was seen by Veeam.

Tape library drivers were as follows
1. Symantec drivers for Tape drives
2. Vendor Drivers for Library / changer element

Veeam could not see the drives but could see the library element, we had the registry change applied to have long frequency between Veeam polling devices....... Issue seen.

Change the Library / Changer to "Unknown Media Changer", as soon as we did this in device manager Veeam automatically listed the library as "Off line". We could then remove the library element from Veeam.

After making this change the backups ran correctly with no performance issues, so even with the registry change Veeam looked to be "polling" the library element for its status which was effecting Tape performance (Don't know how....)

Hope this helps.

NOTE: No changes were required to the BackupExec Environment after these changes were made.

regards

Bernie

[ebarclay5]

I was dealing with the exact same issue. Changing the device driver and removing the library from Veeam worked for me. Prior to upgrading Veeam the Backup Exec tape job was running ~4.5 GB/min and taking around 25 hours. After the update the job dropped to ~2.8 GB/min and ran around 37-38 hours! Thanks Bernie