slow backup speed 02612135

[rmehta]

We are having massive issue trying to backup a server for customer; the server has Windows server veeam agent installed and protects OS and SQL; the backup job when runs takes over 20 hours and then it fails due to different reasons; we only have had very few successful backups and many failures; The server has SQL and some custom script populates the database. The business requirements are the backup should only run between certain hours. Even a weekly restore point for the SQL data is okay.
see below " In this particular sql server, tables are rebuild every day (meaning tables will be cleared and then flush with new sets of data). The main process start at around 1 am in the morning daily and once it finish, database (well… most of the time) become static. In term of backup requirement, I am also not expecting for a restore to a closest time before failure. I just need a valid backup file, it can be from yesterday backup or last few days or last week (even older than that may be ok in some cases). This backup file is used as starting point where I can start refresh job and it will rebuilds tables.
"
Reviewed the windows log on the source server against the failure and found that chages were too much for the VSS snapshot storage area limit and exceeded that causing backup failure; asked the customer to increase the free space; which they did; now the backup failed as the volume appeared to have gone too busy.
"Volsnap error: shadowcopy storage could not grow in time "; latest event log on the source server

So, we split the job and started backing up OS and volume hosting system database in one job and SQL data in another; the first job completes okay however the read is really slow; however the second job has been failing; as per the customer this job also seem to be interfering with the process/script they are running causing the server to slow down significantly and lead to an escalation.

Looking at the VBR server; the stat show the bottleneck is the target which is a decent physical server(22.MGZ 8 core) with memory(64GB) however it backs up 50 servers; and I did notice disk queue on a higher side; however; other jobs run fine although read speed on all the jobs is way less than what we are getting on other VBR servers(20MBs/ to 30MBs). Does anyone has
any idea if this read speed on a job indicate a bottle neck at the customer's environment or could it be influenced by the bottleneck at the repository. The VBR server is manage by us; however the customer environment is managed by a third party.

One of the SQL database is over 500GB

[HannesK]

Hello,
that looks like your support ID... could you please post the support case number (there are many cases and the open ones are not about agents or SQL)

protects OS and SQL; the backup job when runs takes over 20 hours

how much data do they have? How many volumes? Which backup mode is used? Volume based with two jobs?

the stat show the bottleneck

how many %? Is it always the same value (99%)?

decent physical serve

what kind of disks, RAID controller and file-system? Network speed is 10Gbit/s between agent and backup server?

could it be influenced by the bottleneck at the repository.

probably. But of course it could also be network or something else. It's just guessing on a forum... no real investigation

Best regards,
Hannes

[rmehta]

how much data do they have? (SQL data is apprx 1.1 TB(database + log files) including a large SQL database of over 500GB); the script as per the customer flushes the data and fills it with new data (800GB changed data)

How many volumes(3 volumes in SQL job; C:OS : E: has database and one log and I: has system data)? Which backup mode is used? Volume based with two jobs? yes both are volume based

Volume backup for both the jobs

how many %? Is it always the same value (99%)?
Load: Source 0% > Proxy 46% > Network 49% > Target 99%
Load: Source 0% > Proxy 33% > Network 65% > Target 65%

Backup that is completeing on the same error
Load: Source 15% > Proxy 67% > Network 71% > Target 98%
last job
Load: Source 58% > Proxy 17% > Network 39% > Target 79%

[rmehta]

We have a weekly backup configured for the SQL job and customer wants it to start at 8:30AM and stop at 10:30 PM; should not exceed the window however we are unable to backup the SQL volume even in 14 hours;

[rmehta]

I am going to run a backup to a local disk from that server to see what read speeds we get; this will hopefully give better insights on if the read speed is anyways influneced by the MA speed and network. They utilize Vcloud director to host their VMS and could not provide vcenter access hence we are using agent. again their production infra is managed by a third party provider

[rmehta]

unfortunately, it appears we cant run a local backup to local disk; is there a way to do that without affecting the existing backup chain that we have been already running

[HannesK]

1 TB data transferred in 20h is extremely slow... that's only 15MByte/s and sounds like a slow environment

volume backup is good

target at 79 or 65 is not really critical usually. 98 / 99 means relatively for sure that the server hardware is too slow. the software can do more than 1GByte/s . I suggest to check outside Veeam with disk (e.g. diskspd) and network speed test software (e.g. iperf)

with server license multiple jobs can be created. so one job points to the backup server and another job points to a local disk

anyway... as support is also working on the case, it makes little sense that the workload is doubled here and I suggest to continue working out the steps with support. My guess is, that it's an infrastructure issue and not a Veeam issue.

[rmehta]

i did run diskspd on the Veeam B&R server; it was really slow (6MB/s); I am planning on running another test with all the jobs disabled.
RAID 6 with 7 SAS Disk (10.91TB/Drive speed 12G)
Stipe Size: 256KB, Cache policy ARA|WT|DIO

The logic behind running the test with backup jobs and veeam sevrices diabled is that the VBR server has 17 groups of backup jobs ; plus 3 jobs which send all this data to cloud connect;
The CPU utilization is pretty high; 5 jobs run every 15 minutes; Some jobs run every hour; some job run once every night;although SQL is eating over majority of CPU

[HannesK]

WT means "write-though"... sounds like a home-user raid controller without battery / cache protection or misconfiguration.

[rmehta]

yes you were right; we changed the RAID setup to WB; and the issue resolved; the backup that was taking 6 hours is now finising in less than 2 hours. the Full backup on Saturday took just 5 hours.

[HannesK]

I just hope that the RAID controller has a battery cache... otherwise data loss is just a matter of time