Comprehensive data protection for all workloads
Post Reply
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Very slow backups

Post by Yuki »

So we've been battling this issue for the last 2 weeks (case 00168486).

At random, our backups will become very slow. I'm looking at one right now:
Hard Disk 2 (2.0 TB) 438.6GB read at 8Mb/s [CBT] - 16:01:02
Thats 16 hours!

This is blowing our backup window and causes backup-replication scheduling conflicts. I've contacted support and have spoken to 3 different level 1 techs. All of them were very helpful and appeared to be willing to help, but after looking at the system they say it is working correctly.

We are doing Hot-Add backups to a NAS over gigabit link. Backup proxy is mostly idling (20% CPU utilization right now on 8 physical cores). NAS is at 3% CPU utilization. Network throughput is at 9MB/s. Veeam app says target is the bottleneck, but a 12 disk NAS QNAP TS-EC1279-RP has no problems performing any other file operation at much greater speeds.

Is there a way to get in touch with level 2 and hove someone take a deeper look into this?
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow backups

Post by tsightler »

You say this is "at random". That's somewhat concerning as that would imply that something in the environment is unique during those "random" times. Are the stats below from a full backup or an incremental run? What backup mode? What storage optimization settings? How much memory in the proxy?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow backups

Post by Vitaliy S. »

Yuki wrote:Is there a way to get in touch with level 2 and hove someone take a deeper look into this?
Yes, of course. Just ask your engineer to escalate this ticket to a higher tier.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

tsightler wrote:You say this is "at random". That's somewhat concerning as that would imply that something in the environment is unique during those "random" times. Are the stats below from a full backup or an incremental run? What backup mode? What storage optimization settings? How much memory in the proxy?
Reverse Incremental
Compression: Optimal
Optimize for: WAN or LAN (Depending on specific job) (we have one for off-site and one for on-site)
Dedupe: ON

Proxy has 8GB assigned, 2.6GB used.

Backup kicks off in the evening and normally finishes in a few hours before people get back into the office, but it looks like this job is going to run throughout the day in addition to the night and morning it took already.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow backups

Post by tsightler »

Hard Disk 2 (2.0 TB) 438.6GB read at 8Mb/s [CBT] - 16:01:02
Reverse Incremental
So here's my first suspect. You say this is a reverse incremental job, but so far this system has read 438.6GB of changes from this disk. That means almost 25% of this disk blocks are changed from the last backup. Since you're using reverse incremental, this means that 438.6GB of data (well, probably less after compression) was written to the NAS, but and equal number of blocks had to be read from the NAS, and then re-written to the NAS into the rollback file. That's a lot of random I/O on the NAS, and likely why the system is showing the "target" as the bottleneck.

Is there some reason why this particular system would have such a high change rate, perhaps an Exchange or SQL server or something, or maybe a defrag run on a fileserver.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

We've had to skip one or two days of backups while we were performing updates on the Dell server (bug fix in drive firmware) and this VM is a file server.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow backups

Post by tsightler »

OK, but 438.6GB of change on a 2TB server is huge, especially for a file server. Most file servers largely hold files that rarely change. I'd still suspect some process that is causing a lot of change, perhaps a defrag or virus scan. That would explain the "random" nature of it.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

Actually it has 10TB of space spanned over 5 VMDKs (VMware doesn't support more than 2TB in vmfs)., actual space used is 2.4TB. I will check the Anti-virus and defrag settings, but i don't think we got those on the file server.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow backups

Post by tsightler » 1 person likes this post

It may not be defrag or anti-virus, but something had to be triggering the "random" nature of the problem so just pointing out that things that seem "random" rarely are, we just haven't found the pattern/cause yet, and that's one thing to check. When you said earlier that the proxy has 8GB of RAM, but 2.6GB used, where is the "used" figure coming from? Windows task manager? Can you look at the size of the VeeamAgent.exe processes at the time.

Also, I'd suggest looking back through your history and seeing how change there is on a night that finishes normally compared to a night that is too slow.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

1) So the defrag was in fact ON on weekly basis (Win Server 2012).
2) Yes, 2.6GB was for the entire system. VeeamAgent is around 300MB
3) Previous incremental was 200GB+, one before was 240GB, one before was 240GB, one before was 100GB.....
4) This machine is a 2012 VM with dedupe ON. Could this cause any issues?

After combing through the forums i'm converned that ShadowCopy maybe another cause of this? Can anyone confirm? Although i would have to disable it as it provides for much easier and faster restores.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow backups

Post by tsightler » 1 person likes this post

Veeam is an image based backup product, so anything that changes blocks will increase the amount of data that must be backed up. Weekly defrag, nightly dedupe process, shadow copies all increase the number of blocks that change compared to the actual amount of data changed on the filesystem. Defrag is typically and especially big one since it will move lots of block around.

Note that there is nothing particularly "wrong" with this, but that the more change that the is, the slower the backup will be. This is especially true for reverse incremental backups because, for every block that changes on the source, we must read/write 3x that much data on the target. Because this is "random" I/O rather than streaming I/O, this can quickly hit the IOP limit of lower end targets. For example, in the case above, you have 438GB of data, that's almost 1.3TB of data that would be read/written to the target with random I/O.

If you have the space, you can perhaps choose to use forward incremental backups instead. This will likely run much, much faster, since it's a simple streaming write, but will unfortunately use significantly more storage space on the target and require occasional full backups.

Also, I made the assumption that you are writing to the NAS with CIFS, is that true? There are some performance disadvantages to this mode with larger backup files and many NAS devices will perform poorly once the file significantly exceeds it's cache size so perhaps this is a cause. Is this VM in a job by itself? How big is the VBK file at this point?
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

backup is done via CIFS/SMB to a nas that has a single 20TB volume. VBK is at 2.5TB roughtly right now.

In one job it is one of two VMs being backed up in another it is one of 7 VMs being backed up. NAS memory usage doesn't appear to be the problem (600MB used out of 4GB).

Forward incremental with a pereodic full is not an option as we would ahve to push these 2.5TB across 100mbit link to an off-site location. Just not possible.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow backups

Post by tsightler »

So you're backing up locally and then replicating remotely? How are you doing the replication? Just trying to make sure I understand the entire layout.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

we have 2 backups and 2 replications (one local and one remote for each).
Local nas for backups
Local server for replicas
Remote nas for backups
Remote server for replicas


We've looked into doing only one backup and then using rsync or similar to transfer files, that that created more issues than it was solving. We may abandon local backups completely and go to local+remote replicas and remote backup.
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Very slow backups

Post by chrisdearden »

I've seen shadow copies cause large amounts of changed blocks on file servers before.
lohelle
Service Provider
Posts: 77
Liked: 15 times
Joined: Jun 03, 2009 7:45 am
Full Name: Lars O Helle
Contact:

Re: Very slow backups

Post by lohelle »

Windows Server 2012 deduplication will problably change A LOT OF blocks during nightly deduplication jobs.
Hubanero
Lurker
Posts: 1
Liked: never
Joined: May 03, 2012 3:17 pm
Full Name: Chuck Hubbs
Contact:

Re: Very slow backups

Post by Hubanero »

Make sure you have plenty of space on your C:. We had indexes filling up that drive that caused restores and backups to slow to a crawl.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

Do the number of VMs of total BKP file size/SQL data have any effect on reverse incremental speed?

Shell we break out large file servers with a lot of changes into separate jobs? would that have any effect on speed with which the backups are processed for individual VMs?

lohelle - we have space, so we could disable dedupe, but i would like to hear from Veeam that dedupe and Veeam don't play nice together just to make sure before we do this. Right now dedupe on the main file volume claims to save us about 800GB.

Hubanero - on Veeam server/Agent on file server VM?
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow backups

Post by tsightler »

Yuki wrote:...but i would like to hear from Veeam that dedupe and Veeam don't play nice together just to make sure before we do this. Right now dedupe on the main file volume claims to save us about 800GB.
It doesn't have anything to do with "playing nice together" but it's certainly very likely that Windows 2012 dedupe will cause a lot of changed blocks, and the more change blocks, the longer the backups will take. Just like everything in life, it's a balance. Are you willing to have backups that take longer to save 800GB of space. One option might be to run dedupe only once a week rather than nightly.

Job sizes should be kept to a reasonable size. Large VMs should be in a job by itself, and block size can have a significant impact on performance. General recommendations for maximum job sizes:

WAN Optimization - 256KB Blocks - Max of 2TB of VMs per job
LAN Optimization - 512KB Blocks - Max of 4TB of VMs per job
Local Optimization - 1MB Blocks - Max of 8TB of VMs per job

Note that these are not absolute hard limits, but field experience shows that performance will degrade as the amount of data in the job exceeds these amounts. When using CIFS, I actually suggest cutting the sizes above in half.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

WAN Optimization - 256KB Blocks - Max of 2TB of VMs per job
LAN Optimization - 512KB Blocks - Max of 4TB of VMs per job
Local Optimization - 1MB Blocks - Max of 8TB of VMs per job
Is that recommendation for assigned disk space to VM or for resulting backup file (a 2TB assigned disk may end up being 500GB for example).

on last night's reverse incremental got this:

Client error: Banks pool is full. Number of banks in pool: [744]

Now when we try to re-run the job it complains about failing to delete OIB files.

Last time this happened we had to remove all backup files and start a new backup chain.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow backups

Post by tsightler »

I consider this to be the recommendations for the total raw size of the VMs in the job, not the size of the backup file. "Banks pool full" and "Failure to allocate memory" are exactly the type of errors you will see when nearing the limits. The exact limits are somewhat difficult to predict as they seem to vary based on the change rate and length of chain. If you have a lot of change and long retention I would normally suggest being well under these sizes. Actually, I always prefer being well under these as I don't think it's great for performance to be near the maximum, and data sizes will pretty much always grow over time. Of course, I expect that we will also continue to improve our scaling at these sizes, but these suggestions are based on field results.

If you want to use these bigger job sizes, it's really no problem, but I would suggest sticking with Local optimization.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

Ok, so i will need to kill the existing single backup job and then split the VMs across several jobs and set optimization to Local.

The initial seed time is going to take a long time.....
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Very slow backups

Post by yizhar »

HI

Two more things to look for:

1. PLease use a tool like locate32 to search the file server and identify possible cause for the high change rate:
http://locate32.cogit.net/
Look for something like:
files with modified time = last few days, and size > 100mb
I suspect that you are storing some kind(s) of backup files on the file server VM, for example SQL maintanance plan, workstations backup,
or other servers that use the VM shared folders as target for some kind of daily backup (not related to veeam).
Such files and activity will product the large change rate on the VM, and if you also have dedup/defrag it may cause additional impact.
If you have such scenario, you should consider alternative approach.
For example having a dedicated backup/archive server on different VM, which is used only for that purpose but not for "regular" file services.

2. You mention that you have 4 backup processes with Veeam:
local backup, local replica, remote backup, remote replica.
You should make sure that you dont have conflicts or overlapping.
For example - when you backup the big file server locally, is any other process accessing the NAS or using backup proxy at the same time?
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

yizhar,

1) this server stores scientific data, no workstation backups or similar. We've made sure. Our instruments on occasion can produce a lot of data, but the latest jump in data is probably due to people moving files from old server to the new. That's the only explanation we can come up with.

2) they don't overlap, we've got scheduling figured out and will be getting rid of local backup. Only local+remote replication and remote backup will remain.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Very slow backups

Post by veremin »

Moreover, don’t forget that the problem of backup proxy being used at the same time by multiple jobs can be easily solved by specifying the number of concurrent tasks the backup proxy can handle in parallel. If this value is exceeded, the backup proxy will not start a new task until one of the current tasks is finished.

Thanks.
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Very slow backups

Post by yizhar »

Hi.

1. I still suggest a test with tool like locate32 or similar.

2. Also check for "access time" related issues.
for example if you have daily anti virus scan, this will change access time for all files and might cause file system modifications. I think that you can try turning off "access time" tracking in NTFS file system.
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow backups

Post by Yuki »

Our backup jobs run Sun-Friday. It is that Sunday job that takes abnormally long time. It seems that something is caushing changes on weekend. I've disabled defrag, shadow copies, there is no anti-virus or other scanners. The only feature that is enabled is the deduplication on the main file store volume. It gives us 1TB of space savings on 2.4TB data set. So we would like to keep this ON if possible.

Deduplication engine is set to automatic mode where is works when server is mostly idle and works on files older than 7 days. I've ran the locate32 tool as suggested below and included all files changes from friday till Monday and it found about 118GB worth of files. Yet the backup was 236GB (notice how it's twice the size of the changed data). Considering these files have been changed just recently, they would not be a part of the dedupe process (Set for files older than 7 days).

This server's backup took 13 hours for 236GB...
Post Reply

Who is online

Users browsing this forum: Amazon [Bot] and 132 guests