REFS performance issue "workaround"

Availability for the Always-On Enterprise

Re: REFS performance issue "workaround"

Veeam Logoby dellock6 » Tue Jul 11, 2017 11:23 am 1 person likes this post

I believe the biggest benefit here is the monthly active full , which completely breaks the chain and restart a fresh one without involving too many deletions and transforms, which we have all understood by new are the most serious candidates to be the root problem. I don't see per-vm to be an issue to be honest.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com/en/
vExpert 2011-2012-2013-2014-2015-2016-2017-2018
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5297
Liked: 1435 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: REFS performance issue "workaround"

Veeam Logoby mkretzer » Tue Jul 11, 2017 4:17 pm

We did an experiment: We have a volume where fast clone has got really slow. Fast clone for one specific job took 6 hours instead of ~25 minutes it took in the first few weeks. Now after an active full fast clone two days after the active full (it only had to merge 2 backup points!) took 4 1/2 hours. So in our case active full did not help at all!
mkretzer
Expert
 
Posts: 382
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS performance issue "workaround"

Veeam Logoby DaStivi » Tue Jul 11, 2017 9:10 pm

hello
does anyone here having strange CPU issues with refs?
i noticed now a couple of times, that after a backup, when veeam starts to delete old vib files (perVM Backup) the backup repo server starts using all 8vCPUs and than locks up completly... no login is possible... right now i've restarted it a couple of times and after a few minutes the same behavoir started... i've removed the refs disks from my vm now and the server boots normally and i can login and right now i'm installing the latest windows cumulative updates...

btw. i'm on build 14393.1358 (and as far as i understand the "old" fixes should be in there allready: post247790.html#p247790)

i checked last syn full run of this backup repository, i sadly i've to notice the very long "fast clone" too, as it took 2h20min.. :shock:
DaStivi
Service Provider
 
Posts: 93
Liked: 9 times
Joined: Tue Jun 30, 2015 9:13 am
Full Name: Stephan Lang

Re: REFS performance issue "workaround"

Veeam Logoby kubimike » Tue Jul 11, 2017 11:16 pm

@DaStivi same issue I had, contact Veeam open a case get the beta refs driver. In this case it fixed it for me.
kubimike
Expert
 
Posts: 305
Liked: 37 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS performance issue "workaround"

Veeam Logoby kubimike » Wed Jul 12, 2017 2:35 pm

my fastclones are now taking 3 hours after fixing the retention issue. I am going through my backups before July 4th and it only took roughly an hour. Now it takes over 3 hours to make one.
kubimike
Expert
 
Posts: 305
Liked: 37 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS performance issue "workaround"

Veeam Logoby mkretzer » Thu Jul 13, 2017 3:58 am

The worst thing is; fast clones will continue to get slower and will never recover unless you reformat (or perhaps delete all the data). Thats why i had this idea for an workaround in this thread.
mkretzer
Expert
 
Posts: 382
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS performance issue "workaround"

Veeam Logoby jozne » Wed Oct 25, 2017 9:09 am

kubimike wrote:@DaStivi same issue I had, contact Veeam open a case get the beta refs driver. In this case it fixed it for me.

Hi , we are having the same issue. I made ReFS with 64k and stored some backups (4TB) to it for few days as incremental and now deleted all the backups through Veeam GUI
and the Server froze up with high CPU usage.

I try to ask if Support provides this beta driver.
jozne
Influencer
 
Posts: 16
Liked: 6 times
Joined: Wed Apr 18, 2012 6:55 pm
Full Name: Jari Haikonen

Re: REFS performance issue "workaround"

Veeam Logoby veeeammeupscotty » Fri Oct 27, 2017 8:02 pm

kubimike wrote:@DaStivi same issue I had, contact Veeam open a case get the beta refs driver. In this case it fixed it for me.

Veeam is telling me there is no beta driver. Can you tell me your case number?
veeeammeupscotty
Influencer
 
Posts: 15
Liked: 2 times
Joined: Fri May 05, 2017 3:06 pm
Full Name: JP

Re: REFS performance issue "workaround"

Veeam Logoby mkretzer » Sun Oct 29, 2017 9:17 am

From what i understand the beta thing was some months ago and the driver got integrated in the latest updates - which did not help by the way!
mkretzer
Expert
 
Posts: 382
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS performance issue "workaround"

Veeam Logoby veeeammeupscotty » Mon Oct 30, 2017 4:11 pm

Oh so you're referring to an actual Microsoft driver? Have you tried to install the hotfix mentioned in this thread which is supposed to be included in 9.5 U3? The symptoms there are different, high memory consumption, but the behavior makes me think it could be related. I have never seen the "TOO_MANY_REFERENCES" error in my environment, and the CPU peg issue has been sporadic. Each time I try something to fix it, it seems to go away for a while only to rear its ugly head again. It does seem to be triggered by synthetic full backups, which another user corroborates here.
veeeammeupscotty
Influencer
 
Posts: 15
Liked: 2 times
Joined: Fri May 05, 2017 3:06 pm
Full Name: JP

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], derek.davis, dflood, Google Feedfetcher, Weeams and 41 guests