Availability for the Always-On Enterprise
dellock6
Veeam Software
Posts: 5515
Liked: 1526 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: REFS performance issue "workaround"

Post by dellock6 » Jul 11, 2017 11:23 am 1 person likes this post

I believe the biggest benefit here is the monthly active full , which completely breaks the chain and restart a fresh one without involving too many deletions and transforms, which we have all understood by new are the most serious candidates to be the root problem. I don't see per-vm to be an issue to be honest.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com/en/
vExpert 2011-2012-2013-2014-2015-2016-2017-2018
Veeam VMCE #1

mkretzer
Expert
Posts: 403
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS performance issue "workaround"

Post by mkretzer » Jul 11, 2017 4:17 pm

We did an experiment: We have a volume where fast clone has got really slow. Fast clone for one specific job took 6 hours instead of ~25 minutes it took in the first few weeks. Now after an active full fast clone two days after the active full (it only had to merge 2 backup points!) took 4 1/2 hours. So in our case active full did not help at all!

DaStivi
Service Provider
Posts: 93
Liked: 10 times
Joined: Jun 30, 2015 9:13 am
Full Name: Stephan Lang
Contact:

Re: REFS performance issue "workaround"

Post by DaStivi » Jul 11, 2017 9:10 pm

hello
does anyone here having strange CPU issues with refs?
i noticed now a couple of times, that after a backup, when veeam starts to delete old vib files (perVM Backup) the backup repo server starts using all 8vCPUs and than locks up completly... no login is possible... right now i've restarted it a couple of times and after a few minutes the same behavoir started... i've removed the refs disks from my vm now and the server boots normally and i can login and right now i'm installing the latest windows cumulative updates...

btw. i'm on build 14393.1358 (and as far as i understand the "old" fixes should be in there allready: post247790.html#p247790)

i checked last syn full run of this backup repository, i sadly i've to notice the very long "fast clone" too, as it took 2h20min.. :shock:

kubimike
Expert
Posts: 324
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS performance issue "workaround"

Post by kubimike » Jul 11, 2017 11:16 pm

@DaStivi same issue I had, contact Veeam open a case get the beta refs driver. In this case it fixed it for me.

kubimike
Expert
Posts: 324
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS performance issue "workaround"

Post by kubimike » Jul 12, 2017 2:35 pm

my fastclones are now taking 3 hours after fixing the retention issue. I am going through my backups before July 4th and it only took roughly an hour. Now it takes over 3 hours to make one.

mkretzer
Expert
Posts: 403
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS performance issue "workaround"

Post by mkretzer » Jul 13, 2017 3:58 am

The worst thing is; fast clones will continue to get slower and will never recover unless you reformat (or perhaps delete all the data). Thats why i had this idea for an workaround in this thread.

jozne
Influencer
Posts: 17
Liked: 6 times
Joined: Apr 18, 2012 6:55 pm
Full Name: Jari Haikonen
Contact:

Re: REFS performance issue "workaround"

Post by jozne » Oct 25, 2017 9:09 am

kubimike wrote:@DaStivi same issue I had, contact Veeam open a case get the beta refs driver. In this case it fixed it for me.
Hi , we are having the same issue. I made ReFS with 64k and stored some backups (4TB) to it for few days as incremental and now deleted all the backups through Veeam GUI
and the Server froze up with high CPU usage.

I try to ask if Support provides this beta driver.

veeeammeupscotty
Influencer
Posts: 20
Liked: 2 times
Joined: May 05, 2017 3:06 pm
Full Name: JP
Contact:

Re: REFS performance issue "workaround"

Post by veeeammeupscotty » Oct 27, 2017 8:02 pm

kubimike wrote:@DaStivi same issue I had, contact Veeam open a case get the beta refs driver. In this case it fixed it for me.
Veeam is telling me there is no beta driver. Can you tell me your case number?

mkretzer
Expert
Posts: 403
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS performance issue "workaround"

Post by mkretzer » Oct 29, 2017 9:17 am

From what i understand the beta thing was some months ago and the driver got integrated in the latest updates - which did not help by the way!

veeeammeupscotty
Influencer
Posts: 20
Liked: 2 times
Joined: May 05, 2017 3:06 pm
Full Name: JP
Contact:

Re: REFS performance issue "workaround"

Post by veeeammeupscotty » Oct 30, 2017 4:11 pm

Oh so you're referring to an actual Microsoft driver? Have you tried to install the hotfix mentioned in this thread which is supposed to be included in 9.5 U3? The symptoms there are different, high memory consumption, but the behavior makes me think it could be related. I have never seen the "TOO_MANY_REFERENCES" error in my environment, and the CPU peg issue has been sporadic. Each time I try something to fix it, it seems to go away for a while only to rear its ugly head again. It does seem to be triggered by synthetic full backups, which another user corroborates here.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 73 guests