Comprehensive data protection for all workloads
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: REFS performance issue "workaround"

Post by dellock6 » 1 person likes this post

I believe the biggest benefit here is the monthly active full , which completely breaks the chain and restart a fresh one without involving too many deletions and transforms, which we have all understood by new are the most serious candidates to be the root problem. I don't see per-vm to be an issue to be honest.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS performance issue "workaround"

Post by mkretzer »

We did an experiment: We have a volume where fast clone has got really slow. Fast clone for one specific job took 6 hours instead of ~25 minutes it took in the first few weeks. Now after an active full fast clone two days after the active full (it only had to merge 2 backup points!) took 4 1/2 hours. So in our case active full did not help at all!
DaStivi
Service Provider
Posts: 250
Liked: 34 times
Joined: Jun 30, 2015 9:13 am
Full Name: Stephan Lang
Location: Austria
Contact:

Re: REFS performance issue "workaround"

Post by DaStivi »

hello
does anyone here having strange CPU issues with refs?
i noticed now a couple of times, that after a backup, when veeam starts to delete old vib files (perVM Backup) the backup repo server starts using all 8vCPUs and than locks up completly... no login is possible... right now i've restarted it a couple of times and after a few minutes the same behavoir started... i've removed the refs disks from my vm now and the server boots normally and i can login and right now i'm installing the latest windows cumulative updates...

btw. i'm on build 14393.1358 (and as far as i understand the "old" fixes should be in there allready: post247790.html#p247790)

i checked last syn full run of this backup repository, i sadly i've to notice the very long "fast clone" too, as it took 2h20min.. :shock:
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS performance issue "workaround"

Post by kubimike »

@DaStivi same issue I had, contact Veeam open a case get the beta refs driver. In this case it fixed it for me.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS performance issue "workaround"

Post by kubimike »

my fastclones are now taking 3 hours after fixing the retention issue. I am going through my backups before July 4th and it only took roughly an hour. Now it takes over 3 hours to make one.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS performance issue "workaround"

Post by mkretzer »

The worst thing is; fast clones will continue to get slower and will never recover unless you reformat (or perhaps delete all the data). Thats why i had this idea for an workaround in this thread.
jozne
Influencer
Posts: 17
Liked: 7 times
Joined: Apr 18, 2012 6:55 pm
Full Name: Jari Haikonen
Contact:

Re: REFS performance issue "workaround"

Post by jozne »

kubimike wrote:@DaStivi same issue I had, contact Veeam open a case get the beta refs driver. In this case it fixed it for me.
Hi , we are having the same issue. I made ReFS with 64k and stored some backups (4TB) to it for few days as incremental and now deleted all the backups through Veeam GUI
and the Server froze up with high CPU usage.

I try to ask if Support provides this beta driver.
veeeammeupscotty
Enthusiast
Posts: 33
Liked: 2 times
Joined: May 05, 2017 3:06 pm
Full Name: JP
Contact:

Re: REFS performance issue "workaround"

Post by veeeammeupscotty »

kubimike wrote:@DaStivi same issue I had, contact Veeam open a case get the beta refs driver. In this case it fixed it for me.
Veeam is telling me there is no beta driver. Can you tell me your case number?
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS performance issue "workaround"

Post by mkretzer »

From what i understand the beta thing was some months ago and the driver got integrated in the latest updates - which did not help by the way!
veeeammeupscotty
Enthusiast
Posts: 33
Liked: 2 times
Joined: May 05, 2017 3:06 pm
Full Name: JP
Contact:

Re: REFS performance issue "workaround"

Post by veeeammeupscotty »

Oh so you're referring to an actual Microsoft driver? Have you tried to install the hotfix mentioned in this thread which is supposed to be included in 9.5 U3? The symptoms there are different, high memory consumption, but the behavior makes me think it could be related. I have never seen the "TOO_MANY_REFERENCES" error in my environment, and the CPU peg issue has been sporadic. Each time I try something to fix it, it seems to go away for a while only to rear its ugly head again. It does seem to be triggered by synthetic full backups, which another user corroborates here.
Post Reply

Who is online

Users browsing this forum: ThomasIkoula and 173 guests