Host-based backup of Microsoft Hyper-V VMs.
Post Reply
akraker
Enthusiast
Posts: 45
Liked: 5 times
Joined: Feb 11, 2019 6:19 pm
Full Name: Andrew Kraker
Contact:

Monitoring Storage latency for throttling

Post by akraker »

I am wondering if there is any place where I can monitor what Veeam detects the storage latency to be and if/when it is taking throttling actions based on the parameters set in the I/O control section. I have been having users complain about server performance during backup cycles so I lowered the latency values a bit but in the GUI job logs I don't see any indication that throttling is actually occurring.
Egor Yakovlev
Veeam Software
Posts: 2536
Liked: 680 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Monitoring Storage latency for throttling

Post by Egor Yakovlev »

Hi Andrew.

That happens transparently during the session and is hidden from end job results, however during job execution fact of Storage IO Control in action is displayed in the session log. For example, if high latency is detected, new about-to-be-allocated task in session log will be put into "Resource not ready: Primary storage latency is above defined limit " pause state, which changes back to normal when latency goes lower and task continues to be executed as normal.

/Thanks!
akraker
Enthusiast
Posts: 45
Liked: 5 times
Joined: Feb 11, 2019 6:19 pm
Full Name: Andrew Kraker
Contact:

Re: Monitoring Storage latency for throttling

Post by akraker »

Thanks Egor. I found a post from someone a while back that mentions being able to see throttling in the job log next to the hard disk name. They also show entries in the .log file for "LatencyThrottler"
http://vmjoe.blogspot.com/2015/04/veeam ... ue%20worse.

This looks to be an older version of B&R but should I still see the same things in v10? Right now I am not seeing any indication of throttling in either of those locations.
Egor Yakovlev
Veeam Software
Posts: 2536
Liked: 680 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Monitoring Storage latency for throttling

Post by Egor Yakovlev »

Correct, for existing tasks [throttled] shall be still visible next to per-disk statistics under Session log. That is, if throttling took place.
/Cheers!
akraker
Enthusiast
Posts: 45
Liked: 5 times
Joined: Feb 11, 2019 6:19 pm
Full Name: Andrew Kraker
Contact:

Re: Monitoring Storage latency for throttling

Post by akraker »

So the weird thing is that my backups show the source as the bottleneck and I can see that when it makes a snapshot and processes the data my iSCSI connections to our SAN are getting hammered. This is causing noticeable performance issues on the production VM's but when looking at the host machine storage latency metrics it is not going very high at all. I was hoping I could see some readings on what Veeam B&R is seeing the latency as or what method it is using to check. Is it looking at a certain performance monitor counter or using its own test?
Egor Yakovlev
Veeam Software
Posts: 2536
Liked: 680 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Location: Prague, Czech Republic
Contact:

Re: Monitoring Storage latency for throttling

Post by Egor Yakovlev »

Veeam tracks latency on the volume where VM disks reside, so if you have an iSCSI mounted volume for VHDXes, then Hyper-V Host's Drive C: latency is irrelevant.
You can check this KB article to see how to monitor latency with perf monitor. For latency Veeam queries Disk(*)\Avg. Disk sec/Read.

/Cheers!
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Monitoring Storage latency for throttling

Post by wishr »

Hi Andrew,

You may also want to take a look at Veeam ONE (part of Veeam Availability Suite) which provides monitoring tools for precise storage latency control.

Thanks
Post Reply

Who is online

Users browsing this forum: No registered users and 30 guests