Disk/Datastore: Datastore Read Latency" (152.0 Milliseconds) is above a defined threshold (100.0 Milliseconds)
We have high latency while running Veeam backup and replication. What could be causing this?
Our Infrastructure is Cisco UCS and VMWare VSphere 6.5
-
oryx360
- Novice
- Posts: 5
- Liked: never
- Joined: Aug 03, 2021 7:55 pm
- Full Name: Oryx 360
- Contact:
-
Mildur
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Datastore Latency
Veeam needs to read all blocks from the productive storage to do it‘s backup job.
If CBT can be used, then the veeam proxy need only to read changed blocks. If CBT cannot be used (open snapshots, cbt disabled on the vm), then all blocks needs to be checked if they have been changed.
That can put a heavy load (read I/O) on the productive storage itself.
We are having an ucs environment too with Netapp NFS Storage for the esxi environment. As soon the backups starts, the latency goes up. That‘s completely normal.
You can try to limit the parallel tasks on the proxy to reduce the load. Or to limit the snapshots per datastore at the same time.
Or enable storage latency control:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110
If CBT can be used, then the veeam proxy need only to read changed blocks. If CBT cannot be used (open snapshots, cbt disabled on the vm), then all blocks needs to be checked if they have been changed.
That can put a heavy load (read I/O) on the productive storage itself.
We are having an ucs environment too with Netapp NFS Storage for the esxi environment. As soon the backups starts, the latency goes up. That‘s completely normal.
You can try to limit the parallel tasks on the proxy to reduce the load. Or to limit the snapshots per datastore at the same time.
Or enable storage latency control:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110
Product Management Analyst @ Veeam Software
-
oryx360
- Novice
- Posts: 5
- Liked: never
- Joined: Aug 03, 2021 7:55 pm
- Full Name: Oryx 360
- Contact:
Re: Datastore Latency
What about latency that bogs the server that you cannot even login to the server? Can this happen?
-
Mildur
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Datastore Latency
Yeah, definitely possible.
If the read latency is to high, you will „feel“ that, if you want to log in to the vms. If you log in, you windows profil and application needs to be loaded.
Because your datastore is under heavy load from the backup, there could be a very slow login process for you.
You need to optimize your production storage.
On the veeam components, make sure:
1) CBT is enabled and used for each vm
2) play with max tasks on the veeam proxy to reduce parallel load on the storage
3) use the storage latency control feature to reduce iops for the backup jobs
4) Try to reduce max snapshots per Datastore value.
If that is no help, then analyze your production storage and optimize it. You could get help from your storage vendor for that. They can tell you if what can be done on their end.
If the read latency is to high, you will „feel“ that, if you want to log in to the vms. If you log in, you windows profil and application needs to be loaded.
Because your datastore is under heavy load from the backup, there could be a very slow login process for you.
You need to optimize your production storage.
On the veeam components, make sure:
1) CBT is enabled and used for each vm
2) play with max tasks on the veeam proxy to reduce parallel load on the storage
3) use the storage latency control feature to reduce iops for the backup jobs
4) Try to reduce max snapshots per Datastore value.
If that is no help, then analyze your production storage and optimize it. You could get help from your storage vendor for that. They can tell you if what can be done on their end.
Product Management Analyst @ Veeam Software
-
oryx360
- Novice
- Posts: 5
- Liked: never
- Joined: Aug 03, 2021 7:55 pm
- Full Name: Oryx 360
- Contact:
Re: Datastore Latency
Thank you these are valuable inputs.
-
Mildur
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
happy to help.Who is online
Users browsing this forum: No registered users and 39 guests