Comprehensive data protection for all workloads
Post Reply
pirx
Veteran
Posts: 599
Liked: 87 times
Joined: Dec 20, 2015 6:24 pm
Contact:
Contact pirx

Ressource bottlenecks / RTS.ResourcesUsage.log

Post by pirx »

Hi,

we have a rather large Veeam environment with multiple locations worldwide. Our largest environment is in DE, backing up 1650 VMs (Forever Forward Incremental + syn fulls). We are creating multiple copies, including offloading to S3. We use NetApp's as local SMB backup and copy devices.

This environment was designed together with Veeam and a partner 3-4 years ago. The number of VM's didn't change that much but we're seeing resource bottlenecks and I'm trying to find the root cause. Support usually suggests to shrink backup jobs. This is what we are already doing, but it's to that easy to manually load balance jobs.

I was looking into RTS.ResourcesUsage.log and it seems that not the proxies are the problem, instead the repositories are at their concurrent task limit (DE-WBE and DE-WOP are different locations, B01... SOBR backup extend, C01... SOBR copy extend, E00..E02 extends are new ones, E003...E007 are already seald). For proxy and backup repos we have set the limit to 20 tasks. We have 4 physical proxies, each with 20 cores and 128GB RAM. These proxies are the repository server too.

Code: Select all

[20.10.2020 01:35:43] <1093> Info     *******************************************************************************************************************
[20.10.2020 01:35:43] <1093> Info     |--------------------------------------------------------------------------------------------------|
[20.10.2020 01:35:43] <1093> Info     | Resource name                  | Usage | Id                                                      |
[20.10.2020 01:35:43] <1093> Info     |-------------------------------------------------------------------------------- vSphere Proxies -|
[20.10.2020 01:35:43] <1093> Info     | xxxhu2002                      | 4     | {bbb3c82a-dac3-4c25-b636-3e17add6e878}                  |
[20.10.2020 01:35:43] <1093> Info     | yyyyde3522                     | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
[20.10.2020 01:35:43] <1093> Info     |----------------------------------------------------------------------------------- Repositories -|
[20.10.2020 01:35:43] <1093> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
[20.10.2020 01:35:43] <1093> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
[20.10.2020 01:35:43] <1093> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
[20.10.2020 01:35:43] <1093> Info     | DE-WOP-C01-E005                | 18    | {bccdb2c6-3117-4281-897c-ddd7188fccc3}   (preemptible)  |
[20.10.2020 01:35:43] <1093> Info     | DE-WOP-C01-E007                | 1     | {1dffb584-383b-4266-b5f9-19b7a1bc5f40}                  |
[20.10.2020 01:35:43] <1093> Info     | DE-WOP-C01-E007                | 3     | {1dffb584-383b-4266-b5f9-19b7a1bc5f40}   (preemptible)  |
[20.10.2020 01:35:43] <1093> Info     | DE-WOP-C01-E00                 | 10    | {2e05f19a-4a30-4a14-bd05-5737136741db}                  |
[20.10.2020 01:35:43] <1093> Info     | DE-WOP-C01-E00                 | 11    | {2e05f19a-4a30-4a14-bd05-5737136741db}   (preemptible)  |
[20.10.2020 01:35:43] <1093> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
[20.10.2020 01:35:43] <1093> Info     | CZ-SCZS2900-B01-E01            | 3     | {06b3a1ec-8743-4347-a206-ae1935b57821}   (preemptible)  |
[20.10.2020 01:35:43] <1093> Info     | HU-SHUS2001-B01-E01            | 4     | {83b6bdbb-a470-419e-a59a-6826db42e771}                  |
[20.10.2020 01:35:43] <1093> Info     | DE-WBE-B01-E01                 | 19    | {12c5d75c-81cc-4e9a-a942-f9ec425e5af4}                  |
[20.10.2020 01:35:43] <1093> Info     | DE-WBE-B01-E02                 | 17    | {3769735e-725b-40b8-9381-030e88993c0d}                  |
[20.10.2020 01:35:43] <1093> Info     | DE-WBE-B01-E02                 | 3     | {3769735e-725b-40b8-9381-030e88993c0d}   (preemptible)  |
[20.10.2020 01:35:43] <1093> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
[20.10.2020 01:35:43] <1093> Info     | DE-WOP-C01-E01                 | 3     | {92ed449c-fe03-41e5-bdee-3fe800c96b4c}                  |
[20.10.2020 01:35:43] <1093> Info     | DE-WOP-C01-E01                 | 15    | {92ed449c-fe03-41e5-bdee-3fe800c96b4c}   (preemptible)  |
[20.10.2020 01:35:43] <1093> Info     |--------------------------------------------------------------------------------------------------|

This is a grep in RTS file for the max of 20 tasks around the time where we saw problems.


- I was wondering why there is only one of our 4 proxies in this list (yyyyde3522). There was maybe a misconfiguration in the repository config where for some repos not automatic was choose, instead a dedicated server. This is corrected now. Still, it was only for a few repos I'd expect the other 3 servers to show up more often. And the wrong setting was not in the proxy settings, only for repositories.

- a lot of repo extends seem to be at the their maximum of 20 tasks, on the other side, there are no - or just one proxy - in that list.
- is it correct to assume that I can then raise the number of concurrent task on the repos?

And idea how to get better parallelism? The proxies doesn't see to be at their limit looking at our monitoring. Any recommendations for concurrent tasks to try out?

Code: Select all

	Line 61311: [20.10.2020 01:29:18] <1130> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61320: [20.10.2020 01:30:35] <1130> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61322: [20.10.2020 01:30:35] <1130> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61324: [20.10.2020 01:30:35] <1130> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61330: [20.10.2020 01:30:35] <1130> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61333: [20.10.2020 01:30:35] <1130> Info     | DE-WBE-B01-E01                 | 20    | {12c5d75c-81cc-4e9a-a942-f9ec425e5af4}                  |
	Line 61345: [20.10.2020 01:31:00] <862> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61347: [20.10.2020 01:31:00] <862> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61348: [20.10.2020 01:31:00] <862> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61349: [20.10.2020 01:31:00] <862> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61355: [20.10.2020 01:31:00] <862> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61358: [20.10.2020 01:31:00] <862> Info     | DE-WBE-B01-E01                 | 20    | {12c5d75c-81cc-4e9a-a942-f9ec425e5af4}                  |
	Line 61361: [20.10.2020 01:31:00] <862> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61370: [20.10.2020 01:32:00] <1082> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61373: [20.10.2020 01:32:00] <1082> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61374: [20.10.2020 01:32:00] <1082> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61383: [20.10.2020 01:32:00] <1082> Info     | DE-WBE-B01-E01                 | 20    | {12c5d75c-81cc-4e9a-a942-f9ec425e5af4}                  |
	Line 61386: [20.10.2020 01:32:00] <1082> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61395: [20.10.2020 01:32:58] <1088> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61397: [20.10.2020 01:32:58] <1088> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61398: [20.10.2020 01:32:58] <1088> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61399: [20.10.2020 01:32:58] <1088> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61405: [20.10.2020 01:32:58] <1088> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61408: [20.10.2020 01:32:58] <1088> Info     | DE-WBE-B01-E01                 | 20    | {12c5d75c-81cc-4e9a-a942-f9ec425e5af4}                  |
	Line 61411: [20.10.2020 01:32:58] <1088> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61420: [20.10.2020 01:33:24] <1130> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61422: [20.10.2020 01:33:24] <1130> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61424: [20.10.2020 01:33:24] <1130> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61430: [20.10.2020 01:33:24] <1130> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61433: [20.10.2020 01:33:24] <1130> Info     | DE-WBE-B01-E01                 | 20    | {12c5d75c-81cc-4e9a-a942-f9ec425e5af4}                  |
	Line 61445: [20.10.2020 01:34:18] <924> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61447: [20.10.2020 01:34:18] <924> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61448: [20.10.2020 01:34:18] <924> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61449: [20.10.2020 01:34:18] <924> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61455: [20.10.2020 01:34:18] <924> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61458: [20.10.2020 01:34:18] <924> Info     | DE-WBE-B01-E01                 | 20    | {12c5d75c-81cc-4e9a-a942-f9ec425e5af4}                  |
	Line 61461: [20.10.2020 01:34:18] <924> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61470: [20.10.2020 01:34:44] <843> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61473: [20.10.2020 01:34:44] <843> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61474: [20.10.2020 01:34:44] <843> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61486: [20.10.2020 01:34:44] <843> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61495: [20.10.2020 01:35:43] <1093> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61497: [20.10.2020 01:35:43] <1093> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61498: [20.10.2020 01:35:43] <1093> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61499: [20.10.2020 01:35:43] <1093> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61505: [20.10.2020 01:35:43] <1093> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61511: [20.10.2020 01:35:43] <1093> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61520: [20.10.2020 01:36:37] <120> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61545: [20.10.2020 01:38:32] <862> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61547: [20.10.2020 01:38:32] <862> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61548: [20.10.2020 01:38:32] <862> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61555: [20.10.2020 01:38:32] <862> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61558: [20.10.2020 01:38:32] <862> Info     | DE-WBE-B01-E01                 | 20    | {12c5d75c-81cc-4e9a-a942-f9ec425e5af4}                  |
	Line 61561: [20.10.2020 01:38:32] <862> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61570: [20.10.2020 01:38:57] <862> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61573: [20.10.2020 01:38:57] <862> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61586: [20.10.2020 01:38:57] <862> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61595: [20.10.2020 01:39:51] <1364> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61597: [20.10.2020 01:39:51] <1364> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61598: [20.10.2020 01:39:51] <1364> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61605: [20.10.2020 01:39:51] <1364> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61611: [20.10.2020 01:39:51] <1364> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61620: [20.10.2020 01:40:18] <1102> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61624: [20.10.2020 01:40:18] <1102> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61646: [20.10.2020 01:41:42] <1067> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61648: [20.10.2020 01:41:42] <1067> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61649: [20.10.2020 01:41:42] <1067> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61650: [20.10.2020 01:41:42] <1067> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61656: [20.10.2020 01:41:42] <1067> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61663: [20.10.2020 01:41:42] <1067> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61672: [20.10.2020 01:42:07] <901> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61675: [20.10.2020 01:42:07] <901> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61689: [20.10.2020 01:42:07] <901> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61698: [20.10.2020 01:42:33] <955> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61700: [20.10.2020 01:42:33] <955> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61701: [20.10.2020 01:42:33] <955> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61708: [20.10.2020 01:42:33] <955> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61715: [20.10.2020 01:42:33] <955> Info     | DE-WBE-C01-E01                 | 20    | {1e01b76a-a8b4-4c4b-aca2-65287a276769}                  |
	Line 61728: [20.10.2020 01:42:58] <1130> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61750: [20.10.2020 01:43:27] <862> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61754: [20.10.2020 01:43:27] <862> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61776: [20.10.2020 01:43:55] <923> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61778: [20.10.2020 01:43:55] <923> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61780: [20.10.2020 01:43:55] <923> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61786: [20.10.2020 01:43:55] <923> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
	Line 61802: [20.10.2020 01:44:23] <1210> Info     | yyyyde3522             | 20    | {85ef412b-4779-4b58-a81d-40bc06fe7dd2}                  |
	Line 61804: [20.10.2020 01:44:23] <1210> Info     | DE-WBE-C01-E003                | 20    | {7fb44eac-f5db-4200-9603-d0cac93d29c8}                  |
	Line 61805: [20.10.2020 01:44:23] <1210> Info     | DE-WBE-C01-E004                | 20    | {87a554c8-285d-4118-8e4d-8c8d7bb46dd2}                  |
	Line 61806: [20.10.2020 01:44:23] <1210> Info     | DE-WOP-C01-E004                | 20    | {8eb79420-4434-4f27-9489-4f0f86b93016}   (preemptible)  |
	Line 61812: [20.10.2020 01:44:23] <1210> Info     | DE-WBE-C01-E00                 | 20    | {d281a5f5-4c2f-4a6d-843c-20d64be77ae0}                  |
Top
PetrM
Veeam Software
Posts: 3625
Liked: 608 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:
Contact PetrM

Re: Ressource bottlenecks / RTS.ResourcesUsage.log

Post by PetrM »

Hello,

RTS log shows a current load of backup infrastructure resources but does not provide any information why some resources are in use and other resources are not. The more precise logs analysis is required to understand what's happening behind the scenes during distribution of tasks across proxies and repositories. I'd recommend to raise a support request and to escalate it to a higher level in order to determine the root cause of the issue instead of trying out some "random" workarounds without clear understanding of what is the source of the problem. Please paste the support case ID over here for our reference.

I'd recommend to make sure that repositories have enough resources prior to increase the concurrent tasks limit, for example you may refer to our release notes (page 10).

As far as I understand you see just one proxy in RTS log? What about to check proxy affinity rules? Maybe other proxies are simply restricted from working with the SOBR.

Thanks!
Top
Post Reply

Return to “Veeam Backup & Replication”



Who is online

Users browsing this forum: Bing [Bot] and 117 guests