REST API does not allow concurrent connections

[Captnspdr]

The REST API appears to have a bug where multiple concurrent connections are not allowed, even if different user accounts are used to generate tokens. For example, here is a snippet from the REST API service logs if two users try to connect to the API within one minute of each other:

Code: Select all

[01.04.2021 21:58:54] <17> Info         [User: Anonymous][POST] request to [/api/oauth2/token] deserialized. Message: [grant_type=password&username=INTERNAL\SvcDRAutomation&password=****]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive,Keep-Alive; Content-Length: 75; Content-Type: application/x-www-form-urlencoded; Accept: application/json; Expect: 100-continue; Host: localhost:9419; ].
[01.04.2021 21:58:55] <17> Info         Current license edition is EnterprisePlus. Checking expiration and type...
[01.04.2021 21:58:55] <17> Info         Authenticated user: [INTERNAL\SvcDRAutomation]
[01.04.2021 21:58:55] <17> Info         User 'INTERNAL\SvcDRAutomation' authenticated successfully.
[01.04.2021 21:58:55] <17> Info         Using certificate: 0CC2F9F70550AD2BE530C3E4F4AAB1A3FC37B451 for access token
[01.04.2021 22:00:34] <04> Info         [DEBUG] Cache has been prepared successfully
[01.04.2021 22:00:39] <06> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backups] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive,Keep-Alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:39] <21> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backupInfrastructure/repositories] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: Keep-Alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:39] <20> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backups] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive,Keep-Alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:42] <21> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backupInfrastructure/repositories] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:43] <21> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backupInfrastructure/repositories] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:43] <21> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/objectrestorepoints/] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:19] <23> Info         [User: Anonymous][POST] request to [/api/oauth2/token] deserialized. Message: [grant_type=password&username=JonahM&password=****]; Parameters: [x-api-version: 1.0-rev1; Postman-Token: 292ccd38-c7ac-4199-ac7d-747a0cb00ef8; Connection: keep-alive; Content-Length: 106; Content-Type: application/x-www-form-urlencoded; Accept: application/json; Accept-Encoding: gzip, deflate, br; Host: dl-builddev:9419; User-Agent: PostmanRuntime/7.26.10; ].
[01.04.2021 22:01:19] <23> Info         Current license edition is EnterprisePlus. Checking expiration and type...
[01.04.2021 22:01:19] <23> Info         Authenticated user: [.\JonahM]
[01.04.2021 22:01:19] <23> Info         User 'JonahM' authenticated successfully.
[01.04.2021 22:01:19] <23> Info         Using certificate: 0CC2F9F70550AD2BE530C3E4F4AAB1A3FC37B451 for access token
[01.04.2021 22:01:39] <27> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backups] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:39] <27> Error        Token expired
[01.04.2021 22:01:39] <23> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backupInfrastructure/repositories] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:39] <23> Error        Token expired
[01.04.2021 22:01:39] <28> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backups] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:39] <28> Error        Token expired
[01.04.2021 22:01:39] <28> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/objectrestorepoints/] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:39] <28> Error        Token expired

[oleg.feoktistov]

Hi Jonah,

Thanks, that's definitely a bug. Looks like the token for the first user gets expired as soon as another user is authenticated. I have just brought that up with Dev/QA and will keep you posted on your bug report status.

Best regards,
Oleg

[oleg.feoktistov]

Hi Jonah,

Confirmed as a bug with QA. The fix is scheduled to vNext.

Thanks!
Oleg

[Captnspdr]

Thanks for passing it along Oleg. When you say vNext, do you mean the next major release, such as v12 or v11a, or the next cumulative update?

[oleg.feoktistov]

It won't make it to the cumulative patch for sure. So, no sooner than v11a. Thanks!

[zorille]

Hello,
same issue with Veeam Backup & Recovery Manager 11. I need to upgrade my Veeam infrastructure and the same personnal code works with 9.5 at this time for months.

I get session with cURL request and next request is on error.

Code: Select all

[Debug] 14:46:17 (23161) : (Zorille\framework\curl) HTTP/2 201 
* Expire in 0 ms for 6 (transfer 0x561b8b8d8eb0)
*   Trying 10.1.44.12...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x561b8b8d8eb0)
* Connected to 10.1.44.12 (10.1.44.12) port 9398 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=91000-bck-sw002
*  start date: Aug 17 09:08:42 2021 GMT
*  expire date: Aug 15 09:08:42 2031 GMT
*  issuer: CN=91000-bck-sw002
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x561b8b8d8eb0)
> POST /api/sessionMngr/?v=latest HTTP/2
Host: 10.1.44.12:9398
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0
Authorization: Basic LlxhZG1pbmlzdHJhdG9yOlBhNTV3MHJkISQ=
Accept: application/xml
Content-Length: 0
Content-Type: application/xml

* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 201 
< content-length: 6433
< content-type: application/xml; charset=utf-8
< location: https://10.1.44.12:9398/api/sessionMngr/17da4271-d2b3-4587-bef6-011a44cbd8d8
< server: Microsoft-HTTPAPI/2.0
< x-restsvcsessionid: MTdkYTQyNzEtZDJiMy00NTg3LWJlZjYtMDExYTQ0Y2JkOGQ4
< set-cookie: X-RestSvcSessionId=MTdkYTQyNzEtZDJiMy00NTg3LWJlZjYtMDExYTQ0Y2JkOGQ4; path=/api
< date: Tue, 17 Aug 2021 12:43:43 GMT
< 

* Connection #0 to host 10.1.44.12 left intact <===============================================

...
[Debug] 14:46:17 (23161) : SimpleXMLElement Object
(
    [@attributes] => Array
        (
            [Href] => https://10.1.44.12:9398/api/logonSessions/17da4271-d2b3-4587-bef6-011a44cbd8d8
            [Type] => LogonSession
        )
...
[Debug] 14:46:17 (23161) : (Zorille\framework\curl) Zorille\framework\curl::send_curl
* Expire in 0 ms for 6 (transfer 0x561b8b8d8eb0)
*   Trying 10.1.44.12...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x561b8b8d8eb0)
* connect to 10.1.44.12 port 9398 failed: Connection timed out
* Failed to connect to 10.1.44.12 port 9398: Connection timed out
* Closing connection 0
[Debug] 14:46:49 (23161) : (Zorille\framework\curl) 
[Error] 14:46:49 (23161) : (Zorille\veeam\wsclient) Requete https://10.1.44.12:9398/api/querySvc en erreur 
(Zorille\framework\curl) cURL[7] Failed to connect to 10.1.44.12 port 9398: Connection timed out 

So i can't open new connection after the first one.

regards

[oleg.feoktistov]

Hi Damien,

I cannot confirm the same issue in my lab. Have just tried it and it has worked perfectly.
Also, from the curl debug output you shared I couldn't figure clearly if you attempted to create another session.
I just see that you created one, then tried to see the resources references you have under /logonSessions/{id}, which gave you a timeout error. After that you invoked a call to /querySvc, which gave you the same error. Please let me know if these are the steps or walk me through them so that I could say for sure what is happening in your case.

Thanks,
Oleg

[zorille]

I'm using curl for testing (lastest debian version) and my code is on PHP 7.x +CuRL mod.
Code is in production since years.

Code and curl command line has same behavior:
I sent a logon session and it worked but said (despite "Connection; closed" order):
* Connection #0 to host 10.1.44.12 left intact
next request with curl (example for logout but it's same with querySvc) send me :
* Failed to connect to 10.1.44.12 port 9398: Connection timed out
Tcp respond with netcat but i have no application behind the port until previous timeout tcp session. Then i can send 1 new request and need to wait.

This behavior is the same on VB&R and Veeam Manager 11. Can't try VeeamOne at this time.

only latest Veeam Service Provider Console works fine at this time in my testing environnement.

As i said, same code works fine with 9.5 version of Veeam and multiple other API Rest tools.

Regards,
Damien

[oleg.feoktistov]

I just tried the same with curl client in my lab. While I can without any doubts confirm that no concurrent sessions can be created for VBR REST API, for EM REST API I cannot reproduce the behaviour you encountered. I suggest to contact our support engineers and take a good look at the issue together with them. Also, please share the case id here so that we could monitor it closely. Thanks!

[zorille]

Hello,

Thanks for trying on your side.
For information, when i restart veeamRestAPI services, i can send 3 or 4 TCP requests but after, connection timeout restart.
Cas n°04977047 open today.

regards