RESTful knowledge exchange
Post Reply
Captnspdr
Veeam Vanguard
Posts: 27
Liked: 3 times
Joined: Apr 16, 2019 4:50 pm
Full Name: Jonah May
Contact:

REST API does not allow concurrent connections

Post by Captnspdr » 1 person likes this post

The REST API appears to have a bug where multiple concurrent connections are not allowed, even if different user accounts are used to generate tokens. For example, here is a snippet from the REST API service logs if two users try to connect to the API within one minute of each other:

Code: Select all

[01.04.2021 21:58:54] <17> Info         [User: Anonymous][POST] request to [/api/oauth2/token] deserialized. Message: [grant_type=password&username=INTERNAL\SvcDRAutomation&password=****]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive,Keep-Alive; Content-Length: 75; Content-Type: application/x-www-form-urlencoded; Accept: application/json; Expect: 100-continue; Host: localhost:9419; ].
[01.04.2021 21:58:55] <17> Info         Current license edition is EnterprisePlus. Checking expiration and type...
[01.04.2021 21:58:55] <17> Info         Authenticated user: [INTERNAL\SvcDRAutomation]
[01.04.2021 21:58:55] <17> Info         User 'INTERNAL\SvcDRAutomation' authenticated successfully.
[01.04.2021 21:58:55] <17> Info         Using certificate: 0CC2F9F70550AD2BE530C3E4F4AAB1A3FC37B451 for access token
[01.04.2021 22:00:34] <04> Info         [DEBUG] Cache has been prepared successfully
[01.04.2021 22:00:39] <06> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backups] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive,Keep-Alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:39] <21> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backupInfrastructure/repositories] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: Keep-Alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:39] <20> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backups] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive,Keep-Alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:42] <21> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backupInfrastructure/repositories] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:43] <21> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backupInfrastructure/repositories] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:00:43] <21> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/objectrestorepoints/] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:19] <23> Info         [User: Anonymous][POST] request to [/api/oauth2/token] deserialized. Message: [grant_type=password&username=JonahM&password=****]; Parameters: [x-api-version: 1.0-rev1; Postman-Token: 292ccd38-c7ac-4199-ac7d-747a0cb00ef8; Connection: keep-alive; Content-Length: 106; Content-Type: application/x-www-form-urlencoded; Accept: application/json; Accept-Encoding: gzip, deflate, br; Host: dl-builddev:9419; User-Agent: PostmanRuntime/7.26.10; ].
[01.04.2021 22:01:19] <23> Info         Current license edition is EnterprisePlus. Checking expiration and type...
[01.04.2021 22:01:19] <23> Info         Authenticated user: [.\JonahM]
[01.04.2021 22:01:19] <23> Info         User 'JonahM' authenticated successfully.
[01.04.2021 22:01:19] <23> Info         Using certificate: 0CC2F9F70550AD2BE530C3E4F4AAB1A3FC37B451 for access token
[01.04.2021 22:01:39] <27> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backups] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:39] <27> Error        Token expired
[01.04.2021 22:01:39] <23> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backupInfrastructure/repositories] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:39] <23> Error        Token expired
[01.04.2021 22:01:39] <28> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/backups] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:39] <28> Error        Token expired
[01.04.2021 22:01:39] <28> Info         [User: INTERNAL\SvcDRAutomation][GET] request to [/api/v1/objectrestorepoints/] deserialized. Message: [null]; Parameters: [x-api-version: 1.0-rev1; Connection: keep-alive; Accept: application/json; Authorization: Bearer :**** Host: localhost:9419; ].
[01.04.2021 22:01:39] <28> Error        Token expired
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: REST API does not allow concurrent connections

Post by oleg.feoktistov » 2 people like this post

Hi Jonah,

Thanks, that's definitely a bug. Looks like the token for the first user gets expired as soon as another user is authenticated. I have just brought that up with Dev/QA and will keep you posted on your bug report status.

Best regards,
Oleg
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: REST API does not allow concurrent connections

Post by oleg.feoktistov » 1 person likes this post

Hi Jonah,

Confirmed as a bug with QA. The fix is scheduled to vNext.

Thanks!
Oleg
Captnspdr
Veeam Vanguard
Posts: 27
Liked: 3 times
Joined: Apr 16, 2019 4:50 pm
Full Name: Jonah May
Contact:

Re: REST API does not allow concurrent connections

Post by Captnspdr »

Thanks for passing it along Oleg. When you say vNext, do you mean the next major release, such as v12 or v11a, or the next cumulative update?
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: REST API does not allow concurrent connections

Post by oleg.feoktistov »

It won't make it to the cumulative patch for sure. So, no sooner than v11a. Thanks!
zorille
Novice
Posts: 3
Liked: never
Joined: Aug 10, 2021 12:39 pm
Full Name: Damien Vargas
Contact:

Re: REST API does not allow concurrent connections

Post by zorille »

Hello,
same issue with Veeam Backup & Recovery Manager 11. I need to upgrade my Veeam infrastructure and the same personnal code works with 9.5 at this time for months.

I get session with cURL request and next request is on error.

Code: Select all

[Debug] 14:46:17 (23161) : (Zorille\framework\curl) HTTP/2 201 
* Expire in 0 ms for 6 (transfer 0x561b8b8d8eb0)
*   Trying 10.1.44.12...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x561b8b8d8eb0)
* Connected to 10.1.44.12 (10.1.44.12) port 9398 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=91000-bck-sw002
*  start date: Aug 17 09:08:42 2021 GMT
*  expire date: Aug 15 09:08:42 2031 GMT
*  issuer: CN=91000-bck-sw002
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x561b8b8d8eb0)
> POST /api/sessionMngr/?v=latest HTTP/2
Host: 10.1.44.12:9398
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0
Authorization: Basic LlxhZG1pbmlzdHJhdG9yOlBhNTV3MHJkISQ=
Accept: application/xml
Content-Length: 0
Content-Type: application/xml

* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 201 
< content-length: 6433
< content-type: application/xml; charset=utf-8
< location: https://10.1.44.12:9398/api/sessionMngr/17da4271-d2b3-4587-bef6-011a44cbd8d8
< server: Microsoft-HTTPAPI/2.0
< x-restsvcsessionid: MTdkYTQyNzEtZDJiMy00NTg3LWJlZjYtMDExYTQ0Y2JkOGQ4
< set-cookie: X-RestSvcSessionId=MTdkYTQyNzEtZDJiMy00NTg3LWJlZjYtMDExYTQ0Y2JkOGQ4; path=/api
< date: Tue, 17 Aug 2021 12:43:43 GMT
< 

* Connection #0 to host 10.1.44.12 left intact <===============================================

...
[Debug] 14:46:17 (23161) : SimpleXMLElement Object
(
    [@attributes] => Array
        (
            [Href] => https://10.1.44.12:9398/api/logonSessions/17da4271-d2b3-4587-bef6-011a44cbd8d8
            [Type] => LogonSession
        )
...
[Debug] 14:46:17 (23161) : (Zorille\framework\curl) Zorille\framework\curl::send_curl
* Expire in 0 ms for 6 (transfer 0x561b8b8d8eb0)
*   Trying 10.1.44.12...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x561b8b8d8eb0)
* connect to 10.1.44.12 port 9398 failed: Connection timed out
* Failed to connect to 10.1.44.12 port 9398: Connection timed out
* Closing connection 0
[Debug] 14:46:49 (23161) : (Zorille\framework\curl) 
[Error] 14:46:49 (23161) : (Zorille\veeam\wsclient) Requete https://10.1.44.12:9398/api/querySvc en erreur 
(Zorille\framework\curl) cURL[7] Failed to connect to 10.1.44.12 port 9398: Connection timed out 
So i can't open new connection after the first one.

regards
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: REST API does not allow concurrent connections

Post by oleg.feoktistov »

Hi Damien,

I cannot confirm the same issue in my lab. Have just tried it and it has worked perfectly.
Also, from the curl debug output you shared I couldn't figure clearly if you attempted to create another session.
I just see that you created one, then tried to see the resources references you have under /logonSessions/{id}, which gave you a timeout error. After that you invoked a call to /querySvc, which gave you the same error. Please let me know if these are the steps or walk me through them so that I could say for sure what is happening in your case.

Thanks,
Oleg
zorille
Novice
Posts: 3
Liked: never
Joined: Aug 10, 2021 12:39 pm
Full Name: Damien Vargas
Contact:

Re: REST API does not allow concurrent connections

Post by zorille »

I'm using curl for testing (lastest debian version) and my code is on PHP 7.x +CuRL mod.
Code is in production since years.

Code and curl command line has same behavior:
I sent a logon session and it worked but said (despite "Connection; closed" order):
* Connection #0 to host 10.1.44.12 left intact
next request with curl (example for logout but it's same with querySvc) send me :
* Failed to connect to 10.1.44.12 port 9398: Connection timed out
Tcp respond with netcat but i have no application behind the port until previous timeout tcp session. Then i can send 1 new request and need to wait.

This behavior is the same on VB&R and Veeam Manager 11. Can't try VeeamOne at this time.

only latest Veeam Service Provider Console works fine at this time in my testing environnement.

As i said, same code works fine with 9.5 version of Veeam and multiple other API Rest tools.

Regards,
Damien
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: REST API does not allow concurrent connections

Post by oleg.feoktistov »

I just tried the same with curl client in my lab. While I can without any doubts confirm that no concurrent sessions can be created for VBR REST API, for EM REST API I cannot reproduce the behaviour you encountered. I suggest to contact our support engineers and take a good look at the issue together with them. Also, please share the case id here so that we could monitor it closely. Thanks!
zorille
Novice
Posts: 3
Liked: never
Joined: Aug 10, 2021 12:39 pm
Full Name: Damien Vargas
Contact:

Re: REST API does not allow concurrent connections

Post by zorille »

Hello,

Thanks for trying on your side.
For information, when i restart veeamRestAPI services, i can send 3 or 4 TCP requests but after, connection timeout restart.
Cas n°04977047 open today.

regards
Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests