Maintain control of your Microsoft Office 365 email data
Post Reply
kristofpoppe
Enthusiast
Posts: 26
Liked: 2 times
Joined: May 18, 2012 1:19 pm
Full Name: Kristof Poppe
Contact:

Decimal separator REST-API: clarification needed

Post by kristofpoppe » Jan 15, 2019 10:43 am

We're testing the rest-api and are trying to figure out in what format the answers of decimal values are presented.

In our current VB0365 setup (most recent version) , we have some locale settings on the windows machine Belgian Comma, which uses a comma as decimal separator. Are then the values stores also in a comma format ? Or are these values locale-independent ?

This is necessary to find out how we can and will parse the values from the API for further usage in a Grafana dashboard.

Thanks for any input and/or clarification.

nielsengelen
Veeam Software
Posts: 2636
Liked: 540 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Decimal separator REST-API: clarification needed

Post by nielsengelen » Jan 15, 2019 1:42 pm

They are normally locale independent. There is already a Grafana dashboard available, have a look at https://github.com/VeeamHub/automation/ ... 65-grafana
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

kristofpoppe
Enthusiast
Posts: 26
Liked: 2 times
Joined: May 18, 2012 1:19 pm
Full Name: Kristof Poppe
Contact:

Re: Decimal separator REST-API: clarification needed

Post by kristofpoppe » Jan 15, 2019 1:58 pm

Hi Niels, thanks for your prompt response. We are indeed testing with this .sh script, but when analysing the data we get from the server we see comma's.

So we altered the script.: awk -F. '{print $1}') changed to separate on a comma instead of a point, but then we receive
unable to parse..... invalid boolean.

Also when we look at the JSON, the comma's are already there where we expect points, neither if we change the "accepted language" in the curl command.

nielsengelen
Veeam Software
Posts: 2636
Liked: 540 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Decimal separator REST-API: clarification needed

Post by nielsengelen » Jan 15, 2019 2:11 pm

Is the bash script failing? If so can you try using: awk -F',' (with quotes as this could be important)? Or are you getting the parse error somewhere else?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

kristofpoppe
Enthusiast
Posts: 26
Liked: 2 times
Joined: May 18, 2012 1:19 pm
Full Name: Kristof Poppe
Contact:

Re: Decimal separator REST-API: clarification needed

Post by kristofpoppe » Jan 15, 2019 3:50 pm

Indeed, the bash gives now errors on parsing the data.
Below a short sample of the output. For your information, we replaced on all places the awk -F','

HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 5451e386-18d9-11e9-8482-005056ac9cfe
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.3
X-Request-Id: 5451e386-18d9-11e9-8482-005056ac9cfe
Date: Tue, 15 Jan 2019 15:22:11 GMT

HTTP/1.1 400 Bad Request
Content-Type: application/json
Request-Id: 545a6154-18d9-11e9-8483-005056ac9cfe
X-Influxdb-Build: OSS
X-Influxdb-Error: unable to parse 'veeam_office365_restoresession,organization=All organizations,veeamjobname=organizations,type=Vex,result=Success,initiatedBy=VEEAM4OFF365\Administrator itemsProcessed=0,itemsSuccess=0 1532929781': invalid boolean
X-Influxdb-Version: 1.7.3
X-Request-Id: 545a6154-18d9-11e9-8483-005056ac9cfe
Date: Tue, 15 Jan 2019 15:22:11 GMT
Content-Length: 243

{"error":"unable to parse 'veeam_office365_restoresession,organization=All organizations,veeamjobname=organizations,type=Vex,result=Success,initiatedBy=VEEAM4OFF365\\Administrator itemsProcessed=0,itemsSuccess=0 1532929781': invalid boolean"}
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 54625eb4-18d9-11e9-8484-005056ac9cfe
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.3
X-Request-Id: 54625eb4-18d9-11e9-8484-005056ac9cfe
Date: Tue, 15 Jan 2019 15:22:11 GMT

nielsengelen
Veeam Software
Posts: 2636
Liked: 540 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Decimal separator REST-API: clarification needed

Post by nielsengelen » Jan 15, 2019 5:00 pm

I've asked Jorge to jump on here as I don't know the full bash script. It may be needed to rework the parameters with some code so they change the "," into a "." before parsing them.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

jorgedlcruz
Veeam Software
Posts: 260
Liked: 112 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Decimal separator REST-API: clarification needed

Post by jorgedlcruz » Jan 16, 2019 10:27 am

Good morning guys,
Sorry for the delay, if you try the swagger on your browser https://YOURVBOIP:4443/swagger/ui, what values do you see under Job Sessions?

I see the next ones:
Image

I will check the code today and see if I can get rid of that awk anyways using floats when we write back to InfluxDB.
Jorge de la Cruz
Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2019 / Nutanix Technology Champion 2018-2019

wishr
Veeam Software
Posts: 1152
Liked: 117 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Decimal separator REST-API: clarification needed

Post by wishr » Jan 16, 2019 10:45 am

Hi gentlemen,

I'd like to add that our QA team has checked several locales (on both server and client) and confirmed that regardless of a locale set they were getting responses with the "." (dot) separator in decimal values. Most likely that happens due to "JSON features" - in JSON commas are used for separating key-value pairs, while dots are used for decimal values.

Hope this helps. Thanks.

jorgedlcruz
Veeam Software
Posts: 260
Liked: 112 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Decimal separator REST-API: clarification needed

Post by jorgedlcruz » Jan 16, 2019 11:11 am

Hello,
Please try the next script, I've changed the way we retrieve the number: Basically the RESTfulAPI always gives you . which is what I am using now as well to write back to InfluxDB, as it will not accept commas as a value.

In case for whatever reason you want to obtain a comma for the decimals, you can find the change now on the next part of the code:

Code: Select all

awk -F'[^0-9]*' '{print $1 "." $2}'
So you can put your , there, however that will not work for InfluxDB, so I will let it as it is.

Please let us know, also let us know if you have any trouble with the Grafana Dashboard itself, it is working fine for me.
Jorge de la Cruz
Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2019 / Nutanix Technology Champion 2018-2019

kristofpoppe
Enthusiast
Posts: 26
Liked: 2 times
Joined: May 18, 2012 1:19 pm
Full Name: Kristof Poppe
Contact:

Re: Decimal separator REST-API: clarification needed

Post by kristofpoppe » Jan 16, 2019 11:23 am 1 person likes this post

Hi Jorge, many thanks for your input.

In the meanwhile we've tested it with swagger and below is the output: (indeed comma's)
We'll test your updated script and give feedback. We're also figuring out where the parsing of the boolean goes wrong.

Image

kristofpoppe
Enthusiast
Posts: 26
Liked: 2 times
Joined: May 18, 2012 1:19 pm
Full Name: Kristof Poppe
Contact:

Re: Decimal separator REST-API: clarification needed

Post by kristofpoppe » Jan 16, 2019 11:32 am

We're thinking the parser will fail on the GET /v2/RestoreSessions where we have "0 items processed"


"initiatedBy": "VEEAM4OFF365\\Administrator",
"details": "0 items processed",
"_links": {

jorgedlcruz
Veeam Software
Posts: 260
Liked: 112 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Decimal separator REST-API: clarification needed

Post by jorgedlcruz » Jan 16, 2019 11:44 am

Okay, so this is a different problem then, this is on the last part of the Script, has the previous problem been solved?
Jorge de la Cruz
Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2019 / Nutanix Technology Champion 2018-2019

jorgedlcruz
Veeam Software
Posts: 260
Liked: 112 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Decimal separator REST-API: clarification needed

Post by jorgedlcruz » Jan 16, 2019 12:05 pm

I can not find any result like that, for me on my RestoreSessions I have the next:

Code: Select all

{
  "offset": 0,
  "limit": 30,
  "results": [
    {
      "id": "2584e02b-7da2-49d2-9d3f-02150ee4c26f",
      "name": "Veeam Explorer for Exchange (Job: BJ-EXCHANGE-ALL)",
      "organization": "jorgedelacruz.onmicrosoft.com",
      "type": "Vex",
      "creationTime": "2018-12-15T20:13:25.1245205Z",
      "endTime": "2018-12-15T20:15:24.339536Z",
      "state": "Stopped",
      "result": "Success",
      "initiatedBy": "ZIMBRA\\Administrator",
      "details": "16 items processed (16 successes)",
      "_links": {
        "self": {
          "href": "https://192.168.1.33:4443/v2/restoresessions/2584e02b-7da2-49d2-9d3f-02150ee4c26f"
        },
        "organization": {
          "href": "https://192.168.1.33:4443/v2/restoresessions/2584e02b-7da2-49d2-9d3f-02150ee4c26f/organization"
        }
      }
    },
I can try to improve the script in case I had 0 restore sessions but I will need from you that you paste the entire JSON you receive. As said, I think the main problem you had at the start of the thread it is now solved?
Jorge de la Cruz
Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2019 / Nutanix Technology Champion 2018-2019

jorgedlcruz
Veeam Software
Posts: 260
Liked: 112 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Decimal separator REST-API: clarification needed

Post by jorgedlcruz » Jan 16, 2019 12:07 pm

Ha! I just saw your reply with the commas, kristofpoppe, that's fine, the new code should work for that, please let me know about the RestoreSessions, so we can fix it too in case it fails
Jorge de la Cruz
Systems Engineer UKI @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2019 / Nutanix Technology Champion 2018-2019

wishr
Veeam Software
Posts: 1152
Liked: 117 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Decimal separator REST-API: clarification needed

Post by wishr » Jan 16, 2019 1:04 pm

Hi Kristof,

I'd like to jump in as well and ask you what locale do you have installed on the server and what are the locale settings in use. We'd like to perform an internal analysis of that.

Thanks

kristofpoppe
Enthusiast
Posts: 26
Liked: 2 times
Joined: May 18, 2012 1:19 pm
Full Name: Kristof Poppe
Contact:

Re: Decimal separator REST-API: clarification needed

Post by kristofpoppe » Jan 29, 2019 1:52 pm

Update:

Also with the new code, we're still receiving comma's but we were able to 'fix' this

So we added extra ‘missing’ awk for readrate, because there were still comma’s. This is now good interpreted, but also “readrate=” sometimes has no values.
Perhaps this explains the “missing field value”-error ? The only error which now persists is “’invalid Boolean”, so to which variable should this map?
Perhaps related to the inconsistent use of backslashes in the value of “initiatedBy”?

Another thought: we’re running veeam-stats.ps1 on the same linux box, which also uses rest-api and that’s working perfect.

Polina
Veeam Software
Posts: 876
Liked: 136 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Decimal separator REST-API: clarification needed

Post by Polina » Jan 30, 2019 11:57 am

Hi Kristof,

Our engineers are on this issue, but it will take some time to confirm and troubleshoot. Stay tuned for updates.

Polina
Veeam Software
Posts: 876
Liked: 136 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Decimal separator REST-API: clarification needed

Post by Polina » Jan 30, 2019 6:16 pm

Some feedback from our REST API guru:

First, it’s not a JSON issue, but the specifics of the way the server sends data. The values you see under Job Session statistics are the decimal units converted by the server itself from “raw” bytes (and this is where comma comes from). The decimals and their corresponding units of measurement (MB/GB/TB) are then sent as a string, which may cause parsing issues.

This behavior will be improved in the next releases.

Thanks!

b.vanhaastrecht
Service Provider
Posts: 562
Liked: 97 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Decimal separator REST-API: clarification needed

Post by b.vanhaastrecht » Feb 05, 2019 9:39 am

I also stumbled on this, looking forward on being fixed.
========================================
Veeam ProPartner and Cloud Connect Provider

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest