Restful API - listing documents from sharepoint site

[olekl]

Ticket number is #04327042
Customer has more than 100000 documents on sharepoint site, and I don't see any option to list them via Restful API.
If you set "limit=10000" request runs more than 4 hours which is definitely not OK.
If you iterate like "offset=0&limit=500", "offset=500&limit=500", "offset=1000&limit=500", each request runs from 20 to 700 seconds, and for iterating over all you still need 4+ hours...
Trying to iterate with limit=1000, I can reach 80000 documents, then restful service is hanged because memory usage increased for more than 5GB and response become "server 500" with message "Database does not exist or was forcibly closed by server." which stays till restart of the service on VBO...
And the question about restore. The whole size of 80000 elements I got is more than 200 GB... But you still have only option "restore as zip" and send so big zip directly in the response. Hours of transmitting without option to track progress or restart in case of network issue...
How is that planned to be? What is the procedure for a sharepoint site with 100000 documents, how should we iterate over them and do a restore via Restful API?

[Mike Resseler]

Hi @olekl

This is something where we need to work on, but there are limitations. I assume in your case that all of these documents are in 1 single document library. There is indeed a problem with anything above 5000 (if I recall the number correctly). Thanks for letting us know the support case #. I will keep an eye on it also, and I suspect that the information which is provided will be sent to our QA or DEV team for further investigation. I can't say or promise much at this point in time, but making enhancements to these scenario's (where lists or document libraries have basically a crazy amount of items) is on our radar. Unfortunately, solving it isn't that easy...

[k00laid]

Hi Mike, I work with @olekl. One of the problems we see is customers wanting to make use of the Entire Organization feature to make VBO set it and forget it, even breaking that down to workload. Unfortunately that is the only dynamic model that VBO presents today. What would be excellent is if there was a way to say "put all mailboxes|sites|etc. a*-e* into this job" or some other way of dynamically creating jobs.

[nielsengelen]

That would still require a specific offset if you need to go over a large account which can take time as for job creation, we have to query Microsoft as well. We could perform caching but you could still face the moment when someone hits the “refresh” button.

[olekl]

"we have to query Microsoft as well" - but why do you need to query Microsoft if we are exploring restore session? As I understand, content of restore session should be completely local... as you can do backup of only part of account... Have I missed something?

[nielsengelen]

@olekl there was mention about creating jobs dynamically and that is where I replied to.

There is no query to Microsoft for restores.