Collapse
Submit a Service Request
Contact Information for Technical Support
My Service Notifications
Using connectREST API's BULK extraction endpoint to get all incidents
Answer ID 11946 |
Last Review Date 07/22/2022
How do I get all incidents using the connectREST API?
Environment:
connectREST API
Issue:
How do I use the Bulk Extraction API endpoint
Resolution
To use the bulkExtracts API endpoint in conjunction with the bulkExtractResults endpoint:
First create a bulk extract by posting data to:
This post request should contain the information needed to run the extract:
The type of extraction you want to do, such as Incident, Answer, Account, etc; this can include custom objects.
The name of the extraction to completed, this can be used later to get the results.
State the bulkExtracts will start in, this will default to run but can be set to other statues.
Version of ROQL used to call the connectREST endpoint.
First create a bulk extract by posting data to:
https://%3Csite%3E/services/rest/connect/latest/bulkExtractsThis post request should contain the information needed to run the extract:
{'class': 'Incident', The type of extraction you want to do, such as Incident, Answer, Account, etc; this can include custom objects.
'name': 'ExtractExample',The name of the extraction to completed, this can be used later to get the results.
'state': {'lookupName': 'Run'},State the bulkExtracts will start in, this will default to run but can be set to other statues.
'version': 'v1.4'}Version of ROQL used to call the connectREST endpoint.
"filters": [{"leftOperand": "Id", "filterOperator": {"lookupName": ">"}, "rightOperand": "20"}]Filters are also suggested here, as they help to cut down the BULK extract; thus lowering the chances of failure.
POST:
{"class": "Incident", "name": "Extract3", "filters": [{"leftOperand": "Id", "filterOperator": {"lookupName": ">"}, "rightOperand": "20"}], "state": {"lookupName": "Run"}, "version": "v1.4"}Time Related posts will need to still include the incidents ID.
"filters":[{"leftOperand":"createdTime","filterOperator":{"lookupName":">="},"rightOperand":"'2020-01-01 00:00:00'"},{"leftOperand":"createdTime","filterOperator":{"lookupName":"<"},"rightOperand":"'2022-07-24 00:00:00'"},{"leftOperand":"ID","filterOperator":{"lookupName":">="},"rightOperand":"20"}]
This will then start the bulk extract and return a JSON request:
{ 'class': 'Incident', 'columns': { 'links': [ { 'href': 'https://%3Csite%3E/services/rest/connect/latest/bulkExtracts/3/columns', 'rel': 'self'}, { 'href': 'https://%3Csite%3E/services/rest/connect/latest/bulkExtracts/3/columns/{column_id}', 'rel': 'full', 'templated': True}]}, 'createdByAccount': { 'links': [ { 'href': 'https://%3Csite%3E/services/rest/connect/latest/accounts/2', 'rel': 'self'}, { 'href': 'https://%3Csite%3E/services/rest/connect/latest/accounts/2', 'rel': 'canonical'}, { 'href': 'https://%3Csite%3E/services/rest/connect/latest/metadata-catalog/accounts', 'mediaType': 'application/schema+json', 'rel': 'describedby'}]}, 'createdTime': '2021-08-17T18:49:05.000Z', 'filters': { 'links': [ { 'href': 'https://%3Csite%3E/services/rest/connect/latest/bulkExtracts/3/filters', 'rel': 'self'}, { 'href': 'https://%3Csite%3E/services/rest/connect/latest/bulkExtracts/3/filters/{filter_id}', 'rel': 'full', 'templated': True}]}, 'id': 3, 'links': [ { 'href': 'https://%3Csite%3E/services/rest/connect/latest/bulkExtracts/3', 'rel': 'self'}, { 'href': 'https://%3Csite%3E/services/rest/connect/latest/bulkExtracts/3', 'rel': 'canonical'}, { 'href': 'https://%3Csite%3E/services/rest/connect/latest/metadata-catalog/bulkExtracts', 'mediaType': 'application/schema+json', 'rel': 'describedby'}], 'lookupName': 'ExtractExample', 'name': 'ExtractExample', 'state': {'id': 1, 'lookupName': 'Run'}, 'type': {'id': 1, 'lookupName': 'Single'}, 'updatedByAccount': { 'links': [ { 'href': 'https://%3Csite%3E/services/rest/connect/latest/accounts/2', 'rel': 'self'}, { 'href': 'https://%3Csite%3E/services/rest/connect/latest/accounts/2', 'rel': 'canonical'}, { 'href': 'https://%3Csite%3E/services/rest/connect/latest/metadata-catalog/accounts', 'mediaType': 'application/schema+json', 'rel': 'describedby'}]}, 'updatedTime': '2021-08-17T18:49:05.000Z', 'version': 'v1.4'}Of this JSON requests data, save the id here in order to collect the information later.
Then call bulkExtractResults:
When bulkExtractResults is finished the description will read 'Completed Successfully'
After this use the exportedData to get the results, this will return a list of dictionaries with the ids included.
Use this list to download the extracted parts, they will be byte files that comprise a compressed tar.gz.
Then call bulkExtractResults:
https://%3Csite%3EWhen bulkExtractResults is finished the description will read 'Completed Successfully'
After this use the exportedData to get the results, this will return a list of dictionaries with the ids included.
Use this list to download the extracted parts, they will be byte files that comprise a compressed tar.gz.
https://%3Csite%3E/services/rest/connect/latest/bulkExtractResults/{id}/exportedData?downloadNote:
Bulk Extracts are a queued process much like a Custom Process(CPM), it will take time for the bulk extract to run in the queue.
The bulkExtracts endpoint is designed to pull large quantities of information at once, use of the analytics endpoint is suggested for smaller quantities.
The cutoff for this size limitation would be 100,000 rows per request. This is designated via the Rows Scanned.
Documentation:
