How can processing speed for creating or updating objects be increased using the CWS API?

Environment:

Connect Web Services for SOAP API

Resolution:

Please follow these guidelines and don't use values of parameters that exceed maximum allowed numbers to prevent errors. The guidelines are intended to limit customized public API usage to 10% of site/instance capacity available within the hosting environment.

The Batch operation enforces a hard limit of 100 items, as well as the limit of 1,000 objects on each CRUD (create,update or delete) operation. This means that with Batch, you can't have more than 100 batch requests at a time.

A Batch item wraps a bulk request and can contain a mixture of objects if desired. Thus a single Create request (item) could Create 25 Contact, 20 Organization and 22 Incident objects. In this case, this one request has 67 RNObjects. Another Batch request may be an Update request for 10 Contacts and 5 Organizations. Now we are up to 82 RNObjects. You could potentially include 100 of these Batch requests within one SOAP operation (long as you don't exceed 10,000 RNObjects). However, while potentially possible it is also possible that such a request might exceed the memory and/or time limits for the process and will be aborted. In general our processes are limited to no more than 5 minutes of CPU time, so very large requests might exceed that limit.

Due to such operational constraints, it is possible supplying the maximum objects may result in an error. The number of items that may safely be sent in a single CRUD operation request will need to be weighed against the object complexity and size of data. In cases where too much data is supplied in a single request, the server may fail to complete the transaction and the client will receive an unexpected exception from the SOAP toolkit (this may not be a SOAP Fault returned from the server). Further, running large scale API operations on top of mapped internal and customer developed Custom Processes adversely impacts system performance.

A "best practices" bulk import would be to create a Batch request where each Batch Request Item would contain a Create request for somewhere between 50 and 250 objects (e.g. Contact, Organization, Incident). This is somewhat dependent upon the object being created. After each block the Batch Request Item should have its flag "commitAfter" set so that the block just processed is committed to the system. In that way, work can be preserved in reasonable blocks. If there is a failure in the request then only that block of objects is rolled back, and the next block is then processed. There are several reasonable approaches that follow this pattern.

Multiple threads are possible, but please understand that Oracle B2C Service is a shared resource and not an infinite resource. We've taken care to appropriately size our services to support regular use and surge use patterns. It is important that reasonable usage of these shared resources is considered to retain availability for all customers. A maximum of seven simultaneous threads will generally be acceptable. Along these lines, it is also a best practice to perform bulk API operations such as an import during off hours for the site and hosted POD.

For further information on object destroy operations refer to Answer ID 8887: Optimal efficiency when destroying objects using CWS API.

For further information on server-side timeout behavior refer to Answer ID 7970: Server-side timeouts associated with the Connect Web Services for SOAP (CWS) API.

For further information on troubleshooting CWS API issues refer to Answer ID 8190: Troubleshooting SOAP API Issues.