How can we use Data Lifecycle Policy to purge (or archive)?
Environment:
Purging and Archiving incidents
Oracle B2C Service, All Releases
Resolution:
Prior to the 18B release, incident archiving and purging settings were available through a configuration setting and it was also possible to set this value per interface.
Starting from the 18B release, the configuration functionality was migrated to Data Lifecycle Management framework and corresponding configuration settings were removed. If your site had multiple interfaces and incident archive or purge were configured differently per interface, site specific and multiple interface specific Data Lifecycle Cycle policies for incident would have been created after a site update to 18B or later. The incident archive and purge configuration values would have been copied into the appropriate Data Lifecycle Policies during that site update. Documentation of this functionality is available here.
In 18B, it is not possible to create an interface specific incident archive or purge policy using the Data Lifecycle Management framework. However, starting from 18C, you can create custom Data Lifecycle Policies for incident and contact archive/purge for your business needs.
All fields used for configuring the filter conditions on Data Lifecycle Policies need to be indexed as part of a single index. You can create an index on up to four custom attributes through the Object Designer. You cannot however combine standard fields and custom attributes to create an index and consequently use a combination of standard fields and custom attributes to configure the filter conditions in a Data Lifecycle Policy. Custom Fields are also not supported in the Data Lifecycle Management framework.
Please note: In order to create a policy you must start with a filter template and they only exist if there is an associated custom sub-object with indexed fields (more on this below), or you are using one of the following standard Incident filter templates:
The following standard indexes on Incident objects are available to use while creating Data Lifecycle Policies:
- Interface, Status, Closed
- Status, Closed
You can see how these match up to the incidents table by looking at the data dictionary. The indexes there include these two. Keep in mind that even though there are other indexes for the incidents table, these are the only two used for standard filter templates.
You can search for a filter template using the first field of the index or the first letter of that field.
For example, if you want to create a policy to purge all ‘Closed’ status incidents for a given interface after ‘X’ (fixed) number of days after an incident has been closed, you can use the filter template consisting of Interface, Status and Closed by searching for a filter template with the first field i.e. “interface”. If you want a common policy across all interfaces, you can edit the standard policy using Status and Closed fields and search for it with “s” or “Status”.
The true potential of custom Data Lifecycle Policies can be realized when using custom attributes. To use custom attributes as filters in the custom Data Lifecycle Policy, you need to create an index with all the required custom attributes you plan to use as filters in the Data Lifecycle Policy through the Object designer. Once the index is available and deployed, you can create a new custom policy by searching for the filter template consisting of the same set of custom attributes using the first field of the index as key while searching for the filter template and using it to populate the custom policy. If searching for the first field of the index does not work, search for the description of that field as defined within the Object Designer.
For example, if there is a ‘lastContacted’ date data type custom attribute created on the Contact object containing the date field when the Contact was last communicated with and you want to purge Contacts whose last communication was older than one year. You can automate Contact purge by creating a custom Data Lifecycle Policy on Contact object using ‘lastContacted’ as the filter. Note that you will have to first create an index on the ‘lastContacted’ custom attribute through the Object Designer and deploy it before it becomes available as a filter template in the Data Lifecycle Policy workspace.
A data lifecycle policy, once enabled, will continue to exist and run indefinitely. Data, once purged, is irretrievable. So, enable a policy only after due diligence and validation that the business does not require the data to be purged.
FAQs
Q) Will enabling a Data Lifecycle Policy affect production performance?
It could. As with any other data on your site, it is always a best practice to identify how much data is being targeted prior to removal. While the code is written to remove these records efficiently, large datasets can still cause undesired table locks and timeouts. Therefore, if you are looking to remove large portions of data (like transactions) it is recommended you do so in batches.
ex:
The default transaction policies will purge transactions older than 6 months. If that timeframe constitutes millions of records on your site you should add an additional operator for "date created" to reduce the targeted data set. Note, you cannot modify the entry for less than six months but you can add to it to achieve your desired retention policy.
Q) How do I view table sizes for my site?
Table sizes are available in Configuration Assistant. For more information refer to the Configuration Assistant User guide > Data Storage Table
Q) Will the purging start right away and if yes, is it best practice to edit this configuration outside peak hours?
Since the execution of Data Lifecycle Policies is scheduled to execute periodically, it is not determinable when the purging (or archiving) will start. It is better to configure it with the assumption that it will start immediately.
Q) Is there a way to monitor the progress of my Data Lifecycle Policy?
There is no explicit way to monitor execution of Data Lifecycle Policies. However, you can check deleted_recs to see if any records have been purged via Data Lifecycle Management (DLM) using source level as a reference.
Q) How can I add a custom period to purge transactions if I want to keep transactions older than 6 months?
Canned conditions created for canned DLM policies are of two types. In one case, they cannot be edited (e.g. transaction purge period at least older than 6 months) at all. In other cases, only the values can be edited (e.g. incident archive/purge period value in standard policies).
To increase the time period value to purge transactions above 6 months, you can create an additional condition. For example, you can create a filter condition to delete transaction records older than 10 months.
Note that if you give a custom filter value less than 6 months, it will not take effect due to presence of standard filter limiting the purge to older than 6 months.
Q) Why has my Data Lifecycle Policy been disabled automatically?
A Data Lifecycle Policy will be disabled by the system if the filter conditions configured at the time of Data Lifecycle Policy creation does not hold true at runtime. e.g. After the custom policy creation and deployment, if you delete the custom index created on ‘lastContacted’ field on the Contact Object or the condition does not validate due to invalid filters then the custom policy will automatically be disabled at runtime when the policy executes.
Q) How do I setup a new policy for a standard object (Contact, Incident or Transaction)?
Answer ID 10538: Creating a Data Lifecycle Policy for a standard object
Q) How do I setup a new policy for a custom object?
User Guide Link: Creating custom object Data Lifecycle Policies
Q) How do I setup access to Data Lifecycle Management?
Answer ID 10539: Access to Data Lifecycle Policies
Q) How to know if the standard policies for Data Lifecycle Management are enabled?
Answer ID 11243: Viewing the current archive polices on my site
