Data Retention Management Overview

   Journey Manager (JM) Previously known as Transact Manager (TM).  |    System Manager / DevOps  |  v5.1 & Higher This feature is related to v5.1 and higher.

To maintain the system over time, data storage and retention needs to be considered. Manager provides data purging capabilities as well as flexible data storage to allow you to keep your system functioning well.

Why should I purge my data?

Security

Form Transactions can include sensitive information either business or personal depending on how the transaction have been designed. Purging this data after delivery provides a stronger security story.

Performance

In Transaction Manager the Transaction table is a heavy weight table used to store a lot of information about in progress transactions and is not designed for long term storage. When a transaction is completed an entry is created in the Transaction History table which is designed to provide a long term audit trail. Purging data and records quickly from the Transaction table will help maintain performance.

What can be purged?

Transaction Data

This is the forms XML, Data Extracts, Attachments and Receipts. This will include the majority of the users personal information.

Transaction Records

This is the actual record from the Transaction table. The main reason to keep this record in place for short period of time is for troubleshooting.

Transaction History Records

This is the audit log of transaction activity. For systems with high throughput there is a argument for archiving these records for long term storage and purging these records from Transaction Manager.

System Logs

There are numerous areas of system logs whose purge time can the controlled separately for troubleshooting purposes.

Manager comes with highly customizable data retention management capability to suit your company’s data requirements. This is instrumental in achieving the following:

Manager collects important information in relation to data retention management, which includes the status of any individual transaction to keep track of PII data that has been purged or is due for purging. This information can be found for each transaction in the Transaction Status tab as shown below.

This includes details about the data retention stage reached in the life cycle of this transaction. For instance, from this transaction data purging details we can tell that the PII data was purged (as indicated by Transaction Data Deleted being selected) a few minutes after it had been scheduled to be purged (as indicated by Actual Transaction Data Purge and Scheduled Transaction Data Purge). Also, the transaction data record is due to be deleted on the 25 March 2018 (as indicated by Scheduled Transaction Record Purge).

Note that after the 25 March, this transaction along with its submission details will no longer exist. Only a transaction history record remains containing some details about the transaction.

You can check the data retention summary, which is useful for monitoring the overall performance of the data retention management and assists in diagnosing and tuning of any data retention management issues.

You can set the data retention and storage settings globally and override important settings for individual organizations as needed.

Data Storage Configuration (Global)

In addition to data retention, you can also configure submission data storage. This means that you can choose how and where submission-related data (submission data, attachments and submission history data for saved forms) will be stored and retrieved by Transaction Manager.

By default, submission-related data is stored in the Transaction Manager database. However, data storage can be configured and changed at runtime. The functionality is encapsulated in a set of service definitions of type "Submission Data Storage". The storage service that is marked as the default for this service type will be used when storing submission data.

Data Retention and Storage Configuration (per Organization)

You can override important form-related settings for each organization individually.

The data retention policies you can configure are the maximum age of form submissions in general as well as delivered (including undeliverable) and saved (including abandoned) submission data. If you would like the global settings to apply for this organization, select "Use system default".

You can also set the data storage encoding for the organization if it is different to the system-wide setting. We recommend that you use "Compressed/Encrypted" for maximum security and minimal storage space requirements.

Finally, for additional security you can set a rollover interval for the security key for the organization. The security key is used by TM to encrypt submission-related data. All encrypted data is associated with its security key, so data encrypted before a security key change will still be accessible. If you choose a value other than "Never", a new security key will be automatically generated by a background job when required.

Manager has two modes of data retention policy, strict and relaxed, that determines the length of time transactional data is retained. The selected policy can be customized at the environment organization, or form levels.

Note

It's recommended to use the strict policy mode, which enforces less transactional, historical and log data retention as opposed to the relaxed policy mode.

Manager uses various scheduled jobs that automatically purge transactions that have reached their maximum age.

Next, learn about data retention policies.