Dynamic Data in Forms

   MaestroPreviously known as Transact Maestro.  |    Form Builder  |  5.1 & Higher This feature is related to 5.1 and higher.

Maestro forms are made up of HTML, CSS, and JavaScript, and are executed when the form is deployed in a web browser. You can think of Maestro forms as AngularJSAngularJS is a structural framework for dynamic web apps. It lets you use HTML as your template language and lets you extend HTML's syntax to express your application's components clearly and succinctly. AngularJS's data binding and dependency injection eliminate much of the code you would otherwise have to write. For more information, see http://angularjs.org/ single page applications that run in a browser. Almost every form interacts with a form user who, at some stage, is required to enter some data into the form.

Data in a Maestro form representing form field content is formatted as XML. When a form is first rendered, an XML document is generated to contain the data collected by the form. This XML will eventually be populated or updated and saved when the form is submitted back to the Manager server.

Maestro forms include the concept of repeats to model real-world repeatable data, such as lists of previous addresses, lists of applicants and so on. Fields in a form can be wrapped inside repeating components, with various means of controlling the number of repeats dynamically.

A form generally needs access to some type of dynamic data, including personal data contact details, reference data lists of city names, and volatile data, such as current interest or exchange rates. This data is regarded as dynamic as its nature changes according to the information the form is designed to collect, and who is the form.

A well-designed Maestro form should make use of dynamic data that is external to the form itself. That is, you should not hard-code data into the form, but rather access it dynamically when the form is run. Keeping data outside of the form definition leads to a form that can dynamically adapt to changing circumstances. This adaptability can reduce the overall impact of any changes to the data content, or the business rules that govern the data.

A form still needs its supporting data, regardless of where it's kept, so either the form must be given all the data it needs, or the form must collect the data itself. Accordingly, dynamic data makes its way into a form via one of two methods:

  • Prefill – data is given to the form by Manager, which writes values into the form's XML file during rendering.
  • Dynamic Data Services – the form gets its own data when it is required by executing JavaScript code, which calls Manager dynamic data services.

Dynamic data can be used for several different purposes in a form, including:

  • Presentation – to display information to a form user completing the form, for example, displaying terms and conditions that may vary according to circumstance.
  • Calculation – to calculate real values based on volatile variables such as current interest rates, share prices, or currency exchange rates.
  • Experience – to help a form user enjoy a better user experience by allowing them to pick from lists, or provide auto-completion of, for example, country or city names.
  • Validation – to ensure forms contain data that is not only correctly formatted but conforms to business rules and data relationships.
  • Decision-making – to determine the behavior of the form in response to user input, for example, directing users to additional pages for confirming eligibility or entitlement.

This document describes the Prefill and Dynamic Data Services methods of working with dynamic data.

Prefill Dynamic Data

Manager applies form prefills from all enabled prefill sources in the order listed in the table below.

Source Managed By 1 Mapped By 2

XML Seed file (enable) 3

Forms/Forms/form/Form Dashboard/Form Versions box/Data Config link/Form XML Data tab

Input XML Prefill Mapping

Organization properties

Forms/Organizations/organization/Properties tab

Property Prefill Mapping

Form properties

Forms/Forms/form/Form Dashboard/Form Versions box/Properties link

Property Prefill Mapping

URL request parameters

Form Dashboard/Form Versions/Data Config link/Request Param Prefill Mapping tab

Request Param Prefill Mapping

User profile properties

Security/User Accounts/User/User Profiles tab/profile name/User Properties tab

Property Prefill Mapping

Form Prefill Service (XML)

Forms/Forms/form/Form Dashboard/Form Versions box/Services link/Services tab/Form Prefill Data Service dropdown
Services/All Services/service/Groovy Script tab

Not required

Note

1 - The Managed By column shows where to go in Manager to manage the prefill source content.

2 - The Mapped By column shows where to apply mappings for that source. The destination for all prefill data is the form XML.

3 - XML seed prefill is not enabled in a Form Space by default, it needs to be enabled on the Form Space >Properties Edit tab

Prefill data applied at higher rows, shown in the table, will be overwritten by enabled lower rows if there are multiple mappings from prefill sources to the same form field.
A Maestro form designer does not need to do anything extra or special to have a form's fields prefilled. It's just 'there' as though it was given an initial value by the form designer at design time. Manager does all the work to configure mappings and push values into the form XML without the form even being aware of what has occurred.

Form Prefill Service

Form field content in a Maestro form is formatted as XML. The initial XML of a form can be provided at render time by a custom Groovy service – a Form Prefill Service. The service to be used for a form is specified in the Form version Services tab. XML from the Form Prefill Service is directly pushed into the form XML, without the need to map values from source to destination. If the XML contains repeating elements, repeating components are dynamically created in the form.

Prefill Mapping

All other prefill sources require mapping to direct their values into the structure of the form XML. A mapping target form field is specified by an XPath expression. The source field of each mapping is specified differently according to the prefill source:

  • Property – source is specified by property name.
  • Input XML – source is specified by another XPath expression.
  • Request Parameter – source is specified by URL request parameter name.

In the case of input XML mapping, repeating components in a form are automatically set to the number of repeated elements of the input XML, provided that the mapping is made between the parent items of the repeated content.

Property Prefill

Properties are defined at the following scopes, where each scope has slightly different rules regarding properties:

Property Types

Except for untyped form properties, all properties must be associated with a 'Property Type'. A property type defines the nature of a property but does not provide a value for one.
Property types are a combination of:

  • A unique name within the Manager instance.
  • A scope – Organization, Form or User.
  • A data type - Boolean, HTML, Image, JSON, List, Long Text, Number or String.
  • An optional organization – if not provided, the property type will be global and be available to all organizations that want to use it.

Organization Properties

An organization can define only one property of a given property type, but It is not necessary to define a property for each property type if one is not needed.

Form Properties

Each form property may be of a defined property type or may be un-typed. A form can define only one property of a given property type, but It is not necessary to define a property for each property type if one is not needed. An untyped form property must have a property name which is unique within the form, a data type - Boolean, HTML, Image, JSON, List, Long Text, Number or String, and a value.

User Profile Properties

A user profile property must be of a defined property type, or one of the pre-defined property names: Email, Family Name, Given Name, or Mobile.

Prefill Caveats

The list below identifies several of the caveats surrounding the use of form prefill.

  • A form is only prefilled when it is first rendered.
  • A form that is saved and retrieved is not prefilled again, even if property or seed values in Manager have changed since the initial render.
  • You can prefill form fields only with string values.
  • A field cannot be prefilled unless it's represented in the form XML – which means its property 'Include in submission data' must be ticked while designing the form in Maestro.

Dynamic Data Services

Unlike prefill, which happens like magic, a form must implement its own logic to fetch data from a dynamic data service. A form is in absolute control over if, when and how it calls dynamic data services. While it is executing, a Maestro form is always able to stay in communication with the server that rendered it, so it's quite easy for a form to ask for more data when it decides it needs it. Manager supports two types of dynamic data service, both of which are implemented in the Groovy language:

  • Groovy Dynamic Data Services.
  • Fluent Dynamic Data Services.

The difference between the two is in the way that the Groovy APIs are designed. Both choices have no impact on the way that a Maestro form calls them. 'Fluent' is a style of designing APIs. See towards the end of this document for a brief explanation. A Maestro form can call either service type, and in fact, has no knowledge of which type has been used to implement the Dynamic Data Services. When code in a Maestro form makes calls to dynamic data services, complex parameters in JSON format can be passed. Complex values may also be returned in the response, again typically in JSON format, although the exact format returned is up to the service author.
Dynamic data services are called using two different techniques:

  • Dynamic Data Button – a Maestro component that can be configured to call any dynamic data service. It is a visual component you configure with point and click using the Maestro user interface but can be hidden and activated by code instead, if required.
  • DynamicData.call – a Maestro API method that requires more effort to implement, and some more JavaScript programming knowledge to use. The Dynamic Data Button uses this API method to do its work behind the scenes.

Dynamic Data Button

This Maestro component provides all the capability a Maestro form needs to call dynamic data services. You configure it entirely within the Maestro user interface, where you can take advantage of input and output mapping dialogs to collect input parameters and distribute results. You can also write code in the On Success and On Failure rules that the component provides.
The Dynamic Data Button is a visual component, so it will be visible in the Maestro design mode, unless it is excluded, and will be visible at runtime unless it is hidden with a visibility rule.

DynamicData.call

This method calls a Manager dynamic data service, passing in any parameters you provide, and returns a Promise object from which you can obtain the response, so you need to know a little more about JavaScript programming to use it.

You will need to acquire the parameter values, marshal them into method parameters, call the API method, then deal with the returned Promise to retrieve the returned value when it is ready. You will then need to un-marshal the values from the response and distribute them to other components on your form. You will also need to write code to deal with errors. Unless you have a good reason, it's preferable in almost all cases to use Dynamic Data Buttons to call dynamic data services, because it reduces the overall amount of custom JavaScript code you need to write.

One case where you will need to use DynamicData.call is if the response returned from a service call contains nested properties, as the Dynamic Data Button can only map output properties at the top level of the response object. The following example show an JSON response with a nested object bsb, which is a child of data and not a child of the root, so it can't be output-mapped by a Dynamic Data Button.

Nested JSON object:


{
	"success": true,
	"data":{
		"bsb": "012-002",
		"bank": "ANZ"
	}
}

You will need to use DynamicData.call in this case, unless you can have the service changed to return a flat JSON object like:

Flat JSON object:


{
	"success": true,
	"bsb": "012-002",
	"bank": "ANZ"
}

What to fetch

A dynamic data service can return data from absolutely any source, including data held within Manager, as well as data from external sources, such as public or private web services. One source of data that can be easily accessed is organization property values. Complex JSON objects can be returned. from Is there supposed to be more here?

When to fetch dynamic data

From an efficiency aspect, it is preferable as much as possible to fetch data only once, when a form loads, in the Form Load Rule. Such data can then be retained for the lifetime of the form. However, not all data lends itself to this usage pattern.

Consider that when a new form loads, for example, nothing may be known about the person filling the form in, so you cannot yet ask for their personal data. It is important to consider that you may not want to load a list of all city names in the world until you at least know what country you need city names for. There are choices that need to be made about exactly when and how often to fetch dynamic data, which largely depend on the type of dynamic data required.

Note

Do not call DDS in a calculation rule.

Dynamic data can be loosely categorized as:

  • Personal Data.
  • Reference Data.
  • Volatile Data.

How to test dynamic data

When developing a Maestro form, you can't call services for the following reasons:

  1. It is likely that your form is located on one of the shared Maestro instances which will not have the service available. On the shared instances we do not install services as this should be done on your organization Journey Manager instances.
  2. Even if it was available on the same environment, previewing the form generates the form and injects it into an iFrame and only is running in your browser, it is not part of a transaction so Journey Manager would not acknowledge the service call it would not have a valid request key that Manager requires to handle a dynamic service call.

This means that you can't invoke DynamicData.call() in preview without adding preview data.

You can add the following script to your code that does something like:

if (Resource.preview) { Form.addPreviewDDSData("myService", {name: "Glenn", team: "Journey Maestro"}) }

Or

if (Resource.preview) { Form.addPreviewDDSData("myOtherService", function(params) { if (params.name) { return {message: "Hello "+params.name } } else { return {message: "Hello unnamed person"} } }); }

This way you can test your function calls while providing stub data as publishing the form and pushing it to your Journey Manager instance and then testing your form does have a longer round trip than using preview. For more information, see the API documentation for Form.addPreviewDDSData(serviceName, dataOrHandler)

Personal Data

Personal data is related to the individual(s) interacting with the form. By definition, you can only fetch personal data once you have identified the individual(s) concerned. This data could include:

  • Existing policies or accounts.
  • Employment history.
  • Contact details.

In this case, personal data can be fetched only once enough identifying information has been captured to perform a data lookup. It should not be necessary to fetch that same data again during the form's lifetime.

Reference Data

Reference data typically consists of values or lists that don't change all that often, so you can probably safely obtain it when the form first loads and keep it for the entire form session. This type of data is typically used to allow applicants to pick from dropdown lists, to validate form content and to provide fixed values on forms, such as:

  • Lists of countries.
  • Vehicle models.
  • Insurance or finance products.
  • Phone numbers.

It is quite a common scenario to select an initial value from a reference data list of, say countries, and make further calls for more specific data such as city names. Similarly, selecting a vehicle manufacturer may require further calls for models, years, colors or related options.

Volatile Data

This is data that is constantly changing and should be called for as late as possible, just before you need use in a calculation, such as:

  • Share prices.
  • Exchange rates.
  • Interest rates.
  • Number of seats remaining for a flight.
  • Account balances.

In this case, volatile data may need to be fetched more than once, immediately before use each time.

Accessing Prefill Data

It's useful to understand the data processing pipeline that Maestro goes through when populating a form (at least at a high level).

  • The prefill XML (provided by prefill services in Transact Manager) is parsed and turned into an in-memory Javascript object graph with the same structure as the XML.
  • An object graph called "data" is constructed, using the fields in the form to determine the structure of this object graph.
  • The "data" object is initialized from the prefill object.
  • The data object is further modified as calculation rules fire.

The "data" object is live, in that it is continually changing as the user interacts with the form, and calculation rules fire. The "prefill" object never changes. In some scenarios, it may be helpful to refer to the original prefill information.

We may want a display field to show the original value of a prefill value, and not change if that value is changed by the user. In this case, you can directly reference the prefill object like this: {{Form.prefil.TestText}}.

If you would like to see the prefill data object in your browser's debugger, type: maestro.Form.prefill at the console prompt.

Next, learn about component data binding.