Use case: Configure reports for supporting alternative (A/B) testing in ForgeRock Identity Cloud

[lucy.billington]

Use case overview

Providing reports on alternative (A/B) testing for user journeys is a common use case that is easily achieved with ForgeRock Identity Cloud. An example of when you may want to use A/B testing is to compare different journeys for rates of abandonment during registration.

In this use case, we’ll demonstrate how to:

• Use the Analytics dashboard to compare different journeys.

• Create a journey that times different A/B paths, and use Grafana to report on the different paths taken by end users.

Using the Analytics dashboard

You can use the built-in Analytics dashboard to gain valuable insights on your Identity Cloud tenants, including accessing a breakdown of journey failure or success and comparing different journeys.

A comprehensive overview of the Analytics dashboard is provided in the product documentation. See Identity Cloud analytics dashboard for further information.

To access the Analytics dashboard and view journey details:

1. Sign in to the Identity Cloud admin UI using your admin tenant URL, in the format https://<enant-name>/am/XUI/?realm=/#/.

2. Click Dashboard to view the Analytics dashboard for the selected Realm.

   
   
   Example Analytics dashboard

3. Click Last 30 days if you want to change the time period for the trendline charts. All dates and time periods are based on UTC time.

4. Click Compare to last period to compare the trendline for the last time period. A dotted line represents the trendline for the previous time period

5. Scroll down to view the Journeys graph.

   

   Example Journeys graph

   NOTE: To filter the journeys shown, click View All Journeys and select the journeys or categories you want to view.

6. Place your cursor on the point on the trendline and click View Detail to get a metric breakdown, by percentage or number.

   
   
   Example failure and success of journeys by percentage

See Access journey pass/fail details for further information on getting a metric breakdown for journeys.

Creating a user journey that times different A/B paths

In this example we’ll demonstrate how to build a simple journey with two branches, named A and B. The journey includes Timer Start and Timer Stop nodes that provide monitoring endpoints which you can use with Prometheus. This allows you to report on the A and B paths taken by users, via third-party tools such as Grafana.

This use case has three parts:

1. Create a user journey
2. Run through the end user journey taking different A/B paths
3. Configure the Grafana dashboard and view the A/B path results

Prerequisites

First, you’ll need to download and run a Docker-based example Grafana dashboard. The demo requires that you have Docker Desktop installed, and requires macOS.

See Monitor system for performance for further information.

Create a user journey

1. Sign in to the Identity Cloud admin UI using your admin tenant URL, in the format https://<tenant-name>/am/XUI/?realm=/#/.

2. Go to Journeys > New Journey.

3. Enter a unique name for the A/B timer journey, select which identities will authenticate using this journey, (optionally) enter a journey description, and click Save.

4. Create a journey similar to this:

   

   Node descriptions:

   • Platform Username - Prompts the user to enter their username. See Platform Username node for further information.
   • Identity Existing User - Checks the username against the identity repository. See Identify Existing User node for further information.
   • Choice A or B - This is a Message node that prompts the user to choose a path, either A or B.
   • Timer Start nodes - Starts a named timer metric, which you stop with a Timer Stop node. See Timer Start node for further information.
   • Choice Collector nodes - Defines two or more options to present to the user when authenticating. See Choice Collector node for further information.
   • Timer Stop nodes - Records the time elapsed since evaluation passed through the Timer Start node in the specified metric name. See Timer Stop node for further information.

5. Click the Choice A or B node (Message node) and add the message that will be presented to end users, for example, Choose path A or B.

   

6. Click the A Timer Start node and enter the Start Time Property, for example, authentication.A.

   

7. Click the B Timer Start node and enter the Start Time Property, for example, authentication.B.

8. Click A Choice Collector node and configure the following:

   • Choices: A, B, C, D
   • Default Choice: A
   • Prompt: Choose a path

   

9. Click B Choice Collector node and configure the following:

   • Choices: E, F
   • Default Choice: E
   • Prompt: Pick E or F

   

10. For each of the A Timer Stop nodes:

    • Configure the Start Time Property to match the Start Time Property in the A Timer Start node, for example, authentication.A.
    • Configure the Metric Key as the identifier of metric to update when this node is processed, for example, authentication.A.A for A Timer Stop A, authentication.A.B for A Timer Stop B, and so on.

    

11. For each of the B Timer Stop nodes:

    • Configure the Start Time Property to match the Start Time Property in the B Timer Start node, for example, authentication.B.
    • Configure the Metric Key as the identifier of metric to update when this node is processed, for example, authentication.B.E for B Timer Stop E and authentication.B.F for B Timer Stop F.

12. Click Save to save the journey.

Run through the end user journey taking different A/B paths

In this step we’ll run through the end user journey several times, taking different paths, in order to generate test data for reporting purposes.

1. In the Identity Cloud admin UI, go to Journeys.

2. Click the A/B journey you created previously and copy the Preview URL.

   

3. Paste the preview URL into a browser using Incognito or Browsing mode.

4. Enter a test user’s username in the Sign In screen and click Next.

   

5. Click either A or B.

   

6. Select a path, A, D, C, D or E, F (depending on your selection in the previous step).

   

7. Repeat Steps 3 through 6 multiple times using different users and taking different paths.

Configure the Grafana dashboard and view the results

For this use case, we’ll create a simple Grafana dashboard that provides a count based on the metric keys in the A/B journey and displays a count of the different paths taken by end users in a pie chart.

1. Download and run a Docker-based example of a Grafana dashboard (if you haven’t already done so).

2. On the Grafana Home page, go to Dashboards > New Dashboard.

3. Click Add a new Panel.

4. Click Query and select the first metric you want to report on. In this case, we’ll choose am.authenticaton_a_a_count.

   

5. Click + Query to add another query.

6. Repeat Steps 4 and 5 for the remaining counts (am.authenticaton_a_b_count through am.authenticaton_b_f_count).

   

7. Click Last 6 hours and change the time range (optional).

   

8. Click Time Series > Visualizations and select Pie Chart.

   

9. Enter the panel title, for example, “Journey Path Total count over Time”.

10. Click Apply.

    The pie chart is displayed, showing a count of the different paths end users have taken.

    

    TIP: To display the labels on your pie chart as A, B, C, etc. rather than the metric text you can use overrides. Click Overrides > Add field override > Fields with name and choose the metric. Then click Add override property and enter the text you want to display.

    

11. Click Save to save the dashboard.

Additional resources

Documentation:

• Identity Cloud analytics dashboard.
• Journeys
• Monitor system for performance

Training videos:

• Getting Started: Identity Cloud Analytics Dashboard
• Getting Started: Monitor Your Tenant

Acknowledgments: David Lee