Managing data anomaly tests

You can use the Anomaly Test feature to run anomaly tests on their organizations data to identify data items which do not conform to an expected pattern of the dataset. For example, data entries that are considerably different from the remainder of the data. You can run the anomaly tests for individual activities and scheduled to run daily, weekly, monthly, yearly, bi-weekly, bi-monthly, quarterly, or four-weekly. Data anomaly scanning is available for non-interval data activities.

For further information, go to Running a data anomaly test on non-interval data.

Running a data anomaly test on non-interval data

You can configure non-interval tests to scan for data anomalies. This test scans for duplicate entries, overlapping dates, and compares the previous year’s data. Non-interval data activities include types of data, such as invoices or supplier statements, and is the main scan used. You can schedule a non-interval data anomaly test when you are creating the activity or at a later date. You can also update and change these settings at any time.

1. In Diligent ESG, select Data Management and select GHG Activities.

2. Locate the activity that you want to test in the table and select Select in the activity row.

3. In the test details page, select Config to manage the test settings.

4. Select the Anomaly test tab, expand Non-interval data – Anomaly test, and select the Scan non-interval data for anomalies checkbox.

5. In the Activate column, select the Duplicate Entries for Non-interval Data checkbox to identify duplicate data based on matching units, amounts, and dates.

6. Select the Date overlap checkbox to detect overlapping start and end dates within activities or meters.

7. Select the Compare with same period last year checkbox, and in the Tolerance field, enter a tolerance percentage for the data to be above or below the previous years data before it registers as an anomaly. The default tolerance is 25%. Use this feature to identify significant discrepancies in the input quantities between periods and highlights anomalies if the percentage changes exceed predefined thresholds.

8. In the Anomaly test reoccurrence dropdown, select how often the scan runs.

9. In the Number of days after the scan when alerts appear field, enter the number of days after a scan occurs before a user is alerted to data anomalies.

10. In the First scan date field, enter or select the date for the first scan.

11. From the Scan historic data options, select Yes if you would like to scan data from a specific data, and in the First historical scan date field, enter or select the date.

12. Select Save. The test runs immediately.

The system now scans the non-interval upload data files, and you can view the results in the Analysis>Data Anomalies page. For further information about this page, go to Viewing and managing data anomaly tests.

To configure emails to alert users to the data anomalies, go to Managing data anomaly tests.

Viewing and managing data anomaly tests

You can use the Data Anomalies analysis section to check for and correct duplicate data and date overlaps, and compare the input quantity from one period to the same period in the previous year. You can filter results in both the chart and table using the Organisation or Geographic navigation tree, Data Filters, or table column filters.

1. In Diligent ESG, select Analysis and select Data Anomalies.

2. In the Organisation or Geographic navigation tree, select a company, business, or location to filter the tests shown in the table.

3. In the Data Filters section, expand Scope, Source Category, and Activity Type, and select or clear the checkboxes to filter the tests shown in the table.

4. In the data table section, in the fields below the column headings, enter a term or figure to filter the test results shown, such as a percentage in the Variance % column.

5. In the Type of test dropdown, select Duplicate entries, Data overlap, or Compare with same period last year to filter the results shown. Select All to view all test types.

6. (Optional) At the top of the data table section, select the Export to Excel icon or Export to CSV icon to export the table.

7. Select the Select Columns dropdown, select the checkboxes beside other columns that you want to add, and select Show/Hide.

8. Select Ignore to ignore an anomaly.

9. In a test entry row, select the Anomaly ID number to view a test and its results in more detail.

10. In the test details page, manage an anomaly record using one of the following options:

    • Select the Delete icon to delete the appropriate data entry in the table

    • Select an Entry ID number and edit the entries to their correct values

    • Select Ignore to ignore an anomaly

11. (Optional) Select View ignored anomalies or Return to current anomalies to switch between current and ignored anomaly tables.

12. (Optional) Select the Tests tab to view the status and progress of all the data anomaly tests. Select Results to return to the details of all the data anomaly tests.