Python and HCL scripting in Robots

The Robots app provides native, cloud-based scripting that you can use for data analysis or for automating portions of your organization's Diligent One workflow. Robots scripting combines the full power of Python and all its supported libraries with HighBond Command Language (HCL), our custom Python library for domain-focused data analysis and Diligent One automation.

Robots continues to support data analysis and data automation using ACLScript and scripts uploaded from Analytics. So you can benefit from any existing investment in ACL (Audit Command Language) and all the possibilities offered by Python, an industry-standard programming language, and HCL.

Three types of robots

The Robots app contains three types of robots. While similar in many ways, the different robot types support different scripting scenarios:

• ACL robot Supports full-scale data analysis with scripts you upload from Analytics, and scheduled tasks. Scripts can run in the cloud, or on-premise.

• HighBond robot Supports full-scale data analysis with Python/HCL scripts created directly in the robot, and scheduled tasks. Scripts can run in the cloud only.

• Workflow robot Supports light data analysis, or automating portions of your Diligent One workflow, with Python/HCL scripts created directly in the robot, and scheduled or event-based tasks. Scripts can run in the cloud only.

  Note

  To access Workflow robots, you must be assigned the System Admin user type with a Professional subscription.

  The intention is that a System Admin with a Professional subscription will be able to configure tasks in Workflow robots that are triggered by Diligent One events. This functionality is still being developed and not yet available.

Comparison of the robot types

The three robot types are intended for different purposes and are functionally separate from one another. You cannot mix ACLScript and Python/HCL in a single robot.

Robot characteristic	ACL robot	HighBond robot	Workflow robot
Intended purpose	data analysis and automation		Diligent One platform automation
Supported language	ACLScript	Python Pandas, NumPy, Requests, and other standard Python libraries HCL custom Python library (HighBond Command Language)
Script authoring location	Analytics	Robots
Script execution location	cloud on-premise	cloud
Options for running a task	scheduled run ad-hoc (manually)		triggered by Diligent One event scheduled run ad-hoc (manually)
Maximum task run time	cloud-based Robots Agent: 60 minutes on-premise Robots Agent: controlled by customer network	No limit Note A 12-hour limit is planned but not yet implemented. If you need to stop a task that has been running for too long, you can click Cancel task run in the Task runs tab.	15 minutes
Maximum tasks running simultaneously	5	5	50
Required permissions	be a collaborator for the robot degree of access controlled by the collaborator's robot role (Owner, Editor, Reviewer)		System Admin user type with a Professional subscription

Robots script editor

HighBond robots and Workflow robots contain an integrated script editor for authoring Python/HCL scripts and running them interactively (manually). The script editor is modeled on the Jupyter Notebook App, a development environment popular in the data science community. The script editor uses a Cell-based approach to script authoring.

Cell-based approach to script authoring

The most obvious feature of the script editor is the ability to use multiple cells to contain the individual blocks of code that make up a single script. If you are not familiar with Jupyter Notebook, the cell-based approach to script authoring may initially look complicated. In fact, it is easy to use and has benefits you will quickly come to appreciate.

Key characteristics of cell-based script authoring

• Multiple cells are optional You do not have to use multiple cells if you do not want to. You can author and run an entire script using a single cell.

• Logical divisions Cells allow you to divide a script into logical blocks of your own choosing, and to manually run the blocks individually, in any order, or as an entire sequence.

  For example, a basic four-cell arrangement for a script could be:

  • first cell import Python libraries

  • second cell import data

  • third cell perform analysis on the data

  • fourth cell output the results of the analysis

• Benefit of cells Using cells for script development has these benefits:

  • You can work on individual cells in isolation, testing and improving them, while temporarily ignoring script logic contained in other cells.

  • After you run the initial import cells once you do not need to keep running them as you develop and repeatedly test subsequent cells.

  • You can use cells to impose a visual order on your scripts, making the relation between the different logical blocks easier for you, or someone else, to see and understand.

• Sequence indicator Beside each cell is a numeric indicator that displays the order in which the cell has been run during a scripting session. For example: [7].

  When you run a cell, the indicator increments, taking into account any other cells that you have run in the interim. So, for example, an indicator could change from [7] to [10] if you had run two other cells in the interim, or one other cell twice.

• Session memory and stored values Dataframes and variable values held in memory during a scripting session are updated based on the sequence in which you run cells.

  Typically, you structure cells to run sequentially from top to bottom during scheduled, unattended script execution. Certain dataframes or variable values may be progressively updated based on script logic. If you manually run cells out of order, the current state of a dataframe or a variable value may not be what you assume it to be and an error or an inaccurate result could occur. To restore the dataframe or the variable value to its intended state, just re-run all cells referencing the dataframe or the variable in their intended order.

• Display of cell output By default, running a cell displays only the output of the last expression in the cell. If you want to see the output from multiple expressions in a cell, you can put the expressions in separate cells. However, a more convenient approach is to set a configuration option in the script editor.

  • Configure cells to display the output of all expressions:

    from IPython.core.interactiveshell import InteractiveShell
    InteractiveShell.ast_node_interactivity = "all"

  • Reset cells to display the output of only the last expression:

    from IPython.core.interactiveshell import InteractiveShell
    InteractiveShell.ast_node_interactivity = "last_expr"

  Tip

  You can use the configuration setting to switch back and forth between the two modes as required. For additional configuration options for cell output, consult Jupyter or IPython documentation online.

Keyboard shortcuts in the Robots script editor

Keyboard shortcuts in the script editor provide convenience and speed as you develop a script. In the script editor, to see a list of available shortcuts, use either of these methods:

• Press Ctrl+/

• Click the Keyboard shortcut icon in the title bar

The available shortcuts are listed below.

Keyboard shortcut	Action
Ctrl + Shift + Enter	Run selected cell
Ctrl + Shift + M	Run all cells sequentially
Ctrl + Shift + Y	Run selected cell and all cells above
Ctrl + Shift + U	Run selected cell and all cells below
Ctrl + Shift + I	Stop script execution
Ctrl + Shift + K Ctrl + Shift + Up Arrow	Select cell above
Ctrl + Shift + J Ctrl + Shift + Down Arrow	Select cell below
Ctrl + Shift + H Ctrl + Shift + Home	Select first cell
Ctrl + Shift + L Ctrl + Shift + End	Select last cell
Ctrl + Shift + A	Add cell above
Ctrl + Shift + B	Add cell below
Ctrl + Shift + D	Delete selected cell
Ctrl + Shift + Z	Move cell up
Ctrl + Shift + X	Move cell down

Jupyter Notebook features not included in the Robots script editor

As noted above, the Robots script editor is modeled on the Jupyter Notebook App. These Jupyter features are not included in the script editor, although some of them may be included over time as the product continues to evolve:

• Cell-level operations:

  • undo/redo cell operations

  • copy/paste cells

  • split/merge cells

• Markdown cells (for text content)

Saving script output to a log file

When you run a Python/HCL script using a robot task, you have the option of saving the script output to the log file that accompanies each task run. The log is a JSON file with the same name as the task (task_name.json).

To enable the option, select Save script output to the task run log file when you save and commit a version of a script. The option is separately configurable for each version of a script, so if you want to save script output, make sure that you select the option for the specific script version that the task runs.

Script output refers to anything contained in the stdout stream produced by a script. Typically, stdout is anything that the script prints or displays to screen when you run a script interactively. Essentially, by selecting Save script output to the task run log file, you are permanently capturing everything displayed on screen. This captured output can be useful to review when you are developing and troubleshooting scripts.

Additional information:

• The option applies to tasks only Even though you select Save script output to the task run log file in the Robots script editor, the option applies only to scripts that are run by a robot task. The option has no effect when you run a script interactively in the script editor.

• Error messages in the task run log file Python error messages (the stderr stream) are always included in the task run log file, regardless of how you configure Save script output to the task run log file.

• Accessing the task run log file A task run log file is available for each task run in the Task runs tab. Select a task run, and in the Task run details side panel, the log file appears in the Output section. You can download the file to your computer for review.

Making the transition from ACLScript to Python and HCL

Python is a general-purpose programming language and is significantly more complex than ACLScript, which is a relatively straightforward scripting language originally designed for the audit analytics domain.

Our intention is that over time HCL will replicate many of the commands and functions available in ACLScript. To get started with HCL, our custom Python library, requires that you have at least basic Python skills. The advice for those looking to make the transition from ACLScript to Python/HCL is to take a good introductory Python course.

Python coding conventions

You will need to get used to certain coding conventions that are not enforced in ACLScript but are enforced in Python and HCL:

• Case sensitivity Python and HCL syntax is case-sensitive, whereas ACLScript syntax is not.

  In Python:

  • print("Hello World!") works

  • PRINT("Hello World!") gives an error

• No abbreviating Python and HCL function, method, and parameter names must be spelled out in full. Unlike ACLScript, abbreviations are not supported.

• Indentation matters Python uses indentation to implement control structures such as conditional statements and loops. Incorrect indentation causes errors. ACLScript ignores indentation.

Python external libraries

Python supports a large number of external libraries, or packages, that you can install to extended the functionality of the core Python installation and the Python standard library. A Python external library is a collection of related code modules designed to do specific things beyond what you can do with core Python and the standard library. Or it makes certain things quicker and easier to do than they are in core Python.

External libraries included in Diligent One Python

The Diligent One installation of Python includes certain external libraries by default, so you do not need to explicitly install them. With the exception of Diligent's own HCL Python library, you do need to import the pre-installed libraries in order to use them in subsequent Python/HCL script logic.

External libraries pre-installed in the Diligent One Python environment:

Library	Description	Python import statements
HCL	The HCL library, or HighBond Command Language, is Diligent's custom Python library for domain-focused data analysis and Diligent One automation.	not required
Pandas	The Pandas library provides a powerful set of tools for data manipulation and data analysis in Python, including dataframes, which are spreadsheet-like, or table-like, two-dimensional arrays of data.	import pandas or import pandas as pd (optionally creates a short-form alias for the library name)
NumPy	The NumPy library offers fast and powerful scientific computing in Python, including basic algebraic and statistical operations, advanced mathematical operations, and building and performing calculations on multidimensional arrays.	import numpy or import numpy as np (optionally creates a short-form alias for the library name)
Requests	The Requests library provides a lightweight syntax for making HTTP requests in Python and interacting with web-based APIs.	import requests or import requests as req (optionally creates a short-form alias for the library name)
Cryptography	The Cryptography library supports incorporating cryptographic algorithms in Python.	import cryptography or import cryptography as crypt (optionally creates a short-form alias for the library name)

Installing additional external libraries

HighBond robots support installing additional external libraries beyond those that are pre-installed with the Diligent One Python environment. To install an additional library, use pip , a tool for downloading and installing Python packages. By default, pip downloads the packages from the Python Package Index (PyPI), an open-source repository of Python software. You can also use pip to install packages from elsewhere.

For example, to install pyexcel , a Python library for working with Excel files and data, use this command in the Robots script editor:

pip install pyexcel

Even if you install an external library, there is no guarantee that the Robots script editor supports all the functionality of the external library. For example, it may not support displaying charts and plots available in certain external libraries.

Using Python and HCL to work with APIs

You can use Python/HCL scripts in Robots to work with APIs – both third-party APIs and Diligent One's own API.

Work with a third-party API

To work with a third-party API, begin by importing the Python Requests library in the Robots script editor:

import requests

You can now make API requests using Python's request syntax and receive responses. For a simple example of using Python to make a third-party API request in Robots, see Part 3: Extend your reach in Get started with Robots scripting and the Diligent HighBond API .

Explaining in detail how to use Python with third-party APIs is beyond the scope of our documentation. However, many online resources exist that can provide detailed guidance.

Work with the HighBond API

The HighBond API is the developer interface for the Diligent One Platform. You can use the API to automate Diligent One interactions and workflow, build reporting, and integrate Diligent One with other systems in your organization.

HCL includes methods that make interacting with the HighBond API straightforward. For more information, see HCL HighBond API methods.

You can also use the Python Requests library with the HighBond API. However, for ease and simplicity, we recommend using the HCL methods.

For an introductory tutorial that covers using Python and HCL with the HighBond API, see Get started with Robots scripting and the Diligent HighBond API.