Creating Custom Job Steps
If there is no system step that does what you need to do, a developer must develop a custom job step. The developer first writes a task-oriented or chunk-oriented CommonJS script module or a legacy pipeline to describe what the step does. The developer includes the module or pipeline in a cartridge along with a JSON or XML file that describes the step, and then uploads the cartridge to the site. The custom steps are then available to the administrator creating jobs. The topics in this section are.
You can define what a custom step does using a task-oriented CommonJS script module that exposes a function to be called as the main function for the job step. When administrators create jobs using Business Manager, they set parameters that are available as scriptable objects for the module's function and for the dw.job.JobStepExecution
object. The dw.job.JobStepExecution
object allows read-only access to information about the current step execution and job execution.
To control the exit status, the script module's function can return a dw.system.Status
object. If the script finishes with an unhandled exception, the exit status code is ERROR, and the error status flag is true by default. If no status object is returned and no exception occurs, the status code is OK by default.
The following example is a task-oriented CommonJS module.
You can define what a custom step does using a chunk-oriented CommonJS script module that reads and processes items in chunks of size S. If the list contains more items than can be processed in one chunk, a new chunk is started. A chunk-oriented script can include any series of processing steps, not just database transactions. A job step that uses a chunk-oriented script allows fine-grained progress monitoring because the number of elements that are written is updated each time a chunk is finished.
Chunk Processing Example
In this example, a list of 15 orders is exported to a file. Each chunk includes four orders.
-
Chunk 1
- Read Order 1.
- Process Order 1.
- Read Order 2.
- Process Order 2.
- Read Order 3.
- Process Order 3.
- Read Order 4.
- Process Order 4.
- Write Order 1.
- Write Order 2.
- Write Order 3.
- Write Order 4.
-
Chunk 2
- Read Order 5.
- Process Order 5.
- Read Order 6.
- Process Order 6.
- Read Order 7.
- Process Order 7.
- Read Order 8.
- Process Order 8.
- Write Order 5.
- Write Order 6.
- Write Order 7.
- Write Order 8.
-
Chunk 3
- Read Order 9.
- Process Order 9.
- Read Order 10.
- Process Order 10.
- Read Order 11.
- Process Order 11.
- Read Order 12.
- Process Order 12.
- Write Order 9.
- Write Order 10.
- Write Order 11.
- Write Order 12.
-
Chunk 4
- Read Order 13.
- Process Order 13.
- Read Order 14.
- Process Order 14.
- Read Order 15.
- Process Order 15.
- Write Order 13.
- Write Order 14.
- Write Order 15.
This chunk read, processed, and wrote fewer items than the previous chunks, because only three items were left on the list.
Function | Required or Optional | Purpose |
---|---|---|
read-function | Required | Returns one item or nothing if there are no more items. |
process-function | Required | Transforms items and applies business logic to them. It receives the item returned by the read function, performs a process, and returns one item.The item returned can be the same item that the read function returned if no processing logic is necessary, or it can be a new item of a different type. If the process function returns nothing, then the read function item is filtered and doesn't appear in the list of items to be written later. |
write-function | Required | Receives a list of items. The list size matches the chunk size or, if the number of items in the last available chunk is smaller, it is smaller. The write function returns nothing. |
total-count-function | Optional | Returns the total number of items that are available. Called by the framework exactly once before chunk processing begins. A known total count allows better monitoring, for example, to show that 50 of 100 items have already been processed. |
before-step-function | Optional | Executed before a chunk step begins. Implements logic before all items of all chunks are read, processed, and written. |
before-chunk-function | Optional | Executed before a chunk begins. Implements logic before a chunk of S items is read, processed, and written. |
after-chunk-function | Optional | Executed after a chunk finishes. Implements logic after a chunk of S items has been read, processed, and written successfully. |
after-step-function | Optional | Executed after a chunk step finished successfully. Implements logic after all items of all chunks are read, processed, and written successfully. |
When administrators create jobs using Business Manager, they set parameters that are available as scriptable objects for each exposed module function and for the dw.job.JobStepExecution
object. The dw.job.JobStepExecution
object allows read-only access to information about the current step execution and job execution. You can't define the exit status for a chunk-oriented script module. Chunk modules always finish with either OK or ERROR.
The following example shows a chunk-oriented CommonJS module.
In a task-oriented script module or pipeline for a custom job step, you can create an object of type dw.system.Status
to control the exit status and error message for the step. The Status
object includes a boolean flag that indicates whether or not the status is error, and also a status code and message. Messages appear in Business Manager and are written to log files. You can't define the exit status for a chunk-oriented script module. Chunk modules always finish with either OK or ERROR.
The job framework uses the following rules to detect the step status and determine if job execution can continue.
dw.system.Status.isError() | dw.system.Status.getCode() | Status Code | Error Status Flag | Job Execution Behaviour (if not handled by a transition) | Notes |
---|---|---|---|---|---|
n/a | n/a | OK | False | Continue | |
False | n/a | OK | False | Continue | |
False | OK | OK | False | Continue | |
False | <CUSTOM> | <CUSTOM> | False | Continue | Custom status codes can't contain any comma (,) or wildcard (*) characters, include leading or trailing whitespace, or exceed 100 characters. |
True | n/a | ERROR | True | Stop | |
True | ERROR | ERROR | True | Stop | |
True | <CUSTOM> | ERROR | True | Stop | Custom status codes for statuses that represent an error are not supported. If a custom status code is used the code is replaced with ERROR. |
If a pipeline or script module does not return a dw.system.Status
object or if a pipeline or script module's function fails before a Status
object is returned, the following rules apply.
Pipeline | Status Code | Error Status Flag | Job Execution Behaviour (if not handled by a transition) | Notes |
---|---|---|---|---|
Finished with stop node | ERROR | True | Stop | |
Finished with interaction node | ERROR | True | Stop | |
Finished with unhandled exception | ERROR | True | Stop | |
Finished with end node | OK | False | Continue | The name of the end node isn't relevant |
Stores an object that isn't a dw.system.Status object under key ExitStatus in pipeline dictionary | OK | False | Continue | |
Stores no object under key ExitStatus in pipeline dictionary | OK | False | Continue |
Script Module Function | Status Code | Error Status Flag | Job Execution Behaviour (if not handled by a transition) | Notes |
---|---|---|---|---|
Finished with unhandled JavaScript exception | ERROR | True | Stop | |
Runtime exceeds configured (or default) timeout | ERROR | True | Stop | |
Returns an object that isn't a dw.system.Status object | OK | False | Continue | |
Returns no object | OK | False | Continue |
The steptypes.json
file, which describes custom job steps, requires a specific JSON syntax. If a steptypes.json
file contains errors, the errors are logged and the steps are not registered. The system then loads steps from the steptypes.json
files of other cartridges. A typical error message for an invalid steptypes.json
file is Invalid step [Step1]! Type with id [custom.MyCustomStep1] is unknown!
.
To check the syntax of your steptypes.json
file, access the steptypes.schema
file here.
Element | Contained By | Contains | Required or Optional | Notes |
---|---|---|---|---|
@code | status | Required for status | Must not contain leading or trailing white space. | |
@name | parameter | Required for parameter | Must not contain leading or trailing whitespace. | |
@required | parameter | Optional | Must have value true or false. Default is true. If true, parameter is required. If false, parameter is optional. | |
@supports-organization-context | pipeline-step , chunk-script-module-step , and script-module-step | Optional | Must have value true or false. Default is true. If true, steps of this type can be used for flows with organization scope. If false, steps of this type can't be used for flows with organization scope. @supports-organization-context and @supports-site-context cannot both have the same true or false setting. If @supports-site-context is false, @supports-organization-context must be true and vice versa. | |
@supports-parallel-execution | pipeline-step , chunk-script-module-step , and script-module-step | Optional | Must have value true or false. Default is true. If false, split flows that contain steps of this type are always executed sequentially and never in parallel. If true, split flows that contain steps of this type are executed in parallel, as long as:
| |
@supports-site-context | pipeline-step , chunk-script-module-step , and script-module-step | Optional | Must have value true or false. Default is true. If true, steps of this type can be used for flows with one or more sites as scope. If false, steps of this type can't be used for flows with one or more sites as scope. @supports-organization-context and @supports-site-context cannot both have the same true or false setting. If @supports-site-context is false, @supports-organization-context must be true and vice versa. | |
@target-type | parameter | Optional | The type to which the parameter value is converted, either long or date , after the user enters a value in Business Manager. Can be present only when @type is datetime-string , date-string , or time-string . If @target-type is not defined, date is the default value. | |
@trim | parameter | Optional | Specifies whether leading and trailing whitespace for the parameter value entered in Business Manager is removed before the value is validated. Must not contain leading or trailing whitespace. Must have value true or false. Default is true. | |
@type | parameter | Required | Indicates the data type of the parameter value. Must not contain leading or trailing whitespace. Must have value of boolean , string , long , double , datetime-string , date-string , or time-string . | |
@type-id | pipeline-step , chunk-script-module-step , and script-module-step | Required | Identifies the step. This is the name that users will see in Business Manager, so make it descriptive. Must begin with custom. . Must not contain leading or trailing white space or more than 100 characters. Must be unique within the job definition. You can't register multiple steps with the same @type-id in different cartridges. The @type-id is validated as unique by parsing the steptypes.json files from all cartridges on the cartridge path. If there is a step with the same @type-id , the step isn't loaded. The @type-id value can't be the same as any system step, for example, ExecutePipeline or IncludeStepsFromJob . | |
after-chunk-function | chunk-script-module-step | Optional | References the function of the chunk-oriented script module to be executed after a chunk has finished. If not defined, no after-chunk logic is used. | |
after-step-function | chunk-script-module-step | Optional | References the function of the chunk-oriented script module to be executed after the step has finshed. If not defined, no after-step logic is used. | |
before-chunk-function | chunk-script-module-step | Optional | References the function of the chunk-oriented script module to be executed before a chunk begins. If not defined, no before-chunk logic is used. | |
before-step-function | chunk-script-module-step | Optional | References the function of the chunk-oriented script module to be executed before the step begins. If not defined, no before-step logic is used. | |
chunk-module-script-step | step-types |
| Required if no pipeline-step or script-module-step | Defines a chunk-oriented step. |
chunk-size | chunk-script-module-step | Required for status element | Must be a numeric value. Must not contain leading or trailing white space. | |
default-value | parameter | Optional | Default when no value is entered for the parameter. Must have a valid data type value for the parameter (boolean, string, long, double, datetime-string, date-string, or time-string). If the parameter is assigned enum-values , the default-value must match one of the list of enum values. | |
description | parameter | Required | Must not contain leading or trailing whitespace or more than 256 characters. | |
description | pipeline-step , chunk-script-module-step , script-module-step , and status | Optional | Description of the step or status code. Not shown in Business Manager. Must not exceed 4000 characters. | |
enum-values | parameter | value | Optional | Allowed values for the parameter. The value entered by the user in Business Manager is validated against this list. Must not contain leading or trailing white spaces. |
function | script-module-step | Optional | The function of the script module to execute. Must not contain leading or trailing white space. If not defined, the script module must export a function named execute . | |
max-length | parameter | Optional | Maximum length of a string value for this parameter. String entered in Business Manager is validated against this restriction. Only applicable if @type is set to string. Must be at least 1 and greater than or equal tomin-length . If not defined, string length is not restricted other than by the general requirement that parameter values cannot exceed 1000 characters. | |
max-value | parameter | Optional | Maximum numerical value for the parameter. Only applicable if @type is long,double, datetime-string, or time-string. Must be greater than or equal to min-value . The maximum valid value for a long is 9223372036854775807 and for a double is 1.7976931348623157E308. If you do not explicitly define a max-value , those values are considered the max-values for long and double parameters. For datetime-string or time-string parameters, if not defined, there are no restrictions other than the general restriction that parameter values cannot exceed 1000 characters. | |
min-length | parameter | Optional | Minimum length of the string value for the parameter. Only applicable if @type is string. Must be at least 1 and less than max-length . If not defined, there are no restrictions other than the general restriction that parameter values cannot exceed 1000 characters. | |
min-value | parameter | Optional | Minimum value for the parameter. Must not exceed the max-value . The minimum valid value for a long is -9223372036854775808 and for a double is4.9E-324. If you do not explicitly define a min-value , those values are considered the min-values for long and double parameters. For datetime-string or time-string parameters, if you do not define a min-value , there are no restrictions other than the general restriction that parameter values cannot exceed 1000 characters. | |
module | script-module-step and chunk-script-module-step | Required | Path to the script module to be executed. Must not contain leading or trailing white space. | |
parameter | parameters |
| Required if parameters is present | A parameter for the step, which the user configures in Business Manager. |
parameters | pipeline-step and script-module-step | parameter | Required | Parameters for the step, which the user configures in Business Manager. Contains one or more parameter elements. |
pattern | parameter | Optional | Regular expression that defines the allowed values for the parameter. Applicable only if @type is string. | |
pipeline | pipeline-step | Required | Pipeline to execute for the step. Format must be _<PipelineName>_-_<StartNodeName>_ . For example:Search-Show | |
pipeline-step | step-types |
| Required if no script-module-step or chunk-script-module-step | Defines a pipeline for the step. |
process-function | chunk-module-script-step | Optional | Process function of the chunk script. If not defined, script must export a function named process . | |
read-function | chunk-module-script-step | Optional | Read function of the chunk script. If not defined, script must export a function named read . | |
script-module-step | step-types |
| Required if no pipeline-step or chunk-script-module-step | Defines a task-oriented script for the step. |
status | status-codes |
| Required for status-codes . | Defines the error codes and descriptions for the step. |
status-codes | pipeline-step , chunk-script-module-step , and script-module-step | status | Optional | Defines the meta data for status codes returned for the step. |
step-types | Root object. |
| Required | |
timeout-in-seconds | script-module-step | Optional | Must be an integer that sets the timeout in seconds for the script module's function. There is no default timeout, but setting a limit is recommended. | |
total-count-function | chunk-script-module-step | Optional | References the function to be executed to determine the total number of items. If not defined, no total count is used. | |
transactional | script-module-step and chunk-script-module-step | Optional | Indicates if the module requires a database transaction. Must have value true or false. Default is false. If this value is set to true, the job step executes as a single, potentially very large, transaction. To avoid a negative impact on system performance and allow more granular transaction control, keep the default setting of false. Implement transaction handling within the job step using the dw.system.Transaction API. | |
value | enum-values | Required for enum-values | Comma-separated list of values allowed for the parameter | |
write-function | chunk-script-module-step | Optional | References the write function of the chunk script. If not defined, script must export a function named write . |
The following table shows the results of possible settings of the supports-organization-context
and supports-site-context
elements in the steptypes.json
file.
supports-organization-context | supports-site-context | Status |
---|---|---|
true | true | Invalid configuration. |
true | false | Job is executable only if other steps in the same flow can be executed for the entire organization. |
false | true | Job is executable only if other steps in the same flow can be executed for one or more sites. |
false | false | Invalid configuration. |
Use the steptypes.schema.json
file to check the syntax of your steptypes.json
file.
Access the steptypes.schema
file here.
To create a custom job step, you must develop a task-oriented or chunk-oriented CommonJS script module or legacy pipeline to define what the step does. You must also create a JSON or XML steptypes
file that describes the step, and then include the module or pipeline along with the steptypes
file in a cartridge uploaded to the site.
To create a custom job step, you must be familiar with creating and uploading a cartridge.
-
Create a cartridge.
-
Create a task-oriented or chunk-oriented CommonJS module to describe what the step does.
A best practice is to put the CommonJS module in the
cartridge/scripts
directory or subdirectory, for example,my_cartridge/cartridge/scripts/steps
.Instead of using a CommonJS module, you can use the UX Studio plug-in to the Eclipse IDE to create a pipeline. However, we recommend using CommonJS modules for jobs in the new framework.
-
Create a
steptypes.json
orsteptypes.xml
file that describes the job step and put the file in the root directory of the cartridge.You can describe more than one custom job step in the same file. Each cartridge can have only one
steptypes.json
orsteptypes.xml
file. -
Upload the cartridge, and include it in the cartridge path.
-
For a production system, replicate the code to production, and activate the code version that includes your cartridge.
When an administrator creates a job in Business Manager, custom steps defined in the steptypes.json
or steptypes.xml
file are listed as steps available to add to the job.