Creating Custom Job Steps

If there is no system step that does what you need to do, a developer must develop a custom job step. The developer first writes a task-oriented or chunk-oriented CommonJS script module or a legacy pipeline to describe what the step does. The developer includes the module or pipeline in a cartridge along with a JSON or XML file that describes the step, and then uploads the cartridge to the site. The custom steps are then available to the administrator creating jobs. The topics in this section are.

You can define what a custom step does using a task-oriented CommonJS script module that exposes a function to be called as the main function for the job step. When administrators create jobs using Business Manager, they set parameters that are available as scriptable objects for the module's function and for the dw.job.JobStepExecution object. The dw.job.JobStepExecution object allows read-only access to information about the current step execution and job execution.

To control the exit status, the script module's function can return a dw.system.Status object. If the script finishes with an unhandled exception, the exit status code is ERROR, and the error status flag is true by default. If no status object is returned and no exception occurs, the status code is OK by default.

The following example is a task-oriented CommonJS module.

You can define what a custom step does using a chunk-oriented CommonJS script module that reads and processes items in chunks of size S. If the list contains more items than can be processed in one chunk, a new chunk is started. A chunk-oriented script can include any series of processing steps, not just database transactions. A job step that uses a chunk-oriented script allows fine-grained progress monitoring because the number of elements that are written is updated each time a chunk is finished.

Chunk Processing Example

In this example, a list of 15 orders is exported to a file. Each chunk includes four orders.

  1. Chunk 1

    1. Read Order 1.
    2. Process Order 1.
    3. Read Order 2.
    4. Process Order 2.
    5. Read Order 3.
    6. Process Order 3.
    7. Read Order 4.
    8. Process Order 4.
    9. Write Order 1.
    10. Write Order 2.
    11. Write Order 3.
    12. Write Order 4.
  2. Chunk 2

    1. Read Order 5.
    2. Process Order 5.
    3. Read Order 6.
    4. Process Order 6.
    5. Read Order 7.
    6. Process Order 7.
    7. Read Order 8.
    8. Process Order 8.
    9. Write Order 5.
    10. Write Order 6.
    11. Write Order 7.
    12. Write Order 8.
  3. Chunk 3

    1. Read Order 9.
    2. Process Order 9.
    3. Read Order 10.
    4. Process Order 10.
    5. Read Order 11.
    6. Process Order 11.
    7. Read Order 12.
    8. Process Order 12.
    9. Write Order 9.
    10. Write Order 10.
    11. Write Order 11.
    12. Write Order 12.
  4. Chunk 4

    1. Read Order 13.
    2. Process Order 13.
    3. Read Order 14.
    4. Process Order 14.
    5. Read Order 15.
    6. Process Order 15.
    7. Write Order 13.
    8. Write Order 14.
    9. Write Order 15.

    This chunk read, processed, and wrote fewer items than the previous chunks, because only three items were left on the list.

FunctionRequired or OptionalPurpose
read-functionRequiredReturns one item or nothing if there are no more items.
process-functionRequiredTransforms items and applies business logic to them. It receives the item returned by the read function, performs a process, and returns one item.The item returned can be the same item that the read function returned if no processing logic is necessary, or it can be a new item of a different type. If the process function returns nothing, then the read function item is filtered and doesn't appear in the list of items to be written later.
write-functionRequiredReceives a list of items. The list size matches the chunk size or, if the number of items in the last available chunk is smaller, it is smaller. The write function returns nothing.
total-count-functionOptionalReturns the total number of items that are available. Called by the framework exactly once before chunk processing begins. A known total count allows better monitoring, for example, to show that 50 of 100 items have already been processed.
before-step-functionOptionalExecuted before a chunk step begins. Implements logic before all items of all chunks are read, processed, and written.
before-chunk-functionOptionalExecuted before a chunk begins. Implements logic before a chunk of S items is read, processed, and written.
after-chunk-functionOptionalExecuted after a chunk finishes. Implements logic after a chunk of S items has been read, processed, and written successfully.
after-step-functionOptionalExecuted after a chunk step finished successfully. Implements logic after all items of all chunks are read, processed, and written successfully.

When administrators create jobs using Business Manager, they set parameters that are available as scriptable objects for each exposed module function and for the dw.job.JobStepExecution object. The dw.job.JobStepExecutionobject allows read-only access to information about the current step execution and job execution. You can't define the exit status for a chunk-oriented script module. Chunk modules always finish with either OK or ERROR.

The following example shows a chunk-oriented CommonJS module.

In a task-oriented script module or pipeline for a custom job step, you can create an object of type dw.system.Status to control the exit status and error message for the step. The Status object includes a boolean flag that indicates whether or not the status is error, and also a status code and message. Messages appear in Business Manager and are written to log files. You can't define the exit status for a chunk-oriented script module. Chunk modules always finish with either OK or ERROR.

The job framework uses the following rules to detect the step status and determine if job execution can continue.

dw.system.Status.isError()dw.system.Status.getCode()Status CodeError Status FlagJob Execution Behaviour (if not handled by a transition)Notes
n/an/aOKFalseContinue
Falsen/aOKFalseContinue
FalseOKOKFalseContinue
False<CUSTOM><CUSTOM>FalseContinueCustom status codes can't contain any comma (,) or wildcard (*) characters, include leading or trailing whitespace, or exceed 100 characters.
Truen/aERRORTrueStop
TrueERRORERRORTrueStop
True<CUSTOM>ERRORTrueStopCustom status codes for statuses that represent an error are not supported. If a custom status code is used the code is replaced with ERROR.

If a pipeline or script module does not return a dw.system.Status object or if a pipeline or script module's function fails before a Status object is returned, the following rules apply.

PipelineStatus CodeError Status FlagJob Execution Behaviour (if not handled by a transition)Notes
Finished with stop nodeERRORTrueStop 
Finished with interaction nodeERRORTrueStop 
Finished with unhandled exceptionERRORTrueStop 
Finished with end nodeOKFalseContinueThe name of the end node isn't relevant
Stores an object that isn't a dw.system.Status object under key ExitStatus in pipeline dictionaryOKFalseContinue 
Stores no object under key ExitStatus in pipeline dictionaryOKFalseContinue 
Script Module FunctionStatus CodeError Status FlagJob Execution Behaviour (if not handled by a transition)Notes
Finished with unhandled JavaScript exceptionERRORTrueStop 
Runtime exceeds configured (or default) timeoutERRORTrueStop 
Returns an object that isn't a dw.system.Status objectOKFalseContinue 
Returns no objectOKFalseContinue 

The steptypes.json file, which describes custom job steps, requires a specific JSON syntax. If a steptypes.json file contains errors, the errors are logged and the steps are not registered. The system then loads steps from the steptypes.json files of other cartridges. A typical error message for an invalid steptypes.json file is Invalid step [Step1]! Type with id [custom.MyCustomStep1] is unknown!.

To check the syntax of your steptypes.json file, access the steptypes.schema file here.

ElementContained ByContainsRequired or OptionalNotes
@codestatus Required for statusMust not contain leading or trailing white space.
@nameparameter Required for parameterMust not contain leading or trailing whitespace.
@requiredparameter OptionalMust have value true or false. Default is true. If true, parameter is required. If false, parameter is optional.
@supports-organization-contextpipeline-step, chunk-script-module-step, and script-module-step OptionalMust have value true or false. Default is true. If true, steps of this type can be used for flows with organization scope. If false, steps of this type can't be used for flows with organization scope. @supports-organization-context and @supports-site-context cannot both have the same true or false setting. If @supports-site-context is false, @supports-organization-context must be true and vice versa.
@supports-parallel-executionpipeline-step, chunk-script-module-step, and script-module-step OptionalMust have value true or false. Default is true. If false, split flows that contain steps of this type are always executed sequentially and never in parallel. If true, split flows that contain steps of this type are executed in parallel, as long as:
  • The split flows don't contain steps with supports-parallel-execution = false.
  • The split flows are not configured to be executed sequentially.
  • There are enough resources available to do parallel execution.
@supports-site-contextpipeline-step, chunk-script-module-step, and script-module-step OptionalMust have value true or false. Default is true. If true, steps of this type can be used for flows with one or more sites as scope. If false, steps of this type can't be used for flows with one or more sites as scope. @supports-organization-context and @supports-site-context cannot both have the same true or false setting. If @supports-site-context is false, @supports-organization-context must be true and vice versa.
@target-typeparameter OptionalThe type to which the parameter value is converted, either long or date, after the user enters a value in Business Manager. Can be present only when @type is datetime-string, date-string, or time-string. If @target-type is not defined, date is the default value.
@trimparameter OptionalSpecifies whether leading and trailing whitespace for the parameter value entered in Business Manager is removed before the value is validated. Must not contain leading or trailing whitespace. Must have value true or false. Default is true.
@typeparameter RequiredIndicates the data type of the parameter value. Must not contain leading or trailing whitespace. Must have value of boolean, string, long, double, datetime-string, date-string, or time-string.
@type-idpipeline-step, chunk-script-module-step, and script-module-step RequiredIdentifies the step. This is the name that users will see in Business Manager, so make it descriptive. Must begin with custom.. Must not contain leading or trailing white space or more than 100 characters. Must be unique within the job definition. You can't register multiple steps with the same @type-id in different cartridges. The @type-id is validated as unique by parsing the steptypes.json files from all cartridges on the cartridge path. If there is a step with the same @type-id, the step isn't loaded. The @type-id value can't be the same as any system step, for example, ExecutePipeline or IncludeStepsFromJob.
after-chunk-functionchunk-script-module-step OptionalReferences the function of the chunk-oriented script module to be executed after a chunk has finished. If not defined, no after-chunk logic is used.
after-step-functionchunk-script-module-step OptionalReferences the function of the chunk-oriented script module to be executed after the step has finshed. If not defined, no after-step logic is used.
before-chunk-functionchunk-script-module-step OptionalReferences the function of the chunk-oriented script module to be executed before a chunk begins. If not defined, no before-chunk logic is used.
before-step-functionchunk-script-module-step OptionalReferences the function of the chunk-oriented script module to be executed before the step begins. If not defined, no before-step logic is used.
chunk-module-script-stepstep-types
  • @type-id
  • @supports-parallel-execution
  • @supports-site-context
  • @supports-organization-context
  • description
  • module
  • before-step-function
  • total-count-function
  • before-chunk-function
  • read-function
  • process-function
  • write-function
  • after-chunk-function
  • after-step-function
  • chunk-size
  • transactional
  • parameters
  • status-codes
Required if no pipeline-step or script-module-stepDefines a chunk-oriented step.
chunk-sizechunk-script-module-step Required for status elementMust be a numeric value. Must not contain leading or trailing white space.
default-valueparameter OptionalDefault when no value is entered for the parameter. Must have a valid data type value for the parameter (boolean, string, long, double, datetime-string, date-string, or time-string). If the parameter is assigned enum-values, the default-value must match one of the list of enum values.
descriptionparameter RequiredMust not contain leading or trailing whitespace or more than 256 characters.
descriptionpipeline-step, chunk-script-module-step, script-module-step, and status OptionalDescription of the step or status code. Not shown in Business Manager. Must not exceed 4000 characters.
enum-valuesparametervalueOptionalAllowed values for the parameter. The value entered by the user in Business Manager is validated against this list. Must not contain leading or trailing white spaces.
functionscript-module-step OptionalThe function of the script module to execute. Must not contain leading or trailing white space. If not defined, the script module must export a function named execute.
max-lengthparameter OptionalMaximum length of a string value for this parameter. String entered in Business Manager is validated against this restriction. Only applicable if @type is set to string. Must be at least 1 and greater than or equal tomin-length. If not defined, string length is not restricted other than by the general requirement that parameter values cannot exceed 1000 characters.
max-valueparameter OptionalMaximum numerical value for the parameter. Only applicable if @type is long,double, datetime-string, or time-string. Must be greater than or equal to min-value. The maximum valid value for a long is 9223372036854775807 and for a double is 1.7976931348623157E308. If you do not explicitly define a max-value, those values are considered the max-values for long and double parameters. For datetime-string or time-string parameters, if not defined, there are no restrictions other than the general restriction that parameter values cannot exceed 1000 characters.
min-lengthparameter OptionalMinimum length of the string value for the parameter. Only applicable if @type is string. Must be at least 1 and less than max-length. If not defined, there are no restrictions other than the general restriction that parameter values cannot exceed 1000 characters.
min-valueparameter OptionalMinimum value for the parameter. Must not exceed the max-value. The minimum valid value for a long is -9223372036854775808 and for a double is4.9E-324. If you do not explicitly define a min-value, those values are considered the min-values for long and double parameters. For datetime-string or time-string parameters, if you do not define a min-value, there are no restrictions other than the general restriction that parameter values cannot exceed 1000 characters.
modulescript-module-step and chunk-script-module-step RequiredPath to the script module to be executed. Must not contain leading or trailing white space.
parameterparameters
  • name
  • type
  • target-type
  • required
  • trim
  • default-value
  • enum-values
  • description
  • pattern
  • min-length
  • max-length
  • min-value
  • max-value
Required if parameters is presentA parameter for the step, which the user configures in Business Manager.
parameterspipeline-step and script-module-stepparameterRequiredParameters for the step, which the user configures in Business Manager. Contains one or more parameter elements.
patternparameter OptionalRegular expression that defines the allowed values for the parameter. Applicable only if @type is string.
pipelinepipeline-step RequiredPipeline to execute for the step. Format must be _<PipelineName>_-_<StartNodeName>_. For example:Search-Show
pipeline-stepstep-types
  • @type-id
  • @supports-parallel-execution
  • @supports-site-context
  • @supports-organization-context
  • pipeline
  • description
  • parameters
  • status-codes
Required if no script-module-step or chunk-script-module-stepDefines a pipeline for the step.
process-functionchunk-module-script-step OptionalProcess function of the chunk script. If not defined, script must export a function named process.
read-functionchunk-module-script-step OptionalRead function of the chunk script. If not defined, script must export a function named read.
script-module-stepstep-types
  • @type-id
  • @supports-parallel-execution
  • @supports-site-context
  • @supports-organization-context
  • description
  • module
  • function
  • transactional
  • timeout-in-seconds
  • parameters
  • status-codes
Required if no pipeline-step or chunk-script-module-stepDefines a task-oriented script for the step.
statusstatus-codes
  • @code
  • description
Required for status-codes.Defines the error codes and descriptions for the step.
status-codespipeline-step, chunk-script-module-step, and script-module-stepstatusOptionalDefines the meta data for status codes returned for the step.
step-typesRoot object.
  • script-module-step
  • pipeline-step
  • chunk-script-module-step
Required 
timeout-in-secondsscript-module-step OptionalMust be an integer that sets the timeout in seconds for the script module's function. There is no default timeout, but setting a limit is recommended.
total-count-functionchunk-script-module-step OptionalReferences the function to be executed to determine the total number of items. If not defined, no total count is used.
transactionalscript-module-step and chunk-script-module-step OptionalIndicates if the module requires a database transaction. Must have value true or false. Default is false. If this value is set to true, the job step executes as a single, potentially very large, transaction. To avoid a negative impact on system performance and allow more granular transaction control, keep the default setting of false. Implement transaction handling within the job step using the dw.system.Transaction API.
valueenum-values Required for enum-valuesComma-separated list of values allowed for the parameter
write-functionchunk-script-module-step OptionalReferences the write function of the chunk script. If not defined, script must export a function named write.

The following table shows the results of possible settings of the supports-organization-context and supports-site-context elements in the steptypes.json file.

supports-organization-contextsupports-site-contextStatus
truetrueInvalid configuration.
truefalseJob is executable only if other steps in the same flow can be executed for the entire organization.
falsetrueJob is executable only if other steps in the same flow can be executed for one or more sites.
falsefalseInvalid configuration.

Use the steptypes.schema.json file to check the syntax of your steptypes.json file.

Access the steptypes.schema file here.

To create a custom job step, you must develop a task-oriented or chunk-oriented CommonJS script module or legacy pipeline to define what the step does. You must also create a JSON or XML steptypes file that describes the step, and then include the module or pipeline along with the steptypes file in a cartridge uploaded to the site.

To create a custom job step, you must be familiar with creating and uploading a cartridge.

  1. Create a cartridge.

  2. Create a task-oriented or chunk-oriented CommonJS module to describe what the step does.

    A best practice is to put the CommonJS module in the cartridge/scripts directory or subdirectory, for example, my_cartridge/cartridge/scripts/steps.

    Instead of using a CommonJS module, you can use the UX Studio plug-in to the Eclipse IDE to create a pipeline. However, we recommend using CommonJS modules for jobs in the new framework.

  3. Create a steptypes.json or steptypes.xml file that describes the job step and put the file in the root directory of the cartridge.

    You can describe more than one custom job step in the same file. Each cartridge can have only one steptypes.json or steptypes.xml file.

  4. Upload the cartridge, and include it in the cartridge path.

  5. For a production system, replicate the code to production, and activate the code version that includes your cartridge.

When an administrator creates a job in Business Manager, custom steps defined in the steptypes.json or steptypes.xml file are listed as steps available to add to the job.