AWS Data Pipeline

(1)

AWS Data Pipeline

API Reference

API Version 2012-10-29

(2)

AWS Data Pipeline: API Reference

Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be aﬃliated with, connected to, or sponsored by Amazon.

(3)

Welcome

AWS Data Pipeline conﬁgures and manages a data-driven workﬂow called a pipeline. AWS Data Pipeline handles the details of scheduling and ensuring that data dependencies are met so that your application can focus on processing the data.

AWS Data Pipeline provides a JAR implementation of a task runner called AWS Data Pipeline Task Runner. AWS Data Pipeline Task Runner provides logic for common data management scenarios, such as performing database queries and running data analysis using Amazon Elastic MapReduce (Amazon EMR).

You can use AWS Data Pipeline Task Runner as your task runner, or you can write your own task runner to provide custom data management.

AWS Data Pipeline implements two main sets of functionality. Use the first set to create a pipeline and define data sources, schedules, dependencies, and the transforms to be performed on the data. Use the second set in your task runner application to receive the next task ready for processing. The logic for performing the task, such as querying the data, running data analysis, or converting the data from one format to another, is contained within the task runner. The task runner performs the task assigned to it by the web service, reporting progress to the web service as it does so. When the task is done, the task runner reports the final success or failure of the task to the web service.

This document was last published on March 6, 2022.

(8)

Actions

The following actions are supported:

• ActivatePipeline (p. 3)

• AddTags (p. 6)

• CreatePipeline (p. 8)

• DeactivatePipeline (p. 11)

• DeletePipeline (p. 13)

• DescribeObjects (p. 16)

• DescribePipelines (p. 20)

• EvaluateExpression (p. 24)

• GetPipelineDeﬁnition (p. 27)

• ListPipelines (p. 31)

• PollForTask (p. 34)

• PutPipelineDeﬁnition (p. 38)

• QueryObjects (p. 44)

• RemoveTags (p. 48)

• ReportTaskProgress (p. 50)

• ReportTaskRunnerHeartbeat (p. 53)

• SetStatus (p. 56)

• SetTaskStatus (p. 59)

• ValidatePipelineDeﬁnition (p. 63)

(9)

ActivatePipeline

Validates the speciﬁed pipeline and starts processing pipeline tasks. If the pipeline does not pass validation, activation fails.

If you need to pause the pipeline to investigate an issue with a component, such as a data source or script, call DeactivatePipeline (p. 11).

To activate a ﬁnished pipeline, modify the end date for the pipeline and then activate it.

If you activate an on-demand pipeline that is already running, it will cancel all running objects and re-run the pipeline. StartTimestamp does not apply to on-demand pipelines.

Request Syntax

{

"parameterValues": [ {

"id": "string",

"stringValue": "string"

} ],

"pipelineId": "string", "startTimestamp": number }

Request Parameters

For information about the parameters that are common to all actions, see Common Parameters (p. 88).

The request accepts the following data in JSON format.

parameterValues (p. 3)

A list of parameter values to pass to the pipeline at activation.

Type: Array of ParameterValue (p. 76) objects Required: No

pipelineId (p. 3)

The ID of the pipeline.

Type: String

Length Constraints: Minimum length of 1. Maximum length of 1024.

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\n\t]*

Required: Yes startTimestamp (p. 3)

The date and time to resume the pipeline. By default, the pipeline resumes from the last completed

(10)

Response Elements

Required: No

Response Elements

If the action is successful, the service sends back an HTTP 200 response with an empty HTTP body.

Errors

For information about the errors that are common to all actions, see Common Errors (p. 90).

InternalServiceError

An internal service error occurred.

HTTP Status Code: 500 InvalidRequestException

The request was not valid. Verify that your request was properly formatted, that the signature was generated with the correct credentials, and that you haven't exceeded any of the service limits for your account.

HTTP Status Code: 400 PipelineDeletedException

The speciﬁed pipeline has been deleted.

HTTP Status Code: 400 PipelineNotFoundException

The speciﬁed pipeline was not found. Verify that you used the correct user and account identiﬁers.

HTTP Status Code: 400

Examples

Example

This example illustrates one usage of ActivatePipeline.

Sample Request

POST / HTTP/1.1

Content-Type: application/x-amz-json-1.1 X-Amz-Target: DataPipeline.ActivatePipeline Content-Length: 39

Host: datapipeline.us-east-1.amazonaws.com X-Amz-Date: Mon, 12 Nov 2012 17:49:52 GMT Authorization: AuthParams

{"pipelineId": "df-06372391ZG65EXAMPLE"}

(11)

See Also

Sample Response

HTTP/1.1 200

x-amzn-RequestId: ee19d5bf-074e-11e2-af6f-6bc7a6be60d9 Content-Type: application/x-amz-json-1.1

Content-Length: 2

Date: Mon, 12 Nov 2012 17:50:53 GMT {}

AddTags

Adds or modiﬁes tags for the speciﬁed pipeline.

Request Syntax

{

"pipelineId": "string", "tags": [

{

"key": "string", "value": "string"

} ]}

Request Parameters

Type: String

Required: Yes tags (p. 6)

The tags to add, as key/value pairs.

Type: Array of Tag (p. 83) objects

Array Members: Minimum number of 0 items. Maximum number of 10 items.

Required: Yes

Response Elements

Errors

(13)

See Also

CreatePipeline

Creates a new, empty pipeline. Use PutPipelineDeﬁnition (p. 38) to populate the pipeline.

Request Syntax

{

"description": "string", "name": "string",

"tags": [ {

} ],

"uniqueId": "string"

}

Request Parameters

description (p. 8)

The description for the pipeline.

Type: String Required: No name (p. 8)

The name for the pipeline. You can use the same name for multiple pipelines associated with your AWS account, because AWS Data Pipeline assigns each pipeline a unique pipeline identiﬁer.

Type: String

Required: Yes tags (p. 8)

A list of tags to associate with the pipeline at creation. Tags let you control access to pipelines. For more information, see Controlling User Access to Pipelines in the AWS Data Pipeline Developer Guide.

Type: Array of Tag (p. 83) objects

Array Members: Minimum number of 0 items. Maximum number of 10 items.

Required: No uniqueId (p. 8)

A unique identifier. This identifier is not the same as the pipeline identifier assigned by AWS Data Pipeline. You are responsible for defining the format and ensuring the uniqueness of this

(15)

Response Syntax

identiﬁer. You use this parameter to ensure idempotency during repeated calls to CreatePipeline.

For example, if the ﬁrst call to CreatePipeline does not succeed, you can pass in the same unique identiﬁer and pipeline name combination on a subsequent call to CreatePipeline.

CreatePipeline ensures that if a pipeline already exists with the same name and unique identiﬁer, a new pipeline is not created. Instead, you'll receive the pipeline identiﬁer from the previous

attempt. The uniqueness of the name and unique identiﬁer combination is scoped to the AWS account or IAM user credentials.

Type: String

Required: Yes

Response Syntax

{

"pipelineId": "string"

}

Response Elements

If the action is successful, the service sends back an HTTP 200 response.

The following data is returned in JSON format by the service.

The ID that AWS Data Pipeline assigns the newly created pipeline. For example, df-06372391ZG65EXAMPLE.

Type: String

Errors

The request was not valid. Verify that your request was properly formatted, that the signature was generated with the correct credentials, and that you haven't exceeded any of the service limits for

(16)

Examples

Example

This example illustrates one usage of CreatePipeline.

Sample Request

POST / HTTP/1.1

Content-Type: application/x-amz-json-1.1 X-Amz-Target: DataPipeline.CreatePipeline Content-Length: 91

{"name": "myPipeline", "uniqueId": "123456789",

"description": "This is my first pipeline"}

Sample Response

HTTP/1.1 200

x-amzn-RequestId: b16911ce-0774-11e2-af6f-6bc7a6be60d9 Content-Type: application/x-amz-json-1.1

Content-Length: 40

Date: Mon, 12 Nov 2012 17:50:53 GMT {"pipelineId": "df-06372391ZG65EXAMPLE"}

DeactivatePipeline

Deactivates the speciﬁed running pipeline. The pipeline is set to the DEACTIVATING state until the deactivation process completes.

To resume a deactivated pipeline, use ActivatePipeline (p. 3). By default, the pipeline resumes from the last completed execution. Optionally, you can specify the date and time to resume the pipeline.

Request Syntax

{ "cancelActive": boolean, "pipelineId": "string"

}

Request Parameters

cancelActive (p. 11)

Indicates whether to cancel any running objects. The default is true, which sets the state of any running objects to CANCELED. If this value is false, the pipeline is deactivated after all running objects ﬁnish.

Type: Boolean Required: No pipelineId (p. 11)

Type: String

Required: Yes

Response Elements

Errors

(18)

See Also

DeletePipeline

Deletes a pipeline, its pipeline deﬁnition, and its run history. AWS Data Pipeline attempts to cancel instances associated with the pipeline that are currently being processed by task runners.

Deleting a pipeline cannot be undone. You cannot query or restore a deleted pipeline. To temporarily pause a pipeline instead of deleting it, call SetStatus (p. 56) with the status set to PAUSE on individual components. Components that are paused by SetStatus (p. 56) can be resumed.

Request Syntax

{ "pipelineId": "string"

}

Request Parameters

Type: String

Required: Yes

Response Elements

Errors

The request was not valid. Verify that your request was properly formatted, that the signature was generated with the correct credentials, and that you haven't exceeded any of the service limits for

(20)

Examples

PipelineNotFoundException

Examples

Example

This example illustrates one usage of DeletePipeline.

Sample Request

POST / HTTP/1.1

Content-Type: application/x-amz-json-1.1 X-Amz-Target: DataPipeline.DeletePipeline Content-Length: 50

Sample Response

x-amzn-RequestId: b7a88c81-0754-11e2-af6f-6bc7a6be60d9 Content-Type: application/x-amz-json-1.1

Content-Length: 0

Date: Mon, 12 Nov 2012 17:50:53 GMT Unexpected response: 200, OK, undefined

DescribeObjects

Gets the object definitions for a set of objects associated with the pipeline. Object definitions are composed of a set of fields that define the properties of the object.

Request Syntax

{ "evaluateExpressions": boolean, "marker": "string",

"objectIds": [ "string" ], "pipelineId": "string"

}

Request Parameters

evaluateExpressions (p. 16)

Indicates whether any expressions in the object should be evaluated when the object descriptions are returned.

Type: Boolean Required: No marker (p. 16)

The starting point for the results to be returned. For the ﬁrst call, this value should be empty. As long as there are more results, continue to call DescribeObjects with the marker value from the previous call to retrieve the next set of results.

Type: String Required: No objectIds (p. 16)

The IDs of the pipeline objects that contain the deﬁnitions to be described. You can pass as many as 25 identiﬁers in a single call to DescribeObjects.

Type: Array of strings

Required: Yes pipelineId (p. 16)

The ID of the pipeline that contains the object deﬁnitions.

Type: String

(23)

Response Syntax

Required: Yes

Response Syntax

{ "hasMoreResults": boolean, "marker": "string", "pipelineObjects": [ {

"fields": [ {

"key": "string", "refValue": "string", "stringValue": "string"

} ],

"id": "string", "name": "string"

} ]}

Response Elements

hasMoreResults (p. 17)

Indicates whether there are more results to return.

Type: Boolean marker (p. 17)

The starting point for the next page of results. To view the next page of results, call

DescribeObjects again with this marker value. If the value is null, there are no more results.

Type: String pipelineObjects (p. 17)

An array of object deﬁnitions.

Type: Array of PipelineObject (p. 80) objects

Errors

(24)

Examples

InvalidRequestException

Examples

Example

This example illustrates one usage of DescribeObjects.

Sample Request

POST / HTTP/1.1

Content-Type: application/x-amz-json-1.1 X-Amz-Target: DataPipeline.DescribeObjects Content-Length: 98

{"pipelineId": "df-06372391ZG65EXAMPLE", "objectIds":

["Schedule"],

"evaluateExpressions": true}

Sample Response

x-amzn-RequestId: 4c18ea5d-0777-11e2-8a14-21bb8a1f50ef Content-Type: application/x-amz-json-1.1

Content-Length: 1488

Date: Mon, 12 Nov 2012 17:50:53 GMT {"hasMoreResults": false,

"pipelineObjects":

[ {"fields":

[

{"key": "startDateTime",

(25)

See Also

"stringValue": "2012-12-12T00:00:00"}, {"key": "parent",

"refValue": "Default"}, {"key": "@sphere",

"stringValue": "COMPONENT"}, {"key": "type",

"stringValue": "Schedule"}, {"key": "period",

"stringValue": "1 hour"}, {"key": "endDateTime",

"stringValue": "2012-12-21T18:00:00"}, {"key": "@version",

"stringValue": "1"}, {"key": "@status",

"stringValue": "PENDING"}, {"key": "@pipelineId",

"stringValue": "df-06372391ZG65EXAMPLE"}

],

"id": "Schedule", "name": "Schedule"}

] }

DescribePipelines

Retrieves metadata about one or more pipelines. The information retrieved includes the name of the pipeline, the pipeline identiﬁer, its current state, and the user account that owns the pipeline. Using account credentials, you can retrieve metadata about pipelines that you or your IAM users have created.

If you are using an IAM user account, you can retrieve metadata about only those pipelines for which you have read permissions.

To retrieve the full pipeline deﬁnition instead of metadata about the pipeline, call GetPipelineDeﬁnition (p. 27).

Request Syntax

{

"pipelineIds": [ "string" ] }

Request Parameters

pipelineIds (p. 20)

The IDs of the pipelines to describe. You can pass as many as 25 identiﬁers in a single call. To obtain pipeline IDs, call ListPipelines (p. 31).

Required: Yes

Response Syntax

{ "pipelineDescriptionList": [ {

"description": "string", "fields": [

{

} ],

"name": "string", "pipelineId": "string", "tags": [

{

(27)

Response Elements

} ] } ] }

Response Elements

pipelineDescriptionList (p. 20)

An array of descriptions for the speciﬁed pipelines.

Type: Array of PipelineDescription (p. 77) objects

Errors

Examples

Example

This example illustrates one usage of DescribePipelines.

Sample Request

(28)

See Also

POST / HTTP/1.1

Content-Type: application/x-amz-json-1.1 X-Amz-Target: DataPipeline.DescribePipelines Content-Length: 70

{"pipelineIds":

["df-08785951KAKJEXAMPLE"]

}

Sample Response

x-amzn-RequestId: 02870eb7-0736-11e2-af6f-6bc7a6be60d9 Content-Type: application/x-amz-json-1.1

Content-Length: 767

Date: Mon, 12 Nov 2012 17:50:53 GMT {"pipelineDescriptionList":

[

{"description": "This is my first pipeline", "fields":

[

{"key": "@pipelineState", "stringValue": "SCHEDULED"}, {"key": "description",

"stringValue": "This is my first pipeline"}, {"key": "name",

"stringValue": "myPipeline"}, {"key": "@creationTime",

"stringValue": "2012-12-13T01:24:06"}, {"key": "@id",

"stringValue": "df-0937003356ZJEXAMPLE"}, {"key": "@sphere",

"stringValue": "PIPELINE"}, {"key": "@version",

"stringValue": "1"}, {"key": "@userId",

"stringValue": "924374875933"}, {"key": "@accountId",

"stringValue": "924374875933"}, {"key": "uniqueId",

"stringValue": "1234567890"}

],

"name": "myPipeline",

"pipelineId": "df-0937003356ZJEXAMPLE"}

]}

EvaluateExpression

Task runners call EvaluateExpression to evaluate a string in the context of the speciﬁed object. For example, a task runner can evaluate SQL queries stored in Amazon S3.

Request Syntax

{ "expression": "string", "objectId": "string", "pipelineId": "string"

}

Request Parameters

expression (p. 24)

The expression to evaluate.

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Required: Yes objectId (p. 24)

The ID of the object.

Type: String

Required: Yes pipelineId (p. 24)

Type: String

Required: Yes

Response Syntax

{

(31)

Response Elements

"evaluatedExpression": "string"

}

Response Elements

evaluatedExpression (p. 24) The evaluated expression.

Type: String

Pattern: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Errors

HTTP Status Code: 400 TaskNotFoundException

The speciﬁed task was not found.

Examples

Example

(32)

See Also

Sample Request

POST / HTTP/1.1

Content-Type: application/x-amz-json-1.1 X-Amz-Target: DataPipeline.DescribePipelines Content-Length: 164

{"pipelineId": "df-08785951KAKJEXAMPLE", "objectId": "Schedule",

"expression": "Transform started at #{startDateTime} and finished at #{endDateTime}"}

Sample Response

x-amzn-RequestId: 02870eb7-0736-11e2-af6f-6bc7a6be60d9 Content-Type: application/x-amz-json-1.1

Content-Length: 103

Date: Mon, 12 Nov 2012 17:50:53 GMT

{"evaluatedExpression": "Transform started at 2012-12-12T00:00:00 and finished at 2012-12-21T18:00:00"}

GetPipelineDeﬁnition

Gets the definition of the specified pipeline. You can call GetPipelineDefinition to retrieve the pipeline definition that you provided using PutPipelineDefinition (p. 38).

Request Syntax

{ "pipelineId": "string", "version": "string"

}

Request Parameters

Type: String

Required: Yes version (p. 27)

The version of the pipeline definition to retrieve. Set this parameter to latest (default) to use the last definition saved to the pipeline or active to use the last definition that was activated.

Type: String Required: No

Response Syntax

{ "parameterObjects": [ {

"attributes": [ {

"key": "string", "stringValue": "string"

} ],

"id": "string"

} ],

(34)

Response Elements

} ],

"pipelineObjects": [ {

"fields": [ {

} ],

} ] }

Response Elements

parameterObjects (p. 27)

The parameter objects used in the pipeline deﬁnition.

Type: Array of ParameterObject (p. 75) objects parameterValues (p. 27)

The parameter values used in the pipeline deﬁnition.

Type: Array of ParameterValue (p. 76) objects pipelineObjects (p. 27)

The objects deﬁned in the pipeline.

Type: Array of PipelineObject (p. 80) objects

Errors

(35)

Examples

Example

This example illustrates one usage of GetPipelineDeﬁnition.

Sample Request

POST / HTTP/1.1

Content-Type: application/x-amz-json-1.1

X-Amz-Target: DataPipeline.GetPipelineDefinition Content-Length: 40

Sample Response

x-amzn-RequestId: e28309e5-0776-11e2-8a14-21bb8a1f50ef Content-Type: application/x-amz-json-1.1

Content-Length: 890

Date: Mon, 12 Nov 2012 17:50:53 GMT {"pipelineObjects":

[ {"fields":

[

{"key": "workerGroup", "stringValue": "workerGroup"}

],

"id": "Default", "name": "Default"}, {"fields":

[

"stringValue": "2012-09-25T17:00:00"}, {"key": "type",

"stringValue": "1 hour"},

(36)

See Also

"id": "Schedule", "name": "Schedule"}, {"fields":

[

{"key": "schedule", "refValue": "Schedule"}, {"key": "command",

"stringValue": "echo hello"}, {"key": "parent",

"refValue": "Default"}, {"key": "type",

"stringValue": "ShellCommandActivity"}

],

"id": "SayHello", "name": "SayHello"}

] }

ListPipelines

Lists the pipeline identiﬁers for all active pipelines that you have permission to access.

Request Syntax

{ "marker": "string"

}

Request Parameters

marker (p. 31)

The starting point for the results to be returned. For the ﬁrst call, this value should be empty. As long as there are more results, continue to call ListPipelines with the marker value from the previous call to retrieve the next set of results.

Type: String Required: No

Response Syntax

{ "hasMoreResults": boolean, "marker": "string", "pipelineIdList": [ {

} ] }

Response Elements

Indicates whether there are more results that can be obtained by a subsequent call.

Type: Boolean marker (p. 31)

(38)

Errors

Type: String pipelineIdList (p. 31)

The pipeline identifiers. If you require additional information about the pipelines, you can use these identifiers to call DescribePipelines (p. 20) and GetPipelineDefinition (p. 27).

Type: Array of PipelineIdName (p. 79) objects

Errors

Examples

Example

This example illustrates one usage of ListPipelines.

Sample Request

POST / HTTP/1.1

Content-Type: application/x-amz-json-1.1 X-Amz-Target: DataPipeline.ListPipelines Content-Length: 14

{}

Sample Response

Status:

x-amzn-RequestId: b3104dc5-0734-11e2-af6f-6bc7a6be60d9 Content-Type: application/x-amz-json-1.1

Content-Length: 39

Date: Mon, 12 Nov 2012 17:50:53 GMT {"PipelineIdList":

(39)

See Also

[

{"id": "df-08785951KAKJEXAMPLE", "name": "MyPipeline"},

{"id": "df-08662578ISYEXAMPLE", "name": "MySecondPipeline"}

] }

PollForTask

Task runners call PollForTask to receive a task to perform from AWS Data Pipeline. The task runner speciﬁes which tasks it can perform by setting a value for the workerGroup parameter. The task returned can come from any of the pipelines that match the workerGroup value passed in by the task runner and that was launched using the IAM user credentials speciﬁed by the task runner.

If tasks are ready in the work queue, PollForTask returns a response immediately. If no tasks are available in the queue, PollForTask uses long-polling and holds on to a poll connection for up to a 90 seconds, during which time the ﬁrst newly scheduled task is handed to the task runner. To accomodate this, set the socket timeout in your task runner to 90 seconds. The task runner should not call PollForTask again on the same workerGroup until it receives a response, and this can take up to 90 seconds.

Request Syntax

{ "hostname": "string", "instanceIdentity": { "document": "string", "signature": "string"

},

"workerGroup": "string"

}

Request Parameters

hostname (p. 34)

The public DNS name of the calling task runner.

Type: String

Required: No instanceIdentity (p. 34)

Identity information for the EC2 instance that is hosting the task runner. You can get this value from the instance using http://169.254.169.254/latest/meta-data/instance-id. For more information, see Instance Metadata in the Amazon Elastic Compute Cloud User Guide. Passing in this value proves that your task runner is running on an EC2 instance, and ensures the proper AWS Data Pipeline service charges are applied to your pipeline.

Type: InstanceIdentity (p. 71) object Required: No

(41)

Response Syntax

workerGroup (p. 34)

The type of task the task runner is conﬁgured to accept and process. The worker group is set as a ﬁeld on objects in the pipeline when they are created. You can only specify a single value for workerGroup in the call to PollForTask. There are no wildcard values permitted in workerGroup; the string must be an exact, case-sensitive, match.

Type: String Required: Yes

Response Syntax

{

"taskObject": {

"attemptId": "string", "objects": {

"string" : { "fields": [ {

} ],

} },

"pipelineId": "string", "taskId": "string"

} }

Response Elements

taskObject (p. 35)

The information needed to complete the task that is being assigned to the task runner. One of the ﬁelds returned in this object is taskId, which contains an identiﬁer for the task being assigned.

The calling task runner uses taskId in subsequent calls to ReportTaskProgress (p. 50) and SetTaskStatus (p. 59).

Type: TaskObject (p. 84) object

Errors

(42)

Examples

InvalidRequestException

HTTP Status Code: 400 TaskNotFoundException

The speciﬁed task was not found.

Examples

Example

This example illustrates one usage of PollForTask.

Sample Request

POST / HTTP/1.1

Content-Type: application/x-amz-json-1.1 X-Amz-Target: DataPipeline.PollForTask Content-Length: 59

{"workerGroup": "MyworkerGroup", "hostname": "example.com"}

Sample Response

x-amzn-RequestId: 41c713d2-0775-11e2-af6f-6bc7a6be60d9 Content-Type: application/x-amz-json-1.1

Content-Length: 39

Date: Mon, 12 Nov 2012 17:50:53 GMT {"taskObject":

{"attemptId": "@SayHello_2012-12-12T00:00:00_Attempt=1", "objects":

{"@SayHello_2012-12-12T00:00:00_Attempt=1":

{"fields":

[

{"key": "@componentParent", "refValue": "SayHello"}, {"key": "@scheduledStartTime",

"stringValue": "2012-12-12T00:00:00"}, {"key": "parent",

"refValue": "SayHello"}, {"key": "@sphere",

"stringValue": "ATTEMPT"},

(43)

See Also

{"key": "workerGroup",

"stringValue": "workerGroup"}, {"key": "@instanceParent",

"refValue": "@SayHello_2012-12-12T00:00:00"}, {"key": "type",

"stringValue": "ShellCommandActivity"}, {"key": "@status",

"stringValue": "WAITING_FOR_RUNNER"}, {"key": "@version",

"stringValue": "1"}, {"key": "schedule", "refValue": "Schedule"}, {"key": "@actualStartTime",

"stringValue": "2012-12-13T01:40:50"}, {"key": "command",

"stringValue": "echo hello"}, {"key": "@scheduledEndTime",

"stringValue": "2012-12-12T01:00:00"}, {"key": "@activeInstances",

"refValue": "@SayHello_2012-12-12T00:00:00"}, {"key": "@pipelineId",

"stringValue": "df-0937003356ZJEXAMPLE"}

],

"id": "@SayHello_2012-12-12T00:00:00_Attempt=1", "name": "@SayHello_2012-12-12T00:00:00_Attempt=1"}

},

"pipelineId": "df-0937003356ZJEXAMPLE",

"taskId": "2xaM4wRs5zOsIH+g9U3oVHfAgAlbSqU6XduncB0HhZ3xMnmvfePZPn4dIbYXHyWyRK +cU15MqDHwdrvftx/4wv+sNS4w34vJfv7QA9aOoOazW28l1GYSb2ZRR0N0paiQp

+d1MhSKo10hOTWOsVK5S5Lnx9Qm6omFgXHyIvZRIvTlrQMpr1xuUrflyGOfbFOGpOLpvPE172MYdqpZKnbSS4TcuqgQKSWV2833fEubI57DPOP7ghWa2TcYeSIv4pdLYG53fTuwfbnbdc98g2LNUQzSVhSnt7BoqyNwht2aQ6b/

UHg9A80+KVpuXuqmz3m1MXwHFgxjdmuesXNOrrlGpeLCcRWD

+aGo0RN1NqhQRzNAig8V4GlaPTQzMsRCljKqvrIyAoP3Tt2XEGsHkkQo12rEX8Z90957XX2qKRwhruwYzqGkSLWjINoLdAxUJdpRXRc5DJTrBd3D5mdzn7kY1l7NEh4kFHJDt3Cx4Z3Mk8MYCACyCk/

CEyy9DwuPi66cLz0NBcgbCM5LKjTBOwo1m+am

+pvM1kSposE9FPP1+RFGb8k6jQBTJx3TRz1yKilnGXQTZ5xvdOFpJrklIT0OXP1MG3+auM9FlJA

+1dX90QoNJE5z7axmK//MOGXUdkqFe2kiDkorqjxwDvc0Js9pVKfKvAmW8YqUbmI9l0ERpWCXXnLVHNmPWz3jaPY +OBAmuJWDmxB/Z8p94aEDg4BVXQ7LvsKQ3DLYhaB7yJ390CJT+i0mm+EBqY60V6YikPSWDFrYQ/

NPi2b1DgE19mX8zHqw8qprIl4yh1Ckx2Iige4En/N5ktOoIxnASxAw/TzcE2skxdw5KlHDF+UTj71m16CR/

dIaKlXijlfNlNzUBo/bNSadCQn3G5NoO501wPKI:XO50TgDNyo8EXAMPLE/g==:1"}

}

PutPipelineDeﬁnition

Adds tasks, schedules, and preconditions to the speciﬁed pipeline. You can use PutPipelineDefinition to populate a new pipeline.

PutPipelineDefinition also validates the conﬁguration as it adds it to the pipeline. Changes to the pipeline are saved unless one of the following validation errors exist in the pipeline.

1. An object is missing a name or identiﬁer ﬁeld.

2. A string or reference ﬁeld is empty.

3. The number of objects in the pipeline exceeds the allowed maximum number of objects.

4. The pipeline is in a FINISHED state.

Pipeline object deﬁnitions are passed to the PutPipelineDefinition action and returned by the GetPipelineDeﬁnition (p. 27) action.

Request Syntax

{ "parameterObjects": [ {

"attributes": [ {

"key": "string",

} ],

"id": "string"

} ],

"parameterValues": [ {

"id": "string",

} ],

"pipelineId": "string", "pipelineObjects": [ {

"fields": [ {

} ],

} ] }

Request Parameters

(45)

Response Syntax

parameterObjects (p. 38)

The parameter objects used with the pipeline.

Type: Array of ParameterObject (p. 75) objects Required: No

parameterValues (p. 38)

The parameter values used with the pipeline.

Type: Array of ParameterValue (p. 76) objects Required: No

Type: String

Required: Yes pipelineObjects (p. 38)

The objects that deﬁne the pipeline. These objects overwrite the existing pipeline deﬁnition.

Type: Array of PipelineObject (p. 80) objects Required: Yes

Response Syntax

{

"errored": boolean, "validationErrors": [ {

"errors": [ "string" ], "id": "string"

} ],

"validationWarnings": [ {

"id": "string",

"warnings": [ "string" ] }

] }

Response Elements

(46)

Errors

errored (p. 39)

Indicates whether there were validation errors, and the pipeline deﬁnition is stored but cannot be activated until you correct the pipeline and call PutPipelineDefinition to commit the corrected pipeline.

Type: Boolean validationErrors (p. 39)

The validation errors that are associated with the objects deﬁned in pipelineObjects.

Type: Array of ValidationError (p. 86) objects validationWarnings (p. 39)

The validation warnings that are associated with the objects deﬁned in pipelineObjects.

Type: Array of ValidationWarning (p. 87) objects

Errors

Examples

Example 1

This example sets a valid pipeline conﬁguration and returns success.

Sample Request

(47)

Examples

POST / HTTP/1.1

X-Amz-Target: DataPipeline.PutPipelineDefinition Content-Length: 914

{"pipelineId": "df-0937003356ZJEXAMPLE", "pipelineObjects":

[

{"id": "Default", "name": "Default", "fields":

[

{"key": "workerGroup", "stringValue": "workerGroup"}

] },

{"id": "Schedule", "name": "Schedule", "fields":

[

"stringValue": "2012-12-21T18:00:00"}

] },

{"id": "SayHello", "name": "SayHello", "fields":

[

{"key": "type",

"stringValue": "ShellCommandActivity"}, {"key": "command",

"refValue": "Default"}, {"key": "schedule", "refValue": "Schedule"}

] } ]}

Sample Response

HTTP/1.1 200

x-amzn-RequestId: f74afc14-0754-11e2-af6f-6bc7a6be60d9 Content-Type: application/x-amz-json-1.1

Content-Length: 18

Date: Mon, 12 Nov 2012 17:50:53 GMT

(48)

Examples

Example 2

This example sets an invalid pipeline conﬁguration (the value for workerGroup is an empty string) and returns an error message.

Sample Request

POST / HTTP/1.1

X-Amz-Target: DataPipeline.PutPipelineDefinition Content-Length: 903

{"pipelineId": "df-06372391ZG65EXAMPLE", "pipelineObjects":

[

{"id": "Default", "name": "Default", "fields":

[

{"key": "workerGroup", "stringValue": ""}

] },

{"id": "Schedule", "name": "Schedule", "fields":

[

"stringValue": "2012-09-25T18:00:00"}

] },

{"id": "SayHello", "name": "SayHello", "fields":

[

{"key": "type",

"stringValue": "ShellCommandActivity"}, {"key": "command",

"refValue": "Default"}, {"key": "schedule", "refValue": "Schedule"}

] } ]

}

(49)

See Also

Sample Response

HTTP/1.1 200

x-amzn-RequestId: f74afc14-0754-11e2-af6f-6bc7a6be60d9 Content-Type: application/x-amz-json-1.1

Content-Length: 18

Date: Mon, 12 Nov 2012 17:50:53 GMT

{"__type": "com.amazon.setl.webservice#InvalidRequestException",

"message": "Pipeline definition has errors: Could not save the pipeline definition due to FATAL errors: [com.amazon.setl.webservice.ValidationError@108d7ea9] Please call Validate to validate your pipeline"}

QueryObjects

Queries the speciﬁed pipeline for the names of objects that match the speciﬁed set of conditions.

Request Syntax

{ "limit": number, "marker": "string", "pipelineId": "string", "query": {

"selectors": [ {

"fieldName": "string", "operator": {

"type": "string", "values": [ "string" ] }

} ] },

"sphere": "string"

}

Request Parameters

limit (p. 44)

The maximum number of object names that QueryObjects will return in a single call. The default value is 100.

Type: Integer Required: No marker (p. 44)

The starting point for the results to be returned. For the ﬁrst call, this value should be empty. As long as there are more results, continue to call QueryObjects with the marker value from the previous call to retrieve the next set of results.

Type: String Required: No pipelineId (p. 44)

Type: String

Required: Yes

(51)

Response Syntax

query (p. 44)

The query that defines the objects to be returned. The Query object can contain a maximum of ten selectors. The conditions in the query are limited to top-level String fields in the object. These filters can be applied to components, instances, and attempts.

Type: Query (p. 81) object Required: No

sphere (p. 44)

Indicates whether the query applies to components or instances. The possible values are:

COMPONENT, INSTANCE, and ATTEMPT.

Type: String Required: Yes

Response Syntax

{

"hasMoreResults": boolean, "ids": [ "string" ], "marker": "string"

}

Response Elements

Indicates whether there are more results that can be obtained by a subsequent call.

Type: Boolean ids (p. 45)

The identiﬁers that match the query selectors.

marker (p. 45)

The starting point for the next page of results. To view the next page of results, call QueryObjects again with this marker value. If the value is null, there are no more results.

Type: String

AWS Data Pipeline

AWS Data Pipeline

API Reference

API Version 2012-10-29

AWS Data Pipeline: API Reference

Table of Contents

Welcome

Actions

ActivatePipeline

Request Syntax

Request Parameters

Response Elements

Errors

Examples

Example

Sample Request

Sample Response

See Also

AddTags

Request Syntax

Request Parameters

Response Elements

Errors

See Also

CreatePipeline

Request Syntax

Request Parameters

Response Syntax

Response Elements

Errors

Examples

Example

Sample Request

Sample Response

See Also

DeactivatePipeline

Request Syntax

Request Parameters

Response Elements

Errors

See Also

DeletePipeline

Request Syntax

Request Parameters

Response Elements

Errors

Examples

Example

Sample Request

Sample Response

See Also

DescribeObjects

Request Syntax

Request Parameters

Response Syntax

Response Elements

Errors

Examples

Example

Sample Request

Sample Response

See Also

DescribePipelines

Request Syntax

Request Parameters

Response Syntax

Response Elements

Errors

Examples

Example

Sample Request

Sample Response

See Also

EvaluateExpression

Request Syntax

Request Parameters

Response Syntax

Response Elements

Errors

Examples