3. Copy the following command:
aws datasync create-location-s3 bucket-arn arn:aws:s3:::account-b-bucket --s3-config '{"BucketAccessRoleArn":"arn:aws:iam::account-a-id:role/name-of-role"}' 4. Replace account-b-bucket with the name of the S3 bucket in Account B.
5. Replace account-a-id with the AWS account number of Account A.
6. Replace name-of-role with the IAM role that you created for DataSync in Account A (back in Step 1).
7. Run the command.
If the command returns a DataSync location ARN similar to this, you created the location:
{ "LocationArn": "arn:aws:datasync:us-east-2:123456789012:location/loc-abcdef01234567890"
}
8. Switch back to Account A in the AWS Management Console.
9. Open the DataSync console at https://console.aws.amazon.com/datasync/.
10. Select Locations.
You can see the location of the S3 bucket in Account B that you just created with the CLI.
Step 5: Create and start your DataSync task
Before you move your data, let’s recap what you’ve done so far:
• Deployed and activated your DataSync agent in Account A so that the agent can read from your self-managed storage system and communicate with AWS.
• Created an IAM role in Account A so that DataSync can write data to the S3 bucket in Account B.
• Configured your S3 bucket in Account B to ensure your DataSync task works.
• Created your DataSync source and destination locations in Account A.
To create and start your DataSync task
1. While still using the DataSync console in Account A, go to Tasks and select Create task.
2. Choose the source location you’re copying data from (your on-premises storage) and select Next.
3. Choose the destination location you’re copying data to (the S3 bucket in Account B) and select Next.
4. Configure additional settings, such as specifying a CloudWatch log group. Select Next.
5. Select Create task then Start (you can start with defaults or specify overrides).
When your task completes, you'll see the data from your on-premises storage in the S3 bucket. You can now access the bucket data from Account B.
Related
For more information on what you did in this tutorial, see the following:
• Creating a role for an AWS service (console)
• Modifying a role trust policy (console)
• Adding a bucket policy using the Amazon S3 console
• Create an S3 location with the AWS CLI
Transferring data from a self-managed storage array
Additional resources for AWS DataSync
In this section, you can find additional information about and resources for AWS DataSync.
Topics
• Transferring data from a self-managed storage array (p. 146)
• Additional AWS DataSync use cases (p. 146)
Transferring data from a self-managed storage array
You might want to transfer data from a self-managed enterprise storage array to Amazon EFS. In this case, files in the source file system might be modified by another application while the files are being transferred from Network File System (NFS) or Server Message Block (SMB) file share to Amazon EFS.
To ensure that DataSync successfully performs a transfer with full consistency verification, we
recommend that the source location point to a read-only snapshot. This setup ensures that files at the source location can't be modified while the files are being transferred, and makes sure that verification works.
For information about how to take a snapshot in an enterprise storage array, see one of the following:
• EMC VNX: How to create a VNX snapshot and attach it to a server
• NetApp: Snapshot management
• HPE 3PAR: Creating virtual volume snapshots
• HDS: Hitachi Copy-on-Write Snapshot User Guide
Additional AWS DataSync use cases
In this section, you can find information about use cases in AWS DataSync that are not common to most users.
Topics
• Transferring files in opposite directions (p. 146)
• Using multiple tasks to write to the same Amazon S3 bucket (p. 147)
• Allowing Amazon S3 access from a private VPC endpoint (p. 147)
Transferring files in opposite directions
Transferring data in opposite directions allows for workflows where the active application moves between locations. AWS DataSync doesn't support workflows where multiple active applications write to both locations at the same time. Use the steps in the following procedure to configure DataSync to transfer data in opposite directions.
To configure DataSync to data transfers in opposite directions 1. Create a location and name it Location A.
2. Create a second location and name it Location B.
3. Create a task, name it Task A-B, and then configure Location A as the source location and Location B as the destination location.
4. Create a second task, name it Task B-A, and then configure Location B as the source location and Location A as the destination location.
5. To update Location B with data from Location A, run Task A-B.
To update Location A with data from Location B, run Task B-A.
Don't run these two tasks concurrently. DataSync can transfer files in opposite directions
periodically. However, it doesn't support workflows where multiple active applications write to both Location A and Location B simultaneously.
Using multiple tasks to write to the same Amazon S3 bucket
In certain use cases, you might want different tasks to write to the same Amazon S3 bucket. In this case, you create different folders in the S3 bucket for each of the task. This approach prevents file name conflicts between the tasks, and also means that you can set different permissions for each of folders.
For example, you might have three tasks: task1, task2, and task3 write to an S3 bucket named MyBucket.
You create three folders in the bucket:
s3://MyBucket/task1 s3://MyBucket/task2 s3://MyBucket/task3
For each task, you choose the folder in MyBucket that corresponds to the task as the destination, and set different permissions for each of the three folders.
Allowing Amazon S3 access from a private VPC endpoint
In certain cases, you might want to only allow Amazon S3 access from a private endpoint. In that case, you create an IAM policy that allows that access and attach it to the S3 bucket. If you need a policy that restricts your S3 bucket's access to DataSync VPC endpoints, contact AWS DataSync support to get the DataSync VPC endpoint for your AWS Region.
The following is a sample policy that only allows Amazon S3 access from a private endpoint.
{
"Version": "2012-10-17", "Id": "Policy1415115909152", "Statement": [
{
"Sid": "access-to-specific-vpce-only", "Principal": "",
"Action": "s3:",
Allowing Amazon S3 access from a private VPC endpoint
"Effect": "Deny", "Resource": [
"arn:aws:s3:::examplebucket", "arn:aws:s3:::examplebucket/*"
],
"Condition": {
"StringNotEquals": { "aws:sourceVpce": [
"vpce-your-vpc-endpoint",
"vpce-DataSync-vpc-endpoint-for-your-region"
] } } } ] }
For more information, see Controlling access from VPC endpoints with bucket policies in the Amazon S3 User Guide.
API reference
In addition to using the console, you can use the AWS DataSync API to programmatically configure and manage DataSync and its resources. This section describes the AWS DataSync operations and data types and contains the API Reference documentation for AWS DataSync.
Topics
• Actions (p. 149)
• Data Types (p. 280)
• Common Errors (p. 307)
• Common Parameters (p. 309)
Actions
The following actions are supported:
• CancelTaskExecution (p. 151)
• CreateAgent (p. 153)
• CreateLocationEfs (p. 157)
• CreateLocationFsxLustre (p. 161)
• CreateLocationFsxWindows (p. 164)
• CreateLocationHdfs (p. 167)
• CreateLocationNfs (p. 172)
• CreateLocationObjectStorage (p. 176)
• CreateLocationS3 (p. 180)
• CreateLocationSmb (p. 185)
• CreateTask (p. 189)
• DeleteAgent (p. 194)
• DeleteLocation (p. 196)
• DeleteTask (p. 198)
• DescribeAgent (p. 200)
• DescribeLocationEfs (p. 203)
• DescribeLocationFsxLustre (p. 206)
• DescribeLocationFsxWindows (p. 209)
• DescribeLocationHdfs (p. 212)
• DescribeLocationNfs (p. 216)
• DescribeLocationObjectStorage (p. 219)
• DescribeLocationS3 (p. 222)
• DescribeLocationSmb (p. 225)
• DescribeTask (p. 228)
• DescribeTaskExecution (p. 234)
• ListAgents (p. 239)
• ListLocations (p. 241)
• ListTagsForResource (p. 244)
Actions
• ListTaskExecutions (p. 247)
• ListTasks (p. 250)
• StartTaskExecution (p. 253)
• TagResource (p. 257)
• UntagResource (p. 259)
• UpdateAgent (p. 261)
• UpdateLocationHdfs (p. 263)
• UpdateLocationNfs (p. 267)
• UpdateLocationObjectStorage (p. 270)
• UpdateLocationSmb (p. 273)
• UpdateTask (p. 276)
• UpdateTaskExecution (p. 279)
CancelTaskExecution
Cancels execution of a task.
When you cancel a task execution, the transfer of some files is abruptly interrupted. The contents of files that are transferred to the destination might be incomplete or inconsistent with the source files. However, if you start a new task execution on the same task and you allow the task execution to complete, file content on the destination is complete and consistent. This applies to other unexpected failures that interrupt a task execution. In all of these cases, AWS DataSync successfully complete the transfer when you start the next task execution.
Request Syntax
{
"TaskExecutionArn": "string"
}
Request Parameters
For information about the parameters that are common to all actions, see Common Parameters (p. 309).
The request accepts the following data in JSON format.
TaskExecutionArn (p. 151)
The Amazon Resource Name (ARN) of the task execution to cancel.
Type: String
Length Constraints: Maximum length of 128.
Pattern: ^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b):datasync:[a-z\-0-9]*:
[0-9]{12}:task/task-[0-9a-f]{17}/execution/exec-[0-9a-f]{17}$
Required: Yes
Response Elements
If the action is successful, the service sends back an HTTP 200 response with an empty HTTP body.
Errors
For information about the errors that are common to all actions, see Common Errors (p. 307).
InternalException
This exception is thrown when an error occurs in the AWS DataSync service.
HTTP Status Code: 500 InvalidRequestException
This exception is thrown when the client submits a malformed request.
HTTP Status Code: 400
CancelTaskExecution
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS Command Line Interface
• AWS SDK for .NET
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for JavaScript
• AWS SDK for PHP V3
• AWS SDK for Python
• AWS SDK for Ruby V3
CreateAgent
Activates an AWS DataSync agent that you have deployed on your host. The activation process associates your agent with your account. In the activation process, you specify information such as the AWS Region that you want to activate the agent in. You activate the agent in the AWS Region where your target locations (in Amazon S3 or Amazon EFS) reside. Your tasks are created in this AWS Region.
You can activate the agent in a VPC (virtual private cloud) or provide the agent access to a VPC endpoint so you can run tasks without going over the public internet.
You can use an agent for more than one location. If a task uses multiple agents, all of them need to have status AVAILABLE for the task to run. If you use multiple agents for a source location, the status of all the agents must be AVAILABLE for the task to run.
For more information, see Creating and activating an agent in the AWS DataSync User Guide.
Agents are automatically updated by AWS on a regular basis, using a mechanism that ensures minimal interruption to your tasks.
Request Syntax
{ "ActivationKey": "string", "AgentName": "string",
"SecurityGroupArns": [ "string" ], "SubnetArns": [ "string" ], "Tags": [
{
"Key": "string", "Value": "string"
} ],
"VpcEndpointId": "string"
}
Request Parameters
For information about the parameters that are common to all actions, see Common Parameters (p. 309).
The request accepts the following data in JSON format.
ActivationKey (p. 153)
Your agent activation key. You can get the activation key either by sending an HTTP GET request with redirects that enable you to get the agent IP address (port 80). Alternatively, you can get it from the DataSync console.
The redirect URL returned in the response provides you the activation key for your agent in the query string parameter activationKey. It might also include other activation-related parameters;
however, these are merely defaults. The arguments you pass to this API call determine the actual configuration of your agent.
For more information, see Creating and activating an agent in the AWS DataSync User Guide.
Type: String
Length Constraints: Maximum length of 29.
CreateAgent
Pattern: [A-Z0-9]{5}(-[A-Z0-9]{5}){4}
Required: Yes AgentName (p. 153)
The name you configured for your agent. This value is a text reference that is used to identify the agent in the console.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 256.
Pattern: ^[a-zA-Z0-9\s+=._:@/-]+$
Required: No
SecurityGroupArns (p. 153)
The ARNs of the security groups used to protect your data transfer task subnets. See SecurityGroupArns.
Type: Array of strings
Array Members: Fixed number of 1 item.
Length Constraints: Maximum length of 128.
Pattern: ^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b):ec2:[a-z\-0-9]*:[0-9]
{12}:security-group/.*$
Required: No SubnetArns (p. 153)
The Amazon Resource Names (ARNs) of the subnets in which DataSync will create elastic network interfaces for each data transfer task. The agent that runs a task must be private. When you start a task that is associated with an agent created in a VPC, or one that has access to an IP address in a VPC, then the task is also private. In this case, DataSync creates four network interfaces for each task in your subnet. For a data transfer to work, the agent must be able to route to all these four network interfaces.
Type: Array of strings
Array Members: Fixed number of 1 item.
Length Constraints: Maximum length of 128.
Pattern: ^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b):ec2:[a-z\-0-9]*:[0-9]
{12}:subnet/.*$
Required: No Tags (p. 153)
The key-value pair that represents the tag that you want to associate with the agent. The value can be an empty string. This value helps you manage, filter, and search for your agents.
NoteValid characters for key and value are letters, spaces, and numbers representable in UTF-8 format, and the following special characters: + - = . _ : / @.
Type: Array of TagListEntry (p. 301) objects
Array Members: Minimum number of 0 items. Maximum number of 50 items.
Required: No VpcEndpointId (p. 153)
The ID of the VPC (virtual private cloud) endpoint that the agent has access to. This is the client-side VPC endpoint, also called a PrivateLink. If you don't have a PrivateLink VPC endpoint, see Creating a VPC Endpoint Service Configuration in the Amazon VPC User Guide.
VPC endpoint ID looks like this: vpce-01234d5aff67890e1.
Type: String
Pattern: ^vpce-[0-9a-f]{17}$
Required: No
Response Syntax
{
"AgentArn": "string"
}
Response Elements
If the action is successful, the service sends back an HTTP 200 response.
The following data is returned in JSON format by the service.
AgentArn (p. 155)
The Amazon Resource Name (ARN) of the agent. Use the ListAgents operation to return a list of agents for your account and AWS Region.
Type: String
Length Constraints: Maximum length of 128.
Pattern: ^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b):datasync:[a-z\-0-9]+:
[0-9]{12}:agent/agent-[0-9a-z]{17}$
Errors
For information about the errors that are common to all actions, see Common Errors (p. 307).
InternalException
This exception is thrown when an error occurs in the AWS DataSync service.
HTTP Status Code: 500 InvalidRequestException
This exception is thrown when the client submits a malformed request.
HTTP Status Code: 400
CreateAgent
Examples
Example
The following example creates an agent and activates it using an activation key.
Sample Request
{
"ActivationKey": "AAAAA-7AAAA-GG7MC-3I9R3-27COD", "AgentName": "MyAgent",
"Tags": [ { "Key": "Job",
"Value": "TransferJob-1"
} ]}
Example
The response returns the Amazon Resource Name (ARN) of the activated agent.
Sample Response
{ "AgentArn": "arn:aws:datasync:us-east-2:111222333444:agent/agent-0b0addbeef44baca3"
}
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS Command Line Interface
• AWS SDK for .NET
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for JavaScript
• AWS SDK for PHP V3
• AWS SDK for Python
• AWS SDK for Ruby V3
CreateLocationEfs
Creates an endpoint for an Amazon EFS file system.
Request Syntax
{ "Ec2Config": {
"SecurityGroupArns": [ "string" ], "SubnetArn": "string"
},
"EfsFilesystemArn": "string", "Subdirectory": "string", "Tags": [
{
"Key": "string", "Value": "string"
} ] }
Request Parameters
For information about the parameters that are common to all actions, see Common Parameters (p. 309).
The request accepts the following data in JSON format.
Ec2Config (p. 157)
The subnet and security group that the Amazon EFS file system uses. The security group that you provide needs to be able to communicate with the security group on the mount target in the subnet specified.
The exact relationship between security group M (of the mount target) and security group S (which you provide for DataSync to use at this stage) is as follows:
• Security group M (which you associate with the mount target) must allow inbound access for the Transmission Control Protocol (TCP) on the NFS port (2049) from security group S. You can enable inbound connections either by IP address (CIDR range) or security group.
• Security group S (provided to DataSync to access EFS) should have a rule that enables outbound connections to the NFS port on one of the file system’s mount targets. You can enable outbound connections either by IP address (CIDR range) or security group.
For information about security groups and mount targets, see Security Groups for Amazon EC2 Instances and Mount Targets in the Amazon EFS User Guide.
Type: Ec2Config (p. 283) object Required: Yes
EfsFilesystemArn (p. 157)
The Amazon Resource Name (ARN) for the Amazon EFS file system.
Type: String
Length Constraints: Maximum length of 128.
Pattern: ^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b):elasticfilesystem:[a-z
\-0-9]*:[0-9]{12}:file-system/fs-.*$
CreateLocationEfs
Required: Yes Subdirectory (p. 157)
A subdirectory in the location’s path. This subdirectory in the EFS file system is used to read data from the EFS source location or write data to the EFS destination. By default, AWS DataSync uses the root directory.
Note
Subdirectory must be specified with forward slashes. For example, /path/to/folder.
Type: String
Length Constraints: Maximum length of 4096.
Pattern: ^[a-zA-Z0-9_\-\+\./\(\)\p{Zs}]*$
Required: No Tags (p. 157)
The key-value pair that represents a tag that you want to add to the resource. The value can be an empty string. This value helps you manage, filter, and search for your resources. We recommend that you create a name tag for your location.
Type: Array of TagListEntry (p. 301) objects
Array Members: Minimum number of 0 items. Maximum number of 50 items.
Required: No
Response Syntax
{
"LocationArn": "string"
}
Response Elements
If the action is successful, the service sends back an HTTP 200 response.
The following data is returned in JSON format by the service.
LocationArn (p. 158)
The Amazon Resource Name (ARN) of the Amazon EFS file system location that is created.
Type: String
Length Constraints: Maximum length of 128.
Pattern: ^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b):datasync:[a-z\-0-9]+:
[0-9]{12}:location/loc-[0-9a-z]{17}$
Errors
For information about the errors that are common to all actions, see Common Errors (p. 307).
InternalException
This exception is thrown when an error occurs in the AWS DataSync service.
HTTP Status Code: 500 InvalidRequestException
This exception is thrown when the client submits a malformed request.
HTTP Status Code: 400
Examples
Example
The following example creates an endpoint for an Amazon EFS file system.
Sample Request
{ "Ec2Config": {
SecurityGroupArns": ["arn:aws:ec2:us-east-2:11122233344:security-group/
sg-0117195988293d62f"],
"SubnetArn": "arn:aws:ec2:us-east-2:11122233344:subnet/subnet-f45a0e678", }, "EfsFilesystemArn" :"arn:aws:elasticfilesystem:us-east-2:111222333444:file-system/
fs-12345efs",
"Subdirectory": "/MySubdirectory", "Tags": [
{
"Key": "Name",
"Value": "ElasticFileSystem-1"
} ] }
Example
The response returns the Amazon Resource Name (ARN) of the EFS location.
Sample Response
{
"LocationArn": "arn:aws:datasync:us-east-2:111222333444:location/loc-07db7abfc326c50fb"
}
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS Command Line Interface
• AWS SDK for .NET
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for JavaScript
CreateLocationEfs
• AWS SDK for PHP V3
• AWS SDK for Python
• AWS SDK for Ruby V3
CreateLocationFsxLustre
Creates an endpoint for an Amazon FSx for Lustre file system.
Request Syntax
{ "FsxFilesystemArn": "string", "SecurityGroupArns": [ "string" ], "Subdirectory": "string",
"Tags": [ {
"Key": "string", "Value": "string"
} ]}
Request Parameters
For information about the parameters that are common to all actions, see Common Parameters (p. 309).
The request accepts the following data in JSON format.
FsxFilesystemArn (p. 161)
The Amazon Resource Name (ARN) for the FSx for Lustre file system.
Type: String
Length Constraints: Maximum length of 128.
Pattern: ^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b):fsx:[a-z\-0-9]*:[0-9]
{12}:file-system/fs-.*$
Required: Yes
SecurityGroupArns (p. 161)
The Amazon Resource Names (ARNs) of the security groups that are used to configure the FSx for Lustre file system.
Type: Array of strings
Array Members: Minimum number of 1 item. Maximum number of 5 items.
Length Constraints: Maximum length of 128.
Pattern: ^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b):ec2:[a-z\-0-9]*:[0-9]
{12}:security-group/.*$
Required: Yes Subdirectory (p. 161)
A subdirectory in the location's path. This subdirectory in the FSx for Lustre file system is used to read data from the FSx for Lustre source location or write data to the FSx for Lustre destination.
Type: String
Length Constraints: Maximum length of 4096.
CreateLocationFsxLustre
Pattern: ^[a-zA-Z0-9_\-\+\./\(\)\$\p{Zs}]+$
Required: No Tags (p. 161)
The key-value pair that represents a tag that you want to add to the resource. The value can be an empty string. This value helps you manage, filter, and search for your resources. We recommend that you create a name tag for your location.
Type: Array of TagListEntry (p. 301) objects
Array Members: Minimum number of 0 items. Maximum number of 50 items.
Required: No
Response Syntax
{ "LocationArn": "string"
}
Response Elements
If the action is successful, the service sends back an HTTP 200 response.
The following data is returned in JSON format by the service.
LocationArn (p. 162)
The Amazon Resource Name (ARN) of the FSx for Lustre file system location that's created.
Type: String
Length Constraints: Maximum length of 128.
Pattern: ^arn:(aws|aws-cn|aws-us-gov|aws-iso|aws-iso-b):datasync:[a-z\-0-9]+:
[0-9]{12}:location/loc-[0-9a-z]{17}$
Errors
For information about the errors that are common to all actions, see Common Errors (p. 307).
For information about the errors that are common to all actions, see Common Errors (p. 307).