AWS Storage Gateway

(1)

AWS Storage Gateway

User Guide

API Version 2013-06-30

(2)

AWS Storage Gateway: User Guide

Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be aﬃliated with, connected to, or sponsored by Amazon.

(3)

What is Amazon S3 File Gateway

AWS Storage Gateway connects an on-premises software appliance with cloud-based storage to provide seamless integration with data security features between your on-premises IT environment and the AWS storage infrastructure. You can use the service to store data in the AWS Cloud for scalable and cost- effective storage that helps maintain data security. AWS Storage Gateway offers file-based, volume- based, and tape-based storage solutions.

Topics

• Amazon S3 File Gateway (p. 1)

Amazon S3 File Gateway

Amazon S3 File Gateway –Amazon S3 File Gateway supports a file interface into Amazon Simple Storage Service (Amazon S3) and combines a service and a virtual software appliance. By using this combination, you can store and retrieve objects in Amazon S3 using industry-standard file protocols such as Network File System (NFS) and Server Message Block (SMB). The software appliance, or gateway, is deployed into your on-premises environment as a virtual machine (VM) running on VMware ESXi, Microsoft Hyper-V, or Linux Kernel-based Virtual Machine (KVM) hypervisor. The gateway provides access to objects in S3 as files or file share mount points. With a S3 File Gateway, you can do the following:

• You can store and retrieve ﬁles directly using the NFS version 3 or 4.1 protocol.

• You can store and retrieve ﬁles directly using the SMB ﬁle system version, 2 and 3 protocol.

• You can access your data directly in Amazon S3 from any AWS Cloud application or service.

• You can manage your S3 data using lifecycle policies, cross-region replication, and versioning. You can think of a S3 File Gateway as a ﬁle system mount on Amazon S3.

A S3 File Gateway simplifies file storage in Amazon S3, integrates to existing applications through industry-standard file system protocols, and provides a cost-effective alternative to on-premises storage.

It also provides low-latency access to data through transparent local caching. A S3 File Gateway manages data transfer to and from AWS, buﬀers applications from network congestion, optimizes and streams data in parallel, and manages bandwidth consumption. S3 File Gateway integrate with AWS services, for example with the following:

• Common access management using AWS Identity and Access Management (IAM)

• Encryption using AWS Key Management Service (AWS KMS)

• Monitoring using Amazon CloudWatch (CloudWatch)

• Audit using AWS CloudTrail (CloudTrail)

• Operations using the AWS Management Console and AWS Command Line Interface (AWS CLI)

• Billing and cost management

In the following documentation, you can find a Getting Started section that covers setup information common to all gateways and also gateway-specific setup sections. The Getting Started section shows you how to deploy, activate, and configure storage for a gateway. The management section shows you how to manage your gateway and resources:

• provides instructions on how to create and use a S3 File Gateway. It shows you how to create a ﬁle share, map your drive to an Amazon S3 bucket, and upload ﬁles and folders to Amazon S3.

(9)

Amazon S3 File Gateway

• describes how to perform management tasks for all gateway types and resources.

In this guide, you can primarily ﬁnd how to work with gateway operations by using the AWS

Management Console. If you want to perform these operations programmatically, see the AWS Storage Gateway API Reference.

(10)

Amazon S3 File Gateways

How Storage Gateway works (architecture)

Following, you can ﬁnd an architectural overview of the available Storage Gateway solutions.

Topics

• Amazon S3 File Gateways (p. 3)

Amazon S3 File Gateways

To use an S3 File Gateway, you start by downloading a VM image for the gateway. You then activate the gateway from the AWS Management Console or through the Storage Gateway API. You can also create an S3 File Gateway using an Amazon EC2 image.

After the S3 File Gateway is activated, you create and configure your file share and associate that share with your Amazon Simple Storage Service (Amazon S3) bucket. Doing this makes the share accessible by clients using either the Network File System (NFS) or Server Message Block (SMB) protocol. Files written to a file share become objects in Amazon S3, with the path as the key. There is a one-to-one mapping between files and objects, and the gateway asynchronously updates the objects in Amazon S3 as you change the files. Existing objects in the Amazon S3 bucket appear as files in the file system, and the key becomes the path. Objects are encrypted with Amazon S3–server-side encryption keys (SSE-S3). All data transfer is done through HTTPS.

The service optimizes data transfer between the gateway and AWS using multipart parallel uploads or byte-range downloads, to better use the available bandwidth. Local cache is maintained to provide low latency access to the recently accessed data and reduce data egress charges. CloudWatch metrics provide insight into resource use on the VM and data transfer to and from AWS. CloudTrail tracks all API calls.

With S3 File Gateway storage, you can do such tasks as ingesting cloud workloads to Amazon S3, performing backups and archiving, tiering, and migrating storage data to the AWS Cloud. The following diagram provides an overview of ﬁle storage deployment for Storage Gateway.

S3 File Gateway converts files to S3 objects when uploading files to Amazon S3. The interaction between file operations performed against files shares on S3 File Gateway and S3 objects requires certain

operations to be carefully considered when converting between ﬁles and objects.

Common file operations change file metadata, which results in the deletion of the current S3 object and the creation of a new S3 object. The following table shows example file operations and the impact on S3 objects.

File operation S3 object impact Storage class implication

Rename ﬁle Replaces existing S3 object and

creates a new S3 object for each ﬁle

Early deletion fees and retrieval fees may apply

(11)

Amazon S3 File Gateways

File operation S3 object impact Storage class implication

Rename folder Replaces all existing S3 objects and creates new S3 objects for each folder and ﬁles in the folder structure

Change ﬁle/folder permissions Replaces existing S3 object and creates a new S3 object for each ﬁle or folder

Change ﬁle/folder ownership Replaces existing S3 object and creates a new S3 object for each ﬁle or folder

Append to a ﬁle Replaces existing S3 object and creates a new S3 object for each ﬁle

When a file is written to the S3 File Gateway by an NFS or SMB client, the file gateway uploads the file's data to Amazon S3 followed by its metadata, (ownerships, timestamps, etc.). Uploading the file data creates an S3 object, and uploading the metadata for the file updates the metadata for the S3 object.

This process creates another version of the object, resulting in two versions of an object. If S3 Versioning is enabled, both versions will be stored.

When a file is modified in the S3 File Gateway by an NFS or SMB client after it has been uploaded to Amazon S3, the S3 File Gateway uploads the new or modified data instead of uploading the whole file.

The ﬁle modiﬁcation results in a new version of the S3 object being created.

When the S3 File Gateway uploads larger files, it might need to upload smaller chunks of the file before the client is done writing to the S3 File Gateway. Some reasons for this include freeing up cache space or a high rate of writes to a file share. This can result in multiple versions of an object in the S3 bucket.

You should monitor your S3 bucket to determine how many versions of an object exist before setting up lifecycle policies to move objects to diﬀerent storage classes. You should conﬁgure lifecycle expiration for previous versions to minimize the number of versions you have for an object in your S3 bucket. The use of Same-Region replication (SRR) or Cross-Region replication (CRR) between S3 buckets will increase the storage used.

(12)

Sign up for Amazon Web Services

Setting up for Amazon S3 File Gateway

This section provides instructions for getting started with Amazon S3 File Gateway. To get started, you ﬁrst sign up for AWS. If you are a ﬁrst-time user, we recommend that you read the Regions and Requirements sections.

Topics

• Sign up for Amazon Web Services (p. 5)

• Create an IAM user (p. 5)

• File gateway setup requirements (p. 6)

• Accessing AWS Storage Gateway (p. 17)

• Supported AWS Regions (p. 17)

Sign up for Amazon Web Services

If you do not have an AWS account, complete the following steps to create one.

To sign up for an AWS account

1. Open https://portal.aws.amazon.com/billing/signup.

2. Follow the online instructions.

Part of the sign-up procedure involves receiving a phone call and entering a veriﬁcation code on the phone keypad.

Create an IAM user

After you create your AWS account, use the following steps to create an AWS Identity and Access Management (IAM) user for yourself. Then you add that user to a group that has administrative permissions.

To create an administrator user for yourself and add the user to an administrators group (console)

1. Sign in to the IAM console as the account owner by choosing Root user and entering your AWS account email address. On the next page, enter your password.

Note

We strongly recommend that you adhere to the best practice of using the Administrator IAM user that follows and securely lock away the root user credentials. Sign in as the root user only to perform a few account and service management tasks.

2. In the navigation pane, choose Users and then choose Add user.

3. For User name, enter Administrator.

4. Select the check box next to AWS Management Console access. Then select Custom password, and then enter your new password in the text box.

(13)

Requirements

5. (Optional) By default, AWS requires the new user to create a new password when ﬁrst signing in. You can clear the check box next to User must create a new password at next sign-in to allow the new user to reset their password after they sign in.

6. Choose Next: Permissions.

7. Under Set permissions, choose Add user to group.

8. Choose Create group.

9. In the Create group dialog box, for Group name enter Administrators.

10. Choose Filter policies, and then select AWS managed - job function to ﬁlter the table contents.

11. In the policy list, select the check box for AdministratorAccess. Then choose Create group.

Note

You must activate IAM user and role access to Billing before you can use the

AdministratorAccess permissions to access the AWS Billing and Cost Management console. To do this, follow the instructions in step 1 of the tutorial about delegating access to the billing console.

12. Back in the list of groups, select the check box for your new group. Choose Refresh if necessary to see the group in the list.

13. Choose Next: Tags.

14. (Optional) Add metadata to the user by attaching tags as key-value pairs. For more information about using tags in IAM, see Tagging IAM entities in the IAM User Guide.

15. Choose Next: Review to see the list of group memberships to be added to the new user. When you are ready to proceed, choose Create user.

You can use this same process to create more groups and users and to give your users access to your AWS account resources. To learn about using policies that restrict user permissions to speciﬁc AWS resources, see Access management and Example policies.

File gateway setup requirements

Unless otherwise noted, the following requirements are common to all ﬁle gateway types in AWS Storage Gateway. Your setup must meet the requirements in this section. Review the requirements that apply to your gateway setup before you deploy your gateway.

Topics

• Required prerequisites (p. 6)

• Hardware and storage requirements (p. 7)

• Network and ﬁrewall requirements (p. 8)

• Supported hypervisors and host requirements (p. 16)

• Supported NFS clients for a ﬁle gateway (p. 16)

• Supported SMB clients for a ﬁle gateway (p. 17)

• Supported ﬁle system operations for a ﬁle gateway (p. 17)

Required prerequisites

Before you use an Amazon S3 File Gateway (S3 File Gateway), you must meet the following requirements:

• Conﬁgure Microsoft Active Directory (AD).

• Ensure that there is suﬃcient network bandwidth between the gateway and AWS. A minimum of 100 Mbps is required to successfully download, activate, and update the gateway.

(14)

Hardware and storage requirements

• Conﬁgure your private networking, VPN, or AWS Direct Connect between your Amazon Virtual Private Cloud (Amazon VPC) and the on-premises environment where you are deploying your gateway.

• Make sure your gateway can resolve the name of your Active Directory Domain Controller. You can use DHCP in your Active Directory domain to handle resolution, or specify a DNS server manually from the Network Conﬁguration settings menu in the gateway local console.

Hardware and storage requirements

The following sections provide information about the minimum required hardware and settings for your gateway, and the minimum amount of disk space to allocate for the required storage.

For information about best practices for ﬁle gateway performance, see Performance guidance for ﬁle gateways (p. 132).

Hardware requirements for on-premises VMs

When deploying your gateway on-premises, ensure that the underlying hardware on which you deploy the gateway virtual machine (VM) can dedicate the following minimum resources:

• Four virtual processors assigned to the VM

• 16 GiB of reserved RAM for ﬁle gateways

• 80 GiB of disk space for installation of VM image and system data

For more information, see Optimizing Gateway Performance (p. 134). For information about how your hardware aﬀects the performance of the gateway VM, see Quotas for ﬁle shares (p. 210).

Requirements for Amazon EC2 instance types

When deploying your gateway on Amazon Elastic Compute Cloud (Amazon EC2), the instance size must be at least xlarge for your gateway to function. However, for the compute-optimized instance family the size must be at least 2xlarge. Use one of the following instance types recommended for your gateway type.

Recommended for ﬁle gateway types

• General-purpose instance family – m4 or m5 instance type.

• Compute-optimized instance family – c4 or c5 instance types. Choose the 2xlarge instance size or higher to meet the required RAM requirements.

• Memory-optimized instance family – r3 instance types.

• Storage-optimized instance family – i3 instance types.

Note

When you launch your gateway in Amazon EC2 and the instance type you choose supports ephemeral storage, the disks are listed automatically. For more information about Amazon EC2 instance storage, see Instance storage in the Amazon EC2 User Guide.

Application writes are stored in the cache synchronously, and then asynchronously uploaded to durable storage in Amazon S3. If the ephemeral storage is lost because an instance stops before the upload is complete, the data that still resides in the cache and has not yet written to Amazon Simple Storage Service (Amazon S3) can be lost. Before you stop the instance that hosts the gateway, make sure that the CachePercentDirty CloudWatch metric is 0. For information about ephemeral storage, see Using ephemeral storage with EC2 gateways (p. 95). For information about monitoring metrics for your storage gateway, see Monitoring your ﬁle gateway (p. 76).

If you have more than 5 million objects in your S3 bucket and you are using a General Purposes SSD volume, a minimum root EBS volume of 350 GiB is needed for acceptable

(15)

Network and ﬁrewall requirements

performance of your gateway during startup. For information about how to increase the volume size, see Modifying an EBS volume using elastic volumes (console).

Storage requirements

In addition to 80 GiB of disk space for the VM, you also need additional disks for your gateway.

Gateway type Cache

(minimum) Cache (maximum) File gateway 150 GiB 64 TiB

Note

You can conﬁgure one or more local drives for your cache, up to the maximum capacity.

When adding cache to an existing gateway, it's important to create new disks in your host (hypervisor or Amazon EC2 instance). Don't change the size of existing disks if the disks have been previously allocated as a cache.

For information about gateway quotas, see Quotas for ﬁle shares (p. 210).

Network and ﬁrewall requirements

Your gateway requires access to the internet, local networks, Domain Name Service (DNS) servers, ﬁrewalls, routers, and so on.

Network bandwidth requirements vary based on the quantity of data that is uploaded and downloaded by the gateway. A minimum of 100Mbps is required to successfully download, activate, and update the gateway. Your data transfer patterns will determine the bandwidth necessary to support your workload.

Following, you can ﬁnd information about required ports and how to allow access through ﬁrewalls and routers.

Note

In some cases, you might deploy your gateway on Amazon EC2 or use other types of

deployment (including on-premises) with network security policies that restrict AWS IP address ranges. In these cases, your gateway might experience service connectivity issues when the AWS IP range values changes. The AWS IP address range values that you need to use are in the Amazon service subset for the AWS Region that you activate your gateway in. For the current IP range values, see AWS IP address ranges in the AWS General Reference.

Topics

• Port requirements (p. 8)

• Networking and ﬁrewall requirements for the Storage Gateway Hardware Appliance (p. 12)

• Allowing AWS Storage Gateway access through ﬁrewalls and routers (p. 14)

• Conﬁguring security groups for your Amazon EC2 gateway instance (p. 15)

Port requirements

Storage Gateway requires certain ports to be allowed for its operation. The following illustrations show the required ports that you must allow for each type of gateway. Some ports are required by all gateway types, and others are required by speciﬁc gateway types. For more information about port requirements, see Port Requirements (p. 201).

(16)

Common ports for all gateway types

The following ports are common to all gateway types and are required by all gateway types.

Protocol Port Direction Source Destination How used

TCP 443 (HTTPS) Outbound Storage

Gateway AWS For

communication from Storage Gateway to the AWS service endpoint. For information about service endpoints, see Allowing AWS Storage Gateway access through ﬁrewalls and routers (p. 14).

TCP 80 (HTTP) Inbound The host

from which you connect to the AWS Management Console.

Storage

Gateway By local

systems to obtain the storage gateway activation key.

Port 80 is only used during activation of the Storage Gateway appliance.

Storage Gateway does not require port 80 to be publicly accessible.

The required level of access to port 80 depends on your network conﬁguration.

If you activate your gateway from the Storage Gateway console, the host from which you connect to the console must

(17)

have access to your gateway’s port 80.

UDP/UDP 53 (DNS) Outbound Storage

Gateway DNS server For

communication between Storage Gateway and the DNS server.

TCP 22 (Support

channel) Outbound Storage

Gateway AWS Support Allows AWS Support to access your gateway to help you with troubleshooting gateway issues.

You don't need this port open for the normal operation of your gateway, but it is required for troubleshooting.

UDP 123 (NTP) Outbound NTP client NTP server Used by local

systems to synchronize VM time to the host time.

Ports for ﬁle gateways

The following illustration shows the ports to open for an S3 File Gateway.

(18)

Note

For speciﬁc port requirements, see Port Requirements (p. 201).

For S3 File Gateway, you only need to use Microsoft Active Directory when you want to allow domain users to access a Server Message Block (SMB) ﬁle share. You can join your ﬁle gateway to any valid Microsoft Windows domain (resolvable by DNS).

You can also use the AWS Directory Service to create an AWS Managed Microsoft AD in the Amazon Web Services Cloud. For most AWS Managed Microsoft AD deployments, you need to conﬁgure the Dynamic Host Conﬁguration Protocol (DHCP) service for your VPC. For information about creating a DHCP options set, see Create a DHCP options set in the AWS Directory Service Administration Guide.

In addition to the common ports, Amazon S3 File Gateway requires the following ports.

TCP/UDP 2049 (NFS) Inbound NFS clients Storage

Gateway For local systems to connect to NFS shares that your gateway exposes.

TCP/UDP 111 (NFSv3) Inbound NFSv3 client Storage

Gateway For local systems to connect to

(19)

the port mapper that your gateway exposes.

Note

This port is needed onlyfor NFSv3.

TCP/UDP 20048 (NFSv3) Inbound NFSv3 client Storage

Gateway For local systems to connect to mounts that your gateway exposes.

Note

This port is needed only forNFSv3.

Networking and ﬁrewall requirements for the Storage Gateway Hardware Appliance

Each Storage Gateway Hardware Appliance requires the following network services:

• Internet access – an always-on network connection to the internet through any network interface on the server.

• DNS services – DNS services for communication between the hardware appliance and DNS server.

• Time synchronization – an automatically conﬁgured Amazon NTP time service must be reachable.

• IP address – A DHCP or static IPv4 address assigned. You cannot assign an IPv6 address.

There are ﬁve physical network ports at the rear of the Dell PowerEdge R640 server. From left to right (facing the back of the server) these ports are as follows:

1. iDRAC 2. em1 3. em2 4. em3 5. em4

You can use the iDRAC port for remote server management.

(20)

A hardware appliance requires the following ports to operate.

SSH 22 Outbound Hardware

appliance 54.201.223.107 Support channel

DNS 53 Outbound Hardware

appliance DNS servers Name

resolution

UDP/NTP 123 Outbound Hardware

appliance *.amazon.pool.ntp.orgTime

synchronization

HTTPS 443 Outbound Hardware

appliance *.amazonaws.com Data transfer

HTTP 8080 Inbound AWS Hardware

appliance Activation

(only brieﬂy)

To perform as designed, a hardware appliance requires network and ﬁrewall settings as follows:

• Conﬁgure all connected network interfaces in the hardware console.

• Make sure that each network interface is on a unique subnet.

• Provide all connected network interfaces with outbound access to the endpoints listed in the diagram preceding.

• Conﬁgure at least one network interface to support the hardware appliance. For more information, see Conﬁguring network parameters (p. 23).

Note

For an illustration showing the back of the server with its ports, see Rack-mounting your hardware appliance and connecting it to power (p. 20).

All IP addresses on the same network interface (NIC), whether for a gateway or a host, must be on the same subnet. The following illustration shows the addressing scheme.

(21)

For more information about activating and conﬁguring a hardware appliance, see Using the Storage Gateway Hardware Appliance (p. 18).

Allowing AWS Storage Gateway access through ﬁrewalls and routers

Your gateway requires access to the following service endpoints to communicate with AWS. If you use a firewall or router to filter or limit network traffic, you must configure your firewall and router to allow these service endpoints for outbound communication to AWS.

Important

Depending on your gateway's AWS Region, replace region in the service endpoint with the correct Region string.

The following service endpoint is required by all gateways for head-bucket operations.

s3.amazonaws.com:443

The following service endpoints are required by all gateways for control path (anon-cp, client-cp, proxy-app) and data path (dp-1) operations.

anon-cp.storagegateway.region.amazonaws.com:443 client-cp.storagegateway.region.amazonaws.com:443 proxy-app.storagegateway.region.amazonaws.com:443 dp-1.storagegateway.region.amazonaws.com:443

The following gateway service endpoint is required to make API calls.

storagegateway.region.amazonaws.com:443

The following example is a gateway service endpoint in the US West (Oregon) Region (us-west-2).

storagegateway.us-west-2.amazonaws.com:443

The Amazon S3 service endpoint, shown following, is used by file gateways only. A file gateway requires this endpoint to access the Amazon S3 bucket that a file share maps to.

s3.region.amazonaws.com

The following example is an Amazon S3 service endpoint in the US East (Ohio) Region (us-east-2).

s3.us-east-2.amazonaws.com

Note

If your gateway can't determine the AWS Region where your S3 bucket is located, this service endpoint defaults to s3.us-east-1.amazonaws.com. We recommend that you allow access to the US East (N. Virginia) Region (us-east-1) in addition to Regions where your gateway is activated, and where your S3 bucket is located.

The following are Amazon S3 service endpoints for AWS GovCloud (US) Regions.

s3-fips-us-gov-west-1.amazonaws.com (AWS GovCloud (US-West) Region (FIPS)) s3-fips.us-gov-east-1.amazonaws.com (AWS GovCloud (US-East) Region (FIPS))

(22)

s3.us-gov-west-1.amazonaws.com (AWS GovCloud (US-West) Region (Standard)) s3.us-gov-east-1.amazonaws.com (AWS GovCloud (US-East) Region (Standard))

The following example is a FIPS service endpoint for an S3 bucket in the AWS GovCloud (US-West) Region.

bucket-name.s3-fips-us-gov-west-1.amazonaws.com

The Amazon CloudFront endpoint following is required for Storage Gateway to get the list of available AWS Regions.

https://d4kdq0yaxexbo.cloudfront.net/

A Storage Gateway VM is conﬁgured to use the following NTP servers.

0.amazon.pool.ntp.org 1.amazon.pool.ntp.org 2.amazon.pool.ntp.org 3.amazon.pool.ntp.org

• Storage Gateway—For supported AWS Regions and a list of AWS service endpoints that you can use with Storage Gateway, see AWS Storage Gateway endpoints and quotas in the AWS General Reference.

• Storage Gateway Hardware Appliance—For supported AWS Regions that you can use with the hardware appliance, see Storage Gateway hardware appliance Regions in the AWS General Reference.

Conﬁguring security groups for your Amazon EC2 gateway instance

In AWS Storage Gateway, a security group controls traﬃc to your Amazon EC2 gateway instance. When you conﬁgure a security group, we recommend the following:

• The security group should not allow incoming connections from the outside internet. It should allow only instances within the gateway security group to communicate with the gateway.

If you need to allow instances to connect to the gateway from outside its security group, we recommend that you allow connections only on ports 3260 (for iSCSI connections) and 80 (for activation).

• If you want to activate your gateway from an Amazon EC2 host outside the gateway security group, allow incoming connections on port 80 from the IP address of that host. If you cannot determine the activating host's IP address, you can open port 80, activate your gateway, and then close access on port 80 after completing activation.

• Allow port 22 access only if you are using AWS Support for troubleshooting purposes. For more information, see You want AWS Support to help troubleshoot your EC2 gateway (p. 177).

In some cases, you might use an Amazon EC2 instance as an initiator (that is, to connect to iSCSI targets on a gateway that you deployed on Amazon EC2. In such a case, we recommend a two-step approach:

1. You should launch the initiator instance in the same security group as your gateway.

2. You should conﬁgure access so the initiator can communicate with your gateway.

For information about the ports to open for your gateway, see Port Requirements (p. 201).

(23)

Supported hypervisors and host requirements

You can run Storage Gateway on-premises as either a virtual machine (VM) appliance or a physical hardware appliance, or in AWS as an Amazon EC2 instance.

Storage Gateway supports the following hypervisor versions and hosts:

• VMware ESXi Hypervisor (version 6.0, 6.5 or 6.7) – A free version of VMware is available on the VMware website. For this setup, you also need a VMware vSphere client to connect to the host.

• Microsoft Hyper-V Hypervisor (version 2012 R2 or 2016) – A free, standalone version of Hyper-V is available at the Microsoft Download Center. For this setup, you need a Microsoft Hyper-V Manager on a Microsoft Windows client computer to connect to the host.

• Linux Kernel-based Virtual Machine (KVM) – A free, open-source virtualization technology. KVM is included in all versions of Linux version 2.6.20 and newer. Storage Gateway is tested and supported for the CentOS/RHEL 7.7, Ubuntu 16.04 LTS, and Ubuntu 18.04 LTS distributions. Any other modern Linux distribution may work, but function or performance is not guaranteed. We recommend this option if you already have a KVM environment up and running and you are already familiar with how KVM works.

• Amazon EC2 instance – Storage Gateway provides an Amazon Machine Image (AMI) that contains the gateway VM image. For information about how to deploy a gateway on Amazon EC2, see Deploying a ﬁle gateway on an Amazon EC2 host (p. 197).

• Storage Gateway Hardware Appliance – Storage Gateway provides a physical hardware appliance as an on-premises deployment option for locations with limited virtual machine infrastructure.

Note

Storage Gateway doesn’t support recovering a gateway from a VM that was created from a snapshot or clone of another gateway VM or from your Amazon EC2 AMI. If your gateway VM malfunctions, activate a new gateway and recover your data to that gateway. For more information, see Recovering from an unexpected virtual machine shutdown (p. 190).

Storage Gateway doesn’t support dynamic memory and virtual memory ballooning.

Supported NFS clients for a ﬁle gateway

File gateways support the following Network File System (NFS) clients:

• Amazon Linux

• Mac OS X

Note

We recommend setting the rsize and wsize mount options to 64KB to improve performance when mounting NFS ﬁle shares on Mac OS X.

• RHEL 7

• SUSE Linux Enterprise Server 11 and SUSE Linux Enterprise Server 12

• Ubuntu 14.04

• Microsoft Windows 10 Enterprise, Windows Server 2012, and Windows Server 2016. Native clients only support NFS version 3.

• Windows 7 Enterprise and Windows Server 2008.

Native clients only support NFS v3. The maximum supported NFS I/O size is 32 KB, so you might experience degraded performance on these versions of Windows.

Note

You can now use SMB ﬁle shares when access is required through Windows (SMB) clients instead of using Windows NFS clients.

(24)

Supported SMB clients for a ﬁle gateway

File gateways support the following Service Message Block (SMB) clients:

• Microsoft Windows Server 2008 and later

• Windows desktop versions: 10, 8, and 7.

• Windows Terminal Server running on Windows Server 2008 and later

Note

Server Message Block encryption requires clients that support SMB v2.1.

Supported ﬁle system operations for a ﬁle gateway

Your NFS or SMB client can write, read, delete, and truncate ﬁles. When clients send writes to AWS Storage Gateway, it writes to local cache synchronously. Then it writes to Amazon S3 asynchronously through optimized transfers. Reads are ﬁrst served through the local cache. If data is not available, it's fetched through S3 as a read-through cache.

Writes and reads are optimized in that only the parts that are changed or requested are transferred through your gateway. Deletes remove objects from Amazon S3. Directories are managed as folder objects in S3, using the same syntax as in the Amazon S3 console.

HTTP operations such as GET, PUT, UPDATE, and DELETE can modify ﬁles in a ﬁle share. These operations conform to the atomic create, read, update, and delete (CRUD) functions.

Accessing AWS Storage Gateway

You can use the AWS Storage Gateway console to perform various gateway conﬁguration and

management tasks. The Getting Started section and various other sections of this guide use the console to illustrate gateway functionality.

Additionally, you can use the AWS Storage Gateway API to programmatically conﬁgure and manage your gateways. For more information about the API, see API Reference for Storage Gateway (p. 215).

You can also use the AWS SDKs to develop applications that interact with Storage Gateway. The AWS SDKs for Java, .NET, and PHP wrap the underlying Storage Gateway API to simplify your programming tasks. For information about downloading the SDK libraries, see the AWS Developer Center.

For information about pricing, see AWS Storage Gateway pricing.

Supported AWS Regions

• Storage Gateway — For supported AWS Regions and a list of AWS service endpoints that you can use with Storage Gateway, see AWS Storage Gateway endpoints and quotas in the AWS General Reference.

• Storage Gateway Hardware Appliance — For supported Regions that you can use with the hardware appliance, see AWS Storage Gateway Hardware Appliance Regions in the AWS General Reference.

(25)

Using the Storage Gateway Hardware Appliance

The Storage Gateway Hardware Appliance is a physical hardware appliance with the Storage Gateway software preinstalled on a validated server conﬁguration. You can manage your hardware appliance from the Hardware page on the AWS Storage Gateway console.

The hardware appliance is a high-performance 1U server that you can deploy in your data center, or on-premises inside your corporate firewall. When you buy and activate your hardware appliance, the activation process associates your hardware appliance with your AWS account. After activation, your hardware appliance appears in the console as a gateway on the Hardware page. You can configure your hardware appliance as a file gateway, tape gateway, or volume gateway type. The procedure that you use to deploy and activate these gateway types on a hardware appliance is same as on a virtual platform.

The Storage Gateway Hardware Appliance can be ordered directly from the AWS Storage Gateway console.

To order a hardware appliance

1. Open the Storage Gateway console at https://console.aws.amazon.com/storagegateway/home and choose the AWS Region that you want your appliance in.

2. Choose Hardware from the navigation pane.

3. Choose Order appliance, and then choose Proceed. You are redirected to the AWS Elemental Appliances and Software Management Console to request a sales quote.

4. Fill out the necessary information and choose Submit.

Once the information has been reviewed, a sale quote is generated and you are able to proceed with the ordering process and submit a Purchase Order, or arrange for pre-payment.

To view a sales quote or order history for the hardware appliance

1. Open the Storage Gateway console at https://console.aws.amazon.com/storagegateway/home.

2. Choose Hardware from the navigation pane.

3. Choose Quotes and orders, and then choose Proceed. You are redirected to the AWS Elemental Appliances and Software Management Console to review sales quotes and order history.

In the sections that follow, you can ﬁnd instructions about how to set up, conﬁgure, activate, launch, and use an Storage Gateway Hardware Appliance.

Topics

• Supported AWS Regions (p. 19)

• Setting up your hardware appliance (p. 19)

• Rack-mounting your hardware appliance and connecting it to power (p. 20)

• Conﬁguring network parameters (p. 23)

(26)

Supported AWS Regions

• Activating your hardware appliance (p. 25)

• Launching a gateway (p. 27)

• Conﬁguring an IP address for the gateway (p. 27)

• Conﬁguring your gateway (p. 28)

• Removing a gateway from the hardware appliance (p. 28)

• Deleting your hardware appliance (p. 29)

Supported AWS Regions

Storage Gateway Hardware Appliance is available for shipping worldwide where it is legally allowed and permitted for exporting by the US government. For information about supported AWS Regions, see Storage Gateway Hardware Appliance Regions in the AWS General Reference.

Setting up your hardware appliance

After you receive your Storage Gateway Hardware Appliance, you use the hardware appliance console to conﬁgure networking to provide an always-on connection to AWS and activate your appliance. Activation associates your appliance with the AWS account that is used during the activation process. After the appliance is activated, you can launch a ﬁle, volume, or tape gateway from the Storage Gateway console.

To install and conﬁgure your hardware appliance

1. Rack-mount the appliance, and plug in power and network connections. For more information, see Rack-mounting your hardware appliance and connecting it to power (p. 20).

2. Set the Internet Protocol version 4 (IPv4) addresses for both the hardware appliance (the host) and Storage Gateway (the service). For more information, see Conﬁguring network parameters (p. 23).

3. Activate the hardware appliance on the console Hardware page in the AWS Region of your choice.

For more information, see Activating your hardware appliance (p. 25).

4. Install the Storage Gateway on your hardware appliance. For more information, see Conﬁguring your gateway (p. 28).

You set up gateways on your hardware appliance the same way that you set up gateways on VMware ESXi, Microsoft Hyper-V, Linux Kernel-based Virtual Machine (KVM), or Amazon EC2.

Increasing the usable cache storage

You can increase the usable storage on the hardware appliance from 5 TB to 12 TB. Doing this provides a larger cache for low latency access to data in AWS. If you ordered the 5 TB model, you can increase the usable storage to 12 TB by buying ﬁve 1.92 TB SSDs (solid state drives), which are available for ordering on the console Hardware page. You can order the additional SSDs by following the same ordering process as ordering a hardware appliance and requesting a sales quote from the Storage Gateway console.

You can then add them to the hardware appliance before you activate it. If you have already activated the hardware appliance and want to increase the usable storage on the appliance to 12 TB, do the following:

1. Reset the hardware appliance to its factory settings. Contact AWS Support for instructions on how to do this.

(27)

Rack-mounting and connecting the hardware appliance to power 2. Add ﬁve 1.92 TB SSDs to the appliance.

Network interface card options

Depending on the model of appliance you ordered, it may come with a 10G-Base-T copper network card or a 10G DA/SFP+ network card.

• 10G-Base-T NIC conﬁguration:

• Use CAT6 cables for 10G or CAT5(e) for 1G

• 10G DA/SFP+ NIC conﬁguration:

• Use Twinax copper Direct Attach Cables up to 5 meters

• Dell/Intel compatible SFP+ optical modules (SR or LR)

• SFP/SFP+ copper transceiver for 1G-Base-T or 10G-Base-T

Rack-mounting your hardware appliance and connecting it to power

After you unbox your Storage Gateway Hardware Appliance, follow the instructions contained in the box to rack-mount the server. Your appliance has a 1U form factor and ﬁts in a standard International Electrotechnical Commission (IEC) compliant 19-inch rack.

To install your hardware appliance, you need the following components:

• Power cables: one required, two recommended.

• Supported network cabling (depending on which Network Interface Card (NIC) is included in the hardware appliance). Twinax Copper DAC, SFP+ optical module (Intel compatible) or SFP to Base-T copper transceiver.

• Keyboard and monitor, or a keyboard, video, and mouse (KVM) switch solution.

Hardware appliance dimensions

(28)

Hardware appliance dimensions

To connect the hardware appliance to power

Note

Before you perform the following procedure, make sure that you meet all of the requirements for the Storage Gateway Hardware Appliance as described in Networking and ﬁrewall

requirements for the Storage Gateway Hardware Appliance (p. 12).

1. Plug in a power connection to each of the two power supplies. It's possible to plug in to only one power connection, but we recommend power connections to both power supplies.

(29)

Hardware appliance dimensions

In the following image, you can see the hardware appliance with the diﬀerent connections.

2. Plug an Ethernet cable into the em1 port to provide an always-on internet connection. The em1 port is the ﬁrst of the four physical network ports on the rear, from left to right.

Note

The hardware appliance doesn't support VLAN trunking. Set up the switch port to which you are connecting the hardware appliance as a non-trunked VLAN port.

3. Plug in the keyboard and monitor.

4. Power on the server by pressing the Power button on the front panel, as shown in the following image.

After the server boots up, the hardware console appears on the monitor. The hardware console presents a user interface specific to AWS that you can use to configure initial network parameters. You configure these parameters to connect the appliance to AWS and open up a support channel for troubleshooting by AWS Support.

To work with the hardware console, enter text from the keyboard and use the Up, Down, Right, and Left Arrow keys to move about the screen in the indicated direction. Use the Tab key to move forward in order through items on-screen. On some setups, you can use the Shift+Tab keystroke to move sequentially backward. Use the Enter key to save selections, or to choose a button on the screen.

To set a password for the ﬁrst time

1. For Set Password, enter a password, and then press Down arrow.

2. For Conﬁrm, re-enter your password, and then choose Save Password.

(30)

Conﬁguring network parameters

At this point, you are in the hardware console, shown following.

Next step

Conﬁguring network parameters (p. 23)

Conﬁguring network parameters

After the server boots up, you can enter your ﬁrst password in the hardware console as described in Rack-mounting your hardware appliance and connecting it to power (p. 20).

Next, on the hardware console take the following steps to conﬁgure network parameters so your hardware appliance can connect to AWS.

(31)

Conﬁguring network parameters

To set a network address

1. Choose Conﬁgure Network and press the Enter key. The Conﬁgure Network screen shown following appears.

2. For IP Address, enter a valid IPv4 address from one of the following sources:

• Use the IPv4 address assigned by your Dynamic Host Conﬁguration Protocol (DHCP) server to your physical network port.

If you do so, note this IPv4 address for later use in the activation step.

• Assign a static IPv4 address. To do so, choose Static in the em1 section and press Enter to view the Conﬁgure Static IP screen shown following.

The em1 section is at upper left section in the group of port settings.

After you have entered a valid IPv4 address, press the Down arrow or Tab.

Note

If you conﬁgure any other interface, it must provide the same always-on connection to the AWS endpoints listed in the requirements.

(32)

Activating your hardware appliance

3. For Subnet, enter a valid subnet mask, and then press Down arrow.

4. For Gateway, enter your network gateway’s IPv4 address, and then press Down arrow.

5. For DNS1, enter the IPv4 address for your Domain Name Service (DNS) server, and then press Down arrow.

6. (Optional) For DNS2, enter a second IPv4 address, and then press Down arrow. A second DNS server assignment would provide additional redundancy should the ﬁrst DNS server become unavailable.

7. Choose Save and then press Enter to save your static IPv4 address setting for the appliance.

To log out of the hardware console

1. Choose Back to return to the Main screen.

2. Choose Logout to return to the Login screen.

Next step

Activating your hardware appliance (p. 25)

Activating your hardware appliance

After conﬁguring your IP address, you enter this IP address in the console on the Hardware page, as described following. The activation process validates that your hardware appliance has the appropriate security credentials and registers the appliance to your AWS account.

You can choose to activate your hardware appliance in any of the supported AWS Regions. For a list of supported AWS Regions, see Storage Gateway Hardware Appliance Regions in the AWS General Reference.

To activate your appliance for the ﬁrst time or in an AWS Region where you have no gateways deployed

1. Sign in to the AWS Management Console and open the Storage Gateway console at AWS Storage Gateway Management Console with the account credentials to use to activate your hardware.

(33)

Activating your hardware appliance

If this is your ﬁrst gateway in an AWS Region, you see a splash screen. After you create a gateway in this AWS Region, the screen no longer displays.

Note

For activation only, the following must be true:

• Your browser must be on the same network as your hardware appliance.

• Your ﬁrewall must allow HTTP access on port 8080 to the appliance for inbound traﬃc.

2. Choose Get started to view the Create gateway wizard, and then choose Hardware Appliance on the Select host platform page, as shown following.

3. Choose Next to view the Connect to hardware screen shown following.

4. For IP Address in the Connect to hardware appliance section, enter the IPv4 address of your appliance, and then choose Connect to go to the Activate Hardware screen shown following.

5. For Hardware name, enter a name for your appliance. Names can be up to 255 characters long and can't include a slash character.

6. For Hardware time zone, enter your local settings.

The time zone controls when hardware updates take place, with 2 a.m. local time used as the time for updates.

Note

We recommend setting the time zone for your appliance as this determines a standard update time that is out of the usual working day window.

7. (Optional) Keep the RAID Volume Manager set to ZFS.

ZFS is used as the RAID volume manager on the hardware appliance to provide better performance and data protection. ZFS is a software-based, open-source ﬁle system and logical volume manager.

The hardware appliance is speciﬁcally tuned for ZFS RAID. For more information on ZFS RAID, see the ZFS Wikipedia page.

8. Choose Next to ﬁnish activation.

A console banner appears on the Hardware page indicating that the hardware appliance has been successfully activated, as shown following.

At this point, the appliance is associated with your account. The next step is to launch a ﬁle, tape, or cached volume gateway on your appliance.

Next step

Launching a gateway (p. 27)

(34)

Launching a gateway

You can launch any of the three storage gateways on the appliance—ﬁle gateway, volume gateway (cached), or tape gateway.

To launch a gateway on your hardware appliance

1. Sign in to the AWS Management Console and open the Storage Gateway console at https://

console.aws.amazon.com/storagegateway/home.

2. Choose Hardware.

3. For Actions, choose Launch Gateway.

4. For Gateway Type, choose File Gateway, Tape Gateway, or Volume Gateway (Cached).

5. For Gateway name, enter a name for your gateway. Names can be 255 characters long and can't include a slash character.

6. Choose Launch gateway.

The Storage Gateway software for your chosen gateway type installs on the appliance. It can take up to 5–10 minutes for a gateway to show up as online in the console.

To assign a static IP address to your installed gateway, you next conﬁgure the gateway's network interfaces so your applications can use it.

Next step

Conﬁguring an IP address for the gateway (p. 27)

Conﬁguring an IP address for the gateway

Before you activated your hardware appliance, you assigned an IP address to its physical network interface. Now that you have activated the appliance and launched your Storage Gateway on it, you need to assign another IP address to the Storage Gateway virtual machine that runs on the hardware appliance. To assign a static IP address to a gateway installed on your hardware appliance, conﬁgure the IP address from the local console for that gateway. Your applications (such as your NFS or SMB client, your iSCSI initiator, and so on) connect to this IP address. You can access the gateway local console from the hardware appliance console.

To conﬁgure an IP address on your appliance to work with applications

1. On the hardware console, choose Open Service Console to open a login screen for the gateway local console.

2. Enter the localhost login password, and then press Enter.

The default account is admin and the default password is password.

3. Change the default password. Choose Actions then Set Local Password and enter your new credentials in the Set Local Password dialog box.

4. (Optional) Conﬁgure your proxy settings. See Rack-mounting your hardware appliance and connecting it to power (p. 20) for instructions.

5. Navigate to the Network Settings page of the gateway local console as shown following.

(35)

Conﬁguring your gateway

6. Type 2 to go to the Network Conﬁguration page shown following.

7. Conﬁgure a static or DHCP IP address for the network port on your hardware appliance to present a ﬁle, volume, and tape gateway for applications. This IP address must be on the same subnet as the IP address used during hardware appliance activation.

To exit the gateway local console

• Press the Crtl+] (close bracket) keystroke. The hardware console appears.

Note

The keystroke preceding is the only way to exit the gateway local console.

Next step

Conﬁguring your gateway (p. 28)

Conﬁguring your gateway

After your hardware appliance has been activated and conﬁgured, your appliance appears in the console.

Now you can create the type of gateway that you want. Continue the installation for your gateway type.

For instructions, see Conﬁgure your Amazon S3 File Gateway (p. 32).

Removing a gateway from the hardware appliance

To remove gateway software from your hardware appliance, use the following procedure. After you do so, the gateway software is uninstalled from your hardware appliance.

To remove a gateway from a hardware appliance

1. Choose the check box for the gateway.

2. For Actions, choose Remove Gateway.

3. In the Remove gateway from hardware appliance dialog box, choose Conﬁrm.

(36)

Deleting your hardware appliance

Note

When you delete a gateway, you can't undo the action. For certain gateway types, you can lose data on deletion, particularly cached data. For more information on deleting a gateway, see Deleting Your Gateway by Using the AWS Storage Gateway Console and Removing Associated Resources (p. 125).

Deleting a gateway doesn't delete the hardware appliance from the console. The hardware appliance remains for future gateway deployments.

Deleting your hardware appliance

After you activate your hardware appliance in your AWS account, you might have a need to move and activate it in a diﬀerent AWS account. In this case, you ﬁrst delete the appliance from the AWS account and activate it in another AWS account. You might also want to delete the appliance completely from your AWS account because you no longer need it. Follow these instructions to delete your hardware appliance.

To delete your hardware appliance

1. If you have installed a gateway on the hardware appliance, you must ﬁrst remove the gateway before you can delete the appliance. For instructions on how to remove a gateway from your hardware appliance, see Removing a gateway from the hardware appliance (p. 28).

2. On the Hardware page, choose the hardware appliance you want to delete.

3. For Actions, choose Delete Appliance.

4. In the Conﬁrm deletion of resource(s) dialog box, choose the conﬁrmation check box and choose Delete. A message indicating successful deletion is displayed.

When you delete the hardware appliance, all the resources associated with the gateway that is installed on the appliance are delete also, but the data on the hardware appliance itself is not deleted.

(37)

Create an S3 File Gateway

Getting started with AWS Storage Gateway

In this section, you can ﬁnd instructions about how to create and activate a ﬁle gateway in AWS Storage Gateway. Before you get started, make sure that your setup meets the required prerequisites and other requirements described in Setting up for Amazon S3 File Gateway (p. 5).

Topics

• Create and activate an Amazon S3 File Gateway (p. 30)

Create and activate an Amazon S3 File Gateway

In this section, you can ﬁnd instructions on how to create, deploy, and activate a ﬁle gateway in AWS Storage Gateway.

Topics

• Set up an Amazon S3 File Gateway (p. 30)

• Connect your Amazon S3 File Gateway to AWS (p. 31)

• Review settings and activate your Amazon S3 File Gateway (p. 32)

• Conﬁgure your Amazon S3 File Gateway (p. 32)

Set up an Amazon S3 File Gateway

To set up a new S3 File Gateway

1. Open the AWS Management Console at https://console.aws.amazon.com/storagegateway/home/, and choose the AWS Region where you want to create your gateway.

2. Choose Create gateway to open the Set up gateway page.

3. In the Gateway settings section, do the following:

a. For Gateway name, enter a name for your gateway. After your gateway is created, you can search for this name to ﬁnd your gateway on the list pages in the AWS Storage Gateway console.

b. For Gateway time zone, choose the local time zone for the part of the world where you want to deploy your gateway.

4. In the Gateway options section, for Gateway type, choose Amazon S3 File Gateway.

5. In the Platform options section, do the following:

a. For Host platform, choose the platform on which you want to deploy your gateway. Then follow the platform-speciﬁc instructions displayed on the Storage Gateway console page to set up your host platform. You can choose from the following options:

• VMware ESXi – Download, deploy, and conﬁgure the gateway virtual machine using VMware ESXi.

AWS Storage Gateway

AWS Storage Gateway

User Guide

API Version 2013-06-30

AWS Storage Gateway: User Guide

Table of Contents

What is Amazon S3 File Gateway

Amazon S3 File Gateway

How Storage Gateway works (architecture)

Amazon S3 File Gateways

Setting up for Amazon S3 File Gateway

Sign up for Amazon Web Services

To sign up for an AWS account

Create an IAM user

To create an administrator user for yourself and add the user to an administrators group (console)

Note

Note

File gateway setup requirements

Required prerequisites

Hardware and storage requirements

Hardware requirements for on-premises VMs

Requirements for Amazon EC2 instance types

Note

Storage requirements

Note

Network and ﬁrewall requirements

Note

Port requirements

Note

Note

Note

Networking and ﬁrewall requirements for the Storage Gateway Hardware Appliance

Note

Allowing AWS Storage Gateway access through ﬁrewalls and routers

Important

Note

Conﬁguring security groups for your Amazon EC2 gateway instance

Supported hypervisors and host requirements

Note

Supported NFS clients for a ﬁle gateway

Note

Note

Supported SMB clients for a ﬁle gateway

Note

Supported ﬁle system operations for a ﬁle gateway

Accessing AWS Storage Gateway

Supported AWS Regions

Using the Storage Gateway Hardware Appliance

To order a hardware appliance

To view a sales quote or order history for the hardware appliance

Supported AWS Regions

Setting up your hardware appliance

To install and conﬁgure your hardware appliance

Rack-mounting your hardware appliance and connecting it to power

Hardware appliance dimensions

To connect the hardware appliance to power

Note

Note

To set a password for the ﬁrst time

Conﬁguring network parameters

To set a network address

Note

To log out of the hardware console

Activating your hardware appliance

To activate your appliance for the ﬁrst time or in an AWS Region where you have no gateways deployed

Note

Note

Launching a gateway

To launch a gateway on your hardware appliance

Conﬁguring an IP address for the gateway

To conﬁgure an IP address on your appliance to work with applications

To exit the gateway local console

Note

Conﬁguring your gateway

Removing a gateway from the hardware appliance

To remove a gateway from a hardware appliance

Note

Deleting your hardware appliance

To delete your hardware appliance

Getting started with AWS Storage Gateway