Amazon Simple Queue Service
Developer Guide
Amazon Simple Queue Service: Developer Guide
Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved.
Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon.
Table of Contents
What is Amazon SQS? ... 1
Benefits of using Amazon SQS ... 1
Differences between Amazon SQS, Amazon MQ, and Amazon SNS ... 1
Queue types ... 2
Common tasks for getting started with Amazon SQS ... 3
Pricing for Amazon SQS ... 3
Setting up ... 4
Step 1: Create an AWS account ... 4
Step 2: Create an IAM user ... 4
Step 3: Get your access key ID and secret access key ... 5
Step 4: Get ready to use the example code ... 6
Next steps ... 6
Getting started ... 7
Prerequisites ... 7
Step 1: Create a queue ... 7
Step 2: Send a message ... 7
Step 3: Receive and delete your message ... 8
Step 4: Delete your queue ... 9
Next steps ... 9
Configuring Amazon SQS ... 10
Understanding the Amazon SQS console ... 10
Creating a queue ... 11
Editing a queue ... 12
Configuring queue parameters ... 13
Configuring access policy ... 14
Configuring SSE-SQS for a queue ... 14
Configuring SSE-KMS for a queue ... 15
Configuring a dead-letter queue ... 16
Configuring a dead-letter queue redrive ... 16
Configuring tags for a queue ... 17
Subscribing a queue to a topic ... 17
Configuring a Lambda trigger ... 18
Prerequisites ... 18
Managing queues ... 20
Sending messages ... 20
Message attributes ... 21
Receiving and deleting a message ... 22
Purging a queue ... 22
Deleting a queue ... 23
Confirming that a queue is empty ... 23
How Amazon SQS works ... 25
Basic architecture ... 25
Distributed queues ... 25
Message lifecycle ... 26
Standard queues ... 27
Message ordering ... 28
At-least-once delivery ... 28
FIFO queues ... 28
Message ordering ... 29
Key terms ... 29
FIFO delivery logic ... 30
Exactly-once processing ... 31
Moving from a standard queue to a FIFO queue ... 31
High throughput for FIFO queues ... 32
Compatibility ... 37
Queue and message identifiers ... 37
Identifiers for Standard and FIFO queues ... 37
Additional identifiers for FIFO queues ... 38
Message metadata ... 39
Message attributes ... 39
Message system attributes ... 41
Resources required to process messages ... 42
List queue pagination ... 42
Cost allocation tags ... 42
Short and long polling ... 43
Consuming messages using short polling ... 43
Consuming messages using long polling ... 44
Differences between long and short polling ... 44
Dead-letter queues ... 45
How do dead-letter queues work? ... 45
What are the benefits of dead-letter queues? ... 46
How do different queue types handle message failure? ... 46
When should I use a dead-letter queue? ... 47
Moving messages out of a dead-letter queue ... 47
Troubleshooting dead-letter queues ... 48
Visibility timeout ... 49
Inflight messages ... 50
Setting the visibility timeout ... 50
Changing the visibility timeout for a message ... 51
Terminating the visibility timeout for a message ... 51
Delay queues ... 51
Temporary queues ... 52
Virtual queues ... 53
Request-response messaging pattern (virtual queues) ... 53
Example scenario: Processing a login request ... 54
Cleaning up queues ... 55
Message timers ... 56
Best practices ... 57
Recommendations for standard and FIFO queues ... 57
Working with messages ... 57
Reducing costs ... 59
Moving from a Standard queue to a FIFO queue ... 60
Additional recommendations for FIFO queues ... 60
Using the message deduplication ID ... 60
Using the message group ID ... 62
Using the receive request attempt ID ... 63
Java SDK examples ... 64
Using server-side encryption ... 64
Adding SSE to an existing queue ... 64
Disabling SSE for a queue ... 65
Creating a queue with SSE ... 65
Retrieving SSE attributes ... 65
Configuring tags ... 66
Listing tags ... 66
Adding or updating tags ... 66
Removing tags ... 67
Sending message attributes ... 67
Defining attributes ... 67
Sending a message with attributes ... 68
Managing large messages ... 69
Prerequisites ... 69
Example: Using Amazon S3 to manage large Amazon SQS messages ... 69
Working with JMS ... 73
Prerequisites ... 73
Getting started with the Java Messaging Library ... 74
Creating a JMS connection ... 74
Creating an Amazon SQS queue ... 75
Sending messages synchronously ... 75
Receiving messages synchronously ... 76
Receiving messages asynchronously ... 77
Using client acknowledge mode ... 78
Using unordered acknowledge mode ... 79
Using the JMS Client with other Amazon SQS clients ... 79
Working Java example for using JMS with Amazon SQS Standard queues ... 80
ExampleConfiguration.java ... 80
TextMessageSender.java ... 82
SyncMessageReceiver.java ... 83
AsyncMessageReceiver.java ... 85
SyncMessageReceiverClientAcknowledge.java ... 86
SyncMessageReceiverUnorderedAcknowledge.java ... 88
SpringExampleConfiguration.xml ... 91
SpringExample.java ... 92
ExampleCommon.java ... 93
Supported JMS 1.1 implementations ... 94
Supported common interfaces ... 94
Supported message types ... 95
Supported message acknowledgment modes ... 95
JMS-defined headers and reserved properties ... 95
Tutorials ... 96
Creating an Amazon SQS queue (AWS CloudFormation) ... 96
Sending a message from a VPC ... 97
Step 1: Create an Amazon EC2 key pair ... 98
Step 2: Create AWS resources ... 98
Step 3: Confirm that your EC2 instance isn't publicly accessible ... 99
Step 4: Create an Amazon VPC endpoint for Amazon SQS ... 99
Step 5: Send a message to your Amazon SQS queue ... 100
Quotas ... 102
Quotas related to queues ... 102
Quotas related to messages ... 103
Quotas related to policies ... 106
Automating and troubleshooting ... 107
Automating notifications using EventBridge ... 107
Troubleshooting queues using X-Ray ... 107
Security ... 108
Data protection ... 108
Data encryption ... 109
Internetwork traffic privacy ... 116
Identity and access management ... 117
Authentication ... 118
Access control ... 118
Overview ... 119
Using identity-based policies ... 123
Using custom policies with the Access Policy Language ... 131
Using temporary security credentials ... 142
API permissions reference ... 143
Logging and monitoring ... 145
Logging API calls using CloudTrail ... 146
Monitoring queues using CloudWatch ... 149
Compliance validation ... 157
Resilience ... 158
Distributed queues ... 158
Infrastructure security ... 159
Best practices ... 159
Preventative best practices ... 159
Working with APIs ... 162
Making Query API requests ... 162
Constructing an endpoint ... 162
Making a GET request ... 163
Making a POST request ... 163
Authenticating requests ... 164
Interpreting responses ... 167
Batch actions ... 168
Enabling client-side buffering and request batching ... 169
Increasing throughput using horizontal scaling and action batching ... 173
Related resources ... 182
Documentation history ... 183
AWS glossary ... 186
Benefits of using Amazon SQS
What is Amazon Simple Queue Service?
Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components. Amazon SQS offers common constructs such as dead-letter queues (p. 45) and cost allocation tags (p. 42). It provides a generic web services API that you can access using any programming language that the AWS SDK supports.
Amazon SQS supports both standard (p. 27) and FIFO queues (p. 28). For more information, see Queue types (p. 2).
Topics
• Benefits of using Amazon SQS (p. 1)
• Differences between Amazon SQS, Amazon MQ, and Amazon SNS (p. 1)
• Queue types (p. 2)
• Common tasks for getting started with Amazon SQS (p. 3)
• Pricing for Amazon SQS (p. 3)
Benefits of using Amazon SQS
• Security – You control (p. 117) who can send messages to and receive messages from an Amazon SQS queue.
Server-side encryption (SSE) (p. 109) lets you transmit sensitive data by protecting the contents of messages in queues using keys managed in AWS Key Management Service (AWS KMS).
• Durability – For the safety of your messages, Amazon SQS stores them on multiple servers. Standard queues support at-least-once message delivery (p. 28), and FIFO queues support exactly-once message processing (p. 31).
• Availability – Amazon SQS uses redundant infrastructure (p. 25) to provide highly-concurrent access to messages and high availability for producing and consuming messages.
• Scalability – Amazon SQS can process each buffered request (p. 169) independently, scaling transparently to handle any load increases or spikes without any provisioning instructions.
• Reliability – Amazon SQS locks your messages during processing, so that multiple producers can send and multiple consumers can receive messages at the same time.
• Customization – Your queues don't have to be exactly alike—for example, you can set a default delay on a queue (p. 51). You can store the contents of messages larger than 256 KB using Amazon
Simple Storage Service (Amazon S3) (p. 69) or Amazon DynamoDB, with Amazon SQS holding a pointer to the Amazon S3 object, or you can split a large message into smaller messages.
Differences between Amazon SQS, Amazon MQ, and Amazon SNS
Amazon SQS and Amazon SNS are queue and topic services that are highly scalable, simple to use, and don't require you to set up message brokers. We recommend these services for new applications that can benefit from nearly unlimited scalability and simple APIs.
Queue types
Amazon MQ is a managed message broker service that provides compatibility with many popular message brokers. We recommend Amazon MQ for migrating applications from existing message brokers that rely on compatibility with APIs such as JMS or protocols such as AMQP, MQTT, OpenWire, and STOMP.
Queue types
The following table describes the capabilities of standard queues and FIFO queues.
Standard queue FIFO queue
Unlimited Throughput – Standard queues support a nearly unlimited number of API calls per second, per API action (SendMessage, ReceiveMessage, or DeleteMessage).
At-Least-Once Delivery – A message is delivered at least once, but occasionally more than one copy of a message is delivered.
Best-Effort Ordering – Occasionally, messages are delivered in an order different from which they were sent.
High Throughput – If you use batching (p. 168), FIFO queues support up to 3,000 messages per second, per API method (SendMessageBatch, ReceiveMessage, or DeleteMessageBatch).
The 3000 messages per second represent 300 API calls, each with a batch of 10 messages.
To request a quota increase, submit a support request. Without batching, FIFO queues support up to 300 API calls per second, per API method (SendMessage, ReceiveMessage, or DeleteMessage).
Exactly-Once Processing – A message is delivered once and remains available until a consumer processes and deletes it. Duplicates aren't introduced into the queue.
First-In-First-Out Delivery – The order in which messages are sent and received is strictly preserved.
Send data between applications when the throughput is important, for example:
• Decouple live user requests from intensive background work: let users upload media while resizing or encoding it.
• Allocate tasks to multiple worker nodes:
process a high number of credit card validation requests.
• Batch messages for future processing: schedule multiple entries to be added to a database.
Send data between applications when the order of events is important, for example:
• Make sure that user-entered commands are run in the right order.
• Display the correct product price by sending price modifications in the right order.
• Prevent a student from enrolling in a course before registering for an account.
Common tasks for getting started with Amazon SQS
Common tasks for getting started with Amazon SQS
• To create your first queue with Amazon SQS and send, receive, and delete a message, see Getting started with Amazon SQS (p. 7).
• To trigger a Lambda function, see Configuring a queue to trigger an AWS Lambda function (console) (p. 18).
• To discover the functionality and architecture of Amazon SQS, see How Amazon SQS works (p. 25).
• To find out the guidelines and caveats that will help you make the most of Amazon SQS, see Best practices for Amazon SQS (p. 57).
• Explore the Amazon SQS examples for one of the AWS SDKs, such as the AWS SDK for Java 2.x Developer Guide.
• To learn about Amazon SQS actions, see the Amazon Simple Queue Service API Reference.
• To learn about Amazon SQS AWS CLI commands, see the AWS CLI Command Reference.
Pricing for Amazon SQS
Amazon SQS has no upfront costs. The first million monthly requests are free. After that, you pay based on the number and content of requests, and the interactions with Amazon S3 and the AWS Key Management Service.
For information, see Amazon SQS pricing.
Step 1: Create an AWS account
Setting up Amazon SQS
Topics
• Step 1: Create an AWS account (p. 4)
• Step 2: Create an IAM user (p. 4)
• Step 3: Get your access key ID and secret access key (p. 5)
• Step 4: Get ready to use the example code (p. 6)
• Next steps (p. 6)
Before you can use Amazon SQS for the first time, you must complete the following steps.
Step 1: Create an AWS account
To access any AWS service, you first need to create an AWS account, an Amazon.com account that can use AWS products. You can use your AWS account to view your activity and usage reports and to manage authentication and access.
To avoid using your AWS account root user for Amazon SQS actions, it is a best practice to create an IAM user for each person who needs administrative access to Amazon SQS.
To set up a new account
1. Open https://portal.aws.amazon.com/billing/signup.
2. Follow the online instructions.
Part of the sign-up procedure involves receiving a phone call and entering a verification code on the phone keypad.
Step 2: Create an IAM user
To create an administrator user for yourself and add the user to an administrators group (console)
1. Sign in to the IAM console as the account owner by choosing Root user and entering your AWS account email address. On the next page, enter your password.
NoteWe strongly recommend that you adhere to the best practice of using the Administrator IAM user that follows and securely lock away the root user credentials. Sign in as the root user only to perform a few account and service management tasks.
2. In the navigation pane, choose Users and then choose Add user.
3. For User name, enter Administrator.
4. Select the check box next to AWS Management Console access. Then select Custom password, and then enter your new password in the text box.
5. (Optional) By default, AWS requires the new user to create a new password when first signing in. You can clear the check box next to User must create a new password at next sign-in to allow the new user to reset their password after they sign in.
Step 3: Get your access key ID and secret access key
6. Choose Next: Permissions.
7. Under Set permissions, choose Add user to group.
8. Choose Create group.
9. In the Create group dialog box, for Group name enter Administrators.
10. Choose Filter policies, and then select AWS managed - job function to filter the table contents.
11. In the policy list, select the check box for AdministratorAccess. Then choose Create group.
NoteYou must activate IAM user and role access to Billing before you can use the
AdministratorAccess permissions to access the AWS Billing and Cost Management console. To do this, follow the instructions in step 1 of the tutorial about delegating access to the billing console.
12. Back in the list of groups, select the check box for your new group. Choose Refresh if necessary to see the group in the list.
13. Choose Next: Tags.
14. (Optional) Add metadata to the user by attaching tags as key-value pairs. For more information about using tags in IAM, see Tagging IAM entities in the IAM User Guide.
15. Choose Next: Review to see the list of group memberships to be added to the new user. When you are ready to proceed, choose Create user.
You can use this same process to create more groups and users and to give your users access to your AWS account resources. To learn about using policies that restrict user permissions to specific AWS resources, see Access management and Example policies.
Step 3: Get your access key ID and secret access key
To use Amazon SQS actions (for example, using Java or through the AWS Command Line Interface), you need an access key ID and a secret access key.
NoteThe access key ID and secret access key are specific to AWS Identity and Access Management.
Don't confuse them with credentials for other AWS services, such as Amazon EC2 key pairs.
Access keys consist of an access key ID and secret access key, which are used to sign programmatic requests that you make to AWS. If you don't have access keys, you can create them from the AWS Management Console. As a best practice, do not use the AWS account root user access keys for any task where it's not required. Instead, create a new administrator IAM user with access keys for yourself.
The only time that you can view or download the secret access key is when you create the keys. You cannot recover them later. However, you can create new access keys at any time. You must also have permissions to perform the required IAM actions. For more information, see Permissions required to access IAM resources in the IAM User Guide.
To create access keys for an IAM user
1. Sign in to the AWS Management Console and open the IAM console at https://
console.aws.amazon.com/iam/.
2. In the navigation pane, choose Users.
3. Choose the name of the user whose access keys you want to create, and then choose the Security credentials tab.
4. In the Access keys section, choose Create access key.
Step 4: Get ready to use the example code
5. To view the new access key pair, choose Show. You will not have access to the secret access key again after this dialog box closes. Your credentials will look something like this:
• Access key ID: AKIAIOSFODNN7EXAMPLE
• Secret access key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
6. To download the key pair, choose Download .csv file. Store the keys in a secure location. You will not have access to the secret access key again after this dialog box closes.
Keep the keys confidential in order to protect your AWS account and never email them. Do not share them outside your organization, even if an inquiry appears to come from AWS or Amazon.com. No one who legitimately represents Amazon will ever ask you for your secret key.
7. After you download the .csv file, choose Close. When you create an access key, the key pair is active by default, and you can use the pair right away.
Related topics
• What is IAM? in the IAM User Guide
• AWS security credentials in AWS General Reference
Step 4: Get ready to use the example code
This guide includes examples that use the AWS SDK for Java. To run the example code, follow the set-up instructions in Getting Started with AWS SDK for Java 2.0.
You can develop AWS applications in other programming languages, such as Go, JavaScript, Python and Ruby. For more information, see Tools for developing and managing applications on AWS.
Note
You can explore Amazon SQS without writing code with tools such as the AWS Command Line Interface (AWS CLI) or Windows PowerShell. You can find AWS CLI examples in the Amazon SQS section of the AWS CLI Command Reference. You can find Windows PowerShell examples in the Amazon Simple Queue Service section of the AWS Tools for PowerShell Cmdlet Reference.
Next steps
You are now ready for Getting started (p. 7) with managing Amazon SQS queues and messages using the AWS Management Console.
Prerequisites
Getting started with Amazon SQS
This section helps you become more familiar with Amazon SQS by showing you how to manage queues and messages using the Amazon SQS console.
Topics
• Prerequisites (p. 7)
• Step 1: Create a queue (p. 7)
• Step 2: Send a message (p. 7)
• Step 3: Receive and delete your message (p. 8)
• Step 4: Delete your queue (p. 9)
• Next steps (p. 9)
Prerequisites
Before you begin, complete the steps in Setting up Amazon SQS (p. 4).
Step 1: Create a queue
The first and most common Amazon SQS task is creating a queue. This procedure shows how to create and configure a FIFO queue.
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. Choose Create queue.
3. On the Create queue page, specify the correct region.
4. The Standard queue type is selected by default. Choose FIFO.
You can't change the queue type after you create a queue.
5. Enter a Name for your queue. The name of a FIFO queue must end with the .fifo suffix.
6. To create your queue with the default parameters, scroll to the bottom and choose Create Queue.
Amazon SQS creates the queue and displays the queue's Details page.
Amazon SQS propagates information about the new queue across the system. Because Amazon SQS is a distributed system, you may experience a slight delay before the queue is displayed on the Queues page.
Step 2: Send a message
After you create your queue, you can send a message to it.
1. From the left navigation pane, choose Queues. From the queue list, select the queue that you created.
2. From Actions, choose Send and receive messages.
Step 3: Receive and delete your message
The console displays the Send and receive messages page.
3. Enter text in the Message body
4. Enter a Message group id for the queue. For more information, see FIFO delivery logic (p. 30).
5. (Optional) Enter a Message deduplication id. If you enable content-based deduplication, the message deduplication ID is not required. For more information, see FIFO delivery logic (p. 30).
6. Choose Send message.
Your message is sent and the console displays a success message. Choose View details to display information about the sent message.
Step 3: Receive and delete your message
After you send a message to a queue, you can retrieve the message from the queue. When you request messages from a queue, you can't specify which message to retrieve. Instead, you specify the maximum number of messages (up to 10) that you want to retrieve.
1. From the Queues page, select a queue.
2. From Queue Actions, select Send and receive messages.
The console displays the Send and receive messages page.
3. Choose Poll for messages.
Amazon SQS begins to poll servers to find messages in the queue. The progress bar on the right side of the Receive messages section displays the polling duration.
The Messages section displays a list of the received messages. For each message, the list displays the message ID, sent date, size, and receive count.
4. To delete messages, select the messages that you want to delete and then choose Delete.
5. In the Delete Messages dialog box, choose Delete.
Step 4: Delete your queue
Step 4: Delete your queue
1. From the queue list, select the queue that you have created.
2. From the Queues page, select the queue to delete.
3. Choose Delete queue.
The console displays the Delete queue dialog box.
4. In the Delete queue dialog box, confirm the deletion by entering delete.
5. Choose Delete.
Next steps
Now that you've created a queue and learned how to send, receive, and delete messages and how to delete a queue, you might want to try the following:
• Configure queues, including SSE and other features (p. 10).
• Send a message with attributes. (p. 21)
• Send a message from a VPC. (p. 97)
• Learn more about Amazon SQS workflows and processes: Read How Queues Work (p. 25), Best Practices (p. 57), and Quotas (p. 102). You can also explore the Amazon SQS Articles & Tutorials.
If you ever have any questions, browse the Amazon SQS FAQs or participate in the Amazon SQS Developer Forums.
• Learn how to interact with Amazon SQS programmatically: Read Working with APIs (p. 162) and explore the Sample Code and Libraries and the developer centers:
• Java
• JavaScript
• PHP
• Python
• Ruby
• Windows & .NET
• Learn about keeping an eye on costs and resources in the Automating and troubleshooting Amazon SQS queues (p. 107) section.
• Learn about protecting your data and access to it in the Security (p. 108) section.
Understanding the Amazon SQS console
Configuring Amazon SQS queues (console)
Use the Amazon SQS console to configure and manage Amazon Simple Queue Service (Amazon SQS) queues and features. You can also use the console to configure features such as server-side encryption, associate a dead-letter queue with your queue, or set a trigger to invoke an AWS Lambda function.
Topics
• Understanding the Amazon SQS console (p. 10)
• Creating an Amazon SQS queue (console) (p. 11)
• Editing an Amazon SQS queue (console) (p. 12)
• Configuring queue parameters (console) (p. 13)
• Configuring access policy (console) (p. 14)
• Configuring server-side encryption (SSE) for a queue using SQS-owned encryption keys (console) (p. 14)
• Configuring server-side encryption (SSE) for a queue (console) (p. 15)
• Configuring a dead-letter queue (console) (p. 16)
• Configuring a dead-letter queue redrive (console) (p. 16)
• Configuring cost allocation tags for an Amazon SQS queue (console) (p. 17)
• Subscribing an Amazon SQS queue to an Amazon SNS topic (console) (p. 17)
• Configuring a queue to trigger an AWS Lambda function (console) (p. 18)
Understanding the Amazon SQS console
When you open the console, choose Queues from the navigation pane to display the Queues page. The Queues page provides information about all of your queues in the active region.
The entry for each queue shows the queue type and other information about the queue. The Type column helps you distinguish standard queues from First-In-First Out (FIFO) queues at a glance.
From the Queues page, there are two ways to perform actions on a queue. You can choose the option next to the queue name and then choose the action you want to perform on the queue.
Creating a queue
You can also choose the queue name, which open the Details page for the queue. The Details page includes the same actions as the Queues page. In addition, you can choose one of the tabs below the Details section to view additional configuration details and actions.
Creating an Amazon SQS queue (console)
You can use the Amazon SQS console to create standard queues (p. 27) and FIFO queues (p. 28).
The console provides default values for all settings except for the queue name.
Important
Do not add personally identifiable information (PII) or other confidential or sensitive information in queue names. Queue names are accessible to many Amazon Web Services, including billing and CloudWatch logs. Queue names are not intended to be used for private or sensitive data.
To create an Amazon SQS queue (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. Choose Create queue.
3. For Type, the Standard queue type is set by default. To create a FIFO queue, choose FIFO.
NoteYou can't change the queue type after you create the queue.
4. Enter a Name for your queue. The name of a FIFO queue must end with the .fifo suffix.
5. (Optional) The console sets default values for the queue configuration parameters (p. 13). Under Configuration, you can set new values for the following parameters:
a. For Visibility timeout , enter the duration and units. The range is from 0 seconds to 12 hours.
The default value is 30 seconds.
b. For Message retention period, enter the duration and units. The range is from 1 minute to 14 days. The default value is 4 days.
Editing a queue
c. For Delivery delay, enter the duration and units. The range is from 0 seconds to 15 minutes.
The default value is 0 seconds.
d. For Maximum message size, enter a value. The range is from 1 KB to 256 KB. The default value is 256 KB.
e. For Receive message wait time, enter a value. The range is from 0 to 20 seconds. The default value is 0 seconds, which sets short polling (p. 43). Any non-zero value sets long polling.
f. For a FIFO queue, choose Enable content-based deduplication to enable content-based deduplication. The default setting is disabled.
g. (Optional) For a FIFO queue, to enable higher throughput for sending and receiving messages in the queue, choose Enable high throughput FIFO.
Choosing this option changes the related options (Deduplication scope and FIFO throughput limit) to the required settings for enabling high throughput for FIFO queues. If you change any of the settings required for using high throughput FIFO, normal throughput is in effect for the queue, and deduplication occurs as specified. For more information, see High throughput for FIFO queues (p. 32) and Quotas related to messages (p. 103).
h. For Redrive allow policy, choose Allow all (the default), By queue or Deny all. When choosing By queue, specify a list of up to 10 source queues by the Amazon Resource Name (ARN).
6. (Optional) Define an Access policy. The access policy (p. 139) defines the accounts, users, and roles that can access the queue. The access policy also defines the actions (such as SendMessage, ReceiveMessage, or DeleteMessage) that the users can access. The default policy allows only the queue owner to send and receive messages.
To define the access policy, do one of the following:
• Choose Basic to configure who can send messages to the queue and who can receive messages from the queue. The console creates the policy based on your choices and displays the resulting access policy in the read-only JSON panel.
• Choose Advanced to modify the JSON access policy directly. This allows you to specify a custom set of actions that each principal (account, user, or role) can perform.
7. (Optional) To configure encryption (p. 15) for the queue, expand Encryption.
8. (Optional) To configure a dead-letter queue (p. 16) to receive undeliverable messages, expand Dead-letter queue.
9. (Optional) To add tags (p. 17) to the queue, expand Tags.
10. Choose Create queue. Amazon SQS creates the queue and displays the queue's Details page.
Amazon SQS propagates information about the new queue across the system. Because Amazon SQS is a distributed system, you might experience a slight delay before the console displays the queue on the Queues page.
After creating a queue, you can send messages (p. 20) to it, and receive and delete
messages (p. 22). You can also edit (p. 12) any of the queue configuration settings except the queue type.
Editing an Amazon SQS queue (console)
You can use the Amazon SQS console to edit any queue configuration parameters (except the queue type) and add or remove queue features.
To edit an Amazon SQS queue (console)
1. Open the Queues page of the Amazon SQS console.
Configuring queue parameters
2. Select a queue, and then choose Edit.
3. (Optional) Under Configuration, update the queue's configuration parameters (p. 13).
4. (Optional) To update the access policy (p. 14), under Access policy, modify the JSON policy.
5. (Optional) To add, update, or remove encryption (p. 15), expand Encryption.
6. (Optional) To add, update, or remove a dead-letter queue (p. 16) (which allows you to receive undeliverable messages), expand Dead-letter queue.
7. (Optional) To add, update, or remove the tags (p. 17) for the queue, expand Tags.
8. Choose Save.
The console displays the Details page for the queue.
Configuring queue parameters (console)
When you create (p. 11) or edit (p. 12) a queue, you can configure the following parameters:
• Visibility timeout – The length of time that a message received from a queue (by one consumer) won't be visible to the other message consumers. For more information, see Visibility timeout (p. 49).
Note
Using the console to configure the visibility timeout configures the timeout value for all of the messages in the queue. To configure the timeout for single or multiple messages, you must use one of the AWS SDKs.
• Message retention period – The amount of time that Amazon SQS retains messages that remain in the queue. By default, the queue retains messages for four days. You can configure a queue to retain messages for up to 14 days. For more information, see Message retention period.
• Delivery delay – The amount of time that Amazon SQS will delay before delivering a message that is added to the queue. For more information, see Delivery delay (p. 51).
• Maximum message size – The maximum message size for this queue. For more information, see Maximum message size (p. 69).
• Receive message wait time – The maximum amount of time that Amazon SQS waits for messages to become available after the queue gets a receive request. For more information, see Amazon SQS short and long polling (p. 43).
• Enable content-based deduplication – Amazon SQS can automatically create deduplication IDs based on the body of the message. For more information, see Amazon SQS FIFO (First-In-First-Out) queues (p. 28).
• Enable high throughput FIFO – Use to enable high throughput for messages in the queue. Choosing this option changes the related options (Deduplication scope and FIFO throughput limit) to the required settings for enabling high throughput for FIFO queues. For more information, see High throughput for FIFO queues (p. 32) and Quotas related to messages (p. 103).
• Redrive allow policy: defines which source queues can use this queue as the dead-letter queue. For more information, see Amazon SQS dead-letter queues (p. 45).
To configure queue parameters for an existing queue (console) 1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues. Choose a queue and choose Edit.
3. Scroll to the Configuration section.
4. For Visibility timeout , enter the duration and units. The range is 0 seconds to 12 hours. The default value is 30 seconds.
5. For Message retention period, enter the duration and units. The range is 1 minute to 14 days. The default value is 4 days.
Configuring access policy
6. For Delivery delay, enter the duration and units. The range is 0 seconds to 15 minutes. The default value is 0 seconds.
7. For Maximum message size, enter a value. The range is 1 KB to 256 KB. The default value is 256 KB.
8. For a standard queue, enter a value for Receive message wait time. The range is 0 to 20 seconds.
The default value is 0 seconds, which sets short polling (p. 43). Any non-zero value sets long polling.
9. For a FIFO queue, choose Enable content-based deduplication to enable content-based deduplication. The default setting is disabled.
10. (Optional) For a FIFO queue, to enable higher throughput for sending and receiving messages in the queue, choose Enable high throughput FIFO.
Choosing this option changes the related options (Deduplication scope and FIFO throughput limit) to the required settings for enabling high throughput for FIFO queues. If you change any of the settings required for using high throughput FIFO, normal throughput is in effect for the queue, and deduplication occurs as specified. For more information, see High throughput for FIFO queues (p. 32) and Quotas related to messages (p. 103).
11. For Redrive allow policy, choose Allow all (the default), By queue or Deny all. When choosing By queue, specify a list of up to 10 source queues by the Amazon Resource Name (ARN).
12. When you finish configuring the queue parameters, choose Save.
Configuring access policy (console)
When you edit (p. 12) a queue, you can configure its access policy.
The access policy defines the accounts, users, and roles that can access the queue. The access policy also defines the actions (such as SendMessage, ReceiveMessage, or DeleteMessage) that the users can access. The default policy allows only the queue owner to send and receive messages.
To configure the access policy for an existing queue (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. Choose a queue and choose Edit.
4. Scroll to the Access policy section.
5. Edit the access policy statements in the input box.
6. When you finish configuring the access policy, choose Save.
Configuring server-side encryption (SSE) for a
queue using SQS-owned encryption keys (console)
Amazon SQS managed SSE (SSE-SQS) is managed server-side encryption that uses SQS-owned encryption keys to protect sensitive data sent over message queues. With SSE-SQS, you don't need to create and manage encryption keys, or modify your code to encrypt your data. SSE-SQS lets you transmit data securely and helps you meet strict encryption compliance and regulatory requirements at no additional cost.
SSE-SQS protects data at rest using 256-bit Advanced Encryption Standard (AES-256) encryption. SSE encrypts messages as soon as Amazon SQS receives them. Amazon SQS stores messages in encrypted form and decrypts them only when sending them to an authorized consumer.
Configuring SSE-KMS for a queue
To configure SSE-SQS encryption for a queue (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. Choose a queue, and then choose Edit.
4. Expand Encryption.
5. For Server-side encryption, choose Enabled.
6. Select Amazon SQS key (SSE-SQS). There is no additional fee for using this option.
7. Choose Save.
Configuring server-side encryption (SSE) for a queue (console)
To protect the data in a queue's messages, you can enable server-side encryption (SSE) for a queue.
Amazon SQS integrates with the Amazon Web Services Key Management Service (Amazon Web Services KMS) to manage KMS keys for server-side encryption (SSE). For information about using SSE, see Encryption at rest (p. 109).
The KMS key that you assign to your queue must have a key policy that includes permissions for all principals that are authorized to use the queue. For information, see Key Management (p. 111).
If you aren't the owner of the KMS key, or if you log in with an account that doesn't have
kms:ListAliases and kms:DescribeKey permissions, you won't be able to view information about the KMS key on the Amazon SQS console. Ask the owner of the KMS key to grant you these permissions.
For more information, see Key Management (p. 111).
When you create (p. 11) or edit (p. 12) a queue, you can configure SSE-KMS.
To configure SSE-KMS for an existing queue (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. Choose a queue, and then choose Edit.
4. Expand Encryption.
5. For Server-side encryption, choose Enabled.
6. Select AWS Key Management Service key (SSE-KMS).
The console displays the Description, the Account, and the KMS key ARN of the KMS key.
7. Specify the KMS key ID for the queue. For more information, see Key terms (p. 110).
a. Choose the Choose a KMS key alias option.
b. The default key is the Amazon Web Services managed KMS key for Amazon SQS. To use this key, choose it from the KMS key list.
c. To use a custom KMS key from your Amazon Web Services account, choose it from the KMS key list. For instructions on creating custom KMS keys, see Creating Keys in the Amazon Web Services Key Management Service Developer Guide.
d. To use a custom KMS key that is not in the list, or a custom KMS key from another Amazon Web Services account, choose Enter the KMS key alias and enter the KMS key Amazon Resource Name (ARN).
8. (Optional) For Data key reuse period, specify a value between 1 minute and 24 hours. The default is 5 minutes. For more information, see Understanding the data key reuse period (p. 114).
Configuring a dead-letter queue
9. When you finish configuring SSE-KMS, choose Save.
Configuring a dead-letter queue (console)
A dead-letter queue is a queue that one or more source queues can use for messages that are not consumed successfully. For more information, see Amazon SQS dead-letter queues (p. 45).
Amazon SQS does not create the dead-letter queue automatically. You must first create the queue before using it as a dead-letter queue..
The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a standard queue.
When you create (p. 11) or edit (p. 12) a queue, you can configure a dead-letter queue.
To configure a dead-letter queue for an existing queue (console) 1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. Choose a queue and choose Edit.
4. Scroll to the Dead-letter queue section and choose Enabled.
5. Choose the Amazon Resource Name (ARN) of an existing Dead Letter Queue that you want to associate with this source queue.
6. To configure the number of times that a message can be received before being sent to a dead-letter queue, set Maximum receives to a value between 1 and 1,000.
7. When you finish configuring the dead-letter queue, choose Save.
After you save the queue, the console displays the Details page for your queue. On the Details page, the Dead-letter queue tab displays the Maximum Receives and Dead Letter Queue ARN in the Dead-letter queue.
Configuring a dead-letter queue redrive (console)
You can configure a dead-letter queue redrive to move standard unconsumed messages out of an existing dead-letter queue back to their source queues. For more information about dead letter queue redrive, see Moving messages out of a dead-letter queue (p. 47).
To configure a dead-letter queue redrive for an existing standard queue (console) 1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. Choose the name of queue that you have configured as a dead-letter queue (p. 16).
4. Choose Start DLQ redrive.
5. Under Redrive configuration, for Message destination, do either of the following:
• To redrive messages to their source queue, choose Redrive to source queue(s).
• To redrive messages to another queue, choose Redrive to custom destination. Then, enter the Amazon Resource Name (ARN) of an existing destination queue.
6. Under Velocity control settings, choose one of the following:
• System optimized - Redrive dead-letter queue messages at the maximum number of messages per second.
Configuring tags for a queue
• Custom max velocity - Redrive dead-letter queue messages with a custom maximum rate of messages per second. The maximum allowed rate is 500 messages per second.
7. When you finish configuring the dead-letter queue redrive, choose Redrive messages.
NoteA dead-letter queue redrive task can run a maximum of 36 hours. Amazon SQS supports a maximum of 100 active redrive tasks per account.
The redrive task resets the retention period. A new messageID and enqueueTime are assigned to redriven messages.
8. If you want to cancel the message redrive task, on the Details page for your queue, choose Cancel DLQ redrive. When canceling an in progress message redrive, any messages that have already been successfully moved to their move destination queue will remain in the destination queue.
Configuring cost allocation tags for an Amazon SQS queue (console)
To help organize and identify your Amazon SQS queues, you can add cost allocation tags to them. For more information, see Amazon SQS cost allocation tags (p. 42).
On the Details page for a queue, the Tagging tab displays the tags for the queue.
When you create (p. 11) or edit (p. 12) a queue, you can configure tags for it.
To configure tags for an existing queue (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. Choose a queue and choose Edit.
4. Scroll to the Tags section.
5. Add, modify, or remove the queue tags:
a. To add a tag, choose Add new tag, enter a Key and Value, and then choose Add new tag.
b. To update a tag, change its Key and Value.
c. To remove a tag, choose Remove next to its key-value pair.
6. When you finish configuring the tags, choose Save.
Subscribing an Amazon SQS queue to an Amazon SNS topic (console)
You can subscribe one or more Amazon SQS queues to an Amazon Simple Notification Service (Amazon SNS) topic. When you publish a message to a topic, Amazon SNS sends the message to each of the subscribed queues. Amazon SQS manages the subscription and any necessary permissions. For more information about Amazon SNS, see What is Amazon Simple Notification Service? in the Amazon Simple Notification Service Developer Guide.
When you subscribe an Amazon SQS queue to an SNS topic, Amazon SNS uses HTTPS to forward messages to Amazon SQS. For information about using Amazon SNS with encrypted Amazon SQS queues, see Configure KMS permissions for AWS services (p. 112).
Configuring a Lambda trigger
To subscribe a queue to an SNS topic (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. From the list of queues, choose the queue to subscribe to the SNS topic.
4. From Actions, choose Subscribe to Amazon SNS topic.
5. From the Specify an Amazon SNS topic available for this queue menu, choose the SNS topic for your queue.
If the SNS topic isn't listed in the menu, choose Enter Amazon SNS topic ARN and then enter the topic's Amazon Resource Name (ARN).
6. Choose Save.
7. To verify the result of the subscription, publish to the topic and then view the message that the topic sends to the queue. For more information, see Publishing a message to an Amazon SNS topic in the Amazon Simple Notification Service Developer Guide.
If your Amazon SQS queue and SNS topic are in different AWS accounts, the topic owner must first confirm the subscription. For more information, see Confirm the subscription in the Amazon Simple Notification Service Developer Guide.
For information on subscribing to a cross-region SNS topic, see Sending Amazon SNS messages to an Amazon SQS queue or AWS Lambda function in a different Region in the Amazon Simple Notification Service Developer Guide
Configuring a queue to trigger an AWS Lambda function (console)
You can use an AWS Lambda function to process messages in an Amazon SQS queue. Lambda polls the queue and invokes your Lambda function synchronously with an event that contains queue messages.
You can specify another queue to act as a dead-letter queue for messages that your Lambda function can't process.
A Lambda function can process items from multiple queues (using one Lambda event source for each queue). You can use the same queue with multiple Lambda functions.
If you associate an encrypted queue with a Lambda function but Lambda doesn't poll for messages, add the kms:Decrypt permission to your Lambda execution role.
Note the following restrictions:
• Your queue and the Lambda function must be in the same AWS Region.
• An encrypted queue (p. 109) that uses the default key (AWS managed KMS key for Amazon SQS) cannot invoke a Lambda function in a different AWS account.
For information about implementing the Lambda function, see Using AWS Lambda with Amazon SQS in the AWS Lambda Developer Guide.
Prerequisites
To configure Lambda function triggers, you must meet the following requirements:
• If you use an IAM user, your Amazon SQS role must include the following permissions:
Prerequisites
• lambda:CreateEventSourceMapping
• lambda:ListEventSourceMappings
• lambda:ListFunctions
• The Lambda execution role must include the following permissions:
• sqs:DeleteMessage
• sqs:GetQueueAttributes
• sqs:ReceiveMessage
• If you associate an encrypted queue with a Lambda function, add the kms:Decrypt permission to the Lambda execution role.
For more information, see Overview of managing access in Amazon SQS (p. 119).
To configure a queue to trigger a Lambda function (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. On the Queues page, choose the queue to configure.
4. On the queue's page, choose the Lambda triggers tab.
5. On the Lambda triggers page, choose a Lambda trigger.
If the list doesn't include the Lambda trigger that you need, choose Configure Lambda function trigger. Enter the Amazon Resource Name (ARN) of the Lambda function or choose an existing resource. Then choose Save.
6. Choose Save. The console saves the configuration and displays the Details page for the queue.
On the Details page, the Lambda triggers tab displays the Lambda function and its status. It takes approximately 1 minute for the Lambda function to become associated with your queue.
7. To verify the results of the configuration, send a message to your queue (p. 20) and then view the triggered Lambda function in the Lambda console.
Sending messages
Managing Amazon SQS queues (console)
After you create and configure your queue, you can use the Amazon SQS console to send messages to the queue and retrieve them from the queue.
Topics
• Sending messages to a queue (console) (p. 20)
• Sending a message with attributes (console) (p. 21)
• Receiving and deleting messages (console) (p. 22)
• Purging messages from an Amazon SQS queue (console) (p. 22)
• Deleting an Amazon SQS queue (p. 23)
• Confirming that a queue is empty (p. 23)
Sending messages to a queue (console)
After you create your queue, you can send messages to it.
To send a message (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. On the Queues page, choose a queue.
4. From Actions, choose Send and receive messages.
The console displays the Send and receive messages page.
5. In the Message body, enter the message text.
6. For a First-In-First-Out (FIFO) queue, enter a Message group ID. For more information, see FIFO delivery logic (p. 30).
7. (Optional) For a FIFO queue, you can enter a Message deduplication ID. If you enabled content- based deduplication for the queue, the message deduplication ID isn't required. For more information, see FIFO delivery logic (p. 30).
8. (Optional) For a standard queue, you can enter a value for Delivery delay and choose the units. For example, enter 60 and choose seconds. FIFO queues don't support timers on individual messages.
For more information, see Amazon SQS message timers (p. 56).
Message attributes
9. Choose Send message.
When your message is sent, the console displays a success message. Choose View details to display information about the sent message.
Sending a message with attributes (console)
For standard and FIFO queues, you can include structured metadata (such as timestamps, geospatial data, signatures, and identifiers) with messages. For more information, see Amazon SQS message attributes (p. 39).
To send a message with attributes to a queue (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. On the Queues page, choose a queue.
4. From Actions, choose Send and receive messages.
5. Enter the message attribute parameters.
a. In the name text box, enter a unique name of up to 256 characters.
b. For the attribute type, choose String, Number, or Binary.
c. (Optional) Enter a custom data type. For example, you could add byte, int, or float as custom data types for Number.
d. In the value text box, enter the message attribute value.
6. To add another message attribute., choose Add new attribute.
7. You can modify the attribute values any time before sending the message.
8. To delete an attribute, choose Remove. To delete the first attribute, close Message attributes.
9. When you finish adding attributes to the message, choose Send message. Your message is sent and the console displays a success message. To view information about the message attributes of the sent message, choose View details. Choose Done to close the Message details dialog box.
Receiving and deleting a message
Receiving and deleting messages (console)
After you send messages to a queue, you can receive and delete them. When you request messages from a queue, you can't specify which messages to retrieve. Instead, you specify the maximum number of messages (up to 10) that you want to retrieve.
NoteBecause Amazon SQS is a distributed system, a queue with very few messages might display an empty response to a receive request. In this case, rerun the request to get your message.
Depending on your application's needs, you might have to use short or long polling (p. 43) to receive messages.
Amazon SQS doesn't automatically delete a message after retrieving it for you, in case you don't successfully receive the message (for example, if the consumers fail or you lose connectivity). To delete a message, you must send a separate request which acknowledges that you've successfully received and processed the message. Note that you must receive a message before you can delete it.
NoteAfter receiving messages from the Amazon SQS console, the console immediately sets the messages back to visible, so that the messages can be received again.
To receive and delete a message (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. On the Queues page, choose a queue.
4. From Actions, choose Send and receive messages.
The console displays the Send and receive messages page.
5. Choose Poll for messages.
Amazon SQS begins to poll for messages in the queue. The progress bar on the right side of the Receive messages section displays the duration of polling.
The Messages section displays a list of the received messages. For each message, the list displays the message ID, Sent date, Size, and Receive count.
6. To delete messages, choose the messages that you want to delete and choose Delete.
7. In the Delete Messages dialog box, choose Delete.
Purging messages from an Amazon SQS queue (console)
If you don't want to delete an Amazon SQS queue but need to delete all of the messages from it, purge the queue. The message deletion process takes up to 60 seconds. We recommend waiting for 60 seconds regardless of your queue's size.
Deleting a queue
Important
When you purge a queue, you can't retrieve any of the deleted messages.
To purge a queue (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. On the Queues page, choose the queue to purge.
4. Choose Purge.
5. In the Purge queue dialog box, confirm the purge by entering purge and choosing Purge.
All messages are purged from the queue. The console displays a confirmation banner.
Deleting an Amazon SQS queue
If you no longer use an Amazon SQS queue and don't foresee using it in the near future, we recommend deleting it.
TipIf you want to verify that a queue is empty before you delete it, see Confirming that a queue is empty (p. 23).
You can delete a queue even when it isn't empty. To delete the messages in a queue but not the queue itself, purge the queue (p. 22).
To delete a queue (console)
1. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
2. In the navigation pane, choose Queues.
3. On the Queues page, choose the queue to delete.
4. Choose Delete.
5. In the Delete queue dialog box, confirm the deletion by entering delete.
6. Choose Delete.
To delete a queue (AWS CLIAWS API)
You can use one of the following commands to delete a queue:
• AWS CLI: aws sqs delete-queue
• AWS API: DeleteQueue
Confirming that a queue is empty
In most cases, you can use long polling (p. 44) to determine if a queue is empty. In rare cases, you might receive empty responses even when a queue still contains messages, especially if you specified a low value for Receive message wait time when you created the queue. This section describes how to confirm that a queue is empty.
To confirm that a queue is empty (console) 1. Stop all producers from sending messages.
Confirming that a queue is empty
2. Open the Amazon SQS console at https://console.aws.amazon.com/sqs/.
3. In the navigation pane, choose Queues.
4. On the Queues page, choose a queue.
5. Choose the Monitoring tab.
6. At the top right of the Monitoring dashboards, choose the down arrow next to the Refresh symbol.
From the dropdown menu, choose Auto refresh. Leave the Refresh interval at 1 Minute.
7. Observe the following dashboards:
• Approximate Number Of Messages Delayed
• Approximate Number Of Messages Not Visible
• Approximate Number Of Messages Visible
When all of them show 0 values for several minutes, the queue is empty.
To confirm that a queue is empty (AWS CLI, AWS API) 1. Stop all producers from sending messages.
2. Repeatedly run one of the following commands:
• AWS CLI: get-queue-attributes
• AWS API: GetQueueAttributes
3. Observe the metrics for the following attributes:
• ApproximateNumberOfMessagesDelayed
• ApproximateNumberOfMessagesNotVisible
• ApproximateNumberOfMessages
When all of them are 0 for several minutes, the queue is empty.
If you rely on Amazon CloudWatch metrics, make sure that you see multiple consecutive zero data points before considering that queue empty. For more information on CloudWatch metrics, see Available CloudWatch metrics for Amazon SQS (p. 153).
Basic architecture
How Amazon SQS works
This section describes the types of Amazon SQS queues and their basic properties. It also describes the identifiers of queues and messages, and various queue and message management workflows.
Topics
• Basic Amazon SQS architecture (p. 25)
• Amazon SQS Standard queues (p. 27)
• Amazon SQS FIFO (First-In-First-Out) queues (p. 28)
• Amazon SQS queue and message identifiers (p. 37)
• Message metadata (p. 39)
• Resources required to process Amazon SQS messages (p. 42)
• List queue pagination (p. 42)
• Amazon SQS cost allocation tags (p. 42)
• Amazon SQS short and long polling (p. 43)
• Amazon SQS dead-letter queues (p. 45)
• Amazon SQS visibility timeout (p. 49)
• Amazon SQS delay queues (p. 51)
• Amazon SQS temporary queues (p. 52)
• Amazon SQS message timers (p. 56)
Basic Amazon SQS architecture
This section outlines the parts of a distributed messaging system and explains the lifecycle of an Amazon SQS message.
Distributed queues
There are three main parts in a distributed messaging system: the components of your distributed system, your queue (distributed on Amazon SQS servers), and the messages in the queue.
In the following scenario, your system has several producers (components that send messages to the queue) and consumers (components that receive messages from the queue). The queue (which holds messages A through E) redundantly stores the messages across multiple Amazon SQS servers.
Message lifecycle
Message lifecycle
The following scenario describes the lifecycle of an Amazon SQS message in a queue, from creation to deletion.
Standard queues
A producer (component 1) sends message A to a queue, and the message is distributed across the Amazon SQS servers redundantly.
When a consumer (component 2) is ready to process messages, it consumes messages from the queue, and message A is returned. While message A is being processed, it remains in the queue and isn't returned to subsequent receive requests for the duration of the visibility timeout (p. 49).
The consumer (component 2) deletes message A from the queue to prevent the message from being received and processed again when the visibility timeout expires.
NoteAmazon SQS automatically deletes messages that have been in a queue for more than the maximum message retention period. The default message retention period is 4 days. However, you can set the message retention period to a value from 60 seconds to 1,209,600 seconds (14 days) using the SetQueueAttributes action.
Amazon SQS Standard queues
Amazon SQS offers standard as the default queue type. Standard queues support a nearly unlimited number of API calls per second, per API action (SendMessage, ReceiveMessage, or DeleteMessage).
Message ordering
Standard queues support at-least-once message delivery. However, occasionally (because of the highly distributed architecture that allows nearly unlimited throughput), more than one copy of a message might be delivered out of order. Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they're sent.
For information about how to create and configure queues using the Amazon SQS console, see Creating an Amazon SQS queue (console) (p. 11). For Java examples, see Amazon SQS Java SDK examples (p. 64).
You can use standard message queues in many scenarios, as long as your application can process messages that arrive more than once and out of order, for example:
• Decouple live user requests from intensive background work – Let users upload media while resizing or encoding it.
• Allocate tasks to multiple worker nodes – Process a high number of credit card validation requests.
• Batch messages for future processing – Schedule multiple entries to be added to a database.
For quotas related to standard queues, see Quotas related to queues (p. 102).
For best practices of working with standard queues, see Recommendations for Amazon SQS standard and FIFO queues (p. 57).
Message ordering
A standard queue makes a best effort to preserve the order of messages, but more than one copy of a message might be delivered out of order. If your system requires that order be preserved, we recommend using a FIFO (First-In-First-Out) queue (p. 28) or adding sequencing information in each message so you can reorder the messages when they're received.
At-least-once delivery
Amazon SQS stores copies of your messages on multiple servers for redundancy and high availability. On rare occasions, one of the servers that stores a copy of a message might be unavailable when you receive or delete a message.
If this occurs, the copy of the message isn't deleted on that unavailable server, and you might get that message copy again when you receive messages. Design your applications to be idempotent (they should not be affected adversely when processing the same message more than once).
Amazon SQS FIFO (First-In-First-Out) queues
FIFO queues have all the capabilities of the standard queue (p. 27).
FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated. Examples of situations where you might use FIFO queues include the following:
• To make sure that user-entered commands are run in the right order.
• To display the correct product price by sending price modifications in the right order.
• To prevent a student from enrolling in a course before registering for an account.
FIFO queues also provide exactly-once processing but have a limited number of transactions per second (TPS). For information on throughput quotas, see Quotas related to messages (p. 103).
Message ordering
NoteThe name of a FIFO queue must end with the .fifo suffix. The suffix counts towards the 80- character queue name quota. To determine whether a queue is FIFO (p. 28), you can check whether the queue name ends with the suffix.
Amazon SQS FIFO queues are available in all Regions where Amazon SQS is available.
For information about how to create and configure queues using the Amazon SQS console, see Creating an Amazon SQS queue (console) (p. 11). For Java examples, see Amazon SQS Java SDK examples (p. 64).
For best practices of working with FIFO queues, see Additional recommendations for Amazon SQS FIFO queues (p. 60) and Recommendations for Amazon SQS standard and FIFO queues (p. 57).
For information about compatibility of clients and services with FIFO queues, see Compatibility (p. 37).
Topics
• Message ordering (p. 29)
• Key terms (p. 29)
• FIFO delivery logic (p. 30)
• Exactly-once processing (p. 31)
• Moving from a standard queue to a FIFO queue (p. 31)
• High throughput for FIFO queues (p. 32)
• Compatibility (p. 37)
Message ordering
The FIFO queue improves upon and complements the standard queue (p. 27). The most
important features of this queue type are FIFO (First-In-First-Out) delivery (p. 30) and exactly-once processing (p. 31):
• The order in which messages are sent and received is strictly preserved and a message is delivered once and remains available until a consumer processes and deletes it.
• Duplicates aren't introduced into the queue.
In addition, FIFO queues support message groups that allow multiple ordered message groups within a single queue. There is no quota to the number of message groups within a FIFO queue.
Key terms
The following key terms can help you better understand the functionality of FIFO queues. For more information, see the Amazon Simple Queue Service API Reference.
Message deduplication ID
The token used for deduplication of sent messages. If a message with a particular message
deduplication ID is sent successfully, any messages sent with the same message deduplication ID are accepted successfully but aren't delivered during the 5-minute deduplication interval.
NoteMessage deduplication applies to an entire queue, not to individual message groups.
Amazon SQS continues to keep track of the message deduplication ID even after the message is received and deleted.