{
"Effect": "Allow", "Action": [
"kafka-cluster:Connect", "kafka-cluster:CreateTopic", "kafka-cluster:DescribeTopic", "kafka-cluster:WriteData"
],
"Resource": [
"arn:aws:kafka:*:Account-ID:cluster/msk-serverless-tutorial/*", "arn:aws:kafka:*:Account-ID:topic/msk-serverless-tutorial/*"
] } ] }
For instructions on how to write secure policies, see the section called “IAM Access Control” (p. 102).
5. Choose Next: Tags.
6. Choose Next: Review.
7. For the policy name, enter msk-serverless-tutorial.
8. Choose Create policy.
To create an IAM role and attach the policy to it 1. On the navigation pane, choose Roles.
2. Choose Create role.
3. Under Common use cases, choose EC2, then choose Next: Permissions.
4. In the search box, enter msk-serverless-tutorial, then select the box to the left of the policy.
5. Choose Next: Tags.
6. Choose Next: Review.
7. For the role name, enter msk-serverless-tutorial.
8. Choose Create role.
Next Step
Step 3: Create a Client Machine (p. 48)
Step 3: Create a Client Machine
In the step, you perform two tasks. The first task is to create an Amazon EC2 instance to use as an Apache Kafka client machine. The second task is to install Java and Apache Kafka tools on the machine.
To create a client machine
1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. Choose Launch instance.
3. Choose the first Select button.
4. Choose Next: Configure Instance Details.
5. For Network, choose the serverless cluster's virtual private cloud (VPC). This is the VPC based on the Amazon VPC service whose ID you saved after you created the cluster.
Step 4: Create and Write to Topic 8. Choose Next: Add Storage.
9. Choose Next: Add Tags.
10. Choose Add Tag, then under Key enter Name and under Value enter msk-serverless-tutorial.
11. Choose Next: Configure Security Group.
12. Choose the button labeled Select an existing security group.
13. In the list of security groups, select the security group associated with the cluster. This works if that security group has an inbound rule that allows traffic from the security group to itself. Such a rule allows members of the same security group to communicate with each other. For more information, see Security group rules in the Amazon VPC Developer Guide.
14. Choose Review and Launch and then choose Launch.
15. Choose an existing key pair, or create a new key pair.
16. Choose Launch Instances, and then choose View Instances.
17. Choose the check box in the row that represents this newly created Amazon EC2 instance. From this point forward, we call this instance the client machine.
18. Choose Connect and follow the instructions to connect to the client machine.
To set up Apache Kafka client tools on the client machine 1. To install Java, run the following command on the client machine:
sudo yum -y install java-11
2. To get the Apache Kafka tools that we need to create topics and send data, run the following commands:
wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz
tar -xzf kafka_2.12-2.8.1.tgz
3. To make it possible for the client machine to access the cluster, you need to download the Amazon MSK IAM JAR file. Go to the kafka_2.12-2.8.1/libs directory, then run the following command:
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
4. Go to the kafka_2.12-2.8.1/bin directory. Copy the following property settings and paste them into a new file. Name the file client.properties and save it.
security.protocol=SASL_SSL sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
Next Step
Step 4: Create and Write to an Apache Kafka Topic (p. 49)
Step 4: Create and Write to an Apache Kafka Topic
In this step, you use the previously created client machine to create a topic on the serverless cluster. You then write to that topic. The Apache Kafka command that we use to write to the topic is just an example.
You can try out other commands to write to the topic.
Step 5: Delete Resources To create a topic and write data to it
1. In the following export command, replace Endpoint with the bootstrap-server string you that you saved after you created the cluster. Then, go to the kafka_2.12-2.8.1/bin directory on the client machine, and run the export command.
export BS=Endpoint
2. Run the following command to create a topic on the serverless cluster.
./kafka-topics.sh bootstrap-server $BS command-config client.properties create --topic msk-serverless-tutorial --partitions 6
3. To write data to the newly created topic, run the following command.
./kafka-producer-perf-test.sh --topic msk-serverless-tutorial --throughput 2560 num-records 153600 record-size 4096 producer-props bootstrap.servers=$BS acks=all --producer.config client.properties
Next Step
Step 5: Delete Resources (p. 50)
Step 5: Delete Resources
In this step, you delete the resources that you created in this getting-started exercise.
To delete the cluster
1. Open the Amazon MSK console at https://console.aws.amazon.com/msk/home?region=us-east-1#/
home/.
2. In the list of clusters, choose msk-serverless-tutorial.
3. For Actions, choose Delete cluster.
4. Enter delete in the field, then choose Delete.
To stop the client machine
1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
2. In the list of Amazon EC2 instances, choose the client machine, which is the instance named msk-serverless-tutorial.
3. Choose Instance state, then choose Terminate instance.
4. Choose Terminate.
To delete the IAM policy and role
1. Open the IAM console at https://console.aws.amazon.com/iam/.
2. On the navigation pane, choose Roles.
3. In the search box, enter msk-serverless-tutorial.
4. Choose the msk-serverless-tutorial role.
Configuration 7. On the navigation pane, choose Policies.
8. In the search box, enter msk-serverless-tutorial followed by a return.
9. Choose the msk-serverless-tutorial policy.
10. On the policy's Summary page, choose Delete policy.
11. Choose Delete.
Configuration for Serverless Clusters
This is prerelease documentation for a service in preview release. It is subject to change.
Amazon MSK sets broker configuration properties for serverless clusters. You can't change these broker configuration property settings. However, you can set the following topic configuration properties.
Configuration Property Default Editable Maximum Allowed
Value
cleanup.policy Delete Yes, but only at topic
creation time
compression.type Producer Yes
max.message.bytes 1048588 Yes 8 MiB
message.timestamp.difference.max.mslong.max Yes
message.timestamp.type CreateTime Yes
retention.bytes 250 GiB Yes 250 GiB
retention.ms 1 day Yes 1 day
Monitoring for Serverless Clusters
This is prerelease documentation for a service in preview release. It is subject to change.
Amazon MSK integrates with Amazon CloudWatch so that you can collect, view, and analyze metrics for your MSK serverless cluster. The metrics shown in the following table are available for all serverless clusters. As these metrics are published as individual data points for each partition in the topic, we recommend viewing them as a 'SUM' statistic to get the topic-level view.
Metrics available at the DEFAULT monitoring level
Name When Visible Dimensions Description
BytesInPerSec After a producer
writes to a topic Cluster Name,
Topic The number of bytes per second received from clients. This metric is available for each broker and also for each topic.
Monitoring
Name When Visible Dimensions Description
BytesOutPerSec After a consumer group consumes from a topic
Cluster Name,
Topic The number of bytes per second sent to clients. This metric is available for each broker and also for each topic.
FetchMessageConversionsPerSecAfter a consumer group consumes from a topic
Cluster Name,
Topic The number of fetch message
conversions per second for the broker.
MaxEstimatedTimeLagAfter a consumer
A time estimate of the MaxOffsetLag metric.
The maximum offset lag across all partitions in a topic.
MessagesInPerSec After a producer
writes to a topic Cluster Name,
Topic The number of incoming messages per second for the broker.
ProduceMessageConversionsPerSecAfter a producer
writes to a topic Cluster Name,
Topic The number of produce message conversions per second for the broker.
SumOffsetLag After a consumer
The aggregated offset lag for all the partitions in a topic.
To view MSK Serverless metrics
1. Sign in to the AWS Management Console and open the CloudWatch console at https://
console.aws.amazon.com/cloudwatch/.
2. In the navigation pane, under Metrics, choose All metrics.
3. In the metrics search for the term kafka.
4. Choose AWS/Kafka / Cluster Name, Topic or AWS/Kafka / Cluster Name, Consumer Group, Topic to see different metrics.
What is MSK Connect?
MSK Connect
What is MSK Connect?
MSK Connect is a feature of Amazon MSK that makes it easy for developers to stream data to and from their Apache Kafka clusters. MSK Connect uses Kafka Connect 2.7.1, an open-source framework for connecting Apache Kafka clusters with external systems such as databases, search indexes, and file systems. With MSK Connect, you can deploy fully managed connectors built for Kafka Connect that move data into or pull data from popular data stores like Amazon S3 and Amazon OpenSearch Service. You can deploy connectors developed by 3rd parties like Debezium for streaming change logs from databases into an Apache Kafka cluster, or deploy an existing connector with no code changes.
Connectors automatically scale to adjust for changes in load and you pay only for the resources that you use.
Use source connectors to import data from external systems into your topics. With sink connectors, you can export data from your topics to external systems.
MSK Connect supports connectors for any Apache Kafka cluster with connectivity to an Amazon VPC, whether it is an MSK cluster or an independently hosted Apache Kafka cluster.
MSK Connect continuously monitors connector health and delivery state, patches and manages the underlying hardware, and autoscales the connectors to match changes in throughput.
To get started using MSK Connect, see the section called “Getting Started” (p. 53).
To learn about the AWS resources that you can create with MSK Connect, see the section called “Connectors” (p. 59), the section called “Plugins” (p. 62), and the section called
“Workers” (p. 62).
For information about the MSK Connect API, see the Amazon MSK API Reference.
Getting Started Using MSK Connect
This is a step-by-step tutorial that uses the AWS Management Console to create an MSK cluster and a sink connector that sends data from the cluster to an S3 bucket.
Topics
• Step 1: Set up required resources (p. 53)
• Step 2: Create custom plugin (p. 56)
• Step 3: Create client machine and Apache Kafka topic (p. 56)