Amazon Managed Streaming for Apache Kafka Developer Guide

(1)

Amazon Managed

Streaming for Apache Kafka

Developer Guide

(2)

Amazon Managed Streaming for Apache Kafka: Developer Guide

Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be aﬃliated with, connected to, or sponsored by Amazon.

(3)

What Is Amazon MSK?

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. Amazon MSK provides the control-plane operations, such as those for creating, updating, and deleting clusters. It lets you use Apache Kafka data-plane operations, such as those for producing and consuming data. It runs open-source versions of Apache Kafka. This means existing applications, tooling, and plugins from partners and the Apache Kafka community are supported without requiring changes to application code.

You can use Amazon MSK to create clusters that use any of the Apache Kafka versions listed under the section called “Supported Apache Kafka versions” (p. 153).

The following diagram provides an overview of how Amazon MSK works.

(9)

The diagram demonstrates the interaction between the following components:

• Broker nodes — When creating an Amazon MSK cluster, you specify how many broker nodes you want Amazon MSK to create in each Availability Zone. In the example cluster shown in this diagram, there's one broker per Availability Zone. Each Availability Zone has its own virtual private cloud (VPC) subnet.

• ZooKeeper nodes — Amazon MSK also creates the Apache ZooKeeper nodes for you. Apache ZooKeeper is an open-source server that enables highly reliable distributed coordination.

(10)

• Cluster Operations You can use the AWS Management Console, the AWS Command Line Interface (AWS CLI), or the APIs in the SDK to perform control-plane operations. For example, you can create or delete an Amazon MSK cluster, list all the clusters in an account, view the properties of a cluster, and update the number and type of brokers in a cluster.

Amazon MSK detects and automatically recovers from the most common failure scenarios for clusters so that your producer and consumer applications can continue their write and read operations with minimal impact. When Amazon MSK detects a broker failure, it mitigates the failure or replaces the unhealthy or unreachable broker with a new one. In addition, where possible, it reuses the storage from the older broker to reduce the data that Apache Kafka needs to replicate. Your availability impact is limited to the time required for Amazon MSK to complete the detection and recovery. After a recovery, your producer and consumer apps can continue to communicate with the same broker IP addresses that they used before the failure.

To get started using Amazon MSK, see Getting Started (p. 5).

To see the control-plane operations available through Amazon MSK, see the Amazon MSK API Reference.

After you create a cluster, you can use Amazon CloudWatch to monitor it. For more information about monitoring your cluster using metrics, see Monitoring a Cluster (p. 135).

(11)

Sign Up for AWS

Setting Up Amazon MSK

Before you use Amazon MSK for the ﬁrst time, complete the following tasks.

Tasks

• Sign Up for AWS (p. 4)

• Download Libraries and Tools (p. 4)

Sign Up for AWS

When you sign up for AWS, your Amazon Web Services account is automatically signed up for all services in AWS, including Amazon MSK. You are charged only for the services that you use.

If you have an AWS account already, skip to the next task. If you don't have an AWS account, use the following procedure to create one.

To sign up for an Amazon Web Services account 1. Open https://portal.aws.amazon.com/billing/signup.

2. Follow the online instructions.

Part of the sign-up procedure involves receiving a phone call and entering a veriﬁcation code on the phone keypad.

Download Libraries and Tools

The following libraries and tools can help you work with Amazon MSK:

• The AWS Command Line Interface (AWS CLI) supports Amazon MSK. The AWS CLI enables you to control multiple Amazon Web Services from the command line and automate them through scripts.

Upgrade your AWS CLI to the latest version to ensure that it has support for Amazon MSK. For detailed instructions on how to upgrade the AWS CLI, see Installing the AWS Command Line Interface. After you install the AWS CLI, you must configure it. For information on how to configure the AWS CLI, see aws configure.

• The Amazon Managed Streaming for Kafka API Reference documents the API operations that Amazon MSK supports.

• The Amazon Web Services SDKs for Go, Java, JavaScript, .NET, Node.js, PHP, Python, and Ruby include Amazon MSK support and samples.

(12)

Step 1: Create a Cluster

Getting Started Using Amazon MSK

This tutorial shows you an example of how you can create an MSK cluster, produce and consume data, and monitor the health of your cluster using metrics. This example doesn't represent all the options you can choose when you create an MSK cluster. In diﬀerent parts of this tutorial, we choose default options for simplicity. This doesn't mean that they're the only options that work for setting up an MSK cluster or client instances.

Topics

• Step 1: Create an Amazon MSK Cluster (p. 5)

• Step 2: Create a Client Machine (p. 5)

• Step 3: Create a Topic (p. 6)

• Step 4: Produce and Consume Data (p. 7)

• Step 5: Use Amazon CloudWatch to View Amazon MSK Metrics (p. 8)

• Step 6: Delete the AWS Resources Created for This Tutorial (p. 8)

Step 1: Create an Amazon MSK Cluster

In this step of Getting Started Using Amazon MSK (p. 5), you create an Amazon MSK cluster.

To create an Amazon MSK cluster using the AWS Management Console

1. Sign in to the AWS Management Console, and open the Amazon MSK console at https://

console.aws.amazon.com/msk/home?region=us-east-1#/home/.

2. Choose Create cluster.

3. For Cluster name, enter MSKTutorialCluster.

4. From the table under All cluster settings, copy the values of the following settings and save them because you need them later in this tutorial:

• VPC

• Subnets

• Security groups associated with VPC 5. Choose Create cluster.

Next Step

Step 2: Create a Client Machine (p. 5)

Step 2: Create a Client Machine

In this step of Getting Started Using Amazon MSK (p. 5), you create a client machine. You use this client machine to create a topic that produces and consumes data. For simplicity, we'll put this client machine in the same VPC as the MSK cluster. But a client machine doesn't have to be in the same VPC as the cluster.

To create a client machine

1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.

(13)

Step 3: Create a Topic 2. Choose Launch instances.

3. Choose Select to create an instance of Amazon Linux 2 AMI (HVM) - Kernel 5.10, SSD Volume Type.

4. Choose the t2.xlarge instance type by selecting the check box next to it.

5. Choose Next: Conﬁgure Instance Details.

6. In the Network list, choose the VPC whose ID you saved in the section called “Step 1: Create a Cluster” (p. 5).

7. In the Auto-assign Public IP list, choose Enable.

8. In the menu near the top, choose Add Tags.

9. Choose Add Tag.

10. Enter Name for the Key and MSKTutorialClient for the Value.

11. Choose Review and Launch, and then choose Launch.

12. Choose Create a new key pair, enter MSKKeyPair for Key pair name, and then choose Download Key Pair. Alternatively, you can use an existing key pair if you prefer.

13. Read the acknowledgement, select the check box next to it, and choose Launch Instances.

14. Choose View Instances. Then, in the Security Groups column, choose the security group that is associated with the MSKTutorialClient instance.

15. Copy the name of the security group, and save it for later.

16. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.

17. In the navigation pane, choose Security Groups. Find the security group whose ID you saved in the section called “Step 1: Create a Cluster” (p. 5). Choose this row by selecting the check box in the ﬁrst column.

18. In the Inbound Rules tab, choose Edit inbound rules.

19. Choose Add rule.

20. In the new rule, choose All traﬃc in the Type column. In the second ﬁeld in the Source column, select the security group of the client machine. This is the group whose name you saved earlier in this step.

21. Choose Save rules. Now the cluster's security group can accept traﬃc that comes from the client machine's security group.

Next Step

Step 3: Create a Topic (p. 6)

Step 3: Create a Topic

In this step of Getting Started Using Amazon MSK (p. 5), you install Apache Kafka client libraries and tools on the client machine, and then you create a topic.

To create a topic on the client machine

2. In the navigation pane, choose Instances, and then choose MSKTutorialClient by selecting the check box next to it.

3. Choose Actions, and then choose Connect. Follow the instructions to connect to the client machine MSKTutorialClient.

4. Install Java on the client machine by running the following command:

(14)

Step 4: Produce and Consume Data 5. Run the following command to download Apache Kafka.

wget https://archive.apache.org/dist/kafka/2.6.2/kafka_2.12-2.6.2.tgz

NoteIf you want to use a mirror site other than the one used in this command, you can choose a diﬀerent one on the Apache website.

6. Run the following command in the directory where you downloaded the TAR ﬁle in the previous step.

tar -xzf kafka_2.12-2.6.2.tgz 7. Go to the kafka_2.12-2.6.2 directory.

8. Open the Amazon MSK console at https://console.aws.amazon.com/msk/.

9. Wait for the status of MSKTutorialCluster to become Active. This might take several minutes. After the status becomes Active, choose the cluster name. This action takes you to a page where you can see the cluster summary.

10. Choose View client information.

11. Copy the private endpoint for plaintext authentication and the Apache ZooKeeper connection string (also for plaintext communication).

12. Run the following command, replacing ZookeeperConnectString with the string that you obtained in the previous instruction.

bin/kafka-topics.sh --create --zookeeper ZookeeperConnectString --replication-factor 3 --partitions 1 --topic MSKTutorialTopic

If the command succeeds, you see the following message: Created topic MSKTutorialTopic.

Next Step

Step 4: Produce and Consume Data (p. 7)

Step 4: Produce and Consume Data

In this step of Getting Started Using Amazon MSK (p. 5), you produce and consume data.

To produce and consume messages

1. Go to the bin folder of the Apache Kafka installation on the client machine, and create a text ﬁle named client.properties with the following contents.

security.protocol=PLAINTEXT

2. Run the following command in the bin folder, replacing BootstrapBrokerString with the private endpoint string that you obtained in the section called “Step 3: Create a Topic” (p. 6).

./kafka-console-producer.sh --broker-list BootstrapBrokerString --producer.config client.properties --topic MSKTutorialTopic

3. Enter any message that you want, and press Enter. Repeat this step two or three times. Every time you enter a line and press Enter, that line is sent to your Apache Kafka cluster as a separate message.

(15)

Step 5: View Metrics

4. Keep the connection to the client machine open, and then open a second, separate connection to that machine in a new window.

5. In the following command, replace BootstrapBrokerStringTls with the value that you saved earlier. Then, go to the bin folder and run the command using your second connection to the client machine.

./kafka-console-consumer.sh --bootstrap-server BootstrapBrokerStringTls -- consumer.config client.properties --topic MSKTutorialTopic --from-beginning

You start seeing the messages you entered earlier when you used the console producer command.

These messages are TLS encrypted in transit.

6. Enter more messages in the producer window, and watch them appear in the consumer window.

Next Step

Step 5: Use Amazon CloudWatch to View Amazon MSK Metrics (p. 8)

Step 5: Use Amazon CloudWatch to View Amazon MSK Metrics

In this step of Getting Started Using Amazon MSK (p. 5), you look at the Amazon MSK metrics in Amazon CloudWatch.

To view Amazon MSK metrics in CloudWatch

1. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.

2. In the navigation pane, choose Metrics.

3. Choose the All metrics tab, and then choose AWS/Kafka.

4. To view broker-level metrics, choose Broker ID, Cluster Name. For cluster-level metrics, choose Cluster Name.

5. (Optional) In the graph pane, select a statistic and a time period, and then create a CloudWatch alarm using these settings.

Next Step

Step 6: Delete the AWS Resources Created for This Tutorial (p. 8)

Step 6: Delete the AWS Resources Created for This Tutorial

In the ﬁnal step of Getting Started Using Amazon MSK (p. 5), you delete the MSK cluster and the client machine that you created for this tutorial.

To delete the resources using the AWS Management Console

(16)

Step 6: Delete the Resources

5. Choose MSKTutorialClient.

6. Choose Instance state, then choose Terminate instance.

(17)

Creating a Cluster

Amazon MSK: How It Works

An Amazon MSK cluster is the primary Amazon MSK resource that you can create in your account. The topics in this section describe how to perform common Amazon MSK operations. For a list of all the operations that you can perform on an MSK cluster, see the following:

• The AWS Management Console

• The Amazon MSK API Reference

• The Amazon MSK CLI Command Reference

Topics

• Creating an Amazon MSK Cluster (p. 10)

• Deleting an Amazon MSK Cluster (p. 13)

• Getting the Apache ZooKeeper connection string for an Amazon MSK Cluster (p. 14)

• Getting the bootstrap brokers for an Amazon MSK Cluster (p. 15)

• Listing Amazon MSK clusters (p. 16)

• Provisioning storage throughput (p. 17)

• Scaling up broker storage (p. 19)

• Updating the broker type (p. 22)

• Updating the conﬁguration of an Amazon MSK Cluster (p. 24)

• Expanding an Amazon MSK Cluster (p. 25)

• Updating a cluster's security settings (p. 27)

• Rebooting a broker for an Amazon MSK cluster (p. 29)

• Tagging an Amazon MSK Cluster (p. 31)

Creating an Amazon MSK Cluster

Before you can create an Amazon MSK cluster you need to have a VPC and set up subnets within that VPC. You need two subnets in two diﬀerent Availability Zones in the US West (N. California) Region. In all other Regions where Amazon MSK is available, you can specify either two or three subnets. Your subnets must all be in diﬀerent Availability Zones. When you create a cluster, Amazon MSK distributes the broker nodes evenly over the subnets that you specify.

Broker types

When you create an Amazon MSK cluster, you specify the type of brokers that you want it to have.

Amazon MSK supports the following broker types:

• kafka.t3.small

• kafka.m5.large, kafka.m5.xlarge, kafka.m5.2xlarge, kafka.m5.4xlarge, kafka.m5.8xlarge, kafka.m5.12xlarge, kafka.m5.16xlarge, kafka.m5.24xlarge

(18)

Creating a cluster using the AWS Management Console

brokers if you are running larger production-grade workloads or require a greater number of partitions.

To learn more about M5 instance types, see Amazon EC2 M5 Instances.

T3 brokers have the ability to use CPU credits to temporarily burst performance. Use T3 brokers for low-cost development, if you are testing small to medium streaming workloads, or if you have low- throughput streaming workloads that experience temporary spikes in throughput. We recommend that you run a proof-of-concept test to determine if T3 brokers are suﬃcient for production or critical workload. To learn more about T3 instance types, see Amazon EC2 T3 Instances.

For more information on how to choose broker types, see the section called “Right-size your cluster” (p. 165).

Creating a cluster using the AWS Management Console

3. Specify a name for the cluster.

4. In the VPC list, choose the VPC you want to use for the cluster. You can also specify which version of Apache Kafka you want Amazon MSK to use to create the cluster.

5. Specify two subnets if you're using one of the following Regions: South America (São Paulo), Canada (Central), and US West (N. California). In other Regions where Amazon MSK is available, you can specify either two or three subnets. The subnets that you specify must be in diﬀerent Availability Zones.

6. Choose the kind of configuration you want. For information about MSK configurations, see Configuration (p. 33).

7. Specify the type and number of brokers you want MSK to create in each Availability Zone. The minimum is one broker per Availability Zone and the maximum is 30 brokers per cluster.

8. (Optional) Assign tags to your cluster. Tags are optional. For more information, see the section called

“Tagging a Cluster” (p. 31).

9. You can adjust the storage volume per broker. After you create the cluster, you can increase the storage volume per broker but you can't decrease it.

10. Choose the settings you want for encrypting data in transit. By default, MSK encrypts data as it transits between brokers within a cluster. If you don't want to encrypt data as it transits between brokers, clear the check box labeled Enable encryption within the cluster.

11. Choose one of the three settings for encrypting data as it transits between clients and brokers. For more information, see the section called “Encryption in Transit” (p. 87).

12. Choose the kind of CMK that you want to use for encrypting data at rest. For more information, see the section called “Encryption at Rest” (p. 87).

13. If you want to authenticate the identity of clients, choose Enable TLS client authentication by selecting the box next to it. For more information about authentication, see the section called

“Mutual TLS Authentication” (p. 111).

14. Choose the monitoring level you want. This determines the set of metrics you get. For more information, see Monitoring a Cluster (p. 135).

15. (Optional) Choose Advanced settings, and then choose Customize settings. You can specify one or more security groups that you want to give access to your cluster (for example, the security groups of client machines). If you specify security groups that were shared with you, you must ensure that you have permissions to them. Speciﬁcally, you need the ec2:DescribeSecurityGroups permission. For an example, see Amazon EC2: Allows Managing EC2 Security Groups Associated With a Speciﬁc VPC, Programmatically and in the Console.

(19)

Creating a cluster using the AWS CLI

1. Copy the following JSON and save it to a file. Name the file brokernodegroupinfo.json. Replace the subnet IDs in the JSON with the values that correspond to your subnets. These subnets must be in different Availability Zones. Replace "Security-Group-ID" with the ID of one or more security groups of the client VPC. Clients associated with these security groups get access to the cluster. If you specify security groups that were shared with you, you must ensure that you have permissions to them. Specifically, you need the ec2:DescribeSecurityGroups permission. For an example, see Amazon EC2: Allows Managing EC2 Security Groups Associated With a Specific VPC, Programmatically and in the Console. Finally, save the updated JSON file on the computer where you have the AWS CLI installed.

{ "InstanceType": "kafka.m5.large", "ClientSubnets": [

"Subnet-1-ID", "Subnet-2-ID"

], "SecurityGroups": [ "Security-Group-ID"

]}

Important

Specify exactly two subnets if you are using one of the following Regions: South America (São Paulo), Canada (Central), and US West (N. California). For other Regions where Amazon MSK is available, you can specify either two or three subnets. The subnets that you specify must be in distinct Availability Zones. When you create a cluster, Amazon MSK distributes the broker nodes evenly across the subnets that you specify.

2. Run the following AWS CLI command in the directory where you saved the

brokernodegroupinfo.json ﬁle, replacing "Your-Cluster-Name" with a name of your choice. For "Monitoring-Level", you can specify one of the following three values: DEFAULT, PER_BROKER, or PER_TOPIC_PER_BROKER. For information about these three diﬀerent levels of monitoring, see ??? (p. 135). The enhanced-monitoring parameter is optional. If you don't specify it in the create-cluster command, you get the DEFAULT level of monitoring.

aws kafka create-cluster --cluster-name "Your-Cluster-Name" --broker-node-group-info fileb://brokernodegroupinfo.json --kafka-version "2.2.1" --number-of-broker-nodes 3 -- enhanced-monitoring "Monitoring-Level"

The output of the command looks like the following JSON:

{

"ClusterArn": "...",

"ClusterName": "AWSKafkaTutorialCluster", "State": "CREATING"

}

NoteThe create-cluster command might return an error stating that one or more subnets belong to unsupported Availability Zones. When this happens, the error indicates which Availability Zones are unsupported. Create subnets that don't use the unsupported

(20)

Creating a cluster with a custom MSK conﬁguration using the AWS CLI

For information about custom MSK conﬁgurations and how to create them, see Conﬁguration (p. 33).

1. Save the following JSON to a ﬁle, replacing configuration-arn with the ARN of the conﬁguration that you want to use to create the cluster.

{ "Arn": configuration-arn, "Revision": 1

}

2. Run the create-cluster command and use the configuration-info option to point to the JSON ﬁle you saved in the previous step. The following is an example.

aws kafka create-cluster --cluster-name ExampleClusterName --broker-node-group-info fileb://brokernodegroupinfo.json --kafka-version "1.1.1" --number-of-broker-nodes 3 -- enhanced-monitoring PER_TOPIC_PER_BROKER --configuration-info file://configuration.json

The following is an example of a successful response after running this command.

{

"ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/

CustomConfigExampleCluster/abcd1234-abcd-dcba-4321-a1b2abcd9f9f-2", "ClusterName": "CustomConfigExampleCluster",

"State": "CREATING"

}

Creating a cluster using the API

To create a cluster using the API, see CreateCluster.

Deleting an Amazon MSK Cluster

Deleting a cluster using the AWS Management Console

2. Choose the MSK cluster that you want to delete by selecting the check box next to it.

3. Choose Delete, and then conﬁrm deletion.

Deleting a cluster using the AWS CLI

Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can ﬁnd it by listing all clusters. For more information, see the section called “Listing Clusters” (p. 16).

(21)

Deleting a cluster using the API

aws kafka delete-cluster --cluster-arn ClusterArn

Deleting a cluster using the API

To delete a cluster using the API, see DeleteCluster.

Getting the Apache ZooKeeper connection string for an Amazon MSK Cluster

Getting the Apache ZooKeeper connection string using the AWS Management Console

2. The table shows all the clusters for the current region under this account. Choose the name of a cluster to view its description.

3. On the Cluster summary page, choose View client information. This shows you the bootstrap brokers, as well as the Apache ZooKeeper connection string.

Getting the Apache ZooKeeper connection string using the AWS CLI

1. If you don't know the Amazon Resource Name (ARN) of your cluster, you can ﬁnd it by listing all the clusters in your account. For more information, see the section called “Listing Clusters” (p. 16).

2. To get the Apache ZooKeeper connection string, along with other information about your cluster, run the following command, replacing ClusterArn with the ARN of your cluster.

aws kafka describe-cluster --cluster-arn ClusterArn

The output of this describe-cluster command looks like the following JSON example.

{ "ClusterInfo": {

"BrokerNodeGroupInfo": {

"BrokerAZDistribution": "DEFAULT", "ClientSubnets": [

"subnet-0123456789abcdef0", "subnet-2468013579abcdef1", "subnet-1357902468abcdef2"

],

"InstanceType": "kafka.m5.large", "StorageInfo": {

"EbsStorageInfo": { "VolumeSize": 1000 }

}

(22)

Getting the Apache ZooKeeper connection string using the API "ClusterName": "testcluster",

"CreationTime": "2018-12-02T17:38:36.75Z", "CurrentBrokerSoftwareInfo": {

"KafkaVersion": "2.2.1"

},

"CurrentVersion": "K13V1IB3VIYZZH", "EncryptionInfo": {

"EncryptionAtRest": {

"DataVolumeKMSKeyId": "arn:aws:kms:us-east-1:555555555555:key/12345678- abcd-2345-ef01-abcdef123456"

} },

"EnhancedMonitoring": "DEFAULT", "NumberOfBrokerNodes": 3,

"State": "ACTIVE",

"ZookeeperConnectString": "10.0.1.101:2018,10.0.2.101:2018,10.0.3.101:2018"

} }

The previous JSON example shows the ZookeeperConnectString key in the output of the describe-cluster command. Copy the value corresponding to this key and save it for when you need to create a topic on your cluster.

Important

Your Amazon MSK cluster must be in the ACTIVE state for you to be able to obtain the Apache ZooKeeper connection string. When a cluster is still in the CREATING state, the output of the describe-cluster command doesn't include ZookeeperConnectString.

If this is the case, wait a few minutes and then run the describe-cluster again after your cluster reaches the ACTIVE state.

Getting the Apache ZooKeeper connection string using the API

To get the Apache ZooKeeper connection string using the API, see DescribeCluster.

Getting the bootstrap brokers for an Amazon MSK Cluster

Getting the bootstrap brokers using the AWS Management Console

The term bootstrap brokers refers to a list of brokers that an Apache Kafka client can use as a starting point to connect to the cluster. This list doesn't necessarily include all of the brokers in a cluster.

2. The table shows all the clusters for the current region under this account. Choose the name of a cluster to view its description.

3. On the Cluster summary page, choose View client information. This shows you the bootstrap brokers, as well as the Apache ZooKeeper connection string.

(23)

Getting the bootstrap brokers using the AWS CLI

aws kafka get-bootstrap-brokers --cluster-arn ClusterArn

For an MSK cluster that uses the section called “IAM Access Control” (p. 102), the output of this command looks like the following JSON example.

{

"BootstrapBrokerStringSaslIam": "b-1.myTestCluster.123z8u.c2.kafka.us-

west-1.amazonaws.com:9098,b-2.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9098"

}

The following example shows the bootstrap brokers for a cluster that has public access turned on. Use the BootstrapBrokerStringPublicSaslIam for public access, and the BootstrapBrokerStringSaslIam string for access from within AWS.

{

"BootstrapBrokerStringPublicSaslIam": "b-2-public.myTestCluster.v4ni96.c2.kafka- beta.us-east-1.amazonaws.com:9198,b-1-public.myTestCluster.v4ni96.c2.kafka-

beta.us-east-1.amazonaws.com:9198,b-3-public.myTestCluster.v4ni96.c2.kafka-beta.us- east-1.amazonaws.com:9198",

"BootstrapBrokerStringSaslIam": "b-2.myTestCluster.v4ni96.c2.kafka- beta.us-east-1.amazonaws.com:9098,b-1.myTestCluster.v4ni96.c2.kafka-

beta.us-east-1.amazonaws.com:9098,b-3.myTestCluster.v4ni96.c2.kafka-beta.us- east-1.amazonaws.com:9098"

}

The number of bootstrap brokers that you get is equal to the number of zones in which your MSK cluster is deployed.

Getting the bootstrap brokers using the API

To get the bootstrap brokers using the API, see GetBootstrapBrokers.

Listing Amazon MSK clusters

Listing clusters using the AWS Management Console

2. The table shows all the clusters for the current region under this account. Choose the name of a cluster to view its details.

Listing clusters using the AWS CLI

Run the following command.

(24)

Listing clusters using the API

To list clusters using the API, see ListClusters.

Provisioning storage throughput

Amazon MSK brokers persist data on storage volumes. Storage I/O is consumed when producers write to the cluster, when data is replicated between brokers, and when consumers read data that isn't in memory. The volume storage throughput is the rate at which data can be written into and read from a storage volume. Provisioned storage throughput is the ability to specify that rate for the brokers in your cluster.

You can specify the provisioned throughput rate in MiB per second for clusters whose brokers are of type kafka.m5.4xlarge or larger and if the storage volume is 10 GiB or greater. It is possible to specify provisioned throughput during cluster creation. You can also enable or disable provisioned throughput for a cluster that is in the ACTIVE state.

Throughput bottlenecks

There are multiple causes of bottlenecks in broker throughput: volume throughput, EC2-EBS network throughput, and EC2 egress throughput. You can enable provisioned storage throughput to adjust volume throughput. However, broker throughput limitations can be caused by EC2-EBS network throughput and EC2 egress throughput.

EC2 egress throughput is impacted by the number of consumer groups and consumers per consumer groups. Also, both EC2-EBS network throughput and EC2 egress throughput are higher for larger broker types, as shown in the following table.

Broker type EC2-EBS network throughput (MBps)

kafka.m5.4xlarge 593.75

kafka.m5.8xlarge 850

kafka.m5.12xlarge 1187.5

Measuring storage throughput

You can use the VolumeReadBytes and VolumeWriteBytes metrics to measure the average storage throughput of a cluster. The sum of these two metrics gives the average storage throughput in bytes.

To get the average storage throughput for a cluster, set these two metrics to SUM and the period to 1 minute, then use the following formula.

Average storage throughput in MiB/s = (Sum(VolumeReadBytes) + Sum(VolumeWriteBytes)) / (60 * 1024)

For information about the VolumeReadBytes and VolumeWriteBytes metrics, see the section called

“PER_BROKER Level Monitoring” (p. 139).

(25)

Conﬁguration update

When you turn on provisioned throughput for a cluster, you might not see the desired throughput unless you also update the value of the num.replica.fetchers configuration parameter. In the default Amazon MSK configuration, that parameter has a value of 2. To get the benefit of provisioned throughput, update your cluster's configuration by setting num.replica.fetchers to the value that matches the cluster's broker type in the following table. The values in the table are general guidance. We recommend that you adjust these values based on your use case. You can update the configuration either before or after you turn on provisioned throughput. However, you won't see the desired throughput until you perform both actions: update the configuration and turn on provisioned throughput.

Broker type num.replica.fetchers

kafka.m5.4xlarge 4

kafka.m5.8xlarge 8

Provisioning storage throughput using the AWS Management Console

3. Choose Custom create.

4. Specify a name for the cluster.

5. In the Storage section, choose Enable.

6. Choose a value for storage throughput per broker.

7. Choose a VPC, zones and subnets, and a security group.

8. Choose Next.

9. At the bottom of the Security step, choose Next.

10. At the bottom of the Monitoring and tags step, choose Next.

11. Review the cluster settings, then choose Create cluster.

Provisioning storage throughput using the AWS CLI

This section shows an example of how you can use the AWS CLI to create a cluster with provisioned throughput enabled.

1. Copy the following JSON and paste it into a ﬁle. Replace the subnet IDs and security group ID placeholders with values from your account. Name the ﬁle cluster-creation.json and save it.

(26)

Provisioning storage throughput using the API "BrokerNodeGroupInfo":{

"InstanceType":"kafka.m5.4xlarge", "ClientSubnets":[

"Subnet-1-ID", "Subnet-2-ID"

],

"SecurityGroups":[

"Security-Group-ID"

],

"StorageInfo": {

"EbsStorageInfo": { "VolumeSize": 10,

"ProvisionedThroughput": { "Enabled": true, "VolumeThroughput": 250 }

} } },

"EncryptionInfo": {

"EncryptionInTransit": { "InCluster": false,

"ClientBroker": "PLAINTEXT"

} },

"KafkaVersion":"2.2.1", "NumberOfBrokerNodes": 2 },

"ClusterName": "provisioned-throughput-example"

}

2. Run the following AWS CLI command from the directory where you saved the JSON ﬁle in the previous step.

aws kafka create-cluster-v2 --cli-input-json fileb://cluster-creation.json

Provisioning storage throughput using the API

To conﬁgure provisioned storage throughput while creating a cluster, use CreateClusterV2.

Scaling up broker storage

You can increase the amount of EBS storage per broker. You can't decrease the storage.

Storage volumes remain available during this scaling-up operation.

Important

When storage is scaled for an MSK cluster, the additional storage is made available right away.

However, the cluster requires a cool-down period after every storage scaling event. Amazon MSK uses this cool-down period to optimize the cluster before it can be scaled again. This period can range from a minimum of 6 hours to over 24 hours, depending on the cluster’s storage size and utilization and on traﬃc. This is applicable for both auto scaling events and manual scaling using the UpdateBrokerStorage operation. For information about right-sizing your storage, see Right-size your cluster.

Topics

• Automatic scaling (p. 20)

(27)

Automatic scaling

• Manual scaling (p. 21)

Automatic scaling

You can conﬁgure Amazon Managed Streaming for Apache Kafka to automatically expand your cluster's storage in response to increased usage using Application Auto-Scaling policies. Your auto-scaling policy sets the target disk utilization and the maximum scaling capacity.

Note

Amazon MSK does not reduce cluster storage in response to reduced usage. Amazon MSK does not support decreasing the size of storage volumes. If you need to reduce the size of your cluster storage, you must migrate your existing cluster to a cluster with smaller storage. For information about migrating a cluster, see Migration (p. 132).

NoteAmazon MSK does not support auto scaling in the Asia Paciﬁc (Osaka) and Africa (Cape Town) Regions.

MSK Storage auto-expansion details

Your auto scaling policy deﬁnes the following parameters for your cluster:

• Storage Utilization Target: The storage utilization threshold that Amazon MSK uses to trigger an auto-scaling operation. You can set the utilization target between 10% and 80% of the current storage capacity. We recommend that you set the Storage Utilization Target between 50% and 60%.

• Maximum Storage Capacity: This setting is the maximum scaling limit that Amazon MSK can set your broker storage. You can set the maximum storage capacity up to 16 TiB per broker. For more information, see Amazon MSK Quota (p. 150).

When the service detects that your Maximum Disk Utilization metric is equal to or greater than the Storage Utilization Target setting, it will increase your storage capacity automatically. Amazon MSK ﬁrst expands your cluster storage by an amount equal to the larger of two numbers: 10 GiB and 10% of current storage. For example, if you have 1000 GiB, that amount is 100 GiB. Further scaling operations increase storage by a greater amount. The service checks your storage utilization every minute.

You can check to determine if auto scaling operations occurred using the ListClusterOperations operation.

Setting up auto-expansion for your Amazon MSK cluster

You can use the Amazon MSK console or the Amazon MSK API to implement auto-expanding storage.

NoteYou can't implement auto-expansion when you create a cluster. You must ﬁrst create the cluster, and then create and enable an auto-expansion policy for it. However, you can create the policy while the Amazon MSK service creates your cluster.

Setting up auto-expansion using the AWS Management Console

(28)

Manual scaling

4. Create and name an auto-scaling policy. Specify the storage utilization target, the maximum storage capacity, and the target metric.

5. Choose Save changes.

When you save and enable the new policy, the policy becomes active for the cluster. Amazon MSK then expands the cluster's storage when the storage utilization target is reached.

Setting up auto-expansion using the CLI

1. Use the RegisterScalableTarget command to register a storage utilization target.

2. Use the PutScalingPolicy command to create an auto-expansion policy.

Setting up auto-expansion using the API

1. Use the RegisterScalableTarget API to register a storage utilization target.

2. Use the PutScalingPolicy API to create an auto-expansion policy.

Manual scaling

To increase storage, wait for the cluster to be in the ACTIVE state. Storage scaling has a cool-down period of at least six hours between events. Even though the operation makes additional storage available right away, the service performs optimizations on your cluster that can take up to 24 hours or more. The duration of these optimizations is proportional to your storage size.

Scaling up broker storage using the AWS Management Console

2. Choose the MSK cluster for which you want to update broker storage.

3. In the Storage section, choose Edit.

4. Specify the storage volume you want. You can only increase the amount of storage, you can't decrease it.

5. Choose Save changes.

Scaling up broker storage using the AWS CLI

Replace Current-Cluster-Version with the current version of the cluster.

Important

Cluster versions aren't simple integers. To ﬁnd the current version of the cluster, use the DescribeCluster operation or the describe-cluster AWS CLI command. An example version is KTVPDKIKX0DER.

The Target-Volume-in-GiB parameter represents the amount of storage that you want each broker to have. It is only possible to update the storage for all the brokers. You can't specify individual brokers for which to update storage. The value you specify for Target-Volume-in-GiB must be a whole number that is greater than 100 GiB. The storage per broker after the update operation can't exceed 16384 GiB.

(29)

Updating the Broker Type

aws kafka update-broker-storage --cluster-arn ClusterArn --current-version Current- Cluster-Version --target-broker-ebs-volume-info '{"KafkaBrokerNodeId": "All", "VolumeSizeGB": Target-Volume-in-GiB}'

Scaling up broker storage using the API

To update a broker storage using the API, see UpdateBrokerStorage.

Updating the broker type

You can scale your MSK cluster on demand by changing the type (the size or family) of your brokers without reassigning Apache Kafka partitions. Changing the type of your brokers gives you the ﬂexibility to adjust your MSK cluster’s compute capacity based on changes in your workloads, without interrupting your cluster I/O. Amazon MSK uses the same broker type for all the brokers in a given cluster. This section describes how to update the broker type for your MSK cluster. The broker-type update happens in a rolling fashion while the cluster is up and running. This means that Amazon MSK takes down one broker at a time to perform the broker-type update. For information about how to make a cluster highly available during a broker-type update, see the section called “Build highly available clusters” (p. 166).

To further reduce any potential impact on productivity, you can perform the broker-type update during a period of low traﬃc.

During a broker-type update, you can continue to produce and consume data. However, you must wait until the update is done before you can reboot brokers or invoke any of the update operations listed under Amazon MSK operations.

If you want to update your cluster to a smaller broker type, we recommend that you try the update on a test cluster ﬁrst to see how it aﬀects your scenario.

Important

You can't update a cluster to a smaller broker type if the number of partitions per broker exceeds the maximum number speciﬁed in the section called “ Number of partitions per broker” (p. 165).

Updating the broker type using the AWS Management Console

2. Choose the MSK cluster for which you want to update the broker type.

3. On the details page for the cluster, ﬁnd the Brokers summary section, and choose Edit broker type.

4. Choose the broker type you want from the list.

5. Save changes.

Updating the broker type using the AWS CLI

1. Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can ﬁnd it by listing all clusters. For more information, see the section called “Listing Clusters” (p. 16).

Replace with the current version of the cluster and with

(30)

Updating the broker type using the API

aws kafka update-broker-type --cluster-arn ClusterArn --current-version Current- Cluster-Version --target-instance-type TargetType

The following is an example of how to use this command:

aws kafka update-broker-type --cluster-arn "arn:aws:kafka:us-

east-1:0123456789012:cluster/exampleName/abcd1234-0123-abcd-5678-1234abcd-1" --current- version "K1X5R6FKA87" --target-instance-type kafka.m5.large

The output of this command looks like the following JSON example.

{ "ClusterArn": "arn:aws:kafka:us-east-1:0123456789012:cluster/exampleName/

abcd1234-0123-abcd-5678-1234abcd-1",

"ClusterOperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster- operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd- abcd-4f7f-1234-9876543210ef"

}

2. To get the result of the update-broker-type operation, run the following command, replacing ClusterOperationArn with the ARN that you obtained in the output of the update-broker- type command.

aws kafka describe-cluster-operation --cluster-operation-arn ClusterOperationArn

The output of this describe-cluster-operation command looks like the following JSON example.

{

"ClusterOperationInfo": {

"ClientRequestId": "982168a3-939f-11e9-8a62-538df00285db",

"ClusterArn": "arn:aws:kafka:us-east-1:0123456789012:cluster/exampleName/

abcd1234-0123-abcd-5678-1234abcd-1",

"CreationTime": "2021-01-09T02:24:22.198000+00:00",

"OperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster-operation/

exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd- abcd-4f7f-1234-9876543210ef",

"OperationState": "UPDATE_COMPLETE", "OperationType": "UPDATE_BROKER_TYPE", "SourceClusterInfo": {

"InstanceType": "t3.small"

},

"TargetClusterInfo": { "InstanceType": "m5.large"

} }}

If OperationState has the value UPDATE_IN_PROGRESS, wait a while, then run the describe- cluster-operation command again.

Updating the broker type using the API

To update the broker type using the API, see UpdateBrokerType.

(31)

Updating the Conﬁguration of a Cluster

Updating the conﬁguration of an Amazon MSK Cluster

To update the conﬁguration of a cluster, make sure that the cluster is in the ACTIVE state. You must also ensure that the number of partitions per broker on your MSK cluster is under the limits described in the section called “ Number of partitions per broker” (p. 165). You can't update the conﬁguration of a cluster that exceeds these limits.

For information about MSK configuration, including how to create a custom configuration, which properties you can update, and what happens when you update the configuration of an existing cluster, see Configuration (p. 33).

Updating the conﬁguration of a cluster using the AWS CLI

1. Copy the following JSON and save it to a file. Name the file configuration-info.json. Replace ConfigurationArn with the Amazon Resource Name (ARN) of the configuration that you want to use to update the cluster. The ARN string must be in quotes in the following JSON.

Replace Configuration-Revision with the revision of the conﬁguration that you want to use.

Conﬁguration revisions are integers (whole numbers) that start at 1. This integer mustn't be in quotes in the following JSON.

{ "Arn": ConfigurationArn,

"Revision": Configuration-Revision }

2. Run the following command, replacing ClusterArn with the ARN that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can ﬁnd it by listing all clusters.

For more information, see the section called “Listing Clusters” (p. 16).

Replace Path-to-Config-Info-File with the path to your configuration info file. If you named the file that you created in the previous step configuration-info.json and saved it in the current directory, then Path-to-Config-Info-File is configuration-info.json.

Important

aws kafka update-cluster-configuration --cluster-arn ClusterArn --configuration-info fileb://Path-to-Config-Info-File --current-version Current-Cluster-Version

The following is an example of how to use this command:

aws kafka update-cluster-configuration --cluster-arn "arn:aws:kafka:us-

(32)

Updating the conﬁguration of a cluster using the API

The output of this update-cluster-configuration command looks like the following JSON example.

{

"ClusterArn": "arn:aws:kafka:us-east-1:012345678012:cluster/exampleClusterName/

abcdefab-1234-abcd-5678-cdef0123ab01-2",

}

3. To get the result of the update-cluster-configuration operation, run the following command, replacing ClusterOperationArn with the ARN that you obtained in the output of the update- cluster-configuration command.

aws kafka describe-cluster-operation --cluster-operation-arn ClusterOperationArn

{ "ClusterOperationInfo": {

"ClientRequestId": "982168a3-939f-11e9-8a62-538df00285db",

"CreationTime": "2019-06-20T21:08:57.735Z",

"OperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster- operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd- abcd-4f7f-1234-9876543210ef",

"OperationState": "UPDATE_COMPLETE",

"OperationType": "UPDATE_CLUSTER_CONFIGURATION", "SourceClusterInfo": {},

"TargetClusterInfo": { "ConfigurationInfo": {

"Arn": "arn:aws:kafka:us-east-1:123456789012:configuration/

ExampleConfigurationName/abcdabcd-abcd-1234-abcd-abcd123e8e8e-1", "Revision": 1

} } } }

In this output, OperationType is UPDATE_CLUSTER_CONFIGURATION. If OperationState has the value UPDATE_IN_PROGRESS, wait a while, then run the describe-cluster-operation command again.

Updating the conﬁguration of a cluster using the API

To use the API to update the conﬁguration of a cluster, see UpdateClusterConﬁguration.

Expanding an Amazon MSK Cluster

Use this Amazon MSK operation when you want to increase the number of brokers in your MSK cluster.

To expand a cluster, make sure that it is in the ACTIVE state.

(33)

Expanding a cluster using the AWS Management Console Important

If you want to expand an MSK cluster, make sure to use this Amazon MSK operation . Don't try to add brokers to a cluster without using this operation.

For information about how to rebalance partitions after you add brokers to a cluster, see the section called “Reassign partitions” (p. 168).

Expanding a cluster using the AWS Management Console

2. Choose the MSK cluster whose number of brokers you want to increase.

3. On the cluster details page, choose the Edit button next to the Cluster-Level Broker Details heading.

4. Enter the number of brokers that you want the cluster to have per Availability Zone and then choose Save changes.

Expanding a cluster using the AWS CLI

1. Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained when you created your cluster. If you don't have the ARN for your cluster, you can ﬁnd it by listing all clusters. For more information, see the section called “Listing Clusters” (p. 16).

Important

The Target-Number-of-Brokers parameter represents the total number of broker nodes that you want the cluster to have when this operation completes successfully. The value you specify for Target-Number-of-Brokers must be a whole number that is greater than the current number of brokers in the cluster. It must also be a multiple of the number of Availability Zones.

aws kafka update-broker-count --cluster-arn ClusterArn --current-version Current- Cluster-Version --target-number-of-broker-nodes Target-Number-of-Brokers

The output of this update-broker-count operation looks like the following JSON.

{

}

2. To get the result of the update-broker-count operation, run the following command, replacing ClusterOperationArn with the ARN that you obtained in the output of the update-broker- count command.

(34)

Expanding a cluster using the API

{

"ClusterOperationInfo": {

"ClientRequestId": "c0b7af47-8591-45b5-9c0c-909a1a2c99ea",

"CreationTime": "2019-09-25T23:48:04.794Z",

"OperationArn": "arn:aws:kafka:us-east-1:012345678012:cluster- operation/exampleClusterName/abcdefab-1234-abcd-5678-cdef0123ab01-2/0123abcd- abcd-4f7f-1234-9876543210ef",

"OperationState": "UPDATE_COMPLETE", "OperationType": "INCREASE_BROKER_COUNT", "SourceClusterInfo": {

"NumberOfBrokerNodes": 9 },

"TargetClusterInfo": { "NumberOfBrokerNodes": 12 }

} }

In this output, OperationType is INCREASE_BROKER_COUNT. If OperationState has the value UPDATE_IN_PROGRESS, wait a while, then run the describe-cluster-operation command again.

Expanding a cluster using the API

To increase the number of brokers in a cluster using the API, see UpdateBrokerCount.

Updating a cluster's security settings

Use this Amazon MSK operation to update the authentication and client-broker encryption settings of your MSK cluster. You can also update the Private Security Authority used to sign certiﬁcates for mutual TLS authentication. You can't change the in-cluster (broker-to-broker) encryption setting.

The cluster must be in the ACTIVE state for you to update its security settings.

If you turn on authentication using IAM, SASL, or TLS, you must also turn on encryption between clients and brokers. The following table shows the possible combinations.

Authentication Client-Broker Encryption

Options Broker-Broker Encryption

Unauthenticated TLS, PLAINTEXT,

TLS_PLAINTEXT Can be on or oﬀ.

mTLS TLS, TLS_PLAINTEXT Must be on.

SASL/SCRAM TLS Must be on.

SASL/IAM TLS Must be on.

(35)

Updating a cluster's security settings using the AWS Management Console

When client-broker encryption is set to TLS_PLAINTEXT and client-authentication is set to mTLS, Amazon MSK creates two types of listeners for clients to connect to: one listener for clients to connect using mTLS authentication with TLS Encryption, and another for clients to connect without authentication or encryption (plaintext).

For more information about security settings, see Security (p. 86).

Updating a cluster's security settings using the AWS Management Console

2. Choose the MSK cluster that you want to update.

3. In the Security settings section, choose Edit.

4. Choose the authentication and encryption settings that you want for the cluster, then choose Save changes.

Updating a cluster's security settings using the AWS CLI

1. Create a JSON ﬁle that contains the encryption settings that you want the cluster to have. The following is an example.

NoteYou can only update the client-broker encryption setting. You can't update the in-cluster (broker-to-broker) encryption setting.

{"EncryptionInTransit":{"ClientBroker": "TLS"}}

2. Create a JSON ﬁle that contains the authentication settings that you want the cluster to have. The following is an example.

{"Sasl":{"Scram":{"Enabled":true}}}

3. Run the following AWS CLI command:

aws kafka update-security --cluster-arn ClusterArn --current-version Current-Cluster- Version --client-authentication file://Path-to-Authentication-Settings-JSON-File -- encryption-info file://Path-to-Encryption-Settings-JSON-File

The output of this update-security operation looks like the following JSON.

{

}

Amazon Managed Streaming for Apache Kafka Developer Guide

Amazon Managed

Streaming for Apache Kafka

Developer Guide

Amazon Managed Streaming for Apache Kafka: Developer Guide

Table of Contents

What Is Amazon MSK?

Setting Up Amazon MSK

Sign Up for AWS

Download Libraries and Tools

Getting Started Using Amazon MSK

Step 1: Create an Amazon MSK Cluster

Step 2: Create a Client Machine

Step 3: Create a Topic

Step 4: Produce and Consume Data

Step 5: Use Amazon CloudWatch to View Amazon MSK Metrics

Step 6: Delete the AWS Resources Created for This Tutorial

Amazon MSK: How It Works

Creating an Amazon MSK Cluster

Broker types

Creating a cluster using the AWS Management Console

Creating a cluster using the AWS CLI

Creating a cluster with a custom MSK conﬁguration using the AWS CLI

Creating a cluster using the API

Deleting an Amazon MSK Cluster

Deleting a cluster using the AWS Management Console

Deleting a cluster using the AWS CLI

Deleting a cluster using the API

Getting the Apache ZooKeeper connection string for an Amazon MSK Cluster

Getting the Apache ZooKeeper connection string using the AWS Management Console

Getting the Apache ZooKeeper connection string using the AWS CLI

Getting the Apache ZooKeeper connection string using the API

Getting the bootstrap brokers for an Amazon MSK Cluster

Getting the bootstrap brokers using the AWS Management Console

Getting the bootstrap brokers using the AWS CLI

Getting the bootstrap brokers using the API

Listing Amazon MSK clusters

Listing clusters using the AWS Management Console

Listing clusters using the AWS CLI

Listing clusters using the API

Provisioning storage throughput

Throughput bottlenecks

Measuring storage throughput

Conﬁguration update

Provisioning storage throughput using the AWS Management Console

Provisioning storage throughput using the AWS CLI

Provisioning storage throughput using the API

Scaling up broker storage

Automatic scaling

MSK Storage auto-expansion details

Setting up auto-expansion for your Amazon MSK cluster

Setting up auto-expansion using the AWS Management Console

Setting up auto-expansion using the CLI

Setting up auto-expansion using the API

Manual scaling

Scaling up broker storage using the AWS Management Console

Scaling up broker storage using the AWS CLI

Scaling up broker storage using the API

Updating the broker type

Updating the broker type using the AWS Management Console

Updating the broker type using the AWS CLI

Updating the broker type using the API

Updating the conﬁguration of an Amazon MSK Cluster

Updating the conﬁguration of a cluster using the AWS CLI

Updating the conﬁguration of a cluster using the API

Expanding an Amazon MSK Cluster

Expanding a cluster using the AWS Management Console

Expanding a cluster using the AWS CLI

Expanding a cluster using the API

Updating a cluster's security settings

Updating a cluster's security settings using the AWS Management Console

Updating a cluster's security settings using the AWS CLI