Custom Labels Guide Rekognition

(1)

Rekognition

Custom Labels Guide

(2)

Rekognition: Custom Labels Guide

Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be aﬃliated with, connected to, or sponsored by Amazon.

(3)

What Is Amazon Rekognition Custom Labels?

With Amazon Rekognition Custom Labels, you can identify the objects and scenes in images that are speciﬁc to your business needs. For example, you can ﬁnd your logo in social media posts, identify your products on store shelves, classify machine parts in an assembly line, distinguish healthy and infected plants, or detect animated characters in videos.

Developing a custom model to analyze images is a signiﬁcant undertaking that requires time, expertise, and resources. It often takes months to complete. Additionally, it can require thousands or tens of thousands of hand-labeled images to provide the model with enough data to accurately make decisions.

Generating this data can take months to gather, and can require large teams of labelers to prepare it for use in machine learning.

Amazon Rekognition Custom Labels extends Amazon Rekognition’s existing capabilities, which are already trained on tens of millions of images across many categories. Instead of thousands of images, you can upload a small set of training images (typically a few hundred images or less) that are speciﬁc to your use case. You can do this by using the easy-to-use console. If your images are already labeled, Amazon Rekognition Custom Labels can begin training a model in a short time. If not, you can label the images directly within the labeling interface, or you can use Amazon SageMaker Ground Truth to label them for you.

After Amazon Rekognition Custom Labels begins training from your image set, it can produce a custom image analysis model for you in just a few hours. Behind the scenes, Amazon Rekognition Custom Labels automatically loads and inspects the training data, selects the right machine learning algorithms, trains a model, and provides model performance metrics. You can then use your custom model through the Amazon Rekognition Custom Labels API and integrate it into your applications.

Key Beneﬁts

Simpliﬁed data labeling

The Amazon Rekognition Custom Labels console provides a visual interface to make labeling your images fast and simple. The interface allows you to apply a label to the entire image. You can also identify and label speciﬁc objects in images using bounding boxes with a click-and-drag interface. Alternately, if you have a large dataset, you can use Amazon SageMaker Ground Truth to eﬃciently label your images at scale.

Automated machine learning

No machine learning expertise is required to build your custom model. Amazon Rekognition Custom Labels includes automated machine learning (AutoML) capabilities that take care of the machine learning for you. When the training images are provided, Amazon Rekognition Custom Labels can automatically load and inspect the data, select the right machine learning algorithms, train a model, and provide model performance metrics.

Simpliﬁed model evaluation, inference, and feedback

You evaluate your custom model’s performance on your test set. For every image in the test set, you can see the side-by-side comparison of the model’s prediction vs. the label assigned. You can also review detailed performance metrics such as precision, recall, F1 scores, and conﬁdence scores. You can start

(7)

Are You a First-Time Amazon Rekognition Custom Labels User?

using your model immediately for image analysis, or you can iterate and retrain new versions with more images to improve performance. After you start using your model, you track your predictions, correct any mistakes, and use the feedback data to retrain new model versions and improve performance.

Are You a First-Time Amazon Rekognition Custom Labels User?

If you're a ﬁrst-time user of Amazon Rekognition Custom Labels, we recommend that you read the following sections in order:

1.Setting up Amazon Rekognition Custom Labels (p. 4) – In this section, you set your account details.

2.Understanding Amazon Rekognition Custom Labels (p. 27) – In this section, you learn about the workﬂow for creating a model.

3.Getting started with Amazon Rekognition Custom Labels (p. 11) – In this section, you train a model using example projects created by Amazon Rekognition Custom Labels.

4.Tutorial: Classifying images (p. 34) – In this section, you learn how to train a model that classiﬁes images with datasets that you create.

Choosing to Use Amazon Rekognition Custom Labels

You can use Amazon Rekognition Custom Labels to ﬁnd objects, scenes, and concepts in images by using a machine learning model that you create.

Note

Amazon Rekognition Custom Labels is not designed for analyzing faces, detecting text, or ﬁnding unsafe image content in images. To perform these tasks, you can use Amazon Rekognition Image. For more information, see What Is Amazon Rekognition.

Amazon Rekognition Label Detection

Customers across media and entertainment, advertising and marketing, industrial, agricultural, and other segments need to identify, classify, and search for important objects, scenes, and other concepts in their images and videos—at scale. Amazon Rekognition's label detection feature makes it fast and easy to detect thousands of common objects (such as cars and trucks, corn and tomatoes, and basketballs and soccer balls) within images and video by using computer vision and machine learning technologies.

However, you can't find specialized objects using Amazon Rekognition label detection. For example, sports leagues want to identify team logos on player jerseys and helmets during game footage, media networks would like to detect specific sponsor logos to report on advertiser coverage, manufacturers need to distinguish between specific machine parts or products in an assembly line to monitor quality, and other customers want to identify cartoon characters in a media library, or locate products of a specific brand on retail shelves. Amazon Rekognition labels don't help you detect these specialized objects.

Amazon Rekognition Custom Labels

You can use Amazon Rekognition Custom Labels to ﬁnd objects and scenes that are unique to your business needs. Amazon Rekognition Custom Labels supports use cases such as logos, objects, and

(8)

Amazon Rekognition Custom Labels

scenes. You can use it to perform image classiﬁcation (image level predictions) or detection (object/

bounding box level predictions).

For example, while you can ﬁnd plants and leaves by using Amazon Rekognition label detection, you need Amazon Rekognition Custom Labels to distinguish between healthy, damaged, and infected plants.

Similarly, Amazon Rekognition label detection can identify images with machine parts. However, to identify speciﬁc machine parts such as a turbocharger or a torque converter, you need to use Amazon Rekognition Custom Labels. The following are examples of how you can use Amazon Rekognition Custom Labels.

• Detect logos

• Find animation characters

• Find your products

• Identify machine parts

• Classify agricultural produce quality (such as rotten, ripe, or raw)

(9)

Step 1: Create an AWS account

Setting up Amazon Rekognition Custom Labels

In this section, you sign up for an AWS account and then create an IAM user, a security group, and optionally set up access to external Amazon S3 buckets used by the Amazon Rekognition Custom Labels console and SDK.

For information about the AWS Regions that support Amazon Rekognition Custom Labels, see Amazon Rekognition Endpoints and Quotas.

Topics

• Step 1: Create an AWS account (p. 4)

• Step 2: Create an IAM administrator user and group (p. 4)

• Step 3: Set Up the AWS CLI and AWS SDKs (p. 5)

• Step 4: Set up Amazon Rekognition Custom Labels permissions (p. 6)

• Step 5: Create the console bucket (p. 8)

• Step 6: (Optional) Encrypt training ﬁles (p. 9)

• (Optional) Step 7: Associate prior datasets with new projects (p. 10)

Step 1: Create an AWS account

In this section, you sign up for an AWS account. If you already have an AWS account, skip this step.

When you sign up for Amazon Web Services (AWS), your AWS account is automatically signed up for all AWS services, including IAM. You are charged only for the services that you use.

To create an AWS account

1. Open https://portal.aws.amazon.com/billing/signup.

2. Follow the online instructions.

Part of the sign-up procedure involves receiving a phone call and entering a veriﬁcation code on the phone keypad.

Write down your AWS account ID because you'll need it for the next task.

Step 2: Create an IAM administrator user and group

When you create an AWS account, you get a single sign-in identity that has complete access to all of the AWS services and resources in the account. This identity is called the AWS account root user. Signing in

(10)

Step 3: Set Up the AWS CLI and AWS SDKs

to the AWS Management Console by using the email address and password that you used to create the account gives you complete access to all of the AWS resources in your account.

We strongly recommend that you do not use the root user for everyday tasks, even the administrative ones. Instead, adhere to the best practice in Create Individual IAM Users, which is to create an AWS Identity and Access Management (IAM) administrator user. Then securely lock away the root user credentials, and use them to perform only a few account and service management tasks.

To create an administrator user and sign in to the console

1. Create an administrator user in your AWS account. For instructions, see Creating Your First IAM User and Administrators Group in the IAM User Guide.

Note

An IAM user with administrator permissions has unrestricted access to the AWS services in your account. You can restrict permissions as necessary. The code examples in this guide assume that you have a user with the AmazonRekognitionFullAccess permissions. You also have to provide permissions to access the console. For more information, see Step 4:

Set up Amazon Rekognition Custom Labels permissions (p. 6).

2. Sign in to the AWS Management Console.

To sign in to the AWS Management Console as a IAM user, you must use a special URL. For more information, see How Users Sign In to Your Account in the IAM User Guide.

Step 3: Set Up the AWS CLI and AWS SDKs

The following steps show you how to install the AWS Command Line Interface (AWS CLI) and AWS SDKs that the examples in this documentation use. There are a number of diﬀerent ways to authenticate AWS SDK calls. The examples in this guide assume that you're using a default credentials proﬁle for calling AWS CLI commands and AWS SDK API operations.

For a list of available AWS Regions, see Regions and Endpoints in the Amazon Web Services General Reference.

Follow the steps to download and conﬁgure the AWS SDKs.

To set up the AWS CLI and the AWS SDKs

1. Download and install the AWS CLI and the AWS SDKs that you want to use. This guide provides examples for the AWS CLI, Java, and Python. For information about installing AWS SDKs, see Tools for Amazon Web Services.

2. Create an access key for the user you created in Step 2: Create an IAM administrator user and group (p. 4).

a. Sign in to the AWS Management Console and open the IAM console at https://

console.aws.amazon.com/iam/.

b. In the navigation pane, choose Users.

c. Choose the name of the user you created in Step 2: Create an IAM administrator user and group (p. 4).

d. Choose the Security credentials tab.

e. Choose Create access key. Then choose Download .csv file to save the access key ID and secret access key to a CSV file on your computer. Store the file in a secure location. You will not have access to the secret access key again after this dialog box closes. After you have downloaded the CSV file, choose Close.

3. If you have installed the AWS CLI, you can conﬁgure the credentials and Region for most AWS SDKs by entering aws configure at the command prompt. Otherwise, use the following instructions.

(11)

Step 4: Set Up Permissions

4. On your computer, navigate to your home directory, and create an .aws directory. On Unix-based systems, such as Linux or macOS, this is in the following location:

~/.aws

On Windows, this is in the following location:

%HOMEPATH%\.aws

5. In the .aws directory, create a new ﬁle named credentials.

6. Open the credentials CSV ﬁle that you created in step 2. Copy its contents into the credentials ﬁle using the following format:

[default]

aws_access_key_id = your_access_key_id

aws_secret_access_key = your_secret_access_key

Substitute your access key ID and secret access key for your_access_key_id and your_secret_access_key.

7. Save the Credentials ﬁle and delete the CSV ﬁle.

8. In the .aws directory, create a new ﬁle named config.

9. Open the config ﬁle and enter your Region in the following format.

[default]

region = your_aws_region

Substitute your desired AWS Region (for example, us-west-2) for your_aws_region.

Note

If you don't select a Region, then us-east-1 is used by default.

10. Save the config ﬁle.

Step 4: Set up Amazon Rekognition Custom Labels permissions

To use the Amazon Rekognition you need add to have appropriate permissions. If you want to store your training ﬁles in a bucket other than the console bucket, you need additional permissions.

Topics

• Allowing console access (p. 6)

• Accessing external Amazon S3 Buckets (p. 7)

• Policy updates for using the AWS SDK (p. 8)

Allowing console access

The Identity and Access Management (IAM) user or group that uses the Amazon Rekognition Custom Labels consoles needs the following IAM policy that covers Amazon S3, SageMaker Ground Truth, and Amazon Rekognition Custom Labels. To allow console access, use the following IAM policy. For information about adding IAM policies, see Creating IAM Policies.

(12)

Accessing external Amazon S3 Buckets

{

"Version": "2012-10-17", "Statement": [

{

"Effect": "Allow", "Action": [

"s3:ListBucket", "s3:ListAllMyBuckets"

],

"Resource": "*"

}, {

"Sid": "s3Policies", "Effect": "Allow", "Action": [

"s3:ListBucket", "s3:CreateBucket", "s3:GetBucketAcl", "s3:GetBucketLocation", "s3:GetObject",

"s3:GetObjectAcl", "s3:GetObjectVersion", "s3:GetObjectTagging", "s3:GetBucketVersioning", "s3:GetObjectVersionTagging", "s3:PutBucketCORS",

"s3:PutLifecycleConfiguration", "s3:PutBucketPolicy",

"s3:PutObject", "s3:PutObjectTagging", "s3:PutBucketVersioning", "s3:PutObjectVersionTagging"

],

"Resource": [

"arn:aws:s3:::custom-labels-console-*", "arn:aws:s3:::rekognition-video-console-*"

] }, {

"Sid": "rekognitionPolicies", "Effect": "Allow",

"Action": [

"rekognition:*"

],

"Resource": "*"

}, {

"Sid": "groundTruthPolicies", "Effect": "Allow",

"Action": [

"groundtruthlabeling:*"

],

"Resource": "*"

} ] }

Accessing external Amazon S3 Buckets

When you ﬁrst open the Amazon Rekognition Custom Labels console in a new AWS Region, Amazon Rekognition Custom Labels creates a bucket (console bucket) that's used to store project ﬁles.

Alternatively, you can use your own Amazon S3 bucket (external bucket) to upload the images or

(13)

Policy updates for using the AWS SDK

manifest ﬁle to the console. To use an external bucket, add the following policy block to the preceding policy. Replace my-bucket with the name of the bucket.

{ "Sid": "s3ExternalBucketPolicies", "Effect": "Allow",

"Action": [

"s3:GetBucketAcl", "s3:GetBucketLocation", "s3:GetObject",

"s3:GetObjectAcl", "s3:GetObjectVersion", "s3:GetObjectTagging", "s3:ListBucket", "s3:PutObject"

],

"Resource": [

"arn:aws:s3:::my-bucket*"

] }

Policy updates for using the AWS SDK

To use the AWS SDK with the latest release of Amazon Rekognition Custom Labels, you no longer need to give Amazon Rekognition Custom Labels permissions to access the Amazon S3 bucket that contains your training and testing images. If you have previously added permissions, You don't need to remove them. If you choose to, remove any policy from the bucket where the service for the principal is rekognition.amazonaws.com. For example:

"Principal": {

"Service": "rekognition.amazonaws.com"

}

For more information, see Using bucket policies

Step 5: Create the console bucket

You use an Amazon Rekognition Custom Labels project to create and manage your models. When you ﬁrst open the Amazon Rekognition Custom Labels console in a new AWS Region, Amazon Rekognition Custom Labels creates an Amazon S3 bucket (console bucket) to store your projects. You should note the console bucket name somewhere where you can refer to it later because you might need to use the bucket name in AWS SDK operations or console tasks, such as creating a dataset.

The format of the bucket name is custom-labels-console-<region>-<random value>. The random value ensures that there isn't a collision between bucket names.

To create the console bucket

1. Ensure that the IAM user or group you are using has the correct permissions. For more information, see Allowing console access (p. 6).

2. Sign in to the AWS Management Console and open the Amazon S3 console at https://

console.aws.amazon.com/s3/.

3. Choose Get started.

4. If this is the ﬁrst time that you've opened the console in the current AWS Region, do the following in the First Time Set Up dialog box:

(14)

Step 6: (Optional) Encrypt training ﬁles

a. Copy down the name of the Amazon S3 bucket that's shown. You'll need this information later.

b. Choose Create S3 bucket to let Amazon Rekognition Custom Labels create an Amazon S3 bucket (console bucket) on your behalf.

5. Close the browser window.

Step 6: (Optional) Encrypt training ﬁles

You can choose one of the following options to encrypt the Amazon Rekognition Custom Labels manifest ﬁles and image ﬁles that are in a console bucket or an external Amazon S3 bucket.

• Use an Amazon S3 key (SSE-S3).

• Use your AWS KMS key.

Note

The calling IAM principal need permissions to decrypt the ﬁles. For more information, see Decrypting ﬁles encrypted with AWS Key Management Service (p. 9).

For information about encrypting an Amazon S3 bucket, see Setting default server-side encryption behavior for Amazon S3 buckets.

Decrypting ﬁles encrypted with AWS Key Management Service

If you use AWS Key Management Service (KMS) to encrypt your Amazon Rekognition Custom Labels manifest files and image files, add the IAM principal that calls Amazon Rekognition Custom Labels to the key policy of the KMS key. Doing this lets Amazon Rekognition Custom Labels decrypt your manifest and image files before training. For more information, see My Amazon S3 bucket has default encryption using a custom AWS KMS key. How can I allow users to download from and upload to the bucket?

The IAM principal needs the following permissions on the KMS key.

• kms:GenerateDataKey

• kms:Decrypt

For more information, see Protecting Data Using Server-Side Encryption with KMS keys Stored in AWS Key Management Service (SSE-KMS).

Encrypting copied training and test images

To train your model, Amazon Rekognition Custom Labels makes a copy of your source training and test images. By default the copied images are encrypted at rest with a key that AWS owns and manages. You can also choose to use your own AWS KMS key. If you use your own KMS key, you need the following permissions on the KMS key.

• kms:CreateGrant

• kms:DescribeKey

You optionally specify the KMS key when you train the model with the console or when you call the CreateProjectVersion operation. The KMS key you use doesn't need to be the same KMS key that you use to encrypt manifest and image ﬁles in your Amazon S3 bucket. For more information, see Step 6:

(Optional) Encrypt training ﬁles (p. 9).

(15)

Step 7: (Optional) Associate prior datasets

For more information, see AWS Key Management Service concepts. Your source images are unaﬀected.

For information about training a model, see Training an Amazon Rekognition Custom Labels model (p. 89).

(Optional) Step 7: Associate prior datasets with new projects

Amazon Rekognition Custom Labels now manages datasets with projects. Earlier (prior) datasets that you created are read-only and must be associated with a project before you can use them. When you open the details page for a project with the console, we automatically associate the datasets that trained the latest version of the project's model with the project. Automatic association of a dataset with a project doesn't happen if you are using the AWS SDK.

Unassociated prior datasets have never been used to train a model or, were used to train a previous version of a model. The Prior datasets page shows all of your associated and unassociated datasets.

To use an unassociated prior dataset, you create a new project on the Prior datasets page. The dataset becomes the training dataset for the new project. You can also create a project for an already associated dataset as prior datasets can have multiple associations.

To associate a prior dataset to a new project

1. Open the Amazon Rekognition console at https://console.aws.amazon.com/rekognition/.

2. In the left pane, choose Use Custom Labels. The Amazon Rekognition Custom Labels landing page is shown.

3. In the left navigation pane, choose Prior datasets.

4. In the datasets view, choose the prior dataset that you want to associate with a project.

5. Choose Create project with dataset.

6. On the Create project page, enter a name for your new project in Project name.

7. Choose Create project to create the project. The project might take a while to create.

8. Use the project. For more information, see Understanding Amazon Rekognition Custom Labels (p. 27).

Using a prior dataset as a test dataset

You can use a prior dataset as the test dataset for an existing project by ﬁrst associating the prior dataset with a new project. You then copy the training dataset of the new project to the test dataset of the existing project.

To use a prior dataset as a test dataset

1. Follow the instructions at (Optional) Step 7: Associate prior datasets with new projects (p. 10) to associate the prior dataset with a new project.

2. Create the test dataset in the existing project by using copying the training dataset from the new project. For more information, see Existing dataset (p. 64).

3. Follow the instructions at Deleting an Amazon Rekognition Custom Labels project (Console) (p. 180) to delete the new project.

Alternatively, you can create the test dataset by using the manifest ﬁle for prior dataset. For more information, see Creating a manifest ﬁle (p. 226).

(16)

Tutorial videos

Getting started with Amazon Rekognition Custom Labels

You use Amazon Rekognition Custom Labels to train a machine learning model. The trained model analyzes images to ﬁnd the objects, scenes, and concepts that are unique to your business needs. For example, you can train a model to classify images of houses, or ﬁnd the location of electronic parts on a printed circuit board.

To help you get started, Amazon Rekognition Custom Labels includes tutorial videos and example projects.

Topics

• Tutorial videos (p. 11)

• Example projects (p. 11)

• Using the example projects (p. 13)

• Step 1: Choose an example project (p. 14)

• Step 2: Train your model (p. 16)

• Step 3: Start your model (p. 18)

• Step 4: Analyze an image with your model (p. 19)

• Step 5: Stop your model (p. 24)

• Step 6: Next steps (p. 25)

Tutorial videos

The videos show you how to use Amazon Rekognition Custom Labels to train and use a model.

To view the tutorial videos

1. Sign in to the AWS Management Console and open the Amazon Rekognition console at https://

console.aws.amazon.com/rekognition/.

2. In the left pane, choose Use Custom Labels. The Amazon Rekognition Custom Labels landing page is shown. If you don't see Use Custom Labels, check that the AWS Region you are using supports Amazon Rekognition Custom Labels.

3. In the navigation pane, choose Get started.

4. In What is Amazon Rekognition Custom Labels?, choose the video to watch the overview video.

5. In the navigation pane, choose Tutorials.

6. On the Tutorials page, choose the tutorial videos that you want to watch.

Example projects

Amazon Rekognition Custom Labels provides the following example projects.

(17)

Image classiﬁcation

The image classiﬁcation project (Rooms) trains a model that ﬁnds one or more household locations in an image, such as backyard, kitchen, and patio. The training and test images represent a single location.

Each image is labeled with a single image-level label, such as kitchen, patio, or living_space. For an analyzed image, the trained model returns one or more matching labels from the set of image-level labels used for training. For example, the model might ﬁnd the label living_space in the following image.

For more information, see Find objects, scenes, and concepts (p. 55).

Multi-label image classiﬁcation

The multi-label image classification project (Flowers) trains a model that categorizes images of flowers into three concepts (flower type, leaf presence, and growth stage).

The training and test images have image-level labels for each concept, such as camellia for a flower type, with_leaves for a flower with leaves, and fully_grown for a flower that is fully grown.

For an analyzed image, the trained model returns matching labels from the set of image-level labels used for training. For example, the model returns the labels mediterranean_spurge and with_leaves for the following image. For more information, see Find objects, scenes, and concepts (p. 55).

Brand detection

The brand detection project (Logos) trains a model that model ﬁnds the location of certain AWS logos such as Amazon Textract, and AWS lambda. The training images are of the logo only and have a single image level-label, such as lambda or textract. It is also possible to train a brand detection model with training images that have bounding boxes for brand locations. The test images have labeled bounding boxes that represent the location of logos in natural locations, such as an architectural diagram. The trained model ﬁnds the logos and returns a labeled bounding box for each logo found. For more information, see Find brand locations (p. 56).

(18)

Object localization

The object localization project (Circuit boards) trains a model that finds the location of parts on a printed circuit board, such as a comparator or an infra red light emitting diode. The training and test images include bounding boxes that surround the circuit board parts and a label that identifies the part within the bounding box. The label names are ir_phototransistor, ir_led, pot_resistor, and comparator. The trained model finds the circuit board parts and returns a labeled bounding for each circuit part found.

For more information, see Find object locations (p. 56).

Using the example projects

These Getting Started instructions show you how to train a model by using example projects that Amazon Rekognition Custom Labels creates for you. It also shows you how to start the model and use it to analyze an image.

Creating the example project

To get started, decide which project to use. For more information, see Step 1: Choose an example project (p. 14).

Amazon Rekognition Custom Labels uses datasets to train and evaluate (test) a model. A dataset manages images and the labels that identify the contents of images. The example projects include a training dataset and a test dataset in which all images are labeled. You don't need to make any changes before training your model. The example projects show the two ways in which Amazon Rekognition Custom Labels uses labels to train diﬀerent types of models.

• image-level – The label identiﬁes an object, scene, or concept that represents the entire image.

• bounding box – The label identiﬁes the contents of a bounding box. A bounding box is a set of image coordinates that surround an object in an image.

(19)

Training the model

Later, when you create a project with your own images, you must create training and test datasets, and also label your images. For more information, see Decide your model type (p. 27).

Training the model

After Amazon Rekognition Custom Labels creates the example project, you can train the model. For more information, see Step 2: Train your model (p. 16). After training ﬁnishes, you normally evaluate the performance of the model. The images in the example dataset already create a high-performance model, and you don't need to evaluate the model before running the model. For more information, see Improving a trained Amazon Rekognition Custom Labels model (p. 135).

Using the model

Next you start the model. For more information, see Step 3: Start your model (p. 18).

After you start running your model, you can use it to analyze new images. For more information, see Step 4: Analyze an image with your model (p. 19).

You are charged for the amount of time that your model runs. When you ﬁnish using the example model, you should stop the model. For more information, see Step 5: Stop your model (p. 24).

Next steps

When you're ready, you can create your own projects. For more information, see Step 6: Next steps (p. 25).

Step 1: Choose an example project

In this step you use choose an example project. Amazon Rekognition Custom Labels then creates a project and a dataset for you. A project manages the ﬁles used to train your model. For more information, see Managing an Amazon Rekognition Custom Labels project (p. 180). Datasets contain the images, assigned labels, and bounding boxes that you use to train and test a model. For more information, see the section called “Managing datasets” (p. 191).

For information about the example projects, see Example projects (p. 11).

Choose an example project

2. In the left pane, choose Use Custom Labels. The Amazon Rekognition Custom Labels landing page is shown. If you don't see Use Custom Labels, check that the AWS Region you are using supports Amazon Rekognition Custom Labels.

3. Choose Get started.

(20)

Step 1: Choose an example project

4. In Explore example projects, choose Try example projects.

5. Decide which project you want to use and choose Create project "project name" within the example section. Amazon Rekognition Custom Labels then creates the example project for you.

Note

If this is the ﬁrst time that you've opened the console in the current AWS Region, the First Time Set Up dialog box is shown. Do the following:

1. Note the name of the Amazon S3 bucket that's shown.

2. Choose Continue to let Amazon Rekognition Custom Labels create an Amazon S3 bucket (console bucket) on your behalf.

6. After your project is ready, choose Go to dataset.

(21)

Step 2: Train your model

In this step you train your model. The training and test datasets are automatically conﬁgured for you.

After training successfully completes, you can see the overall evaluation results, and evaluation results for individual test images. For more information, see Training an Amazon Rekognition Custom Labels model (p. 89).

To train your model

1. On the dataset page, choose Train model.

2. On the Train model page, Choose Train model. The Amazon Resource Name (ARN) for your project is in the Choose project edit box.

(22)

3. In the Do you want to train your model? dialog box, choose Train model.

4. After training completes, choose the model name. Training is ﬁnished when the model status is TRAINING_COMPLETED.

5. Choose the Evaluate button to see the evaluation results. For information about evaluating a model, see Improving a trained Amazon Rekognition Custom Labels model (p. 135).

6. Choose View test results to see the results for individual test images.

(23)

Step 3: Start your model

7. After viewing the test results, choose the project name to return to the model page.

Step 3: Start your model

In this step you start your model. After your model starts, you can use it to analyze images.

You are charged for the amount of time that your model runs. Stop your model if you don't need to analyze images. You can restart your model at a later time. For more information, see Running a trained Amazon Rekognition Custom Labels model (p. 147).

To start your model

1. Choose the Use model tab on the model page.

(24)

Step 4: Analyze an image with your model

2. In the Start or stop model section do the following:

a. Choose Start.

b. In the Start model dialog box, choose Start.

3. Wait until the model is running. The model is running when the status in the Start or stop model section is Running.

4. Use your model to classify images. For more information, see Step 4: Analyze an image with your model (p. 19).

Step 4: Analyze an image with your model

You analyze an image by calling the DetectCustomLabels API. In this step, you use the detect-custom- labels AWS Command Line Interface (AWS CLI) command to analyze an example image. You get the AWS CLI command from the Amazon Rekognition Custom Labels console. The console conﬁgures the AWS CLI command to use your model. You only need to supply an image that's stored in an Amazon S3 bucket. This topic provides an image that you can use for each example project.

Note

The console also provides Python example code.

The output from detect-custom-labels includes a list of labels found in the image, bounding boxes (if the model ﬁnds object locations), and the conﬁdence that the model has in the accuracy of the predictions.

For more information, see Analyzing an image with a trained model (p. 162).

To analyze an image (console)

1. If you haven't already, set up the AWS CLI. For instructions, see the section called “Step 3: Set Up the AWS CLI and AWS SDKs” (p. 5).

(25)

2. If you haven't already, start running your model. For more information, see Step 2: Train your model (p. 16).

3. Choose the Use Model tab and then choose API code.

4. Choose AWS CLI command.

5. In the Analyze image section, copy the AWS CLI command that calls detect-custom-labels.

(26)

6. Upload an example image to an Amazon S3 bucket. For instructions, see Getting an example image (p. 22).

7. At the command prompt, enter the AWS CLI command that you copied in the previous step. It should look like the following example.

The value of --project-version-arn should be Amazon Resource Name (ARN) of your model.

The value of --region should be the AWS Region in which you created the model.

Change MY_BUCKET and PATH_TO_MY_IMAGE to the Amazon S3 bucket and image that you used in the previous step.

aws rekognition detect-custom-labels \ --project-version-arn "model_arn" \

--image "{"S3Object": {"Bucket": "MY_BUCKET","Name": "PATH_TO_MY_IMAGE"}}" \ --region us-east-1

If the model ﬁnds objects, scenes, and concepts, the JSON output from the AWS CLI command should look similar to the following. Name is the name of the image-level label that the model found. Confidence (0-100) is the model's conﬁdence in the accuracy of the prediction.

{

(27)

Getting an example image

"CustomLabels": [ {

"Name": "living_space",

"Confidence": 83.41299819946289 }

] }

If the model ﬁnds object locations or ﬁnds brand, labeled bounding boxes are returned.

BoundingBox contains the location of a box that surrounds the object. Name is the object that the model found in the bounding box. Confidence is the model's conﬁdence that the bounding box contains the object.

{ "CustomLabels": [ {

"Name": "textract",

"Confidence": 87.7729721069336, "Geometry": {

"BoundingBox": {

"Width": 0.198987677693367, "Height": 0.31296101212501526, "Left": 0.07924537360668182, "Top": 0.4037395715713501 }

} } ] }

8. Continue to use the model to analyze other images. Stop the model if you are no longer using it. For more information, see Step 5: Stop your model (p. 24).

Getting an example image

You can use the following images with the DetectCustomLabels operation. There is one image for each project. To use the images, you upload them to an S3 bucket.

To use an example image

1. Right-click the following image that matches the example project that you are using. Then choose Save image to save the image to your computer. The menu option might be diﬀerent, depending on which browser you are using.

2. Upload the image to an Amazon S3 bucket that's owned by your AWS account and is in the same AWS region in which you are using Amazon Rekognition Custom Labels.

For instructions, see Uploading Objects into Amazon S3 in the Amazon Simple Storage Service User Guide.

(28)

Getting an example image

Image classiﬁcation

Multi-label classiﬁcation

Brand detection

(29)

Step 5: Stop your model

Object localization

Step 5: Stop your model

In this step you stop running your model. You are charged for the amount of time your model is running.

If you have ﬁnished using the model, you should stop it.

To stop your model

1. In the Start or stop model section choose Stop.

2. In the Stop model dialog box, enter stop to conﬁrm that you want to stop the model.

(30)

Step 6: Next steps

3. Choose Stop to stop your model. The model has stopped when the status in the Start or stop model section is Stopped.

Step 6: Next steps

After you ﬁnished trying the examples projects, you can use your own images and datasets to create your own model. For more information, see Understanding Amazon Rekognition Custom Labels (p. 27).

Use the labeling information in the following table to train models similar to the example projects.

Example Training images Test images

Image classiﬁcation (Rooms) 1 Image-level label per image 1 Image-level label per image Multi-label classiﬁcation

(Flowers) Multiple image-level labels per

image Multiple image-level labels per

image

(31)

Step 6: Next steps

Example Training images Test images

Brand detection (Logos) image level-labels (you can also

use Labeled bounding boxes) Labeled bounding boxes Image localization (Circuit

boards) Labeled bounding boxes Labeled bounding boxes

The Tutorial: Classifying images (p. 34) shows you how to create a project, datasets, and models for an Image classiﬁcation model.

For detailed information about creating datasets and training models, see Creating an Amazon Rekognition Custom Labels model (p. 50).

(32)

Decide your model type

Understanding Amazon Rekognition Custom Labels

This section gives you an overview of the workﬂow to train and use an Amazon Rekognition Custom Labels model with the console and the AWS SDK.

Note

Amazon Rekognition Custom Labels now manages datasets within a project. You can create datasets for your projects with the console and with the AWS SDK. If you have previously used Amazon Rekognition Custom Labels, your older datasets might need associating with a new project. For more information, see (Optional) Step 7: Associate prior datasets with new projects (p. 10)

Topics

• Decide your model type (p. 27)

• Create a model (p. 29)

• Improve your model (p. 31)

• Start your model (p. 32)

• Analyze an image (p. 32)

• Stop your model (p. 33)

Decide your model type

You ﬁrst decide which type of model you want to train, which depends on your business goals. For example, you could train a model to ﬁnd your logo in social media posts, identify your products on store shelves, or classify machine parts in an assembly line.

Amazon Rekognition Custom Labels can train the following types of model:

• Find objects, scenes, and concepts (p. 27)

• Find object locations (p. 28)

• Find the location of brands (p. 29)

To help you decide which type of model to train, Amazon Rekognition Custom Labels provides example projects that you can use. For more information, see Getting started with Amazon Rekognition Custom Labels (p. 11).

Find objects, scenes, and concepts

The model predicts objects, scenes, and concepts associated with an entire image. For example, you can train a model that determines if an image contains a tourist attraction, or not.

(33)

Find object locations

Alternatively, you can train a model that categorizes images into multiple categories. For example, the previous image might have categories such as sky color, reﬂection, or lake.

The Image classification (p. 12) and Multi-label image classification (p. 12) example projects show different uses for image-level labels.

Find object locations

The model predicts the location of an object on an image. The prediction includes bounding box information for the object location and a label that identiﬁes the object within the bounding box. For example, the following image shows bounding boxes around various parts of a circuit board, such as a comparator or pot resistor.

(34)

Find the location of brands

The Object localization (p. 13) example project shows how Amazon Rekognition Custom Labels uses labeled bounding boxes to train a model that ﬁnds object locations.

Find the location of brands

Amazon Rekognition Custom Labels can train a model that ﬁnds the location of brands, such as logos, on an image. The prediction includes bounding box information for the brand location and a label that identiﬁes the object within the bounding box.

Create a model

The steps to create a model are creating a project, creating training and test datasets, and training the model.

Create a project

An Amazon Rekognition Custom Labels project is a group of resources needed to create and manage a model. A project manages the following:

• Datasets – The images and image labels used to train a model. A project has a training dataset and a test dataset.

(35)

Create training and test datasets

• Models – The software that you train to ﬁnd the concepts, scenes, and objects unique to your business.

You can have multiple versions of a model in a project.

We recommend that you use a project for a single use case, such as ﬁnding circuit board parts on a circuit board.

You can create a project with the Amazon Rekognition Custom Labels console and with the CreateProject API. For more information, see Creating a project (p. 50).

Create training and test datasets

A dataset is a set of images and labels that describe those images. Within your project, you create a training dataset and a test dataset that Amazon Rekognition Custom Labels uses to train and test your model.

A label identiﬁes an object, scene, concept, or bounding box around an object in an image. Labels are either assigned to an entire image (image-level) or they are assigned to a bounding box that surrounds an object on an image.

Important

How you label the images in your datasets determines the type of model that Amazon Rekognition Custom Labels creates. For example, to train a model that ﬁnds objects, scenes and concepts, you assign image level labels to the images in your training and test datasets. For more information, see Purposing datasets (p. 55).

Images must be in PNG and JPEG format, and you should follow the input images recommendations. For more information, see Preparing images (p. 58).

Create training and test datasets (Console)

You can start a project with a single dataset, or with separate training and test datasets. If you start with a single dataset, Amazon Rekognition Custom Labels splits your dataset during training to create a training dataset (80%) and a test dataset (20%) for your project. Start with a single dataset if you want Amazon Rekognition Custom Labels to decide which images are used for training and testing. For complete control over training, testing, and performance tuning, we recommend that you start your project with separate training and test datasets.

To create the datasets for a project, you import the images in one of the following ways:

• Import images from your local computer.

• Import images from an S3 bucket. Amazon Rekognition Custom Labels can label the images using the folder names that contain the images.

• Import an Amazon SageMaker Ground Truth manifest ﬁle.

• Copy an existing Amazon Rekognition Custom Labels dataset.

For more information, see Creating training and test datasets (Console) (p. 59).

Depending on where you import your images from, your images might be unlabeled. For example, images imported from a local computer aren't labeled. Images imported from an Amazon SageMaker Ground Truth manifest ﬁle are labeled. You can use the Amazon Rekognition Custom Labels console to add, change, and assign labels. For more information, see Labeling images (p. 82).

To create your training and test datasets with the console, see Creating training and test datasets (Console) (p. 59). For a tutorial that includes creating training and test datasets, see Tutorial:

Classifying images (p. 34).

(36)

Train your model

Create training and test datasets (SDK)

To create your training and test datasets, you use the CreateDataset API. You can create a dataset by using an Amazon Sagemaker format manifest file or by copying an existing Amazon Rekognition Custom Labels dataset. For more information, see Create training and test datasets (SDK) (p. 65) If necessary, you can create your own manifest file. For more information, see the section called “Creating a manifest file” (p. 226).

Train your model

Train your model with the training dataset. A new version of a model is created each time it is trained.

Training also evaluates your model by using the test dataset. Training takes a while to complete.

You are only charged for a successful model training. For more information, see Training an Amazon Rekognition Custom Labels model (p. 89). If model training fails, Amazon Rekognition Custom Labels provides debugging information that you can use. For more information, see Debugging a failed model training (p. 99).

Train your model (Console)

To train your model with the console, see Training a model (Console) (p. 90).

Training a model (SDK)

You train an Amazon Rekognition Custom Labels model by calling CreateProjectVersion.

Improve your model

You can use evaluation metrics to help improve your trained model.

Evaluate your model

Evaluate the performance of your model by using the performance metrics created during testing.

Performance metrics, such as F1, precision, and recall, allow you to understand the performance of your trained model, and decide if you're ready to use it in production. For more information, see Metrics for evaluating your model (p. 135).

Evaluate a model (console)

To view performance metrics, see Accessing training results (Console) (p. 137).

Evaluate a model (SDK)

To get performance metrics, you call DescribeProjectVersions to get the training results. For more information, see Accessing Amazon Rekognition Custom Labels training results (SDK) (p. 139). The training results include metrics not available in the console, such as a confusion matrix for classiﬁcation results. The training results are returned in the following formats:

• F1 score – A single value representing the overall performance of precision and recall for the model.

For more information, see F1 (p. 137).

• Summary file location – The training summary includes aggregated evaluation metrics for the entire testing dataset and metrics for each individual label. DescribeProjectVersions returns the S3 bucket and folder location of the summary file. For more information, see Summary file (p. 140).

(37)

Improve your model

• Evaluation manifest snapshot location – The snapshot contains details about the test results, including the conﬁdence ratings and the results of binary classiﬁcation tests, such as false positives.

DescribeProjectVersions returns the S3 bucket and folder location of the snapshot ﬁles. For more information, see Evaluation manifest snapshot (p. 141).

Improve your model

If improvements are needed, you can add more training images or improve dataset labeling. For more information, see Improving an Amazon Rekognition Custom Labels model (p. 145). You can also give feedback on the predictions your model makes and use it to make improvements to your model. For more information, see Model feedback solution (p. 269).

Improve your model (console)

To add images to a dataset, see Adding more images to a dataset (p. 198). To add or change labels, see the section called “Labeling images” (p. 82).

To retrain your model, see Training a model (Console) (p. 90).

Improve your model (SDK)

To add images to a dataset or change the labeling for an image, use the UpdateDatasetEntries API. UpdateDatasetEntries updates or adds JSON lines to a manifest ﬁle. Each JSON line contains information for a single image, such as assigned labels or bounding box information. For more information, see Adding more images (SDK) (p. 198). To view the entries in a dataset, use the ListDatasetEntries API.

To retrain your model, see Training a model (SDK) (p. 93).

Start your model

Before you can use your model, you start the model by using the Amazon Rekognition Custom Labels console or the StartProjectVersion API. You are charged for the amount of time that your model runs. For more information, see Running a trained model (p. 147).

Start your model (console)

To start your model using the console, see Starting an Amazon Rekognition Custom Labels model (Console) (p. 148).

Start your model

You start your model calling StartProjectVersion. For more information, see Starting an Amazon Rekognition Custom Labels model (SDK) (p. 149).

Analyze an image

To analyze an image with your model, you use the DetectCustomLabels API. You can specify a local image, or an image stored in an S3 bucket. The operation also requires the Amazon Resource Name (ARN) of the model that you want to use.

(38)

Stop your model

If your model ﬁnds objects, scenes, and concepts, the response includes a list of image-level labels found in the image. For example, the following image shows the image-level labels found using Rooms example project.

If the model ﬁnds object locations, the response includes list of labeled bounding boxes found in the image. A bounding box represents the location of an object on an image. You can use the bounding box information to draw a bounding box around an object. For example, the following image shows bounding boxes around circuit board parts found using the Circuit boards example project.

For more information, see Analyzing an image with a trained model (p. 162).

Stop your model

You are charged for the time that your model is running. If you are no longer using your model, stop the model by using the Amazon Rekognition Custom Labels console, or by using the StopProjectVersion API. For more information, see Stopping an Amazon Rekognition Custom Labels model (p. 154).

Stop your model (SDK)

To stop a running model with the console, see Stopping an Amazon Rekognition Custom Labels model (Console) (p. 155).

Stop your model (SDK)

To stop a running model, call StopProjectVersion. For more information, see Stopping an Amazon Rekognition Custom Labels model (SDK) (p. 155).

(39)

Step 1: Collect your images

Tutorial: Classifying images

This tutorial shows you how to create the project and datasets for a model that classiﬁes objects, scenes, and concepts found in an image. The model classiﬁes the entire image. For example, by following this tutorial, you can train a model to recognize household locations such as a living room or kitchen. The tutorial also shows you how to use the model to analyze images.

Before starting the tutorial, we recommend that you read Understanding Amazon Rekognition Custom Labels (p. 27).

In this tutorial, you create the training and test datasets by uploading images from your local computer.

Later you assign image-level labels to the images in your training and test datasets.

The model you create classiﬁes images as belonging to the set of image-level labels that you assign to the training dataset images. For example, if the set of image-level labels in your training dataset is kitchen, living_room, patio, and backyard, the model can potentially ﬁnd all of those image-level labels in a single image.

Note

You can create models for diﬀerent purposes such as ﬁnding the location of objects on an image. For more information, see Decide your model type (p. 27).

Step 1: Collect your images

You need two sets of images. One set to add to your training dataset. Another set to add to your test dataset. The images should represent the objects, scenes, and concepts that you want your model to classify. The images must be in PNG or JPEG format. For more information, see Preparing images (p. 58).

You should have at least 10 images for your training dataset and 10 images for your test dataset.

If you don't yet have images, use the images from the Rooms example classiﬁcation project. After creating the project, the training and test images are at the following Amazon S3 bucket locations:

• Training images — s3://custom-labels-console-region-numbers/assets/rooms_version number_test_dataset/

• Test images — s3://custom-labels-console-region-numbers/assets/rooms_version number_test_dataset/

region is the AWS Region in which you are using the Amazon Rekognition Custom Labels console.

numbers is a value that the console assigns to the bucket name. Version number is the version number for the example project, starting at 1.

The following procedure stores images from the Rooms project into local folders on your computer named training and test.

To download the Rooms example project image ﬁles

1. Create the Rooms project. For more information, see Step 1: Choose an example project (p. 14).

(40)

Step 2: Decide your classes

2. Open the command prompt and enter the following command to download the training images.

aws s3 cp s3://custom-labels-console-region-numbers/assets/rooms_version number_training_dataset/ training --recursive

3. At the commend prompt, enter the following command to download the test images.

aws s3 cp s3://custom-labels-console-region-numbers/assets/rooms_version number_test_dataset/ test --recursive

4. Move two of the images from the training folder to a separate folder of your choosing. You'll use the images to try your trained model in Step 9: Analyze an image with your model (p. 45).

Step 2: Decide your classes

Make a list of the classes that you want your model to ﬁnd. For example, if you're training a model to recognize rooms in a house, you can classify the following image as living_room.

Each class maps to an image-level label. Later you assign image-level labels to the images in your training and test datasets.

If you're using the images from the Rooms example project, the image-level labels are backyard, bathroom, bedroom, closet, entry_way, ﬂoor_plan, front_yard, kitchen, living_space, and patio.

Step 3: Create a project

To manage your datasets and models you create a project. Each project should address a single use case, such as recognizing rooms in a house.

To create a project (console)

1. If you haven't already, set up the Amazon Rekognition Custom Labels console. For more information, see Setting up Amazon Rekognition Custom Labels (p. 4).

3. In the left pane, choose Use Custom Labels. The Amazon Rekognition Custom Labels landing page is shown.

4. The Amazon Rekognition Custom Labels landing page, choose Get started

(41)

Step 4: Create training and test datasets

5. In the left navigation pane, choose Projects.

6. On the projects page, choose Create Project.

7. In Project name, enter a name for your project.

8. Choose Create project to create your project.

Step 4: Create training and test datasets

In this step you create a training dataset and a test dataset by uploading images from your local

computer. You can upload as many as 30 images at a time. If you have a lot of images to upload, consider creating the datasets by importing the images from an Amazon S3 bucket. For more information, see Amazon S3 bucket (p. 59).

For more information about datasets, see Managing datasets (p. 191).

To create a dataset using images on a local computer (console)

1. On the project details page, choose Create dataset.

2. In the Starting conﬁguration section, choose Start with a training dataset and a test dataset.

3. In the Training dataset details section, choose Upload images from your computer.

4. In the Test dataset details section, choose Upload images from your computer.

5. Choose Create datasets.

(42)

(43)

6. A dataset page appears with a Training tab and a Test tab for the respective datasets.

7. On the dataset page, choose the Training tab.

8. Choose Actions and then choose Add images to training dataset.

9. In the Add images to training dataset dialog box, choose Choose ﬁles.

10. Choose the images you want to upload to the dataset. You can upload as many as 30 images at a time.

11. Choose Upload images. It might take a few seconds for Amazon Rekognition Custom Labels to add the images to the dataset.

(44)

Step 5: Add labels to the project

12. If you have more images to add to the training dataset, repeat steps 9-12.

13. Choose the Test tab.

14. Repeat steps 8 - 12 to add images to the test dataset. For step 8, choose Actions and then choose Add images to test dataset.

Step 5: Add labels to the project

In this step you add a label to the project for each of the classes you identiﬁed in step Step 2: Decide your classes (p. 35).

(45)

Step 6: Assign image-level labels to training and test datasets

To add a new label (console)

1. On the dataset gallery page, choose Start labeling to enter labeling mode.

2. In the Labels section of the dataset gallery, choose Edit labels to open the Manage labels dialog box.

3. In the edit box, enter a new label name.

4. Choose Add label.

5. Repeat steps 3 and 4 until you have created all the labels you need.

6. Choose Save to save the labels that you added.

Step 6: Assign image-level labels to training and test datasets

In this step you assign a single image level to each image in your training and test datasets. The image- level label is the class that each image represents.

To assign image-level labels to an image (console)

1. On the Datasets page, choose the Training tab.

2. Choose Start labeling to enter labeling mode.

3. Select one or more images that you want to add labels to. You can only select images on a single page at a time. To select a contiguous range of images on a page:

a. Select the ﬁrst image.

b. Press and hold the shift key.

c. Select the second image. The images between the ﬁrst and second image are also selected.

d. Release the shift key.

4. Choose Assign image-level labels.

(46)

5. In Assign image-level labels to selected images dialog box, select a label that you want to assign to the image or images.

6. Choose Assign to assign label to the image.

7. Repeat labeling until every image is annotated with the required labels.

8. Choose the Test tab.

9. Repeat steps to assign image level labels to the test dataset images.

Step 7: Train your model

Use the following steps to train your model. For more information, see Training an Amazon Rekognition Custom Labels model (p. 89).

To train your model (console)

1. On the Dataset page, choose Train model.

2. On the Train model page, choose Train model. The Amazon Resource Name (ARN) for your project is in the Choose project edit box.

(47)

3. In the Do you want to train your model? dialog box, choose Train model.

4. In the Models section of the project page, you can see that training is in progress. You can check the current status by viewing the Model Status column for the model version. Training a model takes a while to complete.

Custom Labels Guide Rekognition

Rekognition