• CREATE_PENDING changes to CREATE_STOPPED.
• CREATE_IN_PROGRESS changes to CREATE_STOPPING and then CREATE_STOPPED.
Step 4: Evaluating a solution version with metrics
You can evaluate the performance of your solution version through offline and online metrics. Online metrics are the empirical results you observe in your users' interactions with real-time recommendations.
For example, you might record your users' click-through rate as they browse your catalog. You are responsible for generating and recording any online metrics.
Offline metrics are the metrics Amazon Personalize generates when you train a solution version. You can use offline metrics to evaluate the performance of the model before you create a campaign and provide recommendations. Offline metrics allow you to view the effects of modifying a solution's hyperparameters or compare results from solutions that use the same training data but use different recipes. For the rest of this section, the term metrics refers to offline metrics.
To get performance metrics, Amazon Personalize splits the input interactions data into a training set and a testing set. The split depends on the type of recipe you choose:
• For USER_SEGMENTATION recipes, the training set consists of 80% of each user's interactions data and the testing set consists of 20% of each user's interactions data.
• For all other recipe types, the training set consists of 90% of each user's interactions data and the testing set consists of the remaining 10% of each user's interactions data.
Amazon Personalize then creates the solution version using the training set. After training completes, Amazon Personalize gives the new solution version the oldest 90% of each user’s data from the testing set as input. Amazon Personalize then calculates metrics by comparing the recommendations the solution version generates to the actual interactions in the newest 10% of each user’s data from the testing set.
To generate a baseline for comparison purposes, we recommend using the Popularity-Count (p. 136) recipe, which recommends the top K most popular items.
Important
In order for Amazon Personalize to generate solution version metrics, you must have at least 10 datapoints in your input dataset group.
Retrieving Metrics
You retrieve the metrics for a specific solution version by calling the GetSolutionMetrics (p. 367) operation.
Retrieve metrics using the AWS Python SDK
1. Create a solution version. For more information, see Creating a solution (p. 125).
2. Use the following code to retrieve metrics.
import boto3
personalize = boto3.client('personalize') response = personalize.get_solution_metrics(
solutionVersionArn = 'solution version arn')
Step 4: Evaluating a solution version with metrics
print(response['metrics'])
The following is an example the output from a solution version created using the User-Personalization (p. 127) recipe with an additional optimization objective.
{ "solutionVersionArn": "arn:aws:personalize:us-west-2:acct-id:solution/MovieSolution/
<version-id>", "metrics": {
"coverage": 0.27,
"mean_reciprocal_rank_at_25": 0.0379,
"normalized_discounted_cumulative_gain_at_5": 0.0405, "normalized_discounted_cumulative_gain_at_10": 0.0513, "normalized_discounted_cumulative_gain_at_25": 0.0828, "precision_at_5": 0.0136,
"precision_at_10": 0.0102, "precision_at_25": 0.0091, "average_rewards_at_k": 0.653 }
}
The above metrics are described below using the following terms:
• Relevant recommendation refers to a recommendation that matches a value in the testing data for the particular user.
• Rank refers to the position of a recommended item in the list of recommendations. Position 1 (the top of the list) is presumed to be the most relevant to the user.
• Query refers to the internal equivalent of a GetRecommendations (p. 423) call.
For each metric, higher numbers are better.
coverage
An evaluation metric that tells you the proportion of unique items that Amazon Personalize might recommend using your model out of the total number of unique items in Interactions and Items datasets. To make sure Amazon Personalize recommends more of your items, use a model with a higher coverage score. Recipes that feature item exploration, such as User-Personalization, have higher coverage than those that don’t, such as popularity-count.
mean reciprocal rank at 25
An evaluation metric that assesses the relevance of a model’s highest ranked recommendation.
Amazon Personalize calculates this metric using the average accuracy of the model when ranking the most relevant recommendation out of the top 25 recommendations over all requests for recommendations.
This metric is useful if you're interested in the single highest ranked recommendation.
normalized discounted cumulative gain (NCDG) at K (5/10/25)
An evaluation metric that tells you about the relevance of your model’s highly ranked
recommendations, where K is a sample size of 5, 10, or 25 recommendations. Amazon Personalize calculates this by assigning weight to recommendations based on their position in a ranked list, where each recommendation is discounted (given a lower weight) by a factor dependent on its position. The normalized discounted cumulative gain at K assumes that recommendations that are lower on a list are less relevant than recommendations higher on the list.
Amazon Personalize uses a weighting factor of 1/log(1 + position), where the top of the list is position 1.
Step 4: Evaluating a solution version with metrics
This metric rewards relevant items that appear near the top of the list, because the top of a list usually draws more attention.
precision at K
An evaluation metric that tells you how relevant your model’s recommendations are based on a sample size of K (5, 10, or 25) recommendations. Amazon Personalize calculates this metric based on the number of relevant recommendations out of the top K recommendations, divided by K, where K is 5, 10, or 25.
This metric rewards precise recommendation of the relevant items.
average_rewards_at_k
When you create a solution version (train a model) for a solution with an optimization objective, Amazon Personalize generates an average_rewards_at_k metric. The score for average_rewards_at_k tells you how well the solution version performs in achieving your objective. To calculate this metric, Amazon Personalize calculates the rewards for each user as follows:
rewards_per_user = total rewards from the user's interactions with their top 25 reward generating recommendations / total rewards from the user's interactions with recommendations
The final average_rewards_at_k is the average of all rewards_per_user normalized to be a decimal value less than or equal to 1 and greater than 0. The closer the value is to 1, the more gains on average per user you can expect from recommendations.
For example, if your objective is to maximize revenue from clicks, Amazon Personalize calculates each user score by dividing total revenue generated by the items the user clicked from their top 25 most expensive recommendations by the revenue from all of the recommended items the user clicked. Amazon Personalize then returns a normalized average of all user scores. The closer the average_rewards_at_k is to 1, the more revenue on average you can expect to gain per user from recommendations.
For more information see Optimizing a solution for an additional objective (p. 161).
hit (hit at K)
If you trained the solution version with a USER_SEGMENTATION recipe, the average number of users in the predicted top relevant K results that match the actual users. Actual users are the users who actually interacted with the items in the test set. K is the top 1% of the most relevant users. The higher the value the more accurate the predictions.
recall (recall at K)
If you trained the solution version with a USER_SEGMENTATION recipe, the average percentage of predicted users in the predicted top relevant K results that match the actual users. Actual users are the users who actually interacted with the items in the test set. K is the top 1% of the most relevant users. The higher the value, the more accurate the predictions.
Example
The following is a simple example where, to generate metrics, a solution version produces a list of recommendations for a specific user. The second and fifth recommendations match records in the testing data for this user. These are the relevant recommendations. If K is set at 5, the following metrics are generated for the user.
reciprocal_rank Calculation: 1/2
Creating a campaign
Result: 0.5000
normalized_discounted_cumulative_gain_at_5
Calculation: (1/log(1 + 2) + 1/log(1 + 5)) / (1/log(1 + 1) + 1/log(1 + 2)) Result: 0.6241
precision_at_5 Calculation: 2/5 Result: 0.4000
Now that you have evaluated your solution version, create a campaign by deploying the optimum solution version. For more information, see Creating a campaign (p. 180).
Creating a campaign
For real-time recommendations, after you complete Preparing and importing data (p. 102) and Creating a solution (p. 125), you are ready to deploy your solution version to generate
recommendations. You deploy a solution version by creating an Amazon Personalize campaign. If you are getting batch recommendations, you don't need to create a campaign. For more information see Getting batch recommendations and user segments (p. 193).
A campaign is a deployed solution version (trained model) with provisioned dedicated transaction capacity for creating real-time recommendations for your application users. After you create a campaign, you use the GetRecommendations (p. 423) or GetPersonalizedRanking (p. 419) API operations to get recommendations.
You create a campaign with the Amazon Personalize console, AWS Command Line Interface (AWS CLI), or AWS SDKs.
Important
If you manually retrain your solution version or want to change your campaign settings, you must update your campaign. For more information see Updating a campaign (p. 184).
Topics
• Minimum provisioned transactions per second and auto-scaling (p. 180)
• Creating a campaign (console) (p. 181)
• Creating a campaign (AWS CLI) (p. 181)
• Creating a campaign (AWS SDKs) (p. 182)
• Updating a campaign (p. 184)
Minimum provisioned transactions per second and auto-scaling
When you create an Amazon Personalize campaign, you specify a dedicated transaction capacity for creating real-time recommendations for your application users. A transaction is a single GetRecommendations or GetPersonalizedRanking call. Transactions per second (TPS) is the throughput and unit of billing for Amazon Personalize. The minimum provisioned TPS
(minProvisionedTPS) specifies the baseline throughput provisioned by Amazon Personalize, and thus, the minimum billing charge.
Creating a campaign (console)
If your TPS increases beyond minProvisionedTPS, Amazon Personalize auto-scales the provisioned capacity up and down, but never below minProvisionedTPS. There's a short time delay while the capacity is increased that might cause loss of transactions.
The actual TPS used is calculated as the average requests/second within a 5-minute window. You pay for maximum of the minimum provisioned TPS or the actual TPS. We recommend starting with a low minProvisionedTPS, track your usage using Amazon CloudWatch metrics, and then increase the minProvisionedTPS as necessary.
Creating a campaign (console)
After your solution version status is Active you are ready to deploy it with an Amazon Personalize campaign.
To create a campaign (console)
1. Open the Amazon Personalize console at https://console.aws.amazon.com/personalize/home and sign into your account.
2. Choose the dataset group with the solution version you want to deploy.
3. In the navigation pane, choose Campaigns.
4. On the Campaigns page, choose Create campaign.
5. On the Create new campaign page, for Campaign details, provide the following information:
• Campaign name: Enter the name of the campaign. The text you enter here appears on the Campaign dashboard and details page.
• Solution: Choose the solution that you just created.
• Solution version ID: Choose the ID of the solution version that you just created.
• Minimum provisioned transactions per second: Set the minimum provisioned transactions per second that Amazon Personalize supports. For more information, see Minimum provisioned transactions per second and auto-scaling (p. 180).
6. If you used the User-Personalization recipe, in Campaign configuration optionally enter values for the Exploration weight and Exploration item age cut off. For more information see User-Personalization (p. 127).
7. Choose Create campaign.
8. On the campaign details page, when the campaign status is Active, you can use the campaign to get recommendations and record impressions. For more information, see Getting recommendations (Custom dataset group) (p. 186).
The campaign is ready when its status is ACTIVE. If you retrain your solution version or want to change your campaign settings, you must update your campaign. For more information see Updating a campaign (p. 184).
Creating a campaign (AWS CLI)
After your solution version status is Active, you are ready to deploy it with an Amazon Personalize campaign. Use the following create-campaign AWS CLI command to create a campaign that deploys a solution version trained using the User-Personalization recipe. Give the campaign a name and specify the solution version ARN (Amazon Resource Name). Optionally change the minProvisionedTPS if your use case requires a higher provisioned capacity. The minimum value is 1.
The campaign-config parameters are specific to the recipe that you used to train the solution version (for more information about recipes see Step 1: Choosing a recipe (p. 125)). The example uses the
Creating a campaign (AWS SDKs)
following User-Personalization recipe specific itemExplorationConfig fields with their default values: explorationWeight and explorationItemAgeCutOff. If you omit the campaign-config parameter, the default values apply. For more information about the itemExplorationConfig fields, see the Properties and hyperparameters (p. 128) for the User-Personalization (p. 127) recipe.
aws personalize create-campaign \ --name campaign name \
--solution-version-arn solution version arn \ --min-provisioned-tps 1 \
--campaign-config "{\"itemExplorationConfig\":{\"explorationWeight\":\"0.3\",
\"explorationItemAgeCutOff\":\"30\"}}"
The campaign is ready when its status is ACTIVE. To get the current status, call DescribeCampaign (p. 339) and check that the status field is ACTIVE.
If you retrain your solution version or want to change your campaign settings, you must update your campaign. For more information see Updating a campaign (p. 184).
Amazon Personalize provides operations for managing campaigns such as ListCampaigns (p. 375) to list the campaigns you have created. You can delete a campaign by calling DeleteCampaign (p. 317). If you delete a campaign, the solution versions that are part of the campaign are not deleted.
After you have created your campaign, use it to make recommendations. For more information, see Getting recommendations (Custom dataset group) (p. 186).
Creating a campaign (AWS SDKs)
After your solution version status is Active you are ready to deploy it with an Amazon Personalize campaign. Use the following code to create a campaign with the AWS SDK for Python (Boto3) or AWS SDK for Java 2.x.
The example code uses the following parameters (for a complete list of parameters, see CreateCampaign (p. 283)):
• A name for the campaign.
• The solution version's ARN (Amazon Resource Name).
• The Minimum provisioned TPS (p. 180) the campaign will support (the minimum value for this parameter is 1).
• Optional campaign configuration parameters itemExplorationWeight and explorationItemAgeCutOff.
The campaign configuration parameters are specific to the recipe that you used to train the solution version (for more information about recipes see Step 1: Choosing a recipe (p. 125)). In this example, the itemExplorationWeight and explorationItemAgeCutOff parameters are specific to the User-Personalization (p. 127) recipe. The default itemExplorationWeight is 0.3 and the default
explorationItemAgeCutOff is 30. If you leave out campaign configuration parameters, the default values apply.
SDK for Python (Boto3)
import boto3
personalize = boto3.client('personalize') response = personalize.create_campaign(
Creating a campaign (AWS SDKs)
name = 'campaign name',
solutionVersionArn = 'solution version arn', minProvisionedTPS = 1,
campaignConfig = {"itemExplorationConfig": {"explorationWeight": "0.3", "explorationItemAgeCutOff": "30"}}
)
arn = response['campaignArn']
description = personalize.describe_campaign(campaignArn = arn)['campaign']
print('Name: ' + description['name']) print('ARN: ' + description['campaignArn']) print('Status: ' + description['status']) SDK for Java 2.x
public static void createCampaign(PersonalizeClient personalizeClient, String campaignName,
//Optional code to instantiate a HashMap and add the explorationWeight and explorationItemAgeCutOff values.
//Remove if you aren't using User-Personaliztion.
Map<String,String> itemExploration = new HashMap<String,String>();
itemExploration.put("explorationWeight", itemExplorationWeight);
itemExploration.put("explorationItemAgeCutOff", explorationItemAgeCutOff);
try {
// Build a User-Personalization recipe specific campaignConfig object with the itemExploration map.
// CampaignConfig construction will vary by recipe.
CampaignConfig campaignConfig = CampaignConfig.builder() .itemExplorationConfig(itemExploration)
.build();
// build the createCampaignRequest
CreateCampaignRequest createCampaignRequest = CreateCampaignRequest.builder() .name(campaignName)
CreateCampaignResponse campaignResponse =
personalizeClient.createCampaign(createCampaignRequest);
String campaignArn = campaignResponse.campaignArn();
DescribeCampaignRequest campaignRequest = DescribeCampaignRequest.builder() .campaignArn(campaignArn)
.build();
DescribeCampaignResponse campaignResponse = personalizeClient.describeCampaign(campaignRequest);
Campaign newCampaign = campaignResponse.campaign();
System.out.println("The Campaign status is " + newCampaign.status());
} catch (PersonalizeException e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
}
Updating a campaign
}
The campaign is ready when its status is ACTIVE. To get the current status, call DescribeCampaign (p. 339) and check that the status field is ACTIVE.
If you manually retrain your solution version or want to change your campaign settings, you must update your campaign. For more information see Updating a campaign (p. 184).
Amazon Personalize provides operations for managing campaigns such as ListCampaigns (p. 375) to list the campaigns you have created. You can delete a campaign by calling DeleteCampaign (p. 317). If you delete a campaign, the solution versions that are part of the campaign are not deleted.
After you have created your campaign, use it to make recommendations. For more information, see Getting recommendations (Custom dataset group) (p. 186).
Updating a campaign
To deploy a retrained solution version with an existing campaign or to change your campaign's Minimum provisioned TPS (p. 180) or campaign configuration, you must manually update the campaign.
With User-Personalization, Amazon Personalize automatically updates your latest solution version (trained with trainingMode set to FULL) every two hours to include new items in recommendations, and your campaign automatically uses the updated solution version. Manually update a campaign only when you manually retrain the solution version with trainingMode set to FULL, or when you want to make changes to your campaign's minProvisionedTPS or campaign configuration. For more information on automatic updates with the User-Personalization recipe see Automatic updates (p. 127).
You manually update a campaign with the Amazon Personalize console, AWS Command Line Interface (AWS CLI), or AWS SDKs.
Topics
• Updating a campaign (console) (p. 184)
• Updating a campaign (AWS CLI) (p. 185)
• Updating a campaign (AWS SDKs) (p. 185)
Updating a campaign (console)
To deploy a manually retrained solution version or make changes to your campaign configuration, you must update your campaign.
To update a campaign (console)
1. Open the Amazon Personalize console at https://console.aws.amazon.com/personalize/home and sign into your account.
2. Choose the dataset group with the campaign you want to update.
3. In the navigation pane, choose Campaigns.
4. On the Campaigns page, choose the campaign you want to update.
5. On the campaign details page, choose Update.
6. On the Update campaign page, make your changes. For example, if you are deploying a retrained solution version, for Solution version ID, choose the identification number for the new solution version.
7. Choose Update. Amazon Personalize updates the campaign to use the new solution version and any changed configurations.
Updating a campaign
Updating a campaign (AWS CLI)
To deploy a new solution version, change your campaign's Minimum provisioned TPS (p. 180), or change your campaign's configuration, you must update your campaign. Use the following update-campaign command to update a update-campaign to use a new solution version with the AWS CLI.
Replace campaign arn with the Amazon Resource Name (ARN) of the campaign you want to update.
Replace new solution version arn with the solution version you want to deploy.
aws personalize update-campaign \ --campaign-arn campaign arn \
--solution-version-arn new solution version arn \ --min-provisioned-tps 1
Updating a campaign (AWS SDKs)
To deploy a new solution version, change your campaign's Minimum provisioned TPS (p. 180) or change your campaign's configuration, you must update your campaign. Use the following code to update a campaign with the SDK for Python (Boto3) or SDK for Java 2.x. For a complete list of parameters, see UpdateCampaign (p. 408).
SDK for Python (Boto3)
Use the following update_campaign method to deploy a new solution version. Replace campaign arn with the Amazon Resource Name (ARN) of the campaign you want to update, replace the new solution version arn with the new solution version ARN and optionally change the minProvisionedTPS.
import boto3
personalize = boto3.client('personalize') response = personalize.update_campaign(
campaignArn = 'campaign arn',
solutionVersionArn = 'new solution version arn', minProvisionedTPS = 1,
)
arn = response['campaignArn']
description = personalize.describe_campaign(campaignArn = arn)['campaign']
print('Name: ' + description['name']) print('ARN: ' + description['campaignArn']) print('Status: ' + description['status'])
SDK for Java 2.x
Use the following updateCampaign method to update a campaign to use a new solution version.
Pass as parameters an Amazon Personalize service client, the new solution version's Amazon Resource Name (ARN), and the Minimum provisioned TPS (p. 180).
public static void updateCampaign(PersonalizeClient personalizeClient, String campaignArn,
String solutionVersionArn, Integer minProvisionedTPS) { try {
// build the updateCampaignRequest
UpdateCampaignRequest updateCampaignRequest = UpdateCampaignRequest.builder()
Getting recommendations
.campaignArn(campaignArn)
.solutionVersionArn(solutionVersionArn) .minProvisionedTPS(minProvisionedTPS) .build();
// update the campaign
personalizeClient.updateCampaign(updateCampaignRequest);
DescribeCampaignRequest campaignRequest = DescribeCampaignRequest.builder() .campaignArn(campaignArn)
.build();
DescribeCampaignResponse campaignResponse = personalizeClient.describeCampaign(campaignRequest);
Campaign updatedCampaign = campaignResponse.campaign();
System.out.println("The Campaign status is " + updatedCampaign.status());
} catch (PersonalizeException e) {
System.err.println(e.awsErrorDetails().errorMessage());
System.exit(1);
} }
Getting recommendations (Custom dataset group)
With Amazon Personalize Custom dataset groups, you can get recommendations in real-time or you can get batch recommendations. For real-time recommendations, you must create a campaign before you get recommendations. For batch recommendations, you don't need to create a campaign. For information on campaigns see Creating a campaign (p. 180).
The following topics explain how and when to use each recommendation type.
The following topics explain how and when to use each recommendation type.