AmazonopensearchserviceBufferingHints
Describes the buffering to perform before delivering data to the Amazon OpenSearch Service destination.
Contents
IntervalInSeconds
Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination. The default value is 300 (5 minutes).
Type: Integer
Valid Range: Minimum value of 60. Maximum value of 900.
Required: No SizeInMBs
Buffer incoming data to the specified size, in MBs, before delivering it to the destination. The default value is 5.
We recommend setting this parameter to a value greater than the amount of data you typically ingest into the delivery stream in 10 seconds. For example, if you typically ingest data at 1 MB/sec, the value should be 10 MB or higher.
Type: Integer
Valid Range: Minimum value of 1. Maximum value of 100.
Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for Ruby V3
AmazonopensearchserviceDestinationConfiguration
AmazonopensearchserviceDestinationConfiguration
Describes the configuration of a destination in Amazon OpenSearch Service
Contents
BufferingHints
The buffering options. If no value is specified, the default values for AmazonopensearchserviceBufferingHints are used.
Type: AmazonopensearchserviceBufferingHints (p. 70) object Required: No
CloudWatchLoggingOptions
Describes the Amazon CloudWatch logging options for your delivery stream.
Type: CloudWatchLoggingOptions (p. 82) object Required: No
ClusterEndpoint
The endpoint to use when communicating with the cluster. Specify either this ClusterEndpoint or the DomainARN field.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: https:.*
Required: No DomainARN
The ARN of the Amazon OpenSearch Service domain. The IAM role must have permissions for DescribeElasticsearchDomain, DescribeElasticsearchDomains, and DescribeElasticsearchDomainConfig after assuming the role specified in RoleARN.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: arn:.*
Required: No IndexName
The ElasticsearAmazon OpenSearch Service index name.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 80.
Pattern: .*
Required: Yes
Contents
IndexRotationPeriod
The Amazon OpenSearch Service index rotation period. Index rotation appends a timestamp to the IndexName to facilitate the expiration of old data.
Type: String
Valid Values: NoRotation | OneHour | OneDay | OneWeek | OneMonth Required: No
ProcessingConfiguration
Describes a data processing configuration.
Type: ProcessingConfiguration (p. 144) object Required: No
RetryOptions
The retry behavior in case Kinesis Data Firehose is unable to deliver documents to Amazon OpenSearch Service. The default value is 300 (5 minutes).
Type: AmazonopensearchserviceRetryOptions (p. 80) object Required: No
RoleARN
The Amazon Resource Name (ARN) of the IAM role to be assumed by Kinesis Data Firehose for calling the Amazon OpenSearch Service Configuration API and for indexing documents.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: arn:.*
Required: Yes S3BackupMode
Defines how documents should be delivered to Amazon S3. When it is set to FailedDocumentsOnly, Kinesis Data Firehose writes any documents that could not be indexed to the configured Amazon S3 destination, with AmazonOpenSearchService-failed/ appended to the key prefix. When set to AllDocuments, Kinesis Data Firehose delivers all incoming records to Amazon S3, and also writes failed documents with AmazonOpenSearchService-failed/ appended to the prefix.
Type: String
Valid Values: FailedDocumentsOnly | AllDocuments Required: No
S3Configuration
Describes the configuration of a destination in Amazon S3.
Type: S3DestinationConfiguration (p. 159) object Required: Yes
TypeName
The Amazon OpenSearch Service type name. For Elasticsearch 6.x, there can be only one type per index. If you try to specify a new type for an existing index that already has another type, Kinesis Data Firehose returns an error during run time.
See Also
Type: String
Length Constraints: Minimum length of 0. Maximum length of 100.
Pattern: .*
Required: No VpcConfiguration
The details of the VPC of the Amazon ES destination.
Type: VpcConfiguration (p. 177) object Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for Ruby V3
AmazonopensearchserviceDestinationDescription
AmazonopensearchserviceDestinationDescription
The destination description in Amazon OpenSearch Service.
Contents
BufferingHints
The buffering options.
Type: AmazonopensearchserviceBufferingHints (p. 70) object Required: No
CloudWatchLoggingOptions
Describes the Amazon CloudWatch logging options for your delivery stream.
Type: CloudWatchLoggingOptions (p. 82) object Required: No
ClusterEndpoint
The endpoint to use when communicating with the cluster. Kinesis Data Firehose uses either this ClusterEndpoint or the DomainARN field to send data to Amazon OpenSearch Service.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: https:.*
Required: No DomainARN
The ARN of the Amazon OpenSearch Service domain.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: arn:.*
Required: No IndexName
The Amazon OpenSearch Service index name.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 80.
Pattern: .*
Required: No IndexRotationPeriod
The Amazon OpenSearch Service index rotation period Type: String
Contents
Valid Values: NoRotation | OneHour | OneDay | OneWeek | OneMonth Required: No
ProcessingConfiguration
Describes a data processing configuration.
Type: ProcessingConfiguration (p. 144) object Required: No
RetryOptions
The Amazon OpenSearch Service retry options.
Type: AmazonopensearchserviceRetryOptions (p. 80) object Required: No
RoleARN
The Amazon Resource Name (ARN) of the AWS credentials.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: arn:.*
Required: No S3BackupMode
The Amazon S3 backup mode.
Type: String
Valid Values: FailedDocumentsOnly | AllDocuments Required: No
S3DestinationDescription
Describes a destination in Amazon S3.
Type: S3DestinationDescription (p. 161) object Required: No
TypeName
The Amazon OpenSearch Service type name. This applies to Elasticsearch 6.x and lower versions. For Elasticsearch 7.x and OpenSearch Service 1.x, there's no value for TypeName.
Type: String
Length Constraints: Minimum length of 0. Maximum length of 100.
Pattern: .*
Required: No
VpcConfigurationDescription
The details of the VPC of the Amazon ES destination.
See Also
Type: VpcConfigurationDescription (p. 179) object Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for Ruby V3
AmazonopensearchserviceDestinationUpdate
AmazonopensearchserviceDestinationUpdate
Describes an update for a destination in Amazon OpenSearch Service.
Contents
BufferingHints
The buffering options. If no value is specified, AmazonopensearchBufferingHints object default values are used.
Type: AmazonopensearchserviceBufferingHints (p. 70) object Required: No
CloudWatchLoggingOptions
Describes the Amazon CloudWatch logging options for your delivery stream.
Type: CloudWatchLoggingOptions (p. 82) object Required: No
ClusterEndpoint
The endpoint to use when communicating with the cluster. Specify either this ClusterEndpoint or the DomainARN field.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: https:.*
Required: No DomainARN
The ARN of the Amazon OpenSearch Service domain. The IAM role must have permissions for DescribeDomain, DescribeDomains, and DescribeDomainConfig after assuming the IAM role specified in RoleARN.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: arn:.*
Required: No IndexName
The Amazon OpenSearch Service index name.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 80.
Pattern: .*
Required: No
Contents
IndexRotationPeriod
The Amazon OpenSearch Service index rotation period. Index rotation appends a timestamp to IndexName to facilitate the expiration of old data.
Type: String
Valid Values: NoRotation | OneHour | OneDay | OneWeek | OneMonth Required: No
ProcessingConfiguration
Describes a data processing configuration.
Type: ProcessingConfiguration (p. 144) object Required: No
RetryOptions
The retry behavior in case Kinesis Data Firehose is unable to deliver documents to Amazon OpenSearch Service. The default value is 300 (5 minutes).
Type: AmazonopensearchserviceRetryOptions (p. 80) object Required: No
RoleARN
The Amazon Resource Name (ARN) of the IAM role to be assumed by Kinesis Data Firehose for calling the Amazon OpenSearch Service Configuration API and for indexing documents.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: arn:.*
Required: No S3Update
Describes an update for a destination in Amazon S3.
Type: S3DestinationUpdate (p. 163) object Required: No
TypeName
The Amazon OpenSearch Service type name. For Elasticsearch 6.x, there can be only one type per index. If you try to specify a new type for an existing index that already has another type, Kinesis Data Firehose returns an error during runtime.
If you upgrade Elasticsearch from 6.x to 7.x and don’t update your delivery stream, Kinesis Data Firehose still delivers data to Elasticsearch with the old index name and type name. If you want to update your delivery stream with a new index name, provide an empty string for TypeName.
Type: String
Length Constraints: Minimum length of 0. Maximum length of 100.
Pattern: .*
See Also
Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for Ruby V3
AmazonopensearchserviceRetryOptions
AmazonopensearchserviceRetryOptions
Configures retry behavior in case Kinesis Data Firehose is unable to deliver documents to Amazon OpenSearch Service.
Contents
DurationInSeconds
After an initial failure to deliver to Amazon OpenSearch Service, the total amount of time during which Kinesis Data Firehose retries delivery (including the first attempt). After this time has elapsed, the failed documents are written to Amazon S3. Default value is 300 seconds (5 minutes). A value of 0 (zero) results in no retries.
Type: Integer
Valid Range: Minimum value of 0. Maximum value of 7200.
Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for Ruby V3
BufferingHints
BufferingHints
Describes hints for the buffering to perform before delivering data to the destination. These options are treated as hints, and therefore Kinesis Data Firehose might choose to use different values when it is optimal. The SizeInMBs and IntervalInSeconds parameters are optional. However, if specify a value for one of them, you must also provide a value for the other.
Contents
IntervalInSeconds
Buffer incoming data for the specified period of time, in seconds, before delivering it to the
destination. The default value is 300. This parameter is optional but if you specify a value for it, you must also specify a value for SizeInMBs, and vice versa.
Type: Integer
Valid Range: Minimum value of 60. Maximum value of 900.
Required: No SizeInMBs
Buffer incoming data to the specified size, in MiBs, before delivering it to the destination. The default value is 5. This parameter is optional but if you specify a value for it, you must also specify a value for IntervalInSeconds, and vice versa.
We recommend setting this parameter to a value greater than the amount of data you typically ingest into the delivery stream in 10 seconds. For example, if you typically ingest data at 1 MiB/sec, the value should be 10 MiB or higher.
Type: Integer
Valid Range: Minimum value of 1. Maximum value of 128.
Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for Ruby V3
CloudWatchLoggingOptions
CloudWatchLoggingOptions
Describes the Amazon CloudWatch logging options for your delivery stream.
Contents
Enabled
Enables or disables CloudWatch logging.
Type: Boolean Required: No LogGroupName
The CloudWatch group name for logging. This value is required if CloudWatch logging is enabled.
Type: String
Length Constraints: Minimum length of 0. Maximum length of 512.
Pattern: [\.\-_/#A-Za-z0-9]*
Required: No LogStreamName
The CloudWatch log stream name for logging. This value is required if CloudWatch logging is enabled.
Type: String
Length Constraints: Minimum length of 0. Maximum length of 512.
Pattern: [^:*]*
Required: No
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for Ruby V3
CopyCommand
CopyCommand
Describes a COPY command for Amazon Redshift.
Contents
CopyOptions
Optional parameters to use with the Amazon Redshift COPY command. For more information, see the "Optional Parameters" section of Amazon Redshift COPY command. Some possible examples that would apply to Kinesis Data Firehose are as follows:
delimiter '\t' lzop; - fields are delimited with "\t" (TAB character) and compressed using lzop.
delimiter '|' - fields are delimited with "|" (this is the default delimiter).
delimiter '|' escape - the delimiter should be escaped.
fixedwidth 'venueid:3,venuename:25,venuecity:12,venuestate:2,venueseats:6' -fields are fixed width in the source, with each width specified after every column in the table.
JSON 's3://mybucket/jsonpaths.txt' - data is in JSON format, and the path specified is the format of the data.
For more examples, see Amazon Redshift COPY command examples.
Type: String
Length Constraints: Minimum length of 0. Maximum length of 204800.
Pattern: .*
Required: No DataTableColumns
A comma-separated list of column names.
Type: String
Length Constraints: Minimum length of 0. Maximum length of 204800.
Pattern: .*
Required: No DataTableName
The name of the target table. The table must already exist in the database.
Type: String
Length Constraints: Minimum length of 1. Maximum length of 512.
Pattern: .*
Required: Yes
See Also
For more information about using this API in one of the language-specific AWS SDKs, see the following:
See Also
• AWS SDK for C++
• AWS SDK for Go
• AWS SDK for Java V2
• AWS SDK for Ruby V3