Each cell has a location in a table, with the first cell being 1,1. In the preceding example, the cell with the value Name is at row 1, column 1. The cell with the value 123 Any Town is at row 2, column 2. A cell block object contains this information in the RowIndex and ColumnIndex fields. The child list contains the IDs for the WORD Block objects that contain the text that's within the cell. The words in the list are in the order in which they're detected, from the top left of the cell to the bottom right of the cell. In the preceding example, the cell has a child ID with the value e9108c8e-0167-4482-989e-8b6cd3c3653e. The following output is for the WORD Block with the ID value of e9108c8e-0167-4482-989e-8b6cd3c3653e:
"Geometry": {...},
"Text": "Name",
"TextType": "Printed",
"BlockType": "WORD",
"Confidence": 99.81139373779297,
"Id": "e9108c8e-0167-4482-989e-8b6cd3c3653e"
},
Selection Elements
Amazon Textract can detect selection elements such as option buttons (radio buttons) and check boxes on a document page. Selection elements can be detected in form data (p. 14) and in tables (p. 16).
For example, when the following table is detected on a form, Amazon Textract detects the check boxes in the table cells.
Agree Neutral Disagree
Good Service ☑ ☐ ☐
Easy to Use ☐ ☑ ☐
Fair Price ☑ ☐ ☐
Detected selection elements are returned as Block (p. 257) objects in the responses from AnalyzeDocument (p. 209) and GetDocumentAnalysis (p. 226).
Note
You can use the FeatureTypes input parameter to retrieve information about key-value pairs, tables, or both. For example, if you filter on tables, the response includes the selection elements that are detected in tables. Selection elements that are detected in key-value pairs aren't included in the response.Information about a selection element is contained in a Block object of type SELECTION_ELEMENT.
To determine the status of a selectable element, use the SelectionStatus field of the
SELECTION_ELEMENT block. The status can be either SELECTED or NOT_SELECTED. For example, the value of SelectionStatus for the previous image is SELECTED.
A SELECTION_ELEMENT Block object is associated with either a key-value pair or a table cell. A SELECTION_ELEMENT Block object contains bounding box information for a selection element in the Geometry field. A SELECTION_ELEMENT Block object isn't a child of a PAGE Block object.
Form Data (Key-Value Pairs)
A key-value pair is used to represent a selection element that's detected on a form. The KEY block contains the text for the selection element. The VALUE block contains the SELECTION_ELEMENT block. The following diagram shows how selection elements are represented by the section called
“Block” (p. 257) objects.
For more information about key-value pairs, see Form Data (Key-Value Pairs) (p. 14).
The following JSON snippet shows the key for a key-value pair that contains a selection element (male
☑). The child ID (Id bd14cfd5-9005-498b-a7f3-45ceb171f0ff) is the ID of the WORD block that contains the text for the selection element (male). The value ID (Id 24aaac7f-fcce-49c7-a4f0-3688b05586d4) is the ID of the VALUE block that contains the SELECTION_ELEMENT block object.
{
"Relationships": [ {
"Type": "VALUE", "Ids": [
"24aaac7f-fcce-49c7-a4f0-3688b05586d4" // Value containing Selection Element
] }, {
"Type": "CHILD", "Ids": [
"bd14cfd5-9005-498b-a7f3-45ceb171f0ff" // WORD - male ]
} ],
"Confidence": 94.15619659423828, "Geometry": {
"BoundingBox": {
"Width": 0.022914813831448555, "Top": 0.08072036504745483, "Left": 0.18966935575008392, "Height": 0.014860388822853565 },
"Polygon": [ {
"Y": 0.08072036504745483, "X": 0.18966935575008392 },
{
"Y": 0.08072036504745483, "X": 0.21258416771888733 },
{
"Y": 0.09558075666427612, "X": 0.21258416771888733 },
{
"Y": 0.09558075666427612, "X": 0.18966935575008392 }
] },
"BlockType": "KEY_VALUE_SET", "EntityTypes": [
"KEY"
],
"Id": "a118dc43-d5f7-49a2-a20a-5f876d9ffd79"
}
The following JSON snippet is the WORD block for the word Male. The WORD block also has a parent LINE block.
{ "Geometry": {
"BoundingBox": {
"Width": 0.022464623674750328, "Top": 0.07842985540628433, "Left": 0.18863198161125183, "Height": 0.01617223583161831 },
"Polygon": [ {
"Y": 0.07842985540628433, "X": 0.18863198161125183 },
{
"Y": 0.07842985540628433, "X": 0.2110965996980667 },
{
"Y": 0.09460209310054779, "X": 0.2110965996980667 },
{
"Y": 0.09460209310054779, "X": 0.18863198161125183 }
] },
"Text": "Male", "BlockType": "WORD",
"Confidence": 54.06439208984375,
"Id": "bd14cfd5-9005-498b-a7f3-45ceb171f0ff"
},
The VALUE block has a child (Id f2f5e8cd-e73a-4e99-a095-053acd3b6bfb) that is the SELECTION_ELEMENT block.
{ "Relationships": [ {
"Type": "CHILD", "Ids": [
"f2f5e8cd-e73a-4e99-a095-053acd3b6bfb" // Selection element ]
} ],
"Confidence": 94.15619659423828, "Geometry": {
"BoundingBox": {
"Width": 0.017281491309404373, "Top": 0.07643391191959381, "Left": 0.2271782010793686, "Height": 0.026274094358086586 },
"Polygon": [ {
"Y": 0.07643391191959381, "X": 0.2271782010793686 },
{
"Y": 0.07643391191959381, "X": 0.24445968866348267 },
{
"Y": 0.10270800441503525, "X": 0.24445968866348267 },
{
"Y": 0.10270800441503525, "X": 0.2271782010793686 }
] },
"BlockType": "KEY_VALUE_SET", "EntityTypes": [
"VALUE"
],
"Id": "24aaac7f-fcce-49c7-a4f0-3688b05586d4"
}, }
The following JSON is the SELECTION_ELEMENT block. The value of SelectionStatus indicates that the check box is selected.
{
"Geometry": {
"BoundingBox": {
"Width": 0.020316146314144135, "Top": 0.07575977593660355, "Left": 0.22590067982673645, "Height": 0.027631107717752457 },
"Polygon": [ {
"Y": 0.07575977593660355, "X": 0.22590067982673645 },
{
"Y": 0.07575977593660355, "X": 0.2462168186903 },
{
"Y": 0.1033908873796463, "X": 0.2462168186903 },
{
"Y": 0.1033908873796463, "X": 0.22590067982673645 }
] },
"BlockType": "SELECTION_ELEMENT", "SelectionStatus": "SELECTED", "Confidence": 74.14942932128906,
"Id": "f2f5e8cd-e73a-4e99-a095-053acd3b6bfb"
}
Table Cells
Amazon Textract can detect selection elements inside a table cell. For example, the cells in the following table have check boxes.
Agree Neutral Disagree
Good Service ☑ ☐ ☐
Easy to Use ☐ ☑ ☐
Fair Price ☑ ☐ ☐
A CELL block can contain child SELECTION_ELEMENT objects for selection elements, as well as child WORD blocks for detected text.
For more information about tables, see Tables (p. 16).
The TABLE Block object for the previous table looks similar to this.
{ "Geometry": {...}, "Relationships": [ {
"Type": "CHILD", "Ids": [
"652c09eb-8945-473d-b1be-fa03ac055928", "37efc5cc-946d-42cd-aa04-e68e5ed4741d", "4a44940a-435a-4c5c-8a6a-7fea341fa295", "2de20014-9a3b-4e26-b453-0de755144b1a", "8ed78aeb-5c9a-4980-b669-9e08b28671d2", "1f8e1c68-2c97-47b2-847c-a19619c02ca9", "9927e1d1-6018-4960-ac17-aadb0a94f4d9", "68f0ed8b-a887-42a5-b618-f68b494a6034", "fcba16e0-6bd7-4ea5-b86e-36e8330b68ea", "2250357c-ae34-4ed9-86da-45dac5a5e903",
"c63ad40d-5a14-4646-a8df-2d4304213dbc", // Cell "2b8417dc-e65f-4fcd-aa0f-61a23f1e8cb0",
"26c62932-72f0-4dc2-9893-1ae27829c060", "27f291cc-abf4-4c23-aa24-676abe99cb1e", "7e5ce028-1bcd-4d9f-ad42-15ac181c5b47", "bf32e3d2-efa2-4fc1-b09b-ab9cc52ff734"
] } ],
"BlockType": "TABLE",
"Confidence": 99.99993896484375,
"Id": "f66eac36-2e74-406e-8032-14d1c14e0b86"
}
The CELL BLOCK object (Id c63ad40d-5a14-4646-a8df-2d4304213dbc) for the cell that contains the check box Good Service looks like the following. It includes a child Block (Id = 26d122fd-c5f4-4b53-92c4-0ae92730ee1e) that is the SELECTION_ELEMENT Block object for the check box.
{
"Geometry": {...}, "Relationships": [ {
"Type": "CHILD", "Ids": [
"26d122fd-c5f4-4b53-92c4-0ae92730ee1e" // Selection Element ]
} ],
"Confidence": 79.741689682006836,
"RowSpan": 1, "RowIndex": 3, "ColumnIndex": 3, "ColumnSpan": 1, "BlockType": "CELL",
"Id": "c63ad40d-5a14-4646-a8df-2d4304213dbc"
}
The SELECTION_ELEMENT Block object for the check box is as follows. The value of SelectionStatus indicates that the check box is selected.
{
"Geometry": {...},
"BlockType": "SELECTION_ELEMENT", "SelectionStatus": "SELECTED", "Confidence": 88.79517364501953,
"Id": "26d122fd-c5f4-4b53-92c4-0ae92730ee1e"
}
Invoice and Receipt Response Objects
When you submit an invoice or a receipt to the AnalyzeExpense API, it returns a series of
ExpenseDocuments objects. Each ExpenseDocument is further separated into LineItemGroups and SummaryFields. Most invoices and receipts contain information such as the vendor name, receipt number, receipt date, or total amount. AnalyzeExpense returns this information under SummaryFields.
Receipts and invoices also contain details about the items purchased. The AnalyzeExpense API returns this information under LineItemGroups. The ExpenseIndex field uniquely identifies the expense, and associates the appropriate SummaryFields and LineItemGroups detected in that expense.
The most granular level of data in the AnalyzeExpense response consists of Type, ValueDetection, and LabelDetection (Optional). The individual entities are:
• Type (p. 24): Refers to what kind of information is detected on a high level.
• LabelDetection (p. 24): Refers to the label of an associated value within the text of the document.
LabelDetection is optional and only returned if the label is written.
• ValueDetection (p. 24): Refers to the value of the label or type returned.
The AnalyzeExpense API also detects ITEM, QUANTITY, and PRICE within line items as normalized fields.
If there is other text in a line item on the receipt image such as SKU or detailed description, it will be included in the JSON as EXPENSE_ROW as shown in the below example:
{
"Type": {
"Text": "EXPENSE_ROW",
"Confidence": 99.95216369628906 },
"ValueDetection": {
"Text": "Banana 5 $2.5", "Geometry": {
… },
"Confidence": 98.11214447021484 }
The example above shows how the AnalyzeExpense API returns the entire row on a receipt that contains line item information about 5 bananas sold for $2.5.
Type
Following is an example of the standard or normalized type of the key-value pair:
{
"PageNumber": 1, "Type": {
"Text": "VENDOR_NAME", "Confidence": 70.0 },
"ValueDetection": { "Geometry": { ... }, "Text": "AMAZON",
"Confidence": 87.89806365966797 }
}
The receipt did not have “Vendor Name” explicitly listed. However, the Analyze Expense API recognized the document as a receipt and categorized the value “AMAZON” as Type VENDOR_NAME.
LabelDetection
Following is an example of text as it is shown on a customer document page:
{
"PageNumber": 1, "Type": {
"Text": "OTHER", "Confidence": 70.0 },
"LabelDetection": { "Geometry": { ... }, "Text": "CASHIER",
"Confidence": 88.19171142578125 },
"ValueDetection": { "Geometry": { ... }, "Text": "Mina",
"Confidence": 87.89806365966797 }
}
The example document contained “CASHIER Mina”. The Analyze Expense API extracted the as-is value and returns it under LabelDetection. For implied values such as “Vendor Name”, where the “key”
is not explicitly shown in the receipt, LabelDetection will not be included in the AnalyzeExpense element. In such cases, the AnalyzeExpense API does not return LabelDetection.
ValueDetection
The following is an example shows the “value” of the key-value pair.
{
"PageNumber": 1, "Type": {
"Text": "OTHER",
"Confidence": 70.0 },
"LabelDetection": { "Geometry": { ... }, "Text": "CASHIER",
"Confidence": 88.19171142578125 },
"ValueDetection": { "Geometry": { ... }, "Text": "Mina",
"Confidence": 87.89806365966797 }
}
In the example, the document contained “CASHIER Mina”. The AnalyzeExpense API detected the Cashier value as Mina and returned it under ValueDetection.