You know a ton about your application, its infrastructure, and the tracing of requests with X-Ray. Now it’s time to dig in a little deeper and augment trace information with logging and metric information that the Bookstore demo stores in Amazon ES.
Amazon Elasticsearch Service is a fully managed service that makes it easy for you to deploy, secure, and operate Elasticsearch at scale with zero down time. The service offers open-source Elasticsearch APIs, managed Kibana, and integrations with Logstash and other AWS Services, enabling you to ingest data securely from any source and search, analyze, and visualize it in real time.
Here are some of the key concepts for Amazon Elasticsearch Service.
Domain: Elasticsearch is a distributed database that runs in a cluster of nodes. Amazon Elasticsearch Service domains comprise a managed, Elasticsearch cluster, along with additional software and infrastructure to provide you with API access to Elasticsearch, and access to Kibana (a web client that we’ll explore in depth).
Index: Data in Elasticsearch is stored in indexes. You can think of an index as being akin to a table in a relational database. When you index data, or send queries, you specify the target index as part of the API call.
Shard: Each index comprises a set of shards - primary or replica. Each shard is an instance of Apache Lucene, which manages storage and processing for a subset of the documents in the index. Shards partition the data in the index so that Elasticsearch can distribute storage and processing across the nodes in the cluster.
Index pattern: When you use Amazon ES for logging and analytics, you use an index per day (usually) to store your data. That way, you can DELETE old indexes for lifecycle management. Your index names contain a root string like “appdata”, along with a timestamp. When you use Kibana, you use the root string with a wildcard to specify the set of indexes - the index pattern.
Document: A document, for logging, is a single log line, as a JSON object. More generally, a document is the main entity that you index and that you retrieve with queries. A document is like a row in a relational database.
Field: A field is a single key from a JSON document. It’s like a column in a relational database.
In this part of the lab, you will build a Kibana dashboard to monitor your application.