v4.6.2.0 RC1 Release Notes
Release Date: 06/03/2020
Notice
The v4.6.2.0 RC1 is a preview release. This release note includes information about supported and unsupported functionalities for Elasticsearch, Logstash, Kibana (ELK), and PostgreSQL. The new configuration properties for v4.6.2.0 RC1 release and the known issues are also included.
Download information
CDH, HDP, and MapR
curl -v https://preview.unraveldata.com/unravel/RPM/4.6.2/unravel-4.6.2.0-onpremise-RC1.rpm -o unravel-4.6.2.0-onpremise-RC1.rpm -uusername:password
MD5SUM: eaecd70edbf3ccd8d745e6a0530431e1
Cloud
curl -v https://preview.unraveldata.com/unravel/RPM/4.6.2/unravel-4.6.2.0-cloud-RC1.rpm -o unravel-4.6.2.0-cloud-RC1.rpm -uusername:password
MD5SUM: fd04cadb8225d8af3ed7a2b7dd7cc8ec
OnDemand - RHEL 7.6
curl -v https://preview.unraveldata.com/unravel/RPM/4.6.2/ondemand-4.6.2.0-rhel7-RC1.tar.gz -o ondemand-4.6.2.0-rhel7-RC1.tar.gz -uusername:password
MD5SUM: ae783cd1e45e131eec4196a4b19cc48c
Elasticsearch, Logstash, Kibana (ELK)
The KPIs of the ELK stack, which comprises of Elasticsearch (search engine), Logstash (ingest pipeline), and Kibana (visualization tool), can be monitored from Unravel. This section lists the supported platform, versions, capabilities, configuration properties, and APIs available for monitoring ELK metrics in Unravel.
Supported platforms and versions
CDH 6.3.3 (CentOS 7.6)
ELK version 6.8 and 5.5
Supported capabilities
Unravel supports metrics and graphically represents the KPI information for Elasticsearch, Logstash, and Kibana.
The following cluster level, node level, as well as the indices level, details, metrics, and graphs are supported for Elasticsearch:
Cluster metrics
Metrics | Description |
|---|---|
Status | Status of the cluster. |
Nodes | Total number of nodes in the cluster. |
Indices | Total count of indices in the cluster. |
Total Shards | Total number of shards in the cluster. |
Unassigned Shards | Total number of unassigned shards in the cluster. |
Documents | Total count of indexed documents in the cluster. |
Successful Shards | Total number of successful shards in the cluster. |
Active Shards | Total number of active shards in the cluster. |
Initializing Shards | Number of initializing shards in the cluster. |
Relocating Shards | Number of relocating shards in the cluster. |
Cluster graphs
Search Rate: This graph plots the rate at which documents are queried in Elasticsearch for the selected cluster.
Search Latency: This graph plots the time taken to execute the search request for the selected cluster.
Indexing Rate: This graph plots the rate at which documents are indexed in the selected cluster.
Indexing Latency: This graph plots the time taken to index the documents in the selected cluster.
Node details
Column | Description |
|---|---|
Name | Name of the node. |
CPU usage | Total CPU usage for the node. |
Disk Total | Total disk space available for the Elasticsearch node. |
Disk Used | Total disk space used by the Elasticsearch node. |
Load Average | The load average of a CPU in an operating system. |
JVM Memory | JVM heap memory used. |
Node metrics
Metrics | Description |
|---|---|
Documents | The total number of indexed documents in the node. |
Free Disk Space | The available disk space in the node. |
Heap Percent | Percentage of heap memory used. |
Flush Latency | Time taken to flush the indices in a node. |
Flush Rate | The rate at which the indices are flushed. |
CPU | The CPU utilization by the operating system in percentage. |
RAM | RAM utilization in percentage. |
Total Read Operations | Total amount of read operations processed. |
Total write Operations | Total amount of write operations processed. |
Uptime | Time since the node is up and active. |
Node graphs
JVM Heap: This graph plots the total heap used by Elasticsearch in the JVM for a selected node.
Index Memory: This graph plots the heap memory used for indexing in a selected node.
CPU Utilization: This graph plots the percentage of CPU usage for the Elasticsearch process in the selected node.
System Load: The graph plots the load average per minute for the selected node.
Segment Count: This graph plots the maximum segment count for the selected node.
Search Latency: This graph plots the average search latency for the selected node.
Indexing Latency: This graph plots the average latency for indexing documents in the selected node.
Documents: This graph plots the count of indexed documents for the selected node.
Index details
Column | Description |
|---|---|
Name | Name of the index. |
Docs | Number of indexed documents. |
Store Size | Storage size of the indexed documents. |
Search Rate | Number of search requests being executed. |
Active Shards | Total number of active shards in an index. |
Unassigned Shards | Total number of unassigned shards in an index. |
Index metrics
Metrics | Description |
|---|---|
Flush Total | Total number of flushed indices. |
Flushed Latency | Time taken to flush an index. |
Flush Rate | The rate at which indices are flushed. |
Segments count | Count of segments. |
Index graphs
Search Rate: This graph plots the number of search requests that are being executed in Elasticsearch.
Indexing Latency: This graph plots the time taken to index the documents in Elasticsearch.
Indexing Rate: This graph plots the rate at which documents are indexed in Elasticsearch
Segments Count: This graph plots the count of indexing segments operation.
Fetch Latency: This graph plots the fetch latency for the indices.
Fetch Rate: This graph plots the time taken to fetch the query results.
Flush Latency: This graph plots the time taken to flush indices.
Flush Rate: This graph plots the rate at which indices are flushed.
The following cluster level, node level, as well as the indices level, details, metrics, and graphs are supported for Logstash:
Cluster metrics
Metrics | Description |
|---|---|
Nodes | Total number of nodes in the cluster. |
Events Received | Average of the number of events flowing into all the nodes, in a selected time period. |
Events Emitted | Average of the number of events flowing out of all the nodes, in a selected time period. |
Memory | The total heap memory used versus total heap memory available (Heap memory used/Total heap memory available). |
Pipelines | Total number of logical pipelines. |
Node details
Column | Description |
|---|---|
Name | Name of the node. |
Jvm Heap Used | JVM heap memory used. |
Events Received | Current number of events flowing into the selected node. |
Events Emitted | Current number of events flowing out of the selected node. |
Events Filtered | Current number of events that are filtered for the selected node. |
Node metrics
Metrics | Description |
|---|---|
Version | Logstash version. |
CPU Usage | Average CPU usage for the selected period of time. |
Uptime | Time since the node has been up and active. |
Reloads | Successful reloads versus the total number of reloads. |
Node graphs
Events Received: This graph plots the average of the total events flowing into the selected node, for the specified time period.
Events Received Rate: This graph plots the average rate at which events flow into the selected node, for the specified time period.
Events Emitted: This graph plots the average of the total events flowing out of the selected node, for the specified time period.
Events Emitted Rate: This graph plots the average rate at which events flow out of the selected node, for the specified time period.
Cpu Percent: This graph plots the percentage of CPU usage for the Logstash process in the selected node.
Jvm Heap: This graph plots the total heap used by Elasticsearch in the JVM for a selected node.
System Load: The graph plots the load average per minute for the selected node.
Pipeline details
Column | Description |
|---|---|
Name | Name of the node. |
Events Received | Current number of events flowing into the selected pipeline. |
Events Emitted | Current number of events flowing out of the selected pipeline. |
Events Filtered | Current number of events that are filtered for the selected pipeline. |
Number of Nodes | Total number of nodes in the pipeline. |
Pipeline metrics
Pipelines Metrics | Description |
|---|---|
Reload Interval | Time interval that specifies how often Logstash checks the config files for changes in the pipeline (in seconds). |
Batch size | Maximum number of events that an individual worker thread collects before executing filters and outputs. |
Reloads | Successful reloads versus total number of reloads. |
Workers | Number of threads to run for filter and output processing. |
Pipeline graphs
Events Received: This graph plots the average of the total events flowing into the selected pipeline, for the specified time period.
Events Received Rate: This graph plots the average rate at which events flow into the selected pipeline, for the specified time period.
Events Emitted: This graph plots the average of the total events flowing out of the selected pipeline, for the specified time period.
Events Emitted Rate: This graph plots the average rate at which events flow out of the selected pipeline, for the specified time period.
The following details, metrics, and graphs are supported for Kibana:
Metrics
Metric | Description |
|---|---|
Status | Status of the cluster. |
Heap Total | Total heap memory available. |
Heap Used | Total heap memory used. |
Requests Disconnects | Number of disconnected client requests. |
Request Total | Total number of client requests received by the Kibana instance. |
Response Time Max | Maximum time taken to respond to the client requests received by the Kibana instance. |
Concurrent Connections | Total number of concurrent connections to the Kibana instance. |
Collection Interval | Time period between data sampling for the metrics. |
Kibana graphs
Heap Used: This graph plots the total heap memory used by Kibana, in a cluster, for the specified period.
Request Total: This graph plots the total number of client requests received by the Kibana instance in the specified period.
Request Disconnects: This graph plots the totals no of disconnected requests in a specified period.
Concurrent Connections: This graph plots the concurrent connections to the Kibana instance in a specified period.
Supported/Unsupported metrics - ELK version 5.5 and 6.8
The following table list the supported/unsupported metrics in Elasticsearch v5.5 and v6.8:
Metrics | v5.5 | v6.8 |
|---|---|---|
Nodes - Disk Total | No | Yes |
Nodes - Disk Used | No | Yes |
Nodes - FREE DISK SPACE | Yes | Yes |
The following table listed the supported/unsupported metrics in Logstash v5.5 and v6.8:
Metrics | v5.5 | v6.8 |
|---|---|---|
All metrics are supported for v5.5 and v6.8. | ||
The following table listed the supported/unsupported metrics in Kibana v5.5 and v6.8:
Metrics | v5.5 | v6.8 |
|---|---|---|
Heap Total | Time series | Yes |
Heap Used | Time series | Yes |
Requests Disconnects | No | Yes |
Requests Tests | No | Yes |
Response Times Max | Time series | Yes |
Concurrent Connections | No | Yes |
Collection Interval | No | Yes |
Graph - Metrics - Heap Used | No | Yes |
Requests Total | Time series | Yes |
Requests Disconnected | No | Yes |
Concurrent Connections | No | Yes |
Configuration properties
ELK APIs
Refer to Elasticsearch, Logstash, Kibana APIs.
PostgreSQL
Supported platforms and versions
HDI 4.0 (RHEL 7.8)
PostgreSQL version 11.4 and 12.3
Platforms
On HDI secure clusters, where Kerberos is enabled, unravel_es daemon does not refresh the Kerberos token after it expires.
Workaround: Navigate to
/usr/local/unravel_es/etc/and add the following property in theunravel.propertiesfile:com.unraveldata.kerberos.kinit_scheduler.enabled=false