v4.6.2.0 RC1 Release Notes

Release Date: 06/03/2020

Notice

The v4.6.2.0 RC1 is a preview release. This release note includes information about supported and unsupported functionalities for Elasticsearch, Logstash, Kibana (ELK), and PostgreSQL. The new configuration properties for v4.6.2.0 RC1 release and the known issues are also included.

Download information

CDH, HDP, and MapR

curl -v https://preview.unraveldata.com/unravel/RPM/4.6.2/unravel-4.6.2.0-onpremise-RC1.rpm -o unravel-4.6.2.0-onpremise-RC1.rpm -u username:password

MD5SUM: eaecd70edbf3ccd8d745e6a0530431e1

Cloud

curl -v https://preview.unraveldata.com/unravel/RPM/4.6.2/unravel-4.6.2.0-cloud-RC1.rpm -o unravel-4.6.2.0-cloud-RC1.rpm -u username:password

MD5SUM: fd04cadb8225d8af3ed7a2b7dd7cc8ec

OnDemand - RHEL 7.6

curl -v https://preview.unraveldata.com/unravel/RPM/4.6.2/ondemand-4.6.2.0-rhel7-RC1.tar.gz -o ondemand-4.6.2.0-rhel7-RC1.tar.gz -u username:password

MD5SUM: ae783cd1e45e131eec4196a4b19cc48c

Elasticsearch, Logstash, Kibana (ELK)

The KPIs of the ELK stack, which comprises of Elasticsearch (search engine), Logstash (ingest pipeline), and Kibana (visualization tool), can be monitored from Unravel. This section lists the supported platform, versions, capabilities, configuration properties, and APIs available for monitoring ELK metrics in Unravel.

Supported platforms and versions

CDH 6.3.3 (CentOS 7.6)
ELK version 6.8 and 5.5

Supported capabilities

Unravel supports metrics and graphically represents the KPI information for Elasticsearch, Logstash, and Kibana.

Elastic Search

The following cluster level, node level, as well as the indices level, details, metrics, and graphs are supported for Elasticsearch:

Cluster metrics

Metrics	Description
Status	Status of the cluster.
Nodes	Total number of nodes in the cluster.
Indices	Total count of indices in the cluster.
Total Shards	Total number of shards in the cluster.
Unassigned Shards	Total number of unassigned shards in the cluster.
Documents	Total count of indexed documents in the cluster.
Successful Shards	Total number of successful shards in the cluster.
Active Shards	Total number of active shards in the cluster.
Initializing Shards	Number of initializing shards in the cluster.
Relocating Shards	Number of relocating shards in the cluster.

Cluster graphs

Search Rate: This graph plots the rate at which documents are queried in Elasticsearch for the selected cluster.
Search Latency: This graph plots the time taken to execute the search request for the selected cluster.
Indexing Rate: This graph plots the rate at which documents are indexed in the selected cluster.
Indexing Latency: This graph plots the time taken to index the documents in the selected cluster.

Node details

Column	Description
Name	Name of the node.
CPU usage	Total CPU usage for the node.
Disk Total	Total disk space available for the Elasticsearch node.
Disk Used	Total disk space used by the Elasticsearch node.
Load Average	The load average of a CPU in an operating system.
JVM Memory	JVM heap memory used.

Node metrics

Metrics	Description
Documents	The total number of indexed documents in the node.
Free Disk Space	The available disk space in the node.
Heap Percent	Percentage of heap memory used.
Flush Latency	Time taken to flush the indices in a node.
Flush Rate	The rate at which the indices are flushed.
CPU	The CPU utilization by the operating system in percentage.
RAM	RAM utilization in percentage.
Total Read Operations	Total amount of read operations processed.
Total write Operations	Total amount of write operations processed.
Uptime	Time since the node is up and active.

Node graphs

JVM Heap: This graph plots the total heap used by Elasticsearch in the JVM for a selected node.
Index Memory: This graph plots the heap memory used for indexing in a selected node.
CPU Utilization: This graph plots the percentage of CPU usage for the Elasticsearch process in the selected node.
System Load: The graph plots the load average per minute for the selected node.
Segment Count: This graph plots the maximum segment count for the selected node.
Search Latency: This graph plots the average search latency for the selected node.
Indexing Latency: This graph plots the average latency for indexing documents in the selected node.
Documents: This graph plots the count of indexed documents for the selected node.

Index details

Column	Description
Name	Name of the index.
Docs	Number of indexed documents.
Store Size	Storage size of the indexed documents.
Search Rate	Number of search requests being executed.
Active Shards	Total number of active shards in an index.
Unassigned Shards	Total number of unassigned shards in an index.

Index metrics

Metrics	Description
Flush Total	Total number of flushed indices.
Flushed Latency	Time taken to flush an index.
Flush Rate	The rate at which indices are flushed.
Segments count	Count of segments.

Index graphs

Search Rate: This graph plots the number of search requests that are being executed in Elasticsearch.
Indexing Latency: This graph plots the time taken to index the documents in Elasticsearch.
Indexing Rate: This graph plots the rate at which documents are indexed in Elasticsearch
Segments Count: This graph plots the count of indexing segments operation.
Fetch Latency: This graph plots the fetch latency for the indices.
Fetch Rate: This graph plots the time taken to fetch the query results.
Flush Latency: This graph plots the time taken to flush indices.
Flush Rate: This graph plots the rate at which indices are flushed.

Logstash

The following cluster level, node level, as well as the indices level, details, metrics, and graphs are supported for Logstash:

Cluster metrics

Metrics	Description
Nodes	Total number of nodes in the cluster.
Events Received	Average of the number of events flowing into all the nodes, in a selected time period.
Events Emitted	Average of the number of events flowing out of all the nodes, in a selected time period.
Memory	The total heap memory used versus total heap memory available (Heap memory used/Total heap memory available).
Pipelines	Total number of logical pipelines.

Node details

Column	Description
Name	Name of the node.
Jvm Heap Used	JVM heap memory used.
Events Received	Current number of events flowing into the selected node.
Events Emitted	Current number of events flowing out of the selected node.
Events Filtered	Current number of events that are filtered for the selected node.

Node metrics

Metrics	Description
Version	Logstash version.
CPU Usage	Average CPU usage for the selected period of time.
Uptime	Time since the node has been up and active.
Reloads	Successful reloads versus the total number of reloads.

Node graphs

Events Received: This graph plots the average of the total events flowing into the selected node, for the specified time period.
Events Received Rate: This graph plots the average rate at which events flow into the selected node, for the specified time period.
Events Emitted: This graph plots the average of the total events flowing out of the selected node, for the specified time period.
Events Emitted Rate: This graph plots the average rate at which events flow out of the selected node, for the specified time period.
Cpu Percent: This graph plots the percentage of CPU usage for the Logstash process in the selected node.
Jvm Heap: This graph plots the total heap used by Elasticsearch in the JVM for a selected node.
System Load: The graph plots the load average per minute for the selected node.

Pipeline details

Column	Description
Name	Name of the node.
Events Received	Current number of events flowing into the selected pipeline.
Events Emitted	Current number of events flowing out of the selected pipeline.
Events Filtered	Current number of events that are filtered for the selected pipeline.
Number of Nodes	Total number of nodes in the pipeline.

Pipeline metrics

Pipelines Metrics	Description
Reload Interval	Time interval that specifies how often Logstash checks the config files for changes in the pipeline (in seconds).
Batch size	Maximum number of events that an individual worker thread collects before executing filters and outputs.
Reloads	Successful reloads versus total number of reloads.
Workers	Number of threads to run for filter and output processing.

Pipeline graphs

Events Received: This graph plots the average of the total events flowing into the selected pipeline, for the specified time period.
Events Received Rate: This graph plots the average rate at which events flow into the selected pipeline, for the specified time period.
Events Emitted: This graph plots the average of the total events flowing out of the selected pipeline, for the specified time period.
Events Emitted Rate: This graph plots the average rate at which events flow out of the selected pipeline, for the specified time period.

Kibana

The following details, metrics, and graphs are supported for Kibana:

Metrics

Metric	Description
Status	Status of the cluster.
Heap Total	Total heap memory available.
Heap Used	Total heap memory used.
Requests Disconnects	Number of disconnected client requests.
Request Total	Total number of client requests received by the Kibana instance.
Response Time Max	Maximum time taken to respond to the client requests received by the Kibana instance.
Concurrent Connections	Total number of concurrent connections to the Kibana instance.
Collection Interval	Time period between data sampling for the metrics.

Kibana graphs

Heap Used: This graph plots the total heap memory used by Kibana, in a cluster, for the specified period.
Request Total: This graph plots the total number of client requests received by the Kibana instance in the specified period.
Request Disconnects: This graph plots the totals no of disconnected requests in a specified period.
Concurrent Connections: This graph plots the concurrent connections to the Kibana instance in a specified period.

Supported/Unsupported metrics - ELK version 5.5 and 6.8

Elasticsearch

The following table list the supported/unsupported metrics in Elasticsearch v5.5 and v6.8:

Metrics	v5.5	v6.8
Nodes - Disk Total	No	Yes
Nodes - Disk Used	No	Yes
Nodes - FREE DISK SPACE	Yes	Yes

Logstash

The following table listed the supported/unsupported metrics in Logstash v5.5 and v6.8:

Metrics	v5.5	v6.8
All metrics are supported for v5.5 and v6.8.

Kibana

The following table listed the supported/unsupported metrics in Kibana v5.5 and v6.8:

Metrics	v5.5	v6.8
Heap Total	Time series	Yes
Heap Used	Time series	Yes
Requests Disconnects	No	Yes
Requests Tests	No	Yes
Response Times Max	Time series	Yes
Concurrent Connections	No	Yes
Collection Interval	No	Yes
Graph - Metrics - Heap Used	No	Yes
Requests Total	Time series	Yes
Requests Disconnected	No	Yes
Concurrent Connections	No	Yes

Configuration properties

Property/Description	Default

Elasticsearch
com.unraveldata.elk.es.monitor.enabled Enable/disable Elasticsearch cluster monitoring.	false
com.unraveldata.elk.es.clusters Specify clusters for Elasticsearch monitoring. Comma-separated in case of multiple clusters.	-
com.unraveldata.elk.es.rest.url.<cluster_name> URL of the REST API for the cluster. Comma-separated in case of multiple URLs.	-
com.unraveldata.elk.es.rest.endpoints.<cluster_name> Rest API endpoints. Comma-separated in case of multiple endpoints.	/_cluster/health?level=indices,/_cluster/stats,/_nodes/stats,/_stats,/_cat/nodes?format=json&amp;amp;h=,/_cat/indices?format=json&amp;amp;h=
com.unraveldata.elk.es.rest.user.<cluster_name> REST API user name.	-
com.unraveldata.elk.es.rest.pwd.<cluster_name> REST API password. This password should be encrypted.	-
com.unraveldata.elk.es.rest.ssl.enabled.<cluster_name> Enable/Disable SSL	false
com.unraveldata.elk.es.metric.poll.interval Intervals to poll Elasticsearch metrics.	90
com.unraveldata.elk.es.http.poll.parallelism Number of cores for parallelism.	2
com.unraveldata.elk.es.http.conn.timeout Time interval before HTTP connection is timeout.	20
com.unraveldata.elk.es.http.read.timeout Time interval before HTTP read is timeout.	120
com.unraveldata.elk.es.metrics.json Path to JSON configuration file.	/usr/local/unravel/etc/elk_es_metrics.json
Logstash
com.unraveldata.elk.logstash.enable Enable/Disable Logstash monitoring.	false
com.unraveldata.elk.logstash.poll_frequency_sec Poll interval for Logstash monitoring.	90
com.unraveldata.elk.logstash.http.conn.timeout Time interval before HTTP connection is timed out.	10
com.unraveldata.elk.logstash.http.read.timeout Time interval before HTTP read is timed out.	60
com.unraveldata.elk.logstash.instance Comma-separated aliases for Logstash instances. For example: com.unraveldata.elk.logstash.instance=LOG,LOG1,LOG2,LOG3	-
com.unraveldata.elk.logstash.<alias>.host Hostname of the Logstash instance. For example:	-
com.unraveldata.elk.logstash.<alias>.port Port used by the Logstash instance for the hostname. For example: com.unraveldata.elk.logstash.LOG.port=9600	-
Kibana
com.unraveldata.elk.kibana.monitor.enabled Enable/Disable Kibana monitoring.	false
com.unraveldata.elk.kibana.instance Instance name. Comma-separated list in case of multiple instances.	-
com.unraveldata.elk.kibana.rest.url.kibana The REST API URL of the Kibana instance. Comma-separated in case of multiple URL values. Provide the URL in the following format: <elasticsearch_cluster_rest_endpoint>:<port>	-
com.unraveldata.elk.kibana.rest.endpoints.kibana Rest API endpoints. Comma-separated in case of multiple endpoints.	-
com.unraveldata.elk.kibana.rest.user.kibana REST API username	-
com.unraveldata.elk.kibana.rest.pwd.kibana REST API password. The password should be encrypted.	-
com.unraveldata.elk.kibana.rest.ssl.enabled.kibana Enable/Disable SSL authentication.	false
com.unraveldata.elk.kibana.metric.poll.interval Intervals to poll the kibana metrics.	90
com.unraveldata.elk.kibana.http.poll.parallelism Number of cores for parallelism.	2
com.unraveldata.elk.kibana.http.conn.timeout Time interval before HTTP connection is timed out.	20
com.unraveldata.elk.kibana.http.read.timeout Time interval before HTTP read is timed out.	120
com.unraveldata.elk.kibana.metrics.json Path to JSON configuration file.	/usr/local/unravel/etc/elk_kibana_metrics.json

ELK APIs

Refer to Elasticsearch, Logstash, Kibana APIs.

PostgreSQL

Supported platforms and versions

HDI 4.0 (RHEL 7.8)
PostgreSQL version 11.4 and 12.3

Known issues

Platforms

On HDI secure clusters, where Kerberos is enabled, unravel_es daemon does not refresh the Kerberos token after it expires.
Workaround: Navigate to /usr/local/unravel_es/etc/ and add the following property in the unravel.properties file:
com.unraveldata.kerberos.kinit_scheduler.enabled=false

For support issues, contact Unravel Support

In this section:

Home