Monitoring API - Cluster

• n::cpuUtilization Current CPU utilisation as a percentage of total available.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpu_utilization
• n::osload Current OS load.
  
  • Available sub-types:
    • last_one_minute Average metric value over 1 minute.
      Prometheus Name: ic_node_osload
    • last_five_minutes Average metric value over 5 minutes.
      Prometheus Name: ic_node_osload
    • last_fifteen_minutes Average metric value over 15 minutes.
      Prometheus Name: ic_node_osload
• n::diskUtilization Total disk space utilisation, by Cassandra, as a percentage of total available.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_disk_utilization
• n::diskAvailable Disk space available in bytes
  
  • Sub-type: value
    Prometheus Name: ic_node_disk_available
• n::diskUsed Disk space used in bytes
  
  • Sub-type: value
    Prometheus Name: ic_node_disk_used
• n::cpuguestpercent Time spent running a virtual CPU for guest OS’ under control of kernel.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpuguestpercent
• n::cpuguestnicepercent Niced processes executing in user mode in virtual OS.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpuguestnicepercent
• n::cpusystempercent Percentage of processes executing in kernel mode.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpusystempercent
• n::cpuidlepercent Percentage of time when one or more kernel threads are executing with the run queue empty and/or no I/O operations are currently cycling.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpuidlepercent
• n::cpuiowaitpercent CPU time the I/O thread spent waiting for a socket ready for reads or writes as a percent.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpuiowaitpercent
• n::cpuirqpercent Number of hardware interrupts the kernel is servicing.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpuirqpercent
• n::cpunicepercent Percentage of processes executing in user mode which have a positive nice value.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpunicepercent
• n::cpusoftirqpercent Number of software interrupts the kernel is servicing.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpusoftirqpercent
• n::cpustealpercent Percentage of time the hypervisor allocated to other tasks external to the one run on the current virtual CPU
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpustealpercent
• n::cpuuserpercent Processes executing in user mode, including application processes.
  
  • Sub-type: percentage
    Prometheus Name: ic_node_cpuuserpercent
• n::memavailable Estimate of how much memory is available to start new applications without swap, taking into account page cache and re-claimability of slab.
  
  • Sub-type: value
    Prometheus Name: ic_node_memavailable
• n::networkindelta Delta count of bytes received.
  
  • Sub-type: value
    Prometheus Name: ic_node_networkindelta
• n::networkoutdelta Delta count of bytes transmitted.
  
  • Sub-type: value
    Prometheus Name: ic_node_networkoutdelta
• n::networkin Count of bytes received.
  
  • Sub-type: value
    Prometheus Name: ic_node_networkin
• n::networkout Count of bytes transmitted.
  
  • Sub-type: value
    Prometheus Name: ic_node_networkout
• n::networkinerrorsdelta Delta count of receive errors detected.
  
  • Sub-type: value
    Prometheus Name: ic_node_networkinerrorsdelta
• n::networkouterrorsdelta Delta count of transmit packets dropped.
  
  • Sub-type: value
    Prometheus Name: ic_node_networkouterrorsdelta
• n::networkindroppeddelta Delta count of receive packets dropped.
  
  • Sub-type: value
    Prometheus Name: ic_node_networkindroppeddelta
• n::networkoutdroppeddelta Delta count of transmit packets dropped.
  
  • Sub-type: value
    Prometheus Name: ic_node_networkoutdroppeddelta
• n::filedescriptorlimit Maximum number of open files limit for the node OS.
  
  • Sub-type: value
    Prometheus Name: ic_node_filedescriptorlimit
• n::filedescriptoropencount Current number of open files in the node OS.
  
  • Sub-type: value
    Prometheus Name: ic_node_filedescriptoropencount
• n::tcpestablished Number of open TCP connections.
  
  • Sub-type: value
    Prometheus Name: ic_node_tcpestablished
• n::tcptimewait Number of TCP sockets waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request.
  
  • Sub-type: value
    Prometheus Name: ic_node_tcptimewait
• n::tcplistening Number of TCP sockets waiting for a connection request from any remote TCP and port.
  
  • Sub-type: value
    Prometheus Name: ic_node_tcplistening
• n::tcpall Total number of TCP connections in all state.
  
  • Sub-type: value
    Prometheus Name: ic_node_tcpall
• n::tcpclosewait Number of TCP sockets which connection is in the process of being closed.
  
  • Sub-type: value
    Prometheus Name: ic_node_tcpclosewait

Additional information on troubleshooting Cassandra metrics is available here.

• csp::shotoverTransformFailuresCount The number of transform failures.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_failures_count
• csp::shotoverTransformTotalCount The number of transforms used.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_total_count
• csp::shotoverTransformPushedTotalCount The number of transforms used to process messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_total_count
• csp::shotoverTransformPushedFailuresCount The number of transform failures while processing messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_failures_count
• csp::shotoverTransformLatencySeconds0th 0th % latency for running the transform.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_latency_seconds0th
• csp::shotoverTransformLatencySeconds50th 50th % latency for running the transform.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_latency_seconds50th
• csp::shotoverTransformLatencySeconds90th 90th % latency for running the transform.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_latency_seconds90th
• csp::shotoverTransformLatencySeconds95th 95th % latency for running the transform.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_latency_seconds95th
• csp::shotoverTransformLatencySeconds99th 99th % latency for running the transform.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_latency_seconds99th
• csp::shotoverTransformLatencySeconds999th 99.9th % latency for running the transform.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_latency_seconds999th
• csp::shotoverTransformLatencySeconds100th 100th % latency for running the transform.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_latency_seconds100th
• csp::shotoverTransformLatencySecondsCount The number of latency for running the transform.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_latency_seconds_count
• csp::shotoverTransformLatencySecondsSum The sum of latency for running the transform.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_latency_seconds_sum
• csp::shotoverTransformPushedLatencySeconds0th 0th % latency for running the transform on messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_latency_seconds0th
• csp::shotoverTransformPushedLatencySeconds50th 50th % latency for running the transform on messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_latency_seconds50th
• csp::shotoverTransformPushedLatencySeconds90th 90th % latency for running the transform on messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_latency_seconds90th
• csp::shotoverTransformPushedLatencySeconds95th 95th % latency for running the transform on messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_latency_seconds95th
• csp::shotoverTransformPushedLatencySeconds99th 99th % latency for running the transform on messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_latency_seconds99th
• csp::shotoverTransformPushedLatencySeconds999th 99.9th % latency for running the transform on messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_latency_seconds999th
• csp::shotoverTransformPushedLatencySeconds100th 100th % latency for running the transform on messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_latency_seconds100th
• csp::shotoverTransformPushedLatencySecondsCount The number of latency for running the transform on messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_latency_seconds_count
• csp::shotoverTransformPushedLatencySecondsSum The sum of latency for running the transform on messages without a corresponding request (events).
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_transform_pushed_latency_seconds_sum
• csp::shotoverSourceToSinkLatencySeconds0th 0th % latency for running the transform from client to cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_source_to_sink_latency_seconds0th
• csp::shotoverSourceToSinkLatencySeconds50th 50th % latency for running the transform from client to cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_source_to_sink_latency_seconds50th
• csp::shotoverSourceToSinkLatencySeconds90th 90th % latency for running the transform from client to cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_source_to_sink_latency_seconds90th
• csp::shotoverSourceToSinkLatencySeconds95th 95th % latency for running the transform from client to cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_source_to_sink_latency_seconds95th
• csp::shotoverSourceToSinkLatencySeconds99th 99th % latency for running the transform from client to cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_source_to_sink_latency_seconds99th
• csp::shotoverSourceToSinkLatencySeconds999th 99.9th % latency for running the transform from client to cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_source_to_sink_latency_seconds999th
• csp::shotoverSourceToSinkLatencySeconds100th 100th % latency for running the transform from client to cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_source_to_sink_latency_seconds100th
• csp::shotoverSourceToSinkLatencySecondsCount The number of latency for running the transform from client to cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_source_to_sink_latency_seconds_count
• csp::shotoverSourceToSinkLatencySecondsSum The sum of latency for running the transform from client to cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_source_to_sink_latency_seconds_sum
• csp::shotoverFailedRequestsCount The number of failed requests.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_failed_requests_count
• csp::shotoverOutOfRackRequestsCount The number of out of rack requests.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_out_of_rack_requests_count
• csp::shotoverAvailableConnectionsCount The number of available connections.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_available_connections_count
• csp::shotoverChainFailuresCount The number of chain failures.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_failures_count
• csp::shotoverChainTotalCount The number of chains used.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_total_count
• csp::shotoverSinkToSourceLatencySeconds0th 0th % latency for running the transform from cluster to client.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_sink_to_source_latency_seconds0th
• csp::shotoverSinkToSourceLatencySeconds50th 50th % latency for running the transform from cluster to client.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_sink_to_source_latency_seconds50th
• csp::shotoverSinkToSourceLatencySeconds90th 90th % latency for running the transform from cluster to client.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_sink_to_source_latency_seconds90th
• csp::shotoverSinkToSourceLatencySeconds95th 95th % latency for running the transform from cluster to client.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_sink_to_source_latency_seconds95th
• csp::shotoverSinkToSourceLatencySeconds99th 99th % latency for running the transform from cluster to client.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_sink_to_source_latency_seconds99th
• csp::shotoverSinkToSourceLatencySeconds999th 99.9th % latency for running the transform from cluster to client.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_sink_to_source_latency_seconds999th
• csp::shotoverSinkToSourceLatencySeconds100th 100th % latency for running the transform from cluster to client.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_sink_to_source_latency_seconds100th
• csp::shotoverSinkToSourceLatencySecondsCount The number of latency for running the transform from cluster to client.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_sink_to_source_latency_seconds_count
• csp::shotoverSinkToSourceLatencySecondsSum The sum of latency for running the transform from cluster to client.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_sink_to_source_latency_seconds_sum
• csp::shotoverChainMessagesPerBatchCount0th 0th % number of messages per batch.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_messages_per_batch_count0th
• csp::shotoverChainMessagesPerBatchCount50th 50th % number of messages per batch.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_messages_per_batch_count50th
• csp::shotoverChainMessagesPerBatchCount90th 90th % number of messages per batch.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_messages_per_batch_count90th
• csp::shotoverChainMessagesPerBatchCount95th 95th % number of messages per batch.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_messages_per_batch_count95th
• csp::shotoverChainMessagesPerBatchCount99th 99th % number of messages per batch.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_messages_per_batch_count99th
• csp::shotoverChainMessagesPerBatchCount999th 99.9th % number of messages per batch.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_messages_per_batch_count999th
• csp::shotoverChainMessagesPerBatchCount100th 100th % number of messages per batch.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_messages_per_batch_count100th
• csp::shotoverChainMessagesPerBatchCountCount The number of messages per batch.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_messages_per_batch_count_count
• csp::shotoverChainMessagesPerBatchCountSum The sum of number of messages per batch.
  
  • Sub-type: value
    Prometheus Name: ic_node_shotover_chain_messages_per_batch_count_sum

• o::memused Percentage of used memory.
  
  • Sub-type: value
    Prometheus Name: ic_node_memused
• o::docsCount Number of non-deleted documents in the segment. This number is based on Lucene documents and may include documents from nested fields.
  
  • Sub-type: value
    Prometheus Name: ic_node_docs_count
• o::docsDeleted Number of deleted documents in the segment. This number is based on Lucene documents. Elasticsearch reclaims the disk space of deleted Lucene documents when a segment is merged.
  
  • Sub-type: value
    Prometheus Name: ic_node_docs_deleted
• o::jvmheappercent Percentage of memory currently in use by the heap.
  
  • Sub-type: value
    Prometheus Name: ic_node_jvmheappercent
• o::jvmthreadscount Number of active threads in use by JVM.
  
  • Sub-type: value
    Prometheus Name: ic_node_jvmthreadscount
• o::indextotalpersec Indices per second.
  
  • Sub-type: value
    Prometheus Name: ic_node_indextotalpersec
• o::querytotalpersec Queries per second.
  
  • Sub-type: value
    Prometheus Name: ic_node_querytotalpersec
• o::indexlatency The latency of new indexing operations measured in milliseconds.
  
  • Sub-type: value
    Prometheus Name: ic_node_indexlatency
• o::querylatency The latency of new query operations measured in milliseconds.
  
  • Sub-type: value
    Prometheus Name: ic_node_querylatency
• o::slasearchlatency Monitors our SLA search latency and alerts when it is above a threshold level. This is the synthetic search query against an Instaclustr canary index.
  
  • Sub-type: value
    Prometheus Name: ic_node_slasearchlatency
• o::slaindexlatency Monitors our SLA indexing latency and alerts when it is above a threshold level. This is the synthetic indexing against an Instaclustr canary index.
  
  • Sub-type: value
    Prometheus Name: ic_node_slaindexlatency

• op::ccr::leaderConnected Indicates the connection status of the connection between follower cluster and leader cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_leader_connected
• op::ccr::followerCheckpoint Indicates the checkpoint at which the follower indices are at. This is a cumulative value across all replicating indices.
  
  • Sub-type: value
    Prometheus Name: ic_node_follower_checkpoint
• op::ccr::leaderCheckpoint Indicates the checkpoint at which the leader indices are at. This is a cumulative value across all replicating indices.
  
  • Sub-type: value
    Prometheus Name: ic_node_leader_checkpoint
• op::ccr::syncingIndicesCount Indicates the number of syncing/replicating indices.
  
  • Sub-type: value
    Prometheus Name: ic_node_syncing_indices_count
• op::ccr::bootstrappingIndicesCount Indicates the number of indices which are at the stage of setting up replication.
  
  • Sub-type: value
    Prometheus Name: ic_node_bootstrapping_indices_count
• op::ccr::pausedIndicesCount Indicates the number of replicating indices which are paused.
  
  • Sub-type: value
    Prometheus Name: ic_node_paused_indices_count
• op::ccr::failedIndicesCount Indicates the number of failed replicating indices.
  
  • Sub-type: value
    Prometheus Name: ic_node_failed_indices_count
• op::ccr::failedReadRequests Indicates the number of read requests failed during replication.
  
  • Sub-type: value
    Prometheus Name: ic_node_failed_read_requests
• op::ccr::failedWriteRequests Indicates the number of write requests failed during replication.
  
  • Sub-type: value
    Prometheus Name: ic_node_failed_write_requests
• op::ccr::throttledReadRequests Indicates the number of read requests throttled during replication.
  
  • Sub-type: value
    Prometheus Name: ic_node_throttled_read_requests
• op::ccr::throttledWriteRequests Indicates the number of write requests throttled during replication.
  
  • Sub-type: value
    Prometheus Name: ic_node_throttled_write_requests
• op::ccr::operationsWritten Indicates the number of operations written during replication.
  
  • Sub-type: value
    Prometheus Name: ic_node_operations_written
• op::ccr::operationsRead Indicates the number of operations read during replication.
  
  • Sub-type: value
    Prometheus Name: ic_node_operations_read
• op::ccr::autoFollowStartSuccess Indicates the number of successful auto follow replication attempts.
  
  • Sub-type: value
    Prometheus Name: ic_node_auto_follow_start_success
• op::ccr::autoFollowStartFailed Indicates the number of failed auto follow replication attempts.
  
  • Sub-type: value
    Prometheus Name: ic_node_auto_follow_start_failed
• op::ccr::autoFollowLeaderCallsFailed Indicates the number of failed replication calls to leader.
  
  • Sub-type: value
    Prometheus Name: ic_node_auto_follow_leader_calls_failed

• e::memused Percentage of used memory.
  
  • Sub-type: value
    Prometheus Name: ic_node_memused
• e::docsCount Number of non-deleted documents in the segment. This number is based on Lucene documents and may include documents from nested fields.
  
  • Sub-type: value
    Prometheus Name: ic_node_docs_count
• e::docsDeleted Number of deleted documents in the segment. This number is based on Lucene documents. Elasticsearch reclaims the disk space of deleted Lucene documents when a segment is merged.
  
  • Sub-type: value
    Prometheus Name: ic_node_docs_deleted
• e::jvmheappercent Percentage of memory currently in use by the heap.
  
  • Sub-type: value
    Prometheus Name: ic_node_jvmheappercent
• e::jvmthreadscount Number of active threads in use by JVM.
  
  • Sub-type: value
    Prometheus Name: ic_node_jvmthreadscount
• e::indextotalpersec Indices per second.
  
  • Sub-type: value
    Prometheus Name: ic_node_indextotalpersec
• e::querytotalpersec Queries per second.
  
  • Sub-type: value
    Prometheus Name: ic_node_querytotalpersec
• e::indexlatency The latency of new indexing operations measured in milliseconds.
  
  • Sub-type: value
    Prometheus Name: ic_node_indexlatency
• e::querylatency The latency of new query operations measured in milliseconds.
  
  • Sub-type: value
    Prometheus Name: ic_node_querylatency
• e::slasearchlatency Monitors our SLA search latency and alerts when it is above a threshold level. This is the synthetic search query against an Instaclustr canary index.
  
  • Sub-type: value
    Prometheus Name: ic_node_slasearchlatency
• e::slaindexlatency Monitors our SLA indexing latency and alerts when it is above a threshold level. This is the synthetic indexing against an Instaclustr canary index.
  
  • Sub-type: value
    Prometheus Name: ic_node_slaindexlatency

• k::activeControllerCount The number of active controllers on the node. In effect it is 0 or 1. The active controller of a cluster is usually the first node to start up in the cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_active_controller_count
• k::offlinePartitions The number of partitions without an active leader. Any partitions that are offline will not be accessible since read and write operations are only performed on the leader of a partition.
  
  • Sub-type: value
    Prometheus Name: ic_node_offline_partitions
• k::activeBrokerCount The number of registered and unfenced brokers.
  
  • Sub-type: value
    Prometheus Name: ic_node_active_broker_count
• k::metadataErrorCount The number of times this controller node has encountered an error during metadata log processing.
  
  • Sub-type: value
    Prometheus Name: ic_node_metadata_error_count
• k::lastCommittedRecordOffset The offset of the last record committed to this Controller. This is always advancing due to the NoOpRecord, and can be used to check cluster availability.
  
  • Sub-type: value
    Prometheus Name: ic_node_last_committed_record_offset
• k::fencedBrokerCount The number of registered but fenced brokers.
  
  • Sub-type: value
    Prometheus Name: ic_node_fenced_broker_count
• k::preferredReplicaImbalanceCount The count of topic partitions for which the leader is not the preferred leader.
  
  • Sub-type: value
    Prometheus Name: ic_node_preferred_replica_imbalance_count
• k::brokerTopicMessagesIn The mean and one minute rate of incoming messages per second.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_broker_topic_messages_in
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_broker_topic_messages_in
    • count
      Prometheus Name: ic_node_broker_topic_messages_in
• k::brokerTopicBytesIn The mean and one minute rate of incoming bytes to the cluster.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_broker_topic_bytes_in
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_broker_topic_bytes_in
    • count
      Prometheus Name: ic_node_broker_topic_bytes_in
• k::brokerTopicBytesOut The mean and one minute rate of outgoing bytes from the cluster.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_broker_topic_bytes_out
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_broker_topic_bytes_out
    • count
      Prometheus Name: ic_node_broker_topic_bytes_out
• k::leaderElectionRate The count, average, max, and one minute rate of leader elections per second.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_leader_election_rate
    • max Maximum value of the metric.
      Prometheus Name: ic_node_leader_election_rate
    • average Average value of the metric.
      Prometheus Name: ic_node_leader_election_rate
    • count
      Prometheus Name: ic_node_leader_election_rate
• k::uncleanLeaderElections The number of failures to elect a suitable leader per second. In the case that no suitable leader can be chosen (ie. no available replicas are in sync), an out-of-sync replica will be elected as leader, resulting in data loss that is proportional to how out-of-sync the newly elected leader is.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_unclean_leader_elections
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_unclean_leader_elections
    • count
      Prometheus Name: ic_node_unclean_leader_elections
• k::partitionLoadTimeAvg The average time of Consumer Group Coordinator to load the Commit Offset partition in 30 seconds interval. This is only available for Kafka 2.4.1+.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_partition_load_time_avg_milliseconds
• k::partitionLoadTimeMax The maximum time of Consumer Group Coordinator to load the Commit Offset partition in 30 seconds interval. This is only available for Kafka 2.4.1+.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_partition_load_time_max_milliseconds
• k::groupCompletedRebalanceCount The number of rebalancing operations triggered by a number of factors as the participants of the group change. The rebalancing leads to the reassignment of partitions across the consumers.
  
  • Sub-type: value
    Prometheus Name: ic_node_group_completed_rebalance_count
• k::groupCompletedRebalanceRate The rate of rebalancing operations.
  
  • Sub-type: value
    Prometheus Name: ic_node_group_completed_rebalance_rate
• k::replicaFetcherMaxLag The max message count lag between all fetchers/topics/partitions.
  
  • Sub-type: value
    Prometheus Name: ic_node_replica_fetcher_max_lag
• k::replicaFetcherFailedPartitionsCount Increment count when partition truncation fails, storage exception is encountered, partition has older epoch than current leader or any other error encountered during fetch request. This is only available for Kafka 2.3.1+.
  
  • Sub-type: value
    Prometheus Name: ic_node_replica_fetcher_failed_partitions_count
• k::replicaFetcherMinFetchRate The minimum number of messages fetched in one minute interval between all fetchers/topics/partitions.
  
  • Sub-type: value
    Prometheus Name: ic_node_replica_fetcher_min_fetch_rate
• k::replicaFetcherDeadThreadCount The number of failed fetcher threads. This is only available for Kafka 2.4.1+.
  
  • Sub-type: value
    Prometheus Name: ic_node_replica_fetcher_dead_thread_count
• k::partitionCount The number of partitions on a node. The number of partitions should be evenly distributed across all nodes in a cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_partition_count
• k::isrShrinkRate The one minute rate, mean rate, and number of decreases in the number of In-Sync Replicas (ISR) per second. This metric is expected to change when adding or removing nodes from a cluster.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_isr_shrink_rate
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_isr_shrink_rate
    • count
      Prometheus Name: ic_node_isr_shrink_rate
• k::isrExpandRate The one minute rate, mean rate, and number of increases in the number of In-Sync Replicas (ISR) per second. This metric is expected to change when adding or removing nodes from a cluster.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_isr_expand_rate
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_isr_expand_rate
    • count
      Prometheus Name: ic_node_isr_expand_rate
• k::underMinIsrPartitions The number of partitions where the number of In-Sync Replicas (ISR) is less than the minimum number of in-sync replicas specified.
  
  • Sub-type: value
    Prometheus Name: ic_node_under_min_isr_partitions
• k::underReplicatedPartitions The number of partitions that do not have enough replicas to meet the desired replication factor.
  
  • Sub-type: value
    Prometheus Name: ic_node_under_replicated_partitions
• k::leaderCount The number of partitions that a node is a leader for. The number of partition leaders should be evenly distributed across all nodes in a cluster.
  
  • Sub-type: value
    Prometheus Name: ic_node_leader_count
• k::kafkaBrokerState The current state of the broker represented as an Integer. Can be one of the following Integer values:
  0. Not running
  1. Starting
  2. Recovering from unclean shutdown
  3. Running as broker
  6. Pending controlled shutdown
  7. Broker shutting down
  
  • Sub-type: value
    Prometheus Name: ic_node_kafka_broker_state
• k::produceRequestTime The count, average, 99th percentile distribution and max time taken to process requests from producers to send data. This is the sum of time spent waiting in request, time spent being processed by the leader, time spent waiting for follower response (if requests.required.acks = 1), and time taken to send the response.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_request_time_milliseconds
    • average
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_request_time_milliseconds
    • count
      Prometheus Name: ic_node_produce_request_time
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_request_time_milliseconds
• k::fetchConsumerRequestTime The count, average, 99th percentile distribution and max amount of time taken while processing, and the number of requests from consumers to get new data. This is the sum of time spent waiting in request, time spent being processed by the leader, time spent waiting for the leader to trigger sending the response (determined by fetch.min.bytes and fetch.wait.max.ms in the consumer configuration), and time taken to send the response.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_request_time_milliseconds
    • average
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_request_time_milliseconds
    • count
      Prometheus Name: ic_node_fetch_consumer_request_time
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_request_time_milliseconds
• k::fetchFollowerRequestTime The count, average, and max amount of time taken while processing requests fromKafka brokers to get new data from partition leaders. This is the sum of time spent waiting in request, time spent being processed by the leader, and time taken to send the response.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_follower_request_time_milliseconds
    • average
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_follower_request_time_milliseconds
    • count
      Prometheus Name: ic_node_fetch_follower_request_time
• k::metadataRequestTime The 99th percentile distribution and max amount of time taken while processing requests from Kafka brokers to retrieve metadata. This is the sum of time spent waiting in request, time spent being processed by the leader, and time taken to send the response.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_request_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_request_time_milliseconds
• k::produceRequestLocalTime The 99th percentile distribution and max amount of time taken by the leader to process requests from producers to send data.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_request_local_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_request_local_time_milliseconds
• k::fetchConsumerRequestLocalTime The 99th percentile distribution and max amount of time spent being processed by the leader from consumer requests to get new data.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_request_local_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_request_local_time_milliseconds
• k::metadataRequestLocalTime The 99th percentile distribution and max amount of time spent being processed by the leader while processing requests from Kafka brokers to retrieve metadata.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_request_local_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_request_local_time_milliseconds
• k::produceRequestRemoteTime The 99th percentile distribution and max amount of time taken waiting for the follower to process requests from producers to send data.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_request_remote_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_request_remote_time_milliseconds
• k::fetchConsumerRequestRemoteTime The 99th percentile distribution and max amount of time waiting for the follower from consumer requests to get new data.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_request_remote_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_request_remote_time_milliseconds
• k::metadataRequestRemoteTime The 99th percentile distribution and max amount of time waiting for the follower while processing requests from Kafka brokers to retrieve metadata.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_request_remote_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_request_remote_time_milliseconds
• k::produceRequestQueueTime The 99th percentile distribution and max amount of time the request waits in the request queue to process requests from producers to send data.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_request_queue_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_request_queue_time_milliseconds
• k::fetchConsumerRequestQueueTime The 99th percentile distribution and max amount of time the request waits in the request queue from consumer requests to get new data.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_request_queue_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_request_queue_time_milliseconds
• k::metadataRequestQueueTime The 99th percentile distribution and max amount of time the request waits in the request queue while processing requests from Kafka brokers to retrieve metadata.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_request_queue_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_request_queue_time_milliseconds
• k::produceResponseQueueTime The 99th percentile distribution and max amount of time the request waits in the response queue to process requests from producers to send data.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_response_queue_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_produce_response_queue_time_milliseconds
• k::fetchConsumerResponseQueueTime The 99th percentile distribution and max amount of time the request waits in the response queue from consumer requests to get new data.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_response_queue_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_fetch_consumer_response_queue_time_milliseconds
• k::metadataResponseQueueTime The 99th percentile distribution and max amount of time the request waits in the response queue while processing requests from Kafka brokers to retrieve metadata.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_response_queue_time_milliseconds
    • 99thPercentile 99th percentile distribution of time.
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_metadata_response_queue_time_milliseconds
• k::producePurgatorySize The number of produce requests currently waiting in purgatory.
  
  • Sub-type: value
    Prometheus Name: ic_node_produce_purgatory_size
• k::fetchPurgatorySize The number of fetch requests currently waiting in purgatory.
  
  • Sub-type: value
    Prometheus Name: ic_node_fetch_purgatory_size
• k::networkProcessorAvgIdlePercent The average percentage of time the network processors are idle, expressed as a number between 0 and 1. Kafka’s network processor threads are responsible for reading and writing data to Kafka clients across the network.
  
  • Sub-type: value
    Prometheus Name: ic_node_network_processor_avg_idle_percent
• k::requestHandlerAvgIdlePercent The average percentage of time Kafka’s request handler threads are idle, expressed as a number between 0 and 1. Kafka’s request handler threads are responsible for servicing client requests, including reading and writing messages to disk.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_request_handler_avg_idle_percent
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_request_handler_avg_idle_percent
    • count
      Prometheus Name: ic_node_request_handler_avg_idle_percent
• k::produceMessageConversionsPerSec The one minute rate, mean rate, and number of produce requests per second that require message format conversion.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_produce_message_conversions_per_sec
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_produce_message_conversions_per_sec
    • count
      Prometheus Name: ic_node_produce_message_conversions_per_sec
• k::fetchMessageConversionsPerSec The one minute rate, mean rate, and number of fetch requests per second that require message format conversion.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_fetch_message_conversions_per_sec
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_fetch_message_conversions_per_sec
    • count
      Prometheus Name: ic_node_fetch_message_conversions_per_sec
• k::slaConsumerLatency The average and maximum time in milliseconds between a synthetic transaction message being sent by the producer and being received by the consumer.
  
  • Available sub-types:
    • average Average value of the metric.
      Prometheus Name: ic_node_sla_consumer_latency
    • max Maximum value of the metric.
      Prometheus Name: ic_node_sla_consumer_latency
• k::slaConsumerRecordsProcessed The number of synthetic transaction messages being successfully consumed and processed on each broker.
  
  • Sub-type: count
    Prometheus Name: ic_node_sla_consumer_records_processed
• k::slaProducerLatencyMs The average and maximum time taken in milliseconds to send a synthetic transaction message to each broker that is successfully replicated to the required number of minimum in-sync replicas.
  
  • Available sub-types:
    • average Average value of the metric.
      Prometheus Name: ic_node_sla_producer_latency_ms
    • max Maximum value of the metric.
      Prometheus Name: ic_node_sla_producer_latency_ms
• k::slaProducerMessagesProcessed The number of synthetic transaction messages being successfully produced to each broker.
  
  • Sub-type: count
    Prometheus Name: ic_node_sla_producer_messages_processed
• k::slaProducerErrors The number of errors encountered when producing synthetic transaction messages.
  
  • Sub-type: count
    Prometheus Name: ic_node_sla_producer_errors
• k::youngGenLastGC Time taken for GC to run young generation during the latest event.
  
  • Sub-type: value
    Prometheus Name: ic_node_young_gen_last_g_c
• k::oldGengcCollectionTime Total time taken for GC to run old generation.
  
  • Sub-type: value
    Prometheus Name: ic_node_old_gengc_collection_time
• k::logFlushRate The total count, one minute rate and mean rate of Kafka log flush.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_log_flush_rate
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_log_flush_rate
    • count
      Prometheus Name: ic_node_log_flush_rate
• k::logFlushTime The average time and maximum time of Kafka log flush.
  
  • Available sub-types:
    • max
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_log_flush_time_milliseconds
    • average
      Unit: milliseconds (ms)
      Prometheus Name: ic_node_log_flush_time_milliseconds
• k::produceRequestsPerSec The one minute rate, mean rate, and number of produce requests, since the beginning of program running. This only works for period below 3h.
  
  • Available sub-types:
    • count
      Prometheus Name: ic_node_produce_requests_per_sec
    • mean_rate
      Prometheus Name: ic_node_produce_requests_per_sec
    • one_minute_rate
      Prometheus Name: ic_node_produce_requests_per_sec
• k::fetchConsumerRequestsPerSec The one minute rate, mean rate, and number of requests from consumer requests to get new data, since the beginning of program running. This only works for period below 3h.
  
  • Available sub-types:
    • count
      Prometheus Name: ic_node_fetch_consumer_requests_per_sec
    • mean_rate
      Prometheus Name: ic_node_fetch_consumer_requests_per_sec
    • one_minute_rate
      Prometheus Name: ic_node_fetch_consumer_requests_per_sec
• k::fetchFollowerRequestsPerSec The one minute rate, mean rate, and number of requests from Kafka brokers to get new data from partition leaders, since the beginning of program running. This only works for period below 3h.
  
  • Available sub-types:
    • count
      Prometheus Name: ic_node_fetch_follower_requests_per_sec
    • mean_rate
      Prometheus Name: ic_node_fetch_follower_requests_per_sec
    • one_minute_rate
      Prometheus Name: ic_node_fetch_follower_requests_per_sec
• k::controlPlaneNetworkProcessorAvgIdlePercent Monitoring the idle percentage of pinned control plane network thread.
  
  • Sub-type: value
    Prometheus Name: ic_node_control_plane_network_processor_avg_idle_percent
• k::brokerFetcherLagConsumerLag The lag in the number of messages per follower replica aggregated at a broker level. Please note that brokers would not report this metric if it is not following a partition. For example all topics in the cluster is created with a replication factor of 1.
  
  • Sub-type: count
    Prometheus Name: ic_node_broker_fetcher_lag_consumer_lag
• k::metadataApplyErrorCount The number of errors encountered by the BrokerMetadataPublisher while applying a new MetadataImage based on the latest MetadataDelta.
  
  • Sub-type: value
    Prometheus Name: ic_node_metadata_apply_error_count
• k::metadataLoadErrorCount The number of errors encountered by the BrokerMetadataListener while loading the metadata log and generating a new MetadataDelta based on it.
  
  • Sub-type: value
    Prometheus Name: ic_node_metadata_load_error_count
• k::commitLatencyAvg The average time in milliseconds to commit an entry in the raft log.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_commit_latency_avg_milliseconds
• k::commitLatencyMax The maximum time in milliseconds to commit an entry in the raft log.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_commit_latency_max_milliseconds
• k::appendRecordsRate The average number of records appended per sec by the leader of the raft quorum.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_append_records_rate
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_append_records_rate
    • count
      Prometheus Name: ic_node_append_records_rate
• k::electionLatencyMax The maximum time in milliseconds spent on electing a new leader.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_election_latency_max_milliseconds
• k::electionLatencyAvg The average time in milliseconds spent on electing a new leader.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_election_latency_avg_milliseconds
• k::pollIdleRatioAvg The average fraction of time the client's poll() is idle as opposed to waiting for the user code to process records.
  
  • Sub-type: value
    Prometheus Name: ic_node_poll_idle_ratio_avg
• k::currentState The current state of this member; possible values are leader, candidate, voted, follower, unattached.
  
  • Sub-type: state
    Prometheus Name: ic_node_current_state
• k::highWatermark The high watermark maintained on this member; -1 if it is unknown.
  
  • Sub-type: value
    Prometheus Name: ic_node_high_watermark
• k::currentLeader The current quorum leader's id; -1 indicates unknown.
  
  • Sub-type: value
    Prometheus Name: ic_node_current_leader
• k::logEndOffset The current raft log end offset.
  
  • Sub-type: value
    Prometheus Name: ic_node_log_end_offset
• k::fetchRecordsRate The average number of records fetched from the leader of the raft quorum.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_fetch_records_rate
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_fetch_records_rate
    • count
      Prometheus Name: ic_node_fetch_records_rate
• k::currentEpoch The current quorum epoch.
  
  • Sub-type: value
    Prometheus Name: ic_node_current_epoch
• k::globalPartitionCount The number of global partitions according to this Controller.
  
  • Sub-type: value
    Prometheus Name: ic_node_global_partition_count
• k::globalTopicCount The number of global topics according to this Controller.
  
  • Sub-type: value
    Prometheus Name: ic_node_global_topic_count
• k::lastAppliedRecordLagMs The difference between current time and the timestamp in milliseconds of the last record from the cluster metadata partition applied by this Controller.
  
  • Sub-type: value
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_last_applied_record_lag_ms_milliseconds
• k::lastAppliedRecordOffset The offset of the last record from the cluster metadata partition applied by this Controller.
  
  • Sub-type: value
    Prometheus Name: ic_node_last_applied_record_offset
• k::lastAppliedRecordTimestamp The timestamp in milliseconds of the last record from the cluster metadata partition applied by this Controller.
  
  • Sub-type: value
    Prometheus Name: ic_node_last_applied_record_timestamp
• k::newActiveControllersCount Counts the number of times this node has seen a new controller elected. A transition to the "no leader" state is not counted here. If the same controller as before becomes active, that still counts. NOTE: This metric is for kraft only
  
  • Sub-type: value
    Prometheus Name: ic_node_new_active_controllers_count
• k::timedOutBrokerHeartbeatCount The number of broker heartbeats that timed out on this controller since the process was started. Note that only active controllers handle heartbeats, so only they will see increases in this metric. NOTE: This metric is for kraft only
  
  • Sub-type: value
    Prometheus Name: ic_node_timed_out_broker_heartbeat_count
• k::currentMetadataVersion Outputs the feature level of the current effective metadata version. NOTE: This metric is for kraft only
  
  • Sub-type: value
    Prometheus Name: ic_node_current_metadata_version
• k::currentControllerId The CurrentControllerId metric shows the ID of the controller, as seen by the node in question. If the current node doesn't think there is an active controller, the value of this metric will be -1. NOTE: This metric is for kraft only
  
  • Sub-type: value
    Prometheus Name: ic_node_current_controller_id
• k::remoteLogReaderTaskQueueSize Size of the queue holding remote storage read tasks
  
  • Sub-type: value
    Prometheus Name: ic_node_remote_log_reader_task_queue_size
• k::remoteLogReaderAvgIdlePercent Average idle percent of thread pool for processing remote storage read tasks.
  
  • Sub-type: value
    Prometheus Name: ic_node_remote_log_reader_avg_idle_percent
• k::remoteLogManagerTasksAvgIdlePercent Average idle percent of thread pool for copying data to remote storage.
  
  • Sub-type: value
    Prometheus Name: ic_node_remote_log_manager_tasks_avg_idle_percent
• k::expiresPerSec Rate of bytes read from remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_node_expires_per_sec
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_node_expires_per_sec

Per-topic metric names follow the format kt::{topic}::{metricName}. Optionally, a ‘sub-type’ may be specified to return a specific part of the metric - kt::{topic}::{metricName}:{subType}

• kt::{topic}::messagesInPerTopic The rate of messages received by the topic. One sub-type must be specified.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_messages_in_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_messages_in_per_topic
• kt::{topic}::bytesInPerTopic The rate of incoming bytes to the topic per second. One sub-type must be specified.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_bytes_in_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_bytes_in_per_topic
• kt::{topic}::bytesOutPerTopic The rate of outgoing bytes from the topic. One sub-type must be specified.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_bytes_out_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_bytes_out_per_topic
• kt::{topic}::fetchMessageConversionsPerTopic The amount and rate of fetch request messages which required message format conversions for the topic. One sub-type must be specified.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_fetch_message_conversions_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_fetch_message_conversions_per_topic
    • count
      Prometheus Name: ic_topic_fetch_message_conversions_per_topic
• kt::{topic}::produceMessageConversionsPerTopic The amount and rate of produce request messages which required message format conversions for the topic. One sub-type must be specified.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_produce_message_conversions_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_produce_message_conversions_per_topic
    • count
      Prometheus Name: ic_topic_produce_message_conversions_per_topic
• kt::{topic}::failedFetchMessagePerTopic The amount and rate of failed fetch requests to the topic. One sub-type must be specified.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_failed_fetch_message_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_failed_fetch_message_per_topic
    • count
      Prometheus Name: ic_topic_failed_fetch_message_per_topic
• kt::{topic}::failedProduceMessagePerTopic The amount and rate of failed produce requests to the topic. One sub-type must be specified.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_failed_produce_message_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_failed_produce_message_per_topic
    • count
      Prometheus Name: ic_topic_failed_produce_message_per_topic
• kt::{topic}::diskUsage The total size fo the files on disk associated with the topic, summed across all partitions.
  
  • Sub-type: disk_usage_kilobytes The total size of the files on disk associated with the topic, summed across all partitions.
    Unit: kilobytes (KB)
    Prometheus Name: ic_topic_disk_usage
• kt::{topic}::remoteCopyLagBytes Rate of bytes read from remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_remote_copy_lag_bytes
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_remote_copy_lag_bytes
• kt::{topic}::remoteDeleteLagBytes Rate of bytes read from remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_remote_delete_lag_bytes
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_remote_delete_lag_bytes
• kt::{topic}::remoteLogSizeBytes Rate of bytes read from remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_remote_log_size_bytes
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_remote_log_size_bytes
• kt::{topic}::remoteFetchBytesPerSecPerTopic Rate of bytes read from remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_remote_fetch_bytes_per_sec_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_remote_fetch_bytes_per_sec_per_topic
• kt::{topic}::remoteFetchRequestsPerSecPerTopic Rate of read requests from remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_remote_fetch_requests_per_sec_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_remote_fetch_requests_per_sec_per_topic
• kt::{topic}::remoteFetchErrorsPerSecPerTopic Rate of read errors from remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_remote_fetch_errors_per_sec_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_remote_fetch_errors_per_sec_per_topic
• kt::{topic}::remoteCopyBytesPerSecPerTopic Rate of bytes copied to remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_remote_copy_bytes_per_sec_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_remote_copy_bytes_per_sec_per_topic
• kt::{topic}::remoteCopyRequestsPerSecPerTopic Rate of write requests to remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_remote_copy_requests_per_sec_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_remote_copy_requests_per_sec_per_topic
• kt::{topic}::remoteCopyErrorsPerSecPerTopic Rate of write errors from remote storage per topic.
  
  • Available sub-types:
    • one_minute_rate One minute rate of the measured metric.
      Prometheus Name: ic_topic_remote_copy_errors_per_sec_per_topic
    • mean_rate The average rate of the measured metric.
      Prometheus Name: ic_topic_remote_copy_errors_per_sec_per_topic

Per-user metric names follow the format ku::{user}::{metricName}. Per-user metric can take up to 50 minutes to be refreshed in case of user removal or user becoming idle. Optionally, a ‘sub-type’ may be specified to return a specific part of the metric - ku::{user}::{metricName}:{subType}

• ku::{user}::produceBandwidthQuotaPerUser Bandwidth quota metrics (produce) per user
  
  • Available sub-types:
    • byte_rate
      Prometheus Name: ic_user_produce_bandwidth_quota_per_user
    • throttle_time
      Prometheus Name: ic_user_produce_bandwidth_quota_per_user
• ku::{user}::fetchBandwidthQuotaPerUser Bandwidth quota metrics (fetch) per user
  
  • Available sub-types:
    • byte_rate
      Prometheus Name: ic_user_fetch_bandwidth_quota_per_user
    • throttle_time
      Prometheus Name: ic_user_fetch_bandwidth_quota_per_user

• r::masterSlotsCount The number of hash slots a master node has been assigned. The number of hash slots of all master nodes should add to 16384.
  
  • Sub-type: value
    Prometheus Name: ic_node_master_slots_count
• r::clusterUnassignedSlotsCount Number of slots which are NOT associated to some node (unbound).
  
  • Sub-type: value
    Prometheus Name: ic_node_cluster_unassigned_slots_count
• r::clusterSlotsNotOkCount Number of hash slots mapping to a node in FAIL or PFAIL state.
  
  • Sub-type: value
    Prometheus Name: ic_node_cluster_slots_not_ok_count
• r::slaWritesLatency The average and maximum time taken in milliseconds by a client to write to a random master node in the cluster.
  
  • Available sub-types:
    • average Average value of the metric.
      Prometheus Name: ic_node_sla_writes_latency
    • max Maximum value of the metric.
      Prometheus Name: ic_node_sla_writes_latency
• r::slaWritesSuccessfulOps Number of successful write operations performed on the cluster. Every 20 seconds, 30 synthetic write transactions are performed on each node.
  
  • Sub-type: count
    Prometheus Name: ic_node_sla_writes_successful_ops
• r::slaWritesFailedOps Number of failed write operations performed on the cluster.
  
  • Sub-type: count
    Prometheus Name: ic_node_sla_writes_failed_ops
• r::slaReadsLatency The average and maximum time taken in milliseconds by a client to read from a random node in the cluster.
  
  • Available sub-types:
    • average Average value of the metric.
      Prometheus Name: ic_node_sla_reads_latency
    • max Maximum value of the metric.
      Prometheus Name: ic_node_sla_reads_latency
• r::slaReadsSuccessfulOps Number of successful read operations performed on the cluster. Every 20 seconds, 30 synthetic read transactions are performed on each node.
  
  • Sub-type: count
    Prometheus Name: ic_node_sla_reads_successful_ops
• r::slaReadsFailedOps Number of failed read operations performed on the cluster.
  
  • Sub-type: count
    Prometheus Name: ic_node_sla_reads_failed_ops
• r::localWritesLatency Tthe average and maximum time taken in milliseconds by a client to write to its local node.
  
  • Available sub-types:
    • average Average value of the metric.
      Prometheus Name: ic_node_local_writes_latency
    • max Maximum value of the metric.
      Prometheus Name: ic_node_local_writes_latency
• r::localWritesSuccessfulOps Number of successful write operations performed on the local node. Every 20 seconds, 30 synthetic write transactions are performed on each node.
  
  • Sub-type: count
    Prometheus Name: ic_node_local_writes_successful_ops
• r::localWritesFailedOps Number of failed write operations performed on the local node.
  
  • Sub-type: count
    Prometheus Name: ic_node_local_writes_failed_ops
• r::localReadsLatency The average and maximum time taken in milliseconds by a client to read from its local node.
  
  • Available sub-types:
    • average Average value of the metric.
      Prometheus Name: ic_node_local_reads_latency
    • max Maximum value of the metric.
      Prometheus Name: ic_node_local_reads_latency
• r::localReadsSuccessfulOps Number of successful read operations performed on the local node. Every 20 seconds, 30 synthetic read transactions are performed on each node.
  
  • Sub-type: count
    Prometheus Name: ic_node_local_reads_successful_ops
• r::localReadsFailedOps Number of failed read operations performed on the local node.
  
  • Sub-type: count
    Prometheus Name: ic_node_local_reads_failed_ops
• r::usedMemory Total memory in megabytes allocated by Redis using its allocator (either standard libc, jemalloc, or an alternative allocator such as tcmalloc).
  
  • Sub-type: value
    Prometheus Name: ic_node_used_memory
• r::usedMemoryRss Memory in megabytes that Redis allocated as seen by the operating system (a.k.a resident set size). This is the number reported by tools such as top(1) and ps(1).
  
  • Sub-type: value
    Prometheus Name: ic_node_used_memory_rss
• r::usedMemoryDataset The size in bytes of the dataset.
  
  • Sub-type: value
    Prometheus Name: ic_node_used_memory_dataset
• r::usedMemoryLua Number of bytes used by the Lua engine.
  
  • Sub-type: value
    Prometheus Name: ic_node_used_memory_lua
• r::memoryFragmentationRatio Ratio between Used Memory Rss and Used Memory.
  
  • Sub-type: value
    Prometheus Name: ic_node_memory_fragmentation_ratio
• r::connectedClients Number of clients connected to the node.
  
  • Sub-type: value
    Prometheus Name: ic_node_connected_clients
• r::operationsPerSec Number of commands processed per second.
  
  • Sub-type: value
    Prometheus Name: ic_node_operations_per_sec
• r::roleIsMaster Is the node the master, will be 1.0 if it is and 0.0 otherwise
  
  • Sub-type: state
    Prometheus Name: ic_node_role_is_master

• z::electionTimeTaken Time taken to complete election.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_election_time_taken_milliseconds
• z::packetsReceived Number of packet operations received.
  
  • Sub-type: value
    Prometheus Name: ic_node_packets_received
• z::txnLogElapsedSyncTime The elapsed sync time of transaction log in milliseconds.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_txn_log_elapsed_sync_time_milliseconds
• z::packetsSent Number of packet operations sent.
  
  • Sub-type: value
    Prometheus Name: ic_node_packets_sent
• z::numAliveConnections Total number of active client connections in the server.
  
  • Sub-type: value
    Prometheus Name: ic_node_num_alive_connections
• z::maxRequestLatency Maximum time it takes for the server to respond to a request.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_max_request_latency_milliseconds
• z::minRequestLatency Minimum time it takes for the server to respond to a request.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_min_request_latency_milliseconds
• z::avgRequestLatency Average time it takes for the server to respond to a request.
  
  • Sub-type: ms
    Unit: milliseconds (ms)
    Prometheus Name: ic_node_avg_request_latency_milliseconds
• z::outstandingRequests Number of pending requests in the server.
  
  • Sub-type: value
    Prometheus Name: ic_node_outstanding_requests
• z::openFileDescriptorCount Number of file descriptors in use.
  
  • Sub-type: value
    Prometheus Name: ic_node_open_file_descriptor_count
• z::lastZxidCounter Last Zookeeper Transaction ID (ZXID) counter value.
  
  • Sub-type: value
    Prometheus Name: ic_node_last_zxid_counter

Summary metric names follow the format cads::{metricName}. Optionally, a ‘sub-type’ may be specified to return a specific part of the metric - cads::{metricName}::{subType}

• cads::frontendV2MemoryHeapInUse The current heap memory usage of the Cadence Frontend service, in bytes.
  
  • Sub-type: value
    Unit: bytes (B)
    Prometheus Name: ic_node_frontend_v2_memory_heap_in_use_bytes
• cads::frontendV2MemoryAllocated The current memory allocation to the Cadence Frontend service, in bytes.
  
  • Sub-type: value
    Unit: bytes (B)
    Prometheus Name: ic_node_frontend_v2_memory_allocated_bytes
• cads::matchingV2MemoryHeapInUse The current heap memory usage of the Cadence Matching service, in bytes.
  
  • Sub-type: value
    Unit: bytes (B)
    Prometheus Name: ic_node_matching_v2_memory_heap_in_use_bytes
• cads::matchingV2MemoryAllocated The current memory allocation to the Cadence Matching service, in bytes.
  
  • Sub-type: value
    Unit: bytes (B)
    Prometheus Name: ic_node_matching_v2_memory_allocated_bytes
• cads::historyV2MemoryHeapInUse The current heap memory usage of the Cadence History service, in bytes.
  
  • Sub-type: value
    Unit: bytes (B)
    Prometheus Name: ic_node_history_v2_memory_heap_in_use_bytes
• cads::historyV2MemoryAllocated The current memory allocation to the Cadence History service, in bytes.
  
  • Sub-type: value
    Unit: bytes (B)
    Prometheus Name: ic_node_history_v2_memory_allocated_bytes
• cads::workerV2MemoryHeapInUse The current heap memory usage of the Cadence Worker service, in bytes.
  
  • Sub-type: value
    Unit: bytes (B)
    Prometheus Name: ic_node_worker_v2_memory_heap_in_use_bytes
• cads::workerV2MemoryAllocated The current memory allocation to the Cadence Worker service, in bytes.
  
  • Sub-type: value
    Unit: bytes (B)
    Prometheus Name: ic_node_worker_v2_memory_allocated_bytes
• cads::slaV2WorkflowSuccess Number of reported Cadence Canary workflow successes, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_sla_v2_workflow_success
• cads::slaV2WorkflowCancel Number of reported Cadence Canary workflow cancellations, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_sla_v2_workflow_cancel
• cads::slaV2WorkflowFail Number of reported Cadence Canary workflow failures, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_sla_v2_workflow_fail
• cads::slaV2WorkflowTimeout Number of reported Cadence Canary workflow time-outs, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_sla_v2_workflow_timeout
• cads::slaV2WorkflowTerminate Number of reported Cadence Canary workflow terminations, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_sla_v2_workflow_terminate
• cads::slaV2WorkflowLatency The average end-to-end latency of the Cadence Canary workflow, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_sla_v2_workflow_latency_seconds
• cads::frontendV2MeanPersistenceRequestRate Average Number of persistence requests made by the Cadence Frontend service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_frontend_v2_mean_persistence_request_rate
• cads::frontendV2MeanPersistenceErrorRate Average Number of internal errors from persistence requests made by the Cadence Frontend service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_frontend_v2_mean_persistence_error_rate
• cads::frontendV2MeanPersistenceLatency Average Latency of persistence requests made by the Cadence Frontend service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_frontend_v2_mean_persistence_latency_seconds
• cads::frontendV2MeanCadenceRequestRate Average Number of Cadence requests made to the Cadence Frontend service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_frontend_v2_mean_cadence_request_rate
• cads::frontendV2MeanCadenceErrorRate Average Number of internal errors from Cadence requests made to the Cadence Frontend service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_frontend_v2_mean_cadence_error_rate
• cads::frontendV2MeanCadenceLatency Average Latency of Cadence requests made to the Cadence Frontend service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_frontend_v2_mean_cadence_latency_seconds
• cads::syncMatchV2Latency Average synchronous match latency of the Cadence Matching service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_sync_match_v2_latency_seconds
• cads::asyncMatchV2Latency Average asynchronous match latency of the Cadence Matching service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_async_match_v2_latency_seconds
• cads::matchingV2MeanPersistenceRequestRate Average Number of persistence requests made by the Cadence Matching service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_matching_v2_mean_persistence_request_rate
• cads::matchingV2MeanPersistenceErrorRate Average Number of internal errors from persistence requests made by the Cadence Matching service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_matching_v2_mean_persistence_error_rate
• cads::matchingV2MeanPersistenceLatency Average Latency of persistence requests made by the Cadence Matching service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_matching_v2_mean_persistence_latency_seconds
• cads::matchingV2MeanCadenceRequestRate Average Number of Cadence requests made to the Cadence Matching service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_matching_v2_mean_cadence_request_rate
• cads::matchingV2MeanCadenceErrorRate Average Number of internal errors from Cadence requests made to the Cadence Matching service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_matching_v2_mean_cadence_error_rate
• cads::matchingV2MeanCadenceLatency Average Latency of Cadence requests made to the Cadence Matching service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_matching_v2_mean_cadence_latency_seconds
• cads::historyV2MeanCadenceRequestRate Average Number of Cadence requests made to the Cadence History service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_cadence_request_rate
• cads::historyV2MeanCadenceErrorRate Average Number of internal errors from Cadence requests made to the Cadence History service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_cadence_error_rate
• cads::historyV2MeanCadenceLatency Average Latency of Cadence requests made to the Cadence History service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_history_v2_mean_cadence_latency_seconds
• cads::historyV2MeanPersistenceRequestRate Average Number of persistence requests made by the Cadence History service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_persistence_request_rate
• cads::historyV2MeanPersistenceErrorRate Average Number of internal errors from persistence requests made by the Cadence History service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_persistence_error_rate
• cads::historyV2MeanPersistenceLatency Average Latency of persistence requests made by the Cadence History service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_history_v2_mean_persistence_latency_seconds
• cads::historyV2MeanTaskRequestRate Average Number of task requests to the Cadence History service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_task_request_rate
• cads::historyV2MeanTaskErrorRate Average Number of errors from task requests to the Cadence History service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_task_error_rate
• cads::historyV2MeanTaskLatency Average Execution latency of tasks in the Cadence History service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_history_v2_mean_task_latency_seconds
• cads::historyV2MeanTaskLatencyQueue Average Queue latency of tasks in the Cadence History service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_history_v2_mean_task_latency_queue_seconds
• cads::historyV2MeanTaskLatencyProcessing Average Processing latency of tasks in the Cadence History service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_history_v2_mean_task_latency_processing_seconds
• cads::historyV2MeanWorkflowSuccess Average Number of successful workflows, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_workflow_success
• cads::historyV2MeanWorkflowCancel Average Number of cancelled workflows, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_workflow_cancel
• cads::historyV2MeanWorkflowFailed Average Number of failed workflows, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_workflow_failed
• cads::historyV2MeanWorkflowTimeout Average Number of timed out workflows, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_workflow_timeout
• cads::historyV2MeanWorkflowTerminate Average Number of terminated workflows, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_workflow_terminate
• cads::historyV2MeanReplicationTasksApplied Average Number of successfully applied replication tasks in the Cadence History service.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_replication_tasks_applied
• cads::historyV2MeanReplicationTasksAppliedLatency Average latency from replication tasks being received to them being applied in the Cadence History service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_history_v2_mean_replication_tasks_applied_latency_seconds
• cads::historyV2MeanReplicationTaskLatency Average latency from replication tasks being created to them being applied in the Cadence History service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_history_v2_mean_replication_task_latency_seconds
• cads::historyV2MeanReplicationTaskCleanupCount Average Number of cleaned up replication tasks after being acknowledged by the standby Cadence clusters in the Cadence History service.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_replication_task_cleanup_count
• cads::historyV2MeanReplicationTaskCleanupFailed Average Number of replication tasks failed to be cleaned up after being acknowledged by the standby Cadence clusters in the Cadence History service.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_replication_task_cleanup_failed
• cads::historyV2ReplicationDlqSize Size of the DLQ of replication tasks that could not be applied after retry in the Cadence History service.
  
  • Sub-type: value
    Prometheus Name: ic_node_history_v2_replication_dlq_size
• cads::historyV2MeanReplicationDlqEnqueueFailed Average Number of replication tasks that could not be applied after retry and are failed to be put into DLQ in the Cadence History service.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_history_v2_mean_replication_dlq_enqueue_failed
• cads::workerV2MeanPersistenceRequestRate Average Number of persistence requests made by the Cadence Worker service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_worker_v2_mean_persistence_request_rate
• cads::workerV2MeanPersistenceErrorRate Average Number of internal errors from persistence requests made by the Cadence Worker service, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_node_worker_v2_mean_persistence_error_rate
• cads::workerV2MeanPersistenceLatency Average Latency of persistence requests made by the Cadence Worker service, in seconds.
  
  • Sub-type: average
    Unit: seconds (s)
    Prometheus Name: ic_node_worker_v2_mean_persistence_latency_seconds

Tag-level metric names follow the format cadt::{tag}::{metricName}. Optionally, a ‘sub-type’ may be specified to return a specific part of the metric - cadt::{tag}::{metricName}::{subType}

• cadt::{tag}::frontendV2PersistenceRequestRate Number of persistence requests made by the Cadence Frontend service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_persistence_request_rate
• cadt::{tag}::frontendV2PersistenceErrorRate Number of internal errors from persistence requests made by the Cadence Frontend service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_persistence_error_rate
• cadt::{tag}::frontendV2PersistenceLatency Latency of persistence requests made by the Cadence Frontend service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_frontend_v2_persistence_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_frontend_v2_persistence_latency_seconds
• cadt::{tag}::frontendV2CadenceRequestRate Number of Cadence requests made to the Cadence Frontend service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_cadence_request_rate
• cadt::{tag}::frontendV2CadenceErrorRate Number of internal errors from Cadence requests made to the Cadence Frontend service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_cadence_error_rate
• cadt::{tag}::frontendV2CadenceClientBadRequestErrorRate Number of client-side errors (bad request) from Cadence requests made to the Cadence Frontend service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_cadence_client_bad_request_error_rate
• cadt::{tag}::frontendV2CadenceClientServiceBusyErrorRate Number of client-side errors (service busy) from Cadence requests made to the Cadence Frontend service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_cadence_client_service_busy_error_rate
• cadt::{tag}::frontendV2CadenceClientCriticalErrorRate Number of client-side errors (critical) from Cadence requests made to the Cadence Frontend service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_cadence_client_critical_error_rate
• cadt::{tag}::frontendV2CadenceClientQueryFailedErrorRate Number of client-side errors (query failed) from Cadence requests made to the Cadence Frontend service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_cadence_client_query_failed_error_rate
• cadt::{tag}::frontendV2CadenceClientLimitExceededErrorRate Number of client-side errors (limit exceeded) from Cadence requests made to the Cadence Frontend service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_cadence_client_limit_exceeded_error_rate
• cadt::{tag}::frontendV2CadenceClientContextTimeoutErrorRate Number of client-side errors (context timeout) from Cadence requests made to the Cadence Frontend service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_cadence_client_context_timeout_error_rate
• cadt::{tag}::frontendV2CadenceClientRetryTaskErrorRate Number of client-side errors (retry task) from Cadence requests made to the Cadence Frontend service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_frontend_v2_cadence_client_retry_task_error_rate
• cadt::{tag}::frontendV2CadenceLatency Latency of Cadence requests made to the Cadence Frontend service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_frontend_v2_cadence_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_frontend_v2_cadence_latency_seconds
• cadt::{tag}::matchingV2CadenceRequestRate Number of Cadence requests made to the Cadence Matching service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_cadence_request_rate
• cadt::{tag}::matchingV2CadenceErrorRate Number of internal errors from Cadence requests made to the Cadence Matching service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_cadence_error_rate
• cadt::{tag}::matchingV2CadenceLatency Latency of Cadence requests made to the Cadence Matching service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_matching_v2_cadence_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_matching_v2_cadence_latency_seconds
• cadt::{tag}::matchingV2CadenceClientBadRequestErrorRate Number of client-side errors (bad request) from Cadence requests made to the Cadence Matching service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_cadence_client_bad_request_error_rate
• cadt::{tag}::matchingV2CadenceClientServiceBusyErrorRate Number of client-side errors (service busy) from Cadence requests made to the Cadence Matching service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_cadence_client_service_busy_error_rate
• cadt::{tag}::matchingV2CadenceClientCriticalErrorRate Number of client-side errors (critical) from Cadence requests made to the Cadence Matching service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_cadence_client_critical_error_rate
• cadt::{tag}::matchingV2CadenceClientQueryFailedErrorRate Number of client-side errors (query failed) from Cadence requests made to the Cadence Matching service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_cadence_client_query_failed_error_rate
• cadt::{tag}::matchingV2CadenceClientLimitExceededErrorRate Number of client-side errors (limit exceeded) from Cadence requests made to the Cadence Matching service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_cadence_client_limit_exceeded_error_rate
• cadt::{tag}::matchingV2CadenceClientContextTimeoutErrorRate Number of client-side errors (context timeout) from Cadence requests made to the Cadence Matching service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_cadence_client_context_timeout_error_rate
• cadt::{tag}::matchingV2CadenceClientRetryTaskErrorRate Number of client-side errors (retry task) from Cadence requests made to the Cadence Matching service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_cadence_client_retry_task_error_rate
• cadt::{tag}::matchingV2SyncMatchLatency The synchronous match latency of the Cadence Matching service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_matching_v2_sync_match_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_matching_v2_sync_match_latency_seconds
• cadt::{tag}::matchingV2AsyncMatchLatency The asynchronous match latency of the Cadence Matching service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_matching_v2_async_match_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_matching_v2_async_match_latency_seconds
• cadt::{tag}::matchingV2PersistenceRequestRate Number of persistence requests made by the Cadence Matching service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_persistence_request_rate
• cadt::{tag}::matchingV2PersistenceErrorRate Number of internal errors from persistence requests made by the Cadence Matching service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_matching_v2_persistence_error_rate
• cadt::{tag}::matchingV2PersistenceLatency Latency of persistence requests made by the Cadence Matching service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_matching_v2_persistence_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_matching_v2_persistence_latency_seconds
• cadt::{tag}::historyV2CadenceRequestRate Number of Cadence requests made to the Cadence History service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_cadence_request_rate
• cadt::{tag}::historyV2CadenceErrorRate Number of internal errors from Cadence requests made to the Cadence History service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_cadence_error_rate
• cadt::{tag}::historyV2CadenceLatency Latency of Cadence requests made to the Cadence History service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_cadence_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_cadence_latency_seconds
• cadt::{tag}::historyV2CadenceClientBadRequestErrorRate Number of client-side errors (bad request) from Cadence requests made to the Cadence History service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_cadence_client_bad_request_error_rate
• cadt::{tag}::historyV2CadenceClientServiceBusyErrorRate Number of client-side errors (service busy) from Cadence requests made to the Cadence History service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_cadence_client_service_busy_error_rate
• cadt::{tag}::historyV2CadenceClientCriticalErrorRate Number of client-side errors (critical) from Cadence requests made to the Cadence History service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_cadence_client_critical_error_rate
• cadt::{tag}::historyV2CadenceClientQueryFailedErrorRate Number of client-side errors (query failed) from Cadence requests made to the Cadence History service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_cadence_client_query_failed_error_rate
• cadt::{tag}::historyV2CadenceClientLimitExceededErrorRate Number of client-side errors (limit exceeded) from Cadence requests made to the Cadence History service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_cadence_client_limit_exceeded_error_rate
• cadt::{tag}::historyV2CadenceClientContextTimeoutErrorRate Number of client-side errors (context timeout) from Cadence requests made to the Cadence History service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_cadence_client_context_timeout_error_rate
• cadt::{tag}::historyV2CadenceClientRetryTaskErrorRate Number of client-side errors (retry task) from Cadence requests made to the Cadence History service, per operation, in seconds.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_cadence_client_retry_task_error_rate
• cadt::{tag}::historyV2PersistenceRequestRate Number of persistence requests made by the Cadence History service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_persistence_request_rate
• cadt::{tag}::historyV2PersistenceErrorRate Number of internal errors from persistence requests made by the Cadence History service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_persistence_error_rate
• cadt::{tag}::historyV2PersistenceLatency Latency of persistence requests made by the Cadence History service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_persistence_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_persistence_latency_seconds
• cadt::{tag}::historyV2TaskRequestRate Number of task requests to the Cadence History service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_task_request_rate
• cadt::{tag}::historyV2TaskErrorRate Number of errors from task requests to the Cadence History service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_task_error_rate
• cadt::{tag}::historyV2TaskLatency Execution latency of tasks in the Cadence History service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_task_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_task_latency_seconds
• cadt::{tag}::historyV2TaskLatencyQueue End-to-end latency of tasks in the Cadence History service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_task_latency_queue_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_task_latency_queue_seconds
• cadt::{tag}::historyV2TaskLatencyProcessing Processing latency of tasks in the Cadence History service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_task_latency_processing_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_task_latency_processing_seconds
• cadt::{tag}::historyV2WorkflowSuccess Number of successful workflows, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_workflow_success
• cadt::{tag}::historyV2WorkflowCancel Number of cancelled workflows, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_workflow_cancel
• cadt::{tag}::historyV2WorkflowFailed Number of failed workflows, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_workflow_failed
• cadt::{tag}::historyV2WorkflowTimeout Number of timed out workflows, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_workflow_timeout
• cadt::{tag}::historyV2WorkflowTerminate Number of terminated workflows, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_workflow_terminate
• cadt::{tag}::historyV2WorkflowFailedCount Number of failed workflows count.
  
  • Sub-type: value
    Prometheus Name: ic_cadence_history_v2_workflow_failed_count
• cadt::{tag}::historyV2ReplicationTasksApplied Average Number of successfully applied replication tasks in the Cadence History service, per operation.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_replication_tasks_applied
• cadt::{tag}::historyV2ReplicationTasksAppliedPerDomain Average Number of successfully applied replication tasks in the Cadence History service, per domain.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_replication_tasks_applied_per_domain
• cadt::{tag}::historyV2ReplicationTasksAppliedLatency Latency from replication tasks being received to them being applied in the Cadence History service, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_replication_tasks_applied_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_replication_tasks_applied_latency_seconds
• cadt::{tag}::historyV2ReplicationTaskLatency Latency from replication tasks being created to them being applied in the Cadence History service, in seconds
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_replication_task_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_history_v2_replication_task_latency_seconds
• cadt::{tag}::historyV2ReplicationTaskCleanupCount Average Number of cleaned up replication tasks after being acknowledged by the standby Cadence clusters in the Cadence History service, per operation.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_replication_task_cleanup_count
• cadt::{tag}::historyV2ReplicationTaskCleanupFailed Average Number of replication tasks failed to be cleaned up after being acknowledged by the standby Cadence clusters in the Cadence History service, per operation.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_replication_task_cleanup_failed
• cadt::{tag}::historyV2ReplicationDlqSize Size of the DLQ of replication tasks that could not be applied after retry in the Cadence History service, per operation.
  
  • Sub-type: value
    Prometheus Name: ic_cadence_history_v2_replication_dlq_size
• cadt::{tag}::historyV2ReplicationDlqEnqueueFailed Average Number of replication tasks that could not be applied after retry and are failed to be put into DLQ in the Cadence History service, per operation.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_history_v2_replication_dlq_enqueue_failed
• cadt::{tag}::workerV2PersistenceRequestRate Number of persistence requests made by the Cadence Worker service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_worker_v2_persistence_request_rate
• cadt::{tag}::workerV2PersistenceErrorRate Number of internal errors from persistence requests made by the Cadence Worker service, per operation, per second.
  
  • Sub-type: count_per_second
    Unit: units per second (1/s)
    Prometheus Name: ic_cadence_worker_v2_persistence_error_rate
• cadt::{tag}::workerV2PersistenceLatency Latency of persistence requests made by the Cadence Worker service, per operation, in seconds.
  
  • Available sub-types:
    • 50thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_worker_v2_persistence_latency_seconds
    • 95thPercentile
      Unit: seconds (s)
      Prometheus Name: ic_cadence_worker_v2_persistence_latency_seconds

• clk::slaAvgWriteLatency Average write latency for 20 writes.
  
  • Sub-type: value
    Prometheus Name: ic_node_sla_avg_write_latency
• clk::slaAvgReadLatency Average read latency 20 reads.
  
  • Sub-type: value
    Prometheus Name: ic_node_sla_avg_read_latency
• clk::slaWriteErrors Number of write request errors.
  
  • Sub-type: value
    Prometheus Name: ic_node_sla_write_errors
• clk::slaReadErrors Number of read request errors.
  
  • Sub-type: value
    Prometheus Name: ic_node_sla_read_errors
• clk::slaKeeperErrors Number of ClickHouse Keeper errors.
  
  • Sub-type: value
    Prometheus Name: ic_node_sla_keeper_errors
• clk::rwLockWaitingReaders Number of threads waiting for read on a table RWLock.
  
  • Sub-type: value
    Prometheus Name: ic_node_rw_lock_waiting_readers
• clk::rwLockWaitingWriters Number of threads waiting for write on a table RWLock.
  
  • Sub-type: value
    Prometheus Name: ic_node_rw_lock_waiting_writers
• clk::merge Number of executing background merges.
  
  • Sub-type: value
    Prometheus Name: ic_node_merge
• clk::readonlyReplica Number of Replicated tables that are currently in readonly state due to re-initialization after ZooKeeper session loss or due to startup without ZooKeeper configured.
  
  • Sub-type: value
    Prometheus Name: ic_node_readonly_replica
• clk::query Number of executing queries.
  
  • Sub-type: value
    Prometheus Name: ic_node_query
• clk::delayedInserts Number of INSERT queries that are throttled due to high number of active data parts for partition in a MergeTree table.
  
  • Sub-type: value
    Prometheus Name: ic_node_delayed_inserts
• clk::s3Requests Number of S3 requests.
  
  • Sub-type: value
    Prometheus Name: ic_node_s3_requests
• clk::distributedFilesToInsert Number of pending files to process for asynchronous insertion into Distributed tables.
  
  • Sub-type: value
    Prometheus Name: ic_node_distributed_files_to_insert
• clk::keeperOutstandingRequests Number of outstanding ClickHouse Keeper requests.
  
  • Sub-type: value
    Prometheus Name: ic_node_keeper_outstanding_requests
• clk::insertQueriesPerSecond Average number of insert queries per second over the last one minute.
  
  • Sub-type: value
    Prometheus Name: ic_node_insert_queries_per_second
• clk::httpConnection Number of connections to HTTP server.
  
  • Sub-type: value
    Prometheus Name: ic_node_http_connection
• clk::totalRows The total number of rows for all active parts.
  
  • Sub-type: value
    Prometheus Name: ic_node_total_rows
• clk::pendingAsyncInsert Number of asynchronous inserts waiting to be flushed.
  
  • Sub-type: value
    Prometheus Name: ic_node_pending_async_insert
• clk::osOpenFiles The total number of opened files on the host machine. This is a system-wide metric, it includes all the processes on the host machine, not just clickhouse-server.
  
  • Sub-type: value
    Prometheus Name: ic_node_os_open_files
• clk::mergesInQueue The total number of merge operations that are waiting in queue.
  
  • Sub-type: value
    Prometheus Name: ic_node_merges_in_queue
• clk::maxInactiveParts The maximum number of inactive parts
  
  • Sub-type: value
    Prometheus Name: ic_node_max_inactive_parts
• clk::znodeCount The number of znodes in ClickHouse Keeper process.
  
  • Sub-type: value
    Prometheus Name: ic_node_znode_count
• clk::totalPartsOfMergeTreeTables Total amount of data parts in all tables of MergeTree family. Numbers larger than 10 000 will negatively affect the server startup time, and it may indicate unreasonable choice of the partition key.
  
  • Sub-type: value
    Prometheus Name: ic_node_total_parts_of_merge_tree_tables
• clk::totalRowsOfMergeTreeTables Total amount of rows (records) stored in all tables of MergeTree family.
  
  • Sub-type: value
    Prometheus Name: ic_node_total_rows_of_merge_tree_tables
• clk::maxPartCountForPartition Maximum number of parts per partition across all partitions of all tables of MergeTree family. Values larger than 300 indicates misconfiguration, overload, or massive data loading.
  
  • Sub-type: value
    Prometheus Name: ic_node_max_part_count_for_partition
• clk::replicasMaxAbsoluteDelay Maximum difference in seconds between the most fresh replicated part and the most fresh data part still to be replicated, across Replicated tables. A very high value indicates a replica with no data.
  
  • Sub-type: value
    Prometheus Name: ic_node_replicas_max_absolute_delay
• clk::remoteStorageUsage Total amount of data stored in remote storage (such as AWS S3), in GiB.
  
  • Sub-type: value
    Prometheus Name: ic_node_remote_storage_usage