Skip to main content

Monitoring JAVA on Kubernetes

Overview#

Java applications running in Kubernetes can be monitored in SnappyFlow using two approaches:

Java monitoring with sfKubeAgent#

In this option, the Java application should be run with Jolokia agent and sfKubeAgent running as a sidecar container and fetches metrics via Jolokia port. Refer to sfKubeAgent Overview

Prerequisites#

  • Copy Jolokia JAR into docker image

  • Run the java application with Jolokia JAR in docker image:

    -javaagent:/<path_jolokia_jar>/jolokia-jvm-<version>-agent.jar

Configurations#

Run sfKubeAgent with JVMJolokia plugin, which is specified using the config map shown below: 

apiVersion: v1 kind: ConfigMap metadata: name: jvm-configmap data: config.yaml: |- key: <profile_key> metrics: plugins: name: jvmjolokia enabled: true interval: 300 config: ip: 127.0.0.1 protocol: http port: <userDefinedJolokiaPort> context: jolokia monitorDeadlocks: false deadLockMonitoringInterval: 300

The example illustrates instantiating sfKubeAgent with jvm-configmap. sfAKubeAgent talks to the Java application via userDefinedJolokiaPort (this example used 8778) 

kind: Pod apiVersion: v1 metadata: name: my-first-pod-1 labels: snappyflow/appname: <app_name> snappyflow/projectname: <project_name> spec:containers: name: java-container image: <docker_id>/<docker_image>:<tag> ports: name: jolokiaport containerPort: <userDefinedJolokiaPort> Snappyflow's sfkubeagent container name: java-sfagent image: snappyflowml/sfagent:latest imagePullPolicy: Always command: /app/sfagent -enable-console-log env: name: APP_NAME value: <app_name> name: PROJECT_NAME value: <project_name> volumeMounts: name: configmap-jvm mountPath: /opt/sfagent/config.yaml subPath: config.yaml volumes: name: configmap-jvm configMap: name: jvm-configmap

Viewing data and dashboards#

  • Data generated by plugin can be viewed in “browse data” page inside the respective application under plugin=jvm_jolokia and documentType=jvm
  • Dashboard for this data can be instantiated by Importing dashboard template “JVM” to the application dashboard

Troubleshooting#

  1. Check if the Jolokia port is accessible

    From inside the application container, run a curl command to the userDefinedJolokiaPort.

    curl http://localhost:<userDefinedJolokiaPort>

  2. Check the logs in sfKubeAgent container for any errors

JVM Monitoring with Prometheus exporter#

Refer to Prometheus Exporter Overview. Prometheus exporter is deployed as a sidecar container in the application pod and connects to the JMX target exposed by the application to scrape the metrics. sfPod polls Prometheus exporter to scrape the metrics.

Pre-requisites#

sfPod can access Prometheus exporter at Service IP: userDefinedPrometheusPort

Configurations#

  • Run Java application with JMX options: 

    -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port= <userDefinedJMXPort> -Dcom.sun.management.jmxremote.authenticate=false - Dcom.sun.management.jmxremote.ssl=false

  • Start the Prometheus exporter with

    java -jar jmx_prometheus_httpserver.jar <userDefinedPrometheusPort> <exporterConfigFile>

    Configurations are passed using config map:

    apiVersion: v1 kind: ConfigMap metadata: labels: snappyflow/appname: <app_name> snappyflow/projectname: <project_name> data: jmx-config.yaml: | --- jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:<userDefinedJMXPort>/jmxrmi ssl: false rules: - pattern: '.*'

  • Prometheus exporter interfaces to JMX via userDefinedJMXPort. Example below uses 555S as the port.

  • Prometheus exporter exposes userDefinedPrometheusPort for scraping. Example uses 5556 as the port

  • Pod definition YAML that illustrates the configuration for Java application and exporter 

    kind: Pod apiVersion: v1 metadata:  name: my-first-pod labels: snappyflow/appname: <app_name> snappyflow/projectname: <project_name> spec: containers: name: app-container image: <docker_id>/<docker_image>:<tag> command: sh -c -x java -jar -Dcom.sun.management.jmxremote - Dcom.sun.management.jmxremote.port=<userDefinedJMXPort> - Dcom.sun.management.jmxremote.authenticate=false - Dcom.sun.management.jmxremote.ssl=false <application_jar> name: "exporter-container" image: "bitnami/jmx-exporter:latest" imagePullPolicy: command: sh -c -x java -jar jmx_prometheus_httpserver.jar <userDefinedPrometheusPort> /tmp/jmx-config.yaml ports: name: exporter-port containerPort: <userDefinedPrometheusPort> volumeMounts: name: configmap-jmx mountPath: /tmp volumes: name: configmap-jmx configMap: name: jmx-configmap

Viewing data and dashboards#

  • Data generated by plugin can be viewed in “browse data” page inside the respective application under plugin=‘kube-prom-jmx‘ and documentType=‘jmxStats’
  • Dashboard for this data can be instantiated by Importing dashboard template “JVM” to the application dashboard.

Troubleshooting#

  1. Check if the JMX port is accessible .From inside the application container, run a curl command to the userDefinedJMXPort.

    curl http://localhost:<userDefinedJMXPort>

  2. Check if metrics are getting scraped. From inside the exporter container, run a curl command to the userDefinedPrometheusPort

    curl  http://localhost:<userDefinedPrometheusPort>/metrics