Overview

You can use a Helm chart to ship k8s logs to Logz.io via Filebeat from Linux and Windows nodes.

Helm is a tool for managing packages of pre-configured Kubernetes resources using Charts. Logzio-k8s-logs allows you to ship logs from your Kubernetes cluster to Logz.io. You can either deploy this Daemonset with the standard Filebeat configuration or with Filebeat Autodiscover. (Learn more about Filebeat Autodiscover from Elastic.)

Filebeat’s basic configuration tends to split longer, multiline logs into multiple logs - 1 log per line. See the relevant tab for details and options for controlling this setting.

Helm 2 reached EOL on November 2020. This document follows the command syntax recommended for Helm 3, but the Chart will work with both Helm 2 and Helm 3.

For troubleshooting this solution, see our user guide.

Sending logs from nodes with taints

If you want to ship logs from any of the nodes that have a taint, make sure that the taint key values are listed in your in your daemonset/deployment configuration as follows:

tolerations:
- key: 
  operator: 
  value: 
  effect:

To determine if a node uses taints as well as to display the taint keys, run:

kubectl get nodes -o json | jq ".items[]|{name:.metadata.name, taints:.spec.taints}"

You need to use Helm client with version v3.9.0 or above.

Standard configuration

Before you begin, you’ll need:

  • Helm CLI installed
  • Outgoing traffic to destination port 5015 allowed
Add logzio-k8s-logs repo to your helm repo list
helm repo add logzio-helm https://logzio.github.io/logzio-helm
helm repo update
Deploy

Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

Replace <<LISTENER-REGION>> with your region’s code (for example, eu). For more information on finding your account’s region, see Account region.

Replace <<CLUSTER-NAME>> with your cluster’s name.

helm install --namespace=kube-system \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioRegion='<<LISTENER-REGION>>' \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
logzio-k8s-logs logzio-helm/logzio-k8s-logs
Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Logz.io.

Autodiscover configuration

Autodiscover allows you to adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.

Add logzio-k8s-logs repo to your helm repo list
helm repo add logzio-helm https://logzio.github.io/logzio-helm
helm repo update
Deploy

In the following commands, make the following changes:

  • Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

  • Replace <<LISTENER-REGION>> with your region’s code (for example, eu). For more information on finding your account’s region, see Account region.

  • Replace <<CLUSTER-NAME>> with your cluster’s name.

  • Use --set configType='<<TYPE>>' for linux based filebeat and/or --set filebeatWindowsConfigType='<<TYPE>>' for windows based filebeat.

This Daemonset’s default autodiscover configuration is hints based. If you wish to deploy it use:

helm install --namespace=kube-system \
--set configType='autodiscover' \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioRegion='<<LISTENER-REGION>>' \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
logzio-k8s-logs logzio-helm/logzio-k8s-logs

If you have a custom configuration, deploy with:

helm install --namespace=kube-system \
--set configType='auto-custom' \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioRegion='<<LISTENER-REGION>>' \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
--set-file filebeatConfig.autoCustomConfig=/path/to/your/config.yaml \
logzio-k8s-logs logzio-helm/logzio-k8s-logs

If you’re using a custom config, please make sure that you’re using a .yaml file with the following structure:

Filebeat requires a file extension specified for the log input.

filebeat.yml: |-
  filebeat.autodiscover:
  #....
    # your autodiscover config
    # ...
  processors:
    - add_cloud_metadata: ~
  fields:
    logzio_codec: ${LOGZIO_CODEC}
    token: ${LOGZIO_LOGS_SHIPPING_TOKEN}
    cluster: ${CLUSTER_NAME}
    type: ${LOGZIO_TYPE}
  fields_under_root: ${FIELDS_UNDER_ROOT}
  ignore_older: ${IGNORE_OLDER}
  output:
    logstash:
      hosts: ["${LOGZIO_LOGS_LISTENER_HOST}:5015"]
      ssl:
        certificate_authorities: ['/etc/pki/tls/certs/SectigoRSADomainValidationSecureServerCA.crt']
Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you still don’t see your logs, see Kubernetes log shipping troubleshooting.

Uninstalling the Chart

The command removes all the k8s components associated with the chart and deletes the release.
To uninstall the logzio-k8s-logs deployment:

helm uninstall --namespace=kube-system logzio-k8s-logs

Autodiscover configuration

Autodiscover allows you to adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.

Add logzio-k8s-logs repo to your helm repo list
helm repo add logzio-helm https://logzio.github.io/logzio-helm
helm repo update
Deploy

In the following commands, make the following changes:

  • Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to.

  • Replace <<LISTENER-REGION>> with your region’s code (for example, eu). For more information on finding your account’s region, see Account region.

  • Replace <<CLUSTER-NAME>> with your cluster’s name.

  • Use --set configType='<<TYPE>>' for linux based filebeat and/or --set filebeatWindowsConfigType='<<TYPE>>' for windows based filebeat.

This Daemonset’s default autodiscover configuration is hints based. If you wish to deploy it use:

helm install --namespace=kube-system \
--set configType='autodiscover' \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioRegion='<<LISTENER-REGION>>' \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
logzio-k8s-logs logzio-helm/logzio-k8s-logs

If you have a custom configuration, deploy with:

helm install --namespace=kube-system \
--set configType='auto-custom' \
--set secrets.logzioShippingToken='<<LOG-SHIPPING-TOKEN>>' \
--set secrets.logzioRegion='<<LISTENER-REGION>>' \
--set secrets.clusterName='<<CLUSTER-NAME>>' \
--set-file filebeatConfig.autoCustomConfig=/path/to/your/config.yaml \
logzio-k8s-logs logzio-helm/logzio-k8s-logs

If you’re using a custom config, please make sure that you’re using a .yaml file with the following structure:

filebeat.yml: |-
  filebeat.autodiscover:
  #....
    # your autodiscover config
    # ...
  processors:
    - add_cloud_metadata: ~
  fields:
    logzio_codec: ${LOGZIO_CODEC}
    token: ${LOGZIO_LOGS_SHIPPING_TOKEN}
    cluster: ${CLUSTER_NAME}
    type: ${LOGZIO_TYPE}
  fields_under_root: ${FIELDS_UNDER_ROOT}
  ignore_older: ${IGNORE_OLDER}
  output:
    logstash:
      hosts: ["${LOGZIO_LOGS_LISTENER_HOST}:5015"]
      ssl:
        certificate_authorities: ['/etc/pki/tls/certs/SectigoRSADomainValidationSecureServerCA.crt']
Check Logz.io for your logs

Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you still don’t see your logs, see Kubernetes log shipping troubleshooting.

Uninstalling the Chart

The command removes all the k8s components associated with the chart and deletes the release.
To uninstall the logzio-k8s-logs deployment:

helm uninstall --namespace=kube-system logzio-k8s-logs

Changing a default

If you wish to change the default values, specify each parameter using the --set key=value argument to helm install. For example,

helm install --namespace=kube-system logzio-k8s-logs logzio-helm/logzio-k8s-logs \
  --set imageTag=7.7.0 \
  --set terminationGracePeriodSeconds=30

Configurations & defaults

Parameter Description Default
image The linux based Filebeat docker image. docker.elastic.co/beats/filebeat
imageTag The linux based Filebeat docker image tag. 7.8.1
filebeatWindowsImage The windows based Filebeat docker image. docker.io/logzio/logzio-filebeat-win
filebeatWindowsImageTag The windows based Filebeat docker image tag. 0.0.1
winglogbeatImage The winlogbeat docker image. docker.io/logzio/logzio-winlogbeat
winglogbeatImageTag The winlogbeat docker image tag. 0.0.1
nameOverride Overrides the Chart name for resources. ""
fullnameOverride Overrides the full name of the resources. filebeat
namespaceOverride Overrides the namespace of the resources. ""
apiVersions.configMap ConfigMap API version. v1
apiVersions.daemonset Daemonset API version. apps/v1
apiVersions.clusterRoleBinding ClusterRoleBinding API version. rbac.authorization.k8s.io/v1
apiVersions.clusterRole ClusterRole API version. rbac.authorization.k8s.io/v1
apiVersions.serviceAccount ServiceAccount API version. v1
apiVersions.secret Secret API version. v1
managedServiceAccount Specifies whether the serviceAccount should be managed by this Helm Chart. Set this to false to manage your own service account and related roles. true
clusterRoleRules Configurable cluster role rules that Filebeat uses to access Kubernetes resources. See values.yaml
logzioCert Logzio public SSL certificate. See values.yaml
configType Specifies which configuration to use for Filebeat. Set to autodiscover to use autodiscover (available only for filebeat). standard
filebeatConfig.standardConfig Standard linux based Filebeat configuration, using filebeat.input. See values.yaml
filebeatConfig.autodiscoverConfig Autodiscover linux based Filebeat configuration, using filebeat.autodiscover. See values.yaml
filebeatConfig.autoCustomConfig Autodiscover linux based Filebeat custom configuration, using filebeat.autodiscover. Should be used if you want to use your customized autodiscover config {}
filebeatWindowsConfig.standardConfig Standard windows based Filebeat configuration, using filebeat.input. See values.yaml
filebeatConfig.autodiscoverConfig Autodiscover windows based Filebeat configuration, using filebeat.autodiscover. See values.yaml
filebeatConfig.autoCustomConfig Autodiscover windows based Filebeat custom configuration, using filebeat.autodiscover. Should be used if you want to use your customized autodiscover config {}
winlogbeatConfig.standardConfig Standard Winlogbeat configuration, using winlogbeat.event_logs. (Currently this is the only available option) See values.yaml
serviceAccount.create Specifies whether a service account should be created. true
serviceAccount.name Name of the service account. filebeat
terminationGracePeriod Termination period (in seconds) to wait before killing Filebeat pod process on pod shutdown. 30
hostNetwork Controls whether the pod may use the node network namespace. true
windowsHostNetwork Controls whether the pod may use the Windows node network namespace. false
dnsPolicy Specifies pod-specific DNS policies. ClusterFirstWithHostNet
daemonset.ignoreOlder Logs older than this will be ignored. (linux based Filebeat) 3h
daemonset.logzioCodec Set to json if shipping JSON logs. Otherwise, set to plain. (linux based Filebeat) json
daemonset.logzioType The log type you’ll use with this Daemonset. This is shown in your logs under the type field in Open Search Dashboards. Logz.io applies parsing based on type. (linux based Filebeat) filebeat
daemonset.fieldsUnderRoot If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. (linux based Filebeat) "true"
daemonset.securityContext Configurable securityContext for Filebeat DaemonSet pod execution environment. (linux based Filebeat) See values.yaml
daemonset.resources Allows you to set the resources for Filebeat Daemonset. See values.yaml (linux based Filebeat)
daemonset.tolerations Set tolerations for all DaemonSet pods. (linux based Filebeat) {}
daemonset.volumes Templatable string of additional volumes to be passed to the DaemonSet. See values.yaml (linux based Filebeat)
daemonset.volumeMounts Templatable string of additional volumeMounts to be passed to the DaemonSet. See values.yaml (linux based Filebeat)
daemonset.priorityClassName Set priorityClassName for all DaemonSet pods. ""
winlogbeatDaemonset.ignoreOlder Logs older than this will be ignored. (Winlogbeat) 3h
winlogbeatDaemonset.logzioCodec Set to json if shipping JSON logs. Otherwise, set to plain. (Winlogbeat) json
winlogbeatDaemonset.logzioType The log type you’ll use with this Daemonset. This is shown in your logs under the type field in Open Search Dashboards. Logz.io applies parsing based on type. (Winlogbeat) winlogbeat
winlogbeatDaemonset.fieldsUnderRoot If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. (Winlogbeat) "true"
winlogbeatDaemonset.securityContext Configurable securityContext for Filebeat DaemonSet pod execution environment. (Winlogbeat) See values.yaml
winlogbeatDaemonset.resources Allows you to set the resources for Filebeat Daemonset. See values.yaml (Winlogbeat)
winlogbeatDaemonset.tolerations Set tolerations for all DaemonSet pods. (Winlogbeat) {}
winlogbeatDaemonset.volumes Templatable string of additional volumes to be passed to the DaemonSet. See values.yaml (Winlogbeat)
winlogbeatDaemonset.volumeMounts Templatable string of additional volumeMounts to be passed to the DaemonSet. See values.yaml (Winlogbeat)
windowsDaemonset.priorityClassName Set priorityClassName for all DaemonSet pods. (windows) ""
filebeatWindowsDaemonset.ignoreOlder Logs older than this will be ignored. (Windows based Filebeat) 3h
filebeatWindowsDaemonset.logzioCodec Set to json if shipping JSON logs. Otherwise, set to plain. (Windows based Filebeat) json
filebeatWindowsDaemonset.logzioType The log type you’ll use with this Daemonset. This is shown in your logs under the type field in Open Search Dashboards. Logz.io applies parsing based on type. (Windows based Filebeat) filebeat-win
filebeatWindowsDaemonset.fieldsUnderRoot If this option is set to true, the custom fields are stored as top-level fields in the output document instead of being grouped under a fields sub-dictionary. (Windows based Filebeat) "true"
filebeatWindowsDaemonset.securityContext Configurable securityContext for Filebeat DaemonSet pod execution environment. (Windows based Filebeat) See values.yaml
filebeatWindowsDaemonset.resources Allows you to set the resources for Filebeat Daemonset. See values.yaml (Windows based Filebeat)
filebeatWindowsDaemonset.tolerations Set tolerations for all DaemonSet pods. (Windows based Filebeat) {}
filebeatWindowsDaemonset.volumes Templatable string of additional volumes to be passed to the DaemonSet. See values.yaml (Windows based Filebeat)
filebeatWindowsDaemonset.volumeMounts Templatable string of additional volumeMounts to be passed to the DaemonSet. See values.yaml (Windows based Filebeat)
secrets.create Boolean to toggle secrets creation. Set to false to disable secrets generation. true
secrets.logzioShippingToken Secret with your logzio shipping token. ""
secrets.logzioRegion Secret with your logzio region. Defaults to US East. " "
secrets.clusterName Secret with your cluster name. ""

Some values, such as daemonset.tolerations, should be set as follows:

--set daemonset.tolerations[0].key='value' \
--set daemonset.tolerations[0].operator='Equal' \

Configuring Filebeat to concatenate multiline logs

Filebeat splits multiline logs by default. If your original logs span multiple lines, you may find that they arrive in your Logz.io account split into several partial logs.

Filebeat offers configuration options that can be used to concatenate multiline logs. The configuration is managed differently, depending on your deployment method:

  • Standard configuration: If you are using a standard configuration (but not autodiscover), use an explicit configuration. Configuration options from Filebeat’s official documentation.

    When using an explicit configuration, you will need to create a single regex expression that covers all of your pods. It also means that Filebeat will need to be reconfigured more often, with the introduction of every new use case.

  • Autodiscover configuration: If you are using autodiscover hints & annotations, add an annotation to your deployment. Configuration options from Filebeat’s official documentation.

    Hints and annotations support the option to manage regex expressions separately for each component. This greatly simplifies the process, making it possible to add a dedicated regex expression to each pod, without needing to change anything on Filebeat itself.

Example

The following is an example of a multiline log sent from a deployment on a k8s cluster:

2021-02-08 09:37:51,031 - errorLogger - ERROR - Traceback (most recent call last):
File "./code.py", line 25, in my_func
1/0
ZeroDivisionError: division by zero

Filebeat’s default configuration will split the above log into 4 logs, 1 for each line of the original log. In other words, each line break (\n) causes a split.

You can overcome this behavior by configuring Filebeat to meet your needs.

Example of an explicit configuration for concatenating multiline logs

To add an explicit configuration to your Filebeat, edit your filebeat.yml file in a text editor and make the appropriate changes under the filebeat.input section.

For the above example, we could use the following regex expression to demarcate the start of our example log. This configuration example is set to identify the first log in a multiline log and concatenate the log lines that follow until it identifies the next log that matches the regex expression. In other words, there is no explicit regex expression to demarcate the end of a multiline log.

filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

See Filebeat’s official documentation for additional configuration options.

Example for using hints & annotations to concatenate multiline logs

If you’re using Filebeat autodiscover hints, you can use annotations to identify multiline logs and concatenate them.

You will need to first configure Filebeat to enable the hints system, and add annotations to the relevant components when you deploy them to your cluster.

Enable Filebeat’s hints system

First, enable Filebeat’s hints system. In your filebeat.yml file, set hints.enabled: true under the filebeat.autodiscover section. For example:

filebeat.autodiscover:
	providers:
	  - type: kubernetes
	    node: ${NODE_NAME}
	    hints.enabled: true # This part enables the hints
	    hints.default_config:
	      type: container
	      paths:
	        - /var/log/containers/*-${data.kubernetes.container.id}.log
Add multiline annotations to your deployment

Whenever you plan to deploy a component to your cluster and want the hints system to detect the multiline logs, you’ll need to add multiline annotations.

For the above log example, you can add the following annotations to your deployment:

annotations:
  co.elastic.logs/multiline.type: 'pattern'
  co.elastic.logs/multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  co.elastic.logs/multiline.negate: 'true'
  co.elastic.logs/multiline.match: 'after'

The above configuration ensures that Filebeat will look for log lines that match the regex under multiline.pattern and concatenate all subsequent lines, until it reaches the next regex match.

See Filebeat’s official documentation for additional configuration options.