Fluentd is an open source data collector and a great option because of its flexibility. This implementation uses a Fluentd DaemonSet to collect Kubernetes logs and send them to Logz.io. The Kubernetes DaemonSet ensures that some or all nodes run a copy of a pod.

The image used in this integration comes pre-configured for Fluentd to gather all logs from the Kubernetes node environment and append the proper metadata to the logs. If you prefer to customize your Fluentd configuration, you can edit it before it’s deployed.

The latest version pulls the image from logzio/logzio-fluentd. Previous versions pulled the image from logzio/logzio-k8s.

Fluentd will fetch all existing logs, as it is not able to ignore older logs.

For troubleshooting this solution, see our user guide.

Sending logs from nodes with taints

If you want to ship logs from any of the nodes that have a taint, make sure that the taint key values are listed in your in your daemonset/deployment configuration as follows:

tolerations:
- key: 
  operator: 
  value: 
  effect: 

To determine if a node uses taints as well as to display the taint keys, run:

kubectl get nodes -o json | jq ".items[]|{name:.metadata.name, taints:.spec.taints}"

You need to use Helm client with version v3.9.0 or above.

K8S version compatibility

Your Kubernetes version may affect your options, as follows:

  • K8S 1.19.3+ - If you’re running on K8S 1.19.3+ or later, be sure to use the DaemonSet that supports a containerd at runtime. It can be downloaded and customized fromlogzio-daemonset-containerd.yaml.

  • K8S 1.16 or earlier - If you’re running K8S 1.16 or earlier, you may need to manually change the API version in your DaemonSet to apiVersion: rbac.authorization.k8s.io/v1beta1.

    The API versions of ClusterRole and ClusterRoleBinding are found in logzio-daemonset-rbac.yaml and logzio-daemonset-containerd.yaml.

    If you are running K8S 1.17 or later, the DaemonSet is set to use apiVersion: rbac.authorization.k8s.io/v1 by default. No change is needed.

  • ARM architecture is supported as of logzio/logzio-fluentd:1.0.2.

Multiline logs

Fluentd’s basic configuration may cause longer, multiline logs to split into multiple logs - 1 log per line. You can use the Fluentd multiline parser plugin to control this behavior.

See the next tab for details about using the Fluentd multiline parser plugin.