This section contains some guidelines for handling errors that you may encounter when trying to run Fluentd to collect Kubernetes logs.

Problem: /file/path.log unreadable

The following error appears when running Fluentd:

/file/path.log unreadable. it is excluded and would be examined next time

Possible cause

You may need to add more volume and volume mount to your Daemonset.

Suggested remedy

Check on which node your pod is running

Find out on which node your Fluentd pod with the errors is running. To do so, use this command:

kubectl -n <<NAMESPACE>> get pod <<FLUENTD-POD-NAME>> -owide
Connect to the node

Connect to the node you found in the previous step (ssh, etc...).

Find the log’s path
  1. Run the following command, to go to the logs directory:
cd /var/log/containers
  1. Run the following command to display the log files symlinks:
ls -ltr

This command should present you a list of your log files and their symlinks, for example:

some-log-file.log -> /var/log/pods/file_name.log
  1. Choose one of those logs, copy the symlink, and run the following command:
ls -ltr /var/log/pods/file_name.log

Again, this command will output the file and its symlink, or example:

/var/log/pods/file_name.log -> /some/other/path/file.log

This directory (/some/other/path) is the directory where your log files are mounted at the host. You’ll need to add that path to your Daemonset.

Add the mount path to your Daemonset
  1. Open your Daemonset in your preffered text editor.
  2. In the volumeMounts section, add the following:
- name: logextramount
  mountPath: <<MOUNT-PATH>>
  readOnly: true

Replace <<MOUNT-PATH>> with the directory path you found in step 3.

  1. In the volumes section, add the following:
- name: logextramount
  hostPath:
    path: <<MOUNT-PATH>>

Replace <<MOUNT-PATH>> with the directory path you found in step 3.

  1. Save the changes.
Deploy your new Daemonset.

Remove your previous Daemonset from the cluster, and apply the new one.

Applying the new Daemonset without removing the old one will not apply the changes.

Check your Fluentd pods to ensure that the error is gone
kubectl -n <<NAMESPACE>> logs <<POD-NAME>>

Problem: You have reached your pull rate limit

vIn some cases (i.e. spot clusters) where the pods or nodes are replaced frequently, they might reach the pull rate limit for images pulled from dockerhub with the following error:

You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: 
https://www.docker.com/increase-rate-limits

Suggested remedy

You can use the following --set commands to use an alternative image repository:

For the monitoring chart and the Telemetry Collector Kubernetes installation:

--set logzio-fluentd.image=public.ecr.aws/logzio/logzio-fluentd

For the fluentd chart:

--set image=public.ecr.aws/logzio/logzio-fluentd