Google Cloud Platform (GCP) Stackdriver collects logs from your cloud services. You can use Google Cloud Pub/Sub to forward your logs from Stackdriver to Logz.io using a continuously runnung Docker container .

Configuration

Before you begin, you’ll need:

Export your logs to Stackdriver

Set up a sink to export your logs to Stackdriver.

For more information, see Exporting with the Logs Viewer from Google Cloud.

Build your credentials file

Create a working directory for this step and cd into it. You’ll need to run this command as root:

mkdir /etc/logzio-pubsub && cd /etc/logzio-pubsub

Next, you’ll need to build a credentials file so Pub/Sub can authenticate and get the right permissions.

You can build it through:

Option 1: Build the credentials file from the command line

In this step, you’ll build your credentials file using your Google Cloud project ID.

Before you begin, you’ll need the gcloud command-line tool (CLI) installed. If it isn’t, follow the steps to install it:

  1. Download the ‘google-cloud-sdk’ to ‘/etc/logzio-pubsub’.
  2. Run source '/etc/logzio-pubsub/google-cloud-sdk/path.bash.inc'. If you’re are not already logged in to gcloud, you will be requested to login through your browser.

Run the following command for each project you’re working with. Replace the placeholder with your project id before running the command:

wget https://raw.githubusercontent.com/logzio/logzio-pubsub/master/create-credentials.py \
&& python create-credentials.py <<project_id>>

If you rename the file, follow these steps as well.

Option 2: Build the credentials file in the Cloud Console

  • In the GCP Console, go to your project’s page. In the left menu, select IAM & admin > Service accounts.

  • At the top of the Service accounts page, click + CREATE SERVICE ACCOUNT.

  • Give a descriptive Service account name, such as “credentials file”. Click CREATE to continue to the Service account permissions page.

  • Add the role: ‘Pub/Sub Editor’.

  • Click CONTINUE to Grant users access to this service account. Click ADD KEY + CREATE NEW KEY to open the Create key panel. Select JSON and click CREATE to save the private key to your machine.

  • Click DONE to return to the Service accounts page.

  • Rename it in the following format: <project-id>-credentials.json - replace to your project id. Move it to the /etc/logzio-pubsub folder you’ve created at the beginning of this step.

Variation
  • If your credentials file name isn’t of the default format <<project_id>>-credentials.json, follow the steps below as well.
Build your Pub/Sub input YAML file

Create a file ‘pubsub-input.yml’ to hold your Pub/Sub input configuration. To create the file run the following command as root. Then open the file in your text editor:

touch /etc/logzio-pubsub/pubsub-input.yml

Paste this code block into your file. Complete configuration instructions are below the code block. 👇

listener: <<LISTENER-HOST>>
pubsubs:
- project_id: PROJECT-1_ID
  topic_id: TOPIC-1_ID
  token: <<LOG-SHIPPING-TOKEN>>
  credentials_file: ./credentials-file.json
  subscriptions: [SUB1_ID, SUB2_ID, SUB3_ID]
  type: stackdriver

- project_id: PROJECT-1_ID
  topic_id: TOPIC-2_ID
  token: <<LOG-SHIPPING-TOKEN>>
  credentials_file: ./credentials-file.json
  subscriptions: [SUB1_ID, SUB2_ID, SUB3_ID]
  type: stackdriver

- project_id: PROJECT-3_ID
  topic_id: TOPIC-1_ID
  token: <<LOG-SHIPPING-TOKEN>>
  credentials_file: ./credentials-file.json
  subscriptions: [SUB1_ID, SUB2_ID, SUB3_ID]
  type: stackdriver

** Note that YAML files are sensitive to spaces and tabs. We recommend using a YAML validator to make sure that the file structure is correct.

Click here for more information about filebeat for Google Cloud Pub/Sub.

Configuration instructions
Parameter Description
listener The Logz.io listener host. Replace <<LISTENER-HOST>> with the host for your region. For example, listener.logz.io if your account is hosted on AWS US East, or listener-nl.logz.io if hosted on Azure West Europe. The required port depends whether HTTP or HTTPS is used: HTTP = 8070, HTTPS = 8071.
pubsubs This is an array of one or more GCP subscriptions. For each subscription, provide topic and subscription IDs, as given from Pub/Sub.
token Your Logz.io shipping token. For each project under pubsubs. Replace <<LOG-SHIPPING-TOKEN>> with the token of the account you want to ship to. You can send your logs to different accounts that are in the same region, you can do that by inserting a different token per project.
credentials_file (Not required, Default value: ‘-credentials.json') This field is only required if your credentials file is named differently than the default value. For an example of adding this field go to input example file.
Pull the Docker image

Download the logzio/logzio-pubsub image:

docker pull logzio/logzio-pubsub
Run the container

Run the following command after you replace <<PROJECT_ID>> with your details.

docker run --name logzio-pubsub \
-v /etc/logzio-pubsub/pubsub-input.yml:/logzio-pubsub/pubsub-input.yml \
-v /etc/logzio-pubsub/<<PROJECT_ID>>-credentials.json:/logzio-pubsub/<<PROJECT_ID>>-credentials.json \
logzio/logzio-pubsub
Variations

  • If you’re working with multiple topics, add this line for every credentials file you’ve created. Insert your project id instead of the parameters:

      -v /etc/logzio-pubsub/<<PROJECT_ID>>-credentials.json:/logzio-pubsub/<<PROJECT_ID>>-credentials-file.json \

  • If your credentials file name isn’t of the default format <<project_id>>-credentials.json, follow the steps below as well.

  • If you’re using a Mac, you’ll need to fix issues with mounting files from root directory. Add the path ‘/etc/logzio-pubsub’ to your Docker File Sharing. Click here for a guide on how to fix this issue - you can use docker desktop or manually edit your Docker configuration file. For more information about mounting files from the root directory click here.

Check Logz.io for your logs

Spin up your Docker containers if you haven’t done so already. Give your logs some time to get from your system to ours, and then open Open Search Dashboards.

If you’ve renamed the credentials file

The default naming convention for the credentials file is: <<project_id>>-credentials.json.

When you create the credentials file through the command line, it is automatically named as per the default.

If you create the credentials file using the GCP Console, you’ll have the option to select the file name. We strongly recommend that you stick to the default format: <<project_id>>-credentials.json.

If you decide to give the credentials file another name, please follow these instructions:

  1. On step 3 - building your ‘pubsub-input.yml’ file, add the field ‘credentials_file’ with your credentials file’s name as the value.

    Go to the github project to see an example of an input file.

  2. On step 5 - running the docker, add the following line for every credentials file you’ve created:

    ‘-v /etc/logzio-pubsub/<>.json:/logzio-pubsub/<>.json \'.

    Replace <<credentials-file-name>> with your credentials file’s name.