Parsing is the process of breaking down your log message into smaller chunks of data, placing each chunk into its own specific named field, and enriching data with additional information such as geolocation. Parsed logs can be more easily analyzed than raw data, allowing you to create rich visualizations and helpful alerts.
Parsing is not necessary for all types of logs. But if you use a custom or uncommon log type, parsing can be an invaluable tool.
You can analyze a set of sample logs for a Community (free) account in the data parsing wizard, to simplify the process.
When you’re logged into a Community account, you can find the data parsing wizard by selecting Data Hub > Data Parsing from the menu.
Deprecation notice
With the release of Logz.io Data Parsing, the parsing wizard is deprecated.
To use the data parsing wizard
Choose a data source
Choose the log type you want to parse from the Select log type list.
If the log type you want to parse is disabled, Logz.io automatically parses it. If you want to override the default parsing, change the log type in your log shipper.
Click Next to continue.
Configure your parse settings
If you want to change the sample log lines, click Select. You can choose up to 5 sample lines.
Type your grok pattern in the Parse method text box.
- As you type, your parsed log lines are shown in the Parse results table. Use the colors to help you match the fields in the log lines with your parsing results.
- To omit data, do not name the fields.
- To override the log’s existing message field,
name one or more fields
messagein the grok pattern. If more than one is used, all message fields are concatenated into a single message field. If you don’t use this field name, the log’s existing message field will be used. - After entering your grok pattern, you can define a field type for each field that you parse.
- You can let Logz.io detect each field’s data type by leaving the default Automatic settings. Otherwise, you can define other data types, such as boolean, date, IP, and byte. For example: For geo-enrichment, you need to select the Geo-Enrichment field type.
To help make the best grok pattern for your logs, use the Grok Debugger. For reference, see grok patterns from Elastic.
Click Next to continue.
Enrich
If any fields are parsed as geo IP, choose which geo enrichment fields to add to your logs, such as continent_code or country_name.
Configure any timestamp fields. If there are more than one timestamp field, choose a Leading timestamp.
Click Next to continue.
Validate
In both of the tabs, review Unparsed logs and All logs to troubleshoot any problems with your grok pattern.
If everything looks good, click Apply to parse future logs using these settings. Otherwise, click Back to make changes to your settings.