Rsyslog - oVirt Support

Background

In the logging role context, oVirt is a special input type which handles inputs from the remote RHV systems and the files input (imfile) on the target host. The log messages are formatted in the ovirt input and forwarded to the elasticsearch. Plus, in the stable-0.1 version, it depended on the default rsyslog.conf which was deployed by setting the rsyslog_default variable to true.

The goal of the task “RHELPLAN-42379 - Logging - Support oVirt input” is providing the same logging experience using the logging_inputs, logging_outputs, and logging_flows configuration. Then, the rsyslog_default flag (with etc/rsyslog.conf.j2 template) and the send-target-only config file are safely dropped.

Implementation Changes

Note: in the effort of “RHELPLAN-41572 - Logging - Convert each subrole to a pair of task/var files”, ovirt is also converted from a subrole to a pair of task/var files in roles/rsyslog/{tasks,vars}/inputs.

ovirt task

This is an example of ovirt_input sends the log messages to the elasticsearch_output.

logging_outputs:
  - name: elasticsearch_output
    type: elasticsearch
    server_host: logging-es
    server_port: 9200
    index_prefix: project.
    input_type: ovirt
    retryfailures: false
    ca_cert: "/etc/rsyslog.d/es-ca.crt"
    cert: "/etc/rsyslog.d/es-cert.pem"
    key: "/etc/rsyslog.d/es-key.pem"
logging_inputs:
  - name: ovirt_collectd
    type: ovirt
    subtype: collectd
  - name: ovirt_engine
    type: ovirt
    subtype: engine
  - name: ovirt_vdsm
    type: ovirt
    subtype: vdsm
logging_flows:
  - name: flow_0
    inputs: [ovirt_collectd, ovirt_engine, ovirt_vdsm]
    outputs: [elasticsearch_output]

The ovirt logging_inputs are divided into 3 subtypes, collectd, engine and vdsm. They are separately configured to allow each subtype send its log messages to the different logging_outputs using logging_flows. The configuration files are generated by the ovirt task Create oVirt input configuration files in /etc/rsyslog. using ovirt_input_template.j2.

These are the snippet of the generated configuration files.

==> 90-input-ovirt-ovirt_collectd.conf <==
input (name="ovirt_collectd" type="imtcp" port="44514")
if $inputname == "ovirt_collectd" then {
    set $!original_raw_message = $msg;
    action(name="collectd_mmjsonparse" type="mmjsonparse" cookie="") # parse entire message as json

==> 90-input-ovirt-ovirt_engine.conf <==
input(type="imfile" file="/var/log/ovirt-engine/engine.log" tag="ovirt_engine"
      reopenOnTruncate="on" startmsg.regex="^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}")
if $syslogtag == "ovirt_engine" then {
  action(name="ovirt-engine_mmnormalize" type="mmnormalize"

==> 90-input-ovirt-ovirt_vdsm.conf <==
input(type="imfile" file="/var/log/vdsm/vdsm.log" tag="ovirt_vdsm"
      reopenOnTruncate="on" startmsg.regex="^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}")
if $syslogtag == "ovirt_vdsm" then {
  action(name="ovirt-vdsm_mmnormalize" type="mmnormalize"

As shown in this example, each name in the ovirt logging_inputs is used in the 2 places.

in the file name, the 2 digits 90 means these configuration files have the input type. The next input is a fixed prefix, ovirt is from the input type, which is followed by the name - ovirt_collectd, ovirt_engine, and ovirt_vdsm.
in the input tasks, the name is either set to $inputname or $syslogtag through the rsyslog input call. The imtcp input sets $inputname to the name value, while the imfile does $syslogtag to the tag value.

The elasticsearch output prepares ruleset ruleset(name="elasticsearch_output") to catch these calls.

oVirt parameters

common

name - unique name
type - ovirt
subtype - one of collectd, engine, vdsm; if any other value is set, the section is ignored.
ovirt_env_name - default to engine
ovirt_env_uuid - default to none
subtype collectd
ovirt_collectd_port - port listening to the collectd inputs; default to 44514.
ovirt_elasticsearch_index_prefix - index prefix for the elasticsearch; default to project.ovirt-metrics
subtype engine
ovirt_input_file - input log file to read; default to /var/log/ovirt-engine/engine.log
ovirt_elasticsearch_index_prefix - index prefix for the elasticsearch; default to project.ovirt-logs
ovirt_vds_cluster_name - default to none
ovirt_engine_fqdn - ovirt engine fqdn; default to none
subtype vdsm
ovirt_input_file - input log file to read; default to /var/log/vdsm/vdsm.log
ovirt_elasticsearch_index_prefix - index prefix for the elasticsearch; default to project.ovirt-logs
ovirt_vds_cluster_name - default to none
ovirt_engine_fqdn - ovirt engine fqdn; default to none

Replacement of rsyslog_default

The functionality is replaced with the basics input + default files output combination. The basic input, the default files output, and the flow from the basic input to the default files output are added to the example in the previous section. The basic input reads the system logs via the rsyslog imjournal, then the logs are stored in the local files, e.g., /var/log/messages by the default files output.

logging_outputs:
  - name: default_files
    type: file
  - name: elasticsearch_output
    type: elasticsearch
    server_host: logging-es
    server_port: 9200
    index_prefix: project.
    input_type: ovirt
    retryfailures: false
    ca_cert: "/etc/rsyslog.d/es-ca.crt"
    cert: "/etc/rsyslog.d/es-cert.pem"
    key: "/etc/rsyslog.d/es-key.pem"
logging_inputs:
  - name: basic_input
    type: basics
  - name: ovirt_collectd
    type: ovirt
    subtype: collectd
  - name: ovirt_engine
    type: ovirt
    subtype: engine
  - name: ovirt_vdsm
    type: ovirt
    subtype: vdsm
logging_flows:
  - name: flow_0
    inputs: [ovirt_collectd, ovirt_engine, ovirt_vdsm]
    outputs: [elasticsearch_output]
  - name: flow_1
    inputs: [basic_input]
    outputs: [default_files]

Test case

There are 2 test playbooks tests_ovirt_elasticsearch.yml and tests_ovirt_elasticsearch_params.yml. The test playbooks configure a basic input and three ovirt inputs, a default files output and two elasticsearch outputs, and a flow from the ovirt inputs to the elasticsearch outputs and the basic input to the default files output.

The test generates these config files in /etc/rsyslog.d.

00-global.conf                       30-output-files-default_files.conf
05-common-defaults.conf              31-output-elasticsearch-elasticsearch_output.conf
10-input-basics-modules.conf         31-output-elasticsearch-elasticsearch_output_ops.conf
10-input-ovirt-local_modules.conf    90-input-ovirt-ovirt_collectd.conf
10-input-ovirt-main-modules.conf     90-input-ovirt-ovirt_engine.conf
10-output-elasticsearch-module.conf  90-input-ovirt-ovirt_vdsm.conf
10-output-files-modules.conf         ovirt_engine.rulebase
11-input-basics-basic_input.conf     ovirt_vdsm.rulebase
30-output-elasticsearch.conf         parse_json.rulebase

The caller part from the ovirt inputs and the callees in the elasticsearch outputs look like this.

==> 90-input-ovirt-ovirt_collectd.conf <==
<<snip>>
if
  ($inputname == "ovirt_collectd")
  then {
    call elasticsearch_output
}
if
  ($inputname == "ovirt_collectd")
  then {
    call elasticsearch_output_ops
}

==> 90-input-ovirt-ovirt_engine.conf <==
<<snip>>
if
  ($syslogtag == "ovirt_engine")
  then {
    call elasticsearch_output
}
if
  ($syslogtag == "ovirt_engine")
  then {
    call elasticsearch_output_ops
}

==> 90-input-ovirt-ovirt_vdsm.conf <==
<<snip>>
if
  ($syslogtag == "ovirt_vdsm")
  then {
    call elasticsearch_output
}
if
  ($syslogtag == "ovirt_vdsm")
  then {
    call elasticsearch_output_ops
}

==> 31-output-elasticsearch-elasticsearch_output.conf <==
ruleset(name="elasticsearch_output") {
    if (strlen($.omes) > 0) and (strlen($.omes!status) > 0) then {
        # retry case
    <<snip>>

==> 31-output-elasticsearch-elasticsearch_output_ops.conf <==
ruleset(name="elasticsearch_output_ops") {
    if (strlen($.omes) > 0) and (strlen($.omes!status) > 0) then {
        # retry case
    <<snip>>

Cleaning up

The variable rsyslog_default and roles/rsyslog/template/etc/rsyslog.conf.j2 are eliminated.
40-send-targets-only.conf is no longer generated.