Rsyslog - Flow Controls between Inputs and Outputs

History - Before the flow control and rsyslog_default removal

The logging role defines rsyslog_default variable to deploy the original, all-in-one rsyslog configuration file rsyslog.conf. It configures to get inputs from imjournal and output to the local files in /var/log. It was originally added to provide “noop” to the logging role. Then, it was utilized to support for the local system logs to store into the local files combining with the openshift logging in the ovirt logging. In the linux-system-roles review process, some critical questions on rsyslog_default were raised, such as the original intention - providing “noop” is not the style ansible recommends and no other linux-system-roles do not nor will not have such a default variable. Considering the inputs, the functionality that rsyslog_default offers has to be achieved by the logging role itself instead of copying the hardcoded rsyslog.conf to the target host.

This proposal will discuss about these items:

how to replace the rsyslog_default variable with the logging role inputs/outputs configuration.
how to support more generic way to connect inputs and outputs.

Post Flow Control Implementation

The primary playbook variables are made from 3 dicts - logging_inputs, logging_outputs, and logging_flows.

logging_inputs supports 3 types. [*]

basics input (imjournal, imuxsock)
files input (imfile)
ovirt handling ovirt inputs logging_outputs supports 3 types. [*]
files output (omfile)
forwards output (omfwd)
elasticsearch (omelasticsearch).

[*] - These are the input and output types implemented at the time this design doc was written. More inputs and outputs are added now and will be in the future.

How the flow control works

Outputs

Each output action is in the ruleset which name is the logging_outputs name.

Output config file name is constructed of prefix 2 digits, output, type of the output, and type of configuration or the unique name of the output item concatenated by - with suffix .conf. For instance, an elasticsearch config file defining the module is 10-output-elasticsearch-module.conf, where elasticsearch is the type of output, and module is the type of configuration. Another example, a output files config file which item has the unique name files_output is 30-output-files-files_output.conf. The 2 digits are defined in rsyslog_weight_map as described in templates. Also, it could be reassigned in each internal __rsyslog_rules.

Looking into the contents of the action configuration file, one configuration file contains one action wrapped in a ruleset with a unique name. The following ruleset example has the name files_output and is for logging the messages in the local file /var/log/messages. This output type is files. Log messages are passed to the ruleset by “call files_output” from an input.

=> 30-output-files-files_output.conf <==
ruleset(name="files_output") {
  *.info;authpriv.none;auth.none;cron.none;mail.none /var/log/messages
}

This is generated by this set of configuration parameters, which would be explained below.

logging_outputs
  - name: files_output
    type: files
    severity: info
    exclude: [authpriv.none, auth.none, cron.none, mail.none]
    path: /var/log/messages

Inputs

The logging_inputs variables in the inventory specifies the input methods (plugins) to get the log messages. Currently, basics type (imjournal and imtcp, imptcp, imudp) and files type (`imfiles) are supported.

The file’s naming convention is similar to the outputs. When the input type can have multiple logging inputs such has reading from log files, the same number of input configuration files are generated and its name is constructed of prefix 2 digits, input, type of the input, and the unique name of the input item concatenated by - with suffix .conf.

The contents of an input configuration file has the caller point to the ruleset name in the output configuration based on the flow definition described next.

Flows to define the Input and Output Relationships

Based upon the flows definition in the inventory that customers provide, input config jumps to the ruleset. The next imjournal (basics) example shows log messages read from journald by the imjournal plugin are logged in the local files.

imjournal example Log messages from journald are read by imjournal. Then, they are passed to the files output by calling files_output0 and files_output1 ruleset.

=> 90-input-basics-basic_input.conf <==
module(load="imuxsock"    # provides support for local system logging (e.g. via logger command)
       SysSock.Use="off") # Turn off message reception via local log socket.
module(load="imjournal"
       StateFile="/var/lib/rsyslog/imjournal.state"
       RateLimit.Burst="20000"
       RateLimit.Interval="600"
       PersistStateInterval="10")
if
  ($inputname == "imjournal" or $inputname == "imuxsock")
  then {
    call files_test0
}
if
  ($inputname == "imjournal" or $inputname == "imuxsock")
  then {
    call files_test1
}
if
  ($inputname == "imjournal" or $inputname == "imuxsock")
  then {
    call forwards_severity_and_facility
}
if
  ($inputname == "imjournal" or $inputname == "imuxsock")
  then {
    call forwards_facility_only
}

imfile example The next imfile example shows log messages read from /var/log/inputdirectory/*.log by the imfile plugin are tagged with the input name files_input0 then passed to the output ruleset specified by files_output0 and files_output1.

==> 90-input-files-files_input0.conf <==
input(type="imfile" file="/var/log/inputdirectory/*.log" tag="files_input")
if
  ($syslogtag == "files_input")
  then {
    call files_test0
}
if
  ($syslogtag == "files_input")
  then {
    call files_test1
}

Inventory variable changes

Here is an example that the inputs from imjournal (basics) are logged in the local file; the inputs from imfile (files) are forwarded to the remote host in addition to the local file,

New style

logging_inputs:
  - name: basic-input
    type: basics
  - name: file-input
    type: files
logging_outputs:
  - name: forward-output
    type: forwards
    target: remote_host_name.remote_domain_name
    tcp_port: 514
  - name: file-output
    type: files
logging_flows:
  - name: flow0
    inputs: [basic-input]
    outputs: [file-output]
  - name: flow1
    inputs: [file-input]
    outputs: [file-output, forward-output]

In the new configuration style, logging_inputs (input tasks) do not belong to logging_outputs (output tasks), but in the same level. The relationship between the inputs and the outputs are represented by logging_flow with the subvariables inputs and outputs. The inputs variable stores the list of the names of inputs and outputs does the list of the names of outputs.

For the comparison, here is the previous style, in which logging_inputs (input roles) used to belong to logging_outputs (output roles) to specify the relationship between the input and the output roles.

Previous style

logging_outputs:
  - name: forward-output
    type: forwards
    rsyslog_forwards_actions:
      - name: to-remote
        target: remote_host_name.remote_domain_name
        tcp_port: 514
    logging_inputs:
      - name: basic-input
        type: basics
  - name: file-output
    type: files
    logging_inputs:
      - name: file-input
        type: files

In the above example, the inputs from imjournal (basic-input) are forwarded to the remote host (forward-output); the inputs from imfile (file-input) are stored in the local file (file-output).

With this style, it is not straightforward to implement one input to send multiple outputs in the logging roles. Moreover, to configure the input from imjournal to send both the remote host and the local file, the basics input would be placed under both forwards and files outputs as follows. It could give an impression basic-input0 and basic-input1 could be configured independently, but it is not. There is only one imjournal input and most likely the first one is wiped out. If possible, we should avoid this type of confusions in the designing phase.

Previous style example with uncertainty

logging_outputs:
  - name: forward-output
    type: forwards
    rsyslog_forwards_actions:
      - name: to-remote
        target: remote_host_name.remote_domain_name
        tcp_port: 514
    logging_inputs:
      - name: basic-input0
        type: basics
        journal_persist_state_interval: 1000
        ratelimit_interval: 10
        ratelimit_burst: 100000
  - name: file-output
    type: files
    logging_inputs:
      - name: basic-input1
        type: basics
        persist_state_interval: 3000
        ratelimit_interval: 30
        ratelimit_burst: 300000

logging_outputs, logging_inputs and logging_flows

This is a typical example of the configuration. In logging_outputs, the to-be-configured output roles are listed. In logging_inputs, the to-be-configure input roles are listed. In logging_flows, define which inputs are related to which outputs.

logging_outputs:
  - name: forward-output
    type: forwards
    target: remote_host_name.remote_domain_name
    tcp_port: 514
  - name: file-output
    type: files
logging_inputs:
  - name: basic-input
    type: basics
  - name: file-input
    type: files
logging_flows:
  - name: flow0
    inputs: [basic-input]
    outputs: [file-output]
  - name: flow1
    inputs: [file-input]
    outputs: [file-output, forward-output]

There are some exceptions.

The files output is configured, by default. That is, even if there is no logging_outputs is given, the files output config file is deployed. The filters and the actions in the configuration file (30-output-files.conf) are equivalent to the ones in the original rsyslog.conf.

To Do’s

Input viaq + viaq-k8s

The input viaq and viaq-k8s are tightly coupled, where viaq is made from the imfile module call and the viaq formatting; viaq-k8s contains openshift specific code such as the location of the logging and code to get the meta data from kubernetes. Elasticsearch is only an expected output, for now. So, viaq + viaq-k8s also could skip the output configuration in the inventory file and just do call elasticsearch.