Syslog Structured Data for Microsoft Extensions Logging(4 min read)

The first part of logging I have polished up Microsoft.Extensions.Logging is structured data support with a Syslog Structured Data package that contains a component which will render as syslog RFC 5424 structured data.

To use the Syslog StructuredData component, install the nuget package:

dotnet add package Syslog.StructuredData

You can then use the structured data via BeginScope() on an ILogger:

using (_logger.BeginScope(new StructuredData
{
    Id = "origin", ["ip"] = ipAddress
}))
{
    // ...
}

For default logger providers, that don't understand structured data, the ToString() method on the StructuredData object will render out the data in RFC 5424 format. This format can still be easily parsed by log analyzers, although the surrounding context won't be a syslog message.

Changes from System Diagnostics

Microsoft.Extensions.Logging already includes an semantic logging overload of BeginScope() that accepts a format message and values, including named placeholder support for semantic logging. Internally this uses a FormattedLogValues class, which implements IReadOnlyList<KeyValuePair<string, object>> that allows logger providers to extract out the semantic properties.

This is similar to the usage of structured data and logical operation scope in older System.Diagnostics extensions, i.e. semantic data is already supported. All that semantic logger providers need to do is handle the named values, and those that don't will render out the string.

So, this structured data class is slightly different, with the purpose of rendering into the specific format used in RFC 5424 (syslog).

Usage

The StructuredData class supports collection initializers for the parameter values, which can be used at the same time as the Id property initializer for a compact representation. See the above code for an example.

If you don't include the Id value, then the parameters will still be rendered, with a nil ID. This isn't strictly part of the RFC, although the nil Id is a valid ABNF value. (See the output rendered above)

using (_logger.BeginScope(new StructuredData
{
    ["CustomerId"] = customerId, ["OrderId"] = orderId, ["DueDate"] = dueDate
}))
{
    // ...
}

An overload of BeginScope() extension method is also provided:

using (_logger.BeginScope("userevent@-", 
    new Dictionary<string, object> {["UserId"] = userId, ["EventId"] = eventId}))
{
    // ...
}

Structured logging

For logger providers that do understand structured data (semantic logging), the StructuredData class implements the IReadOnlyList<KeyValuePair<string, object>> interface to be compatible with FormattedLogValues, allowing individual structured parameters to be extracted and logged as individual fields.

For data with a specified SD-ID the value is prefixed to the individual parameter names, e.g. see "origin:ip" in the example below, and the SD-ID value is included with the name "SD-ID" (specified in the field StructuredData.IdKey).

For example, sending the values to the Seq logger provider produces the following, with the individual parameter values.

Examples

Examples are provided in the syslog-structureddata project on github.

The examples use the generic host builder, a background service worker, and the Microsoft LoggerMessage pattern for high-performance logging.

These can easily be run from the command line (output shown above):

dotnet run --project ./examples/DefaultConsoleLogging

Or using the console 'Systemd' format:

dotnet run --project ./examples/SyslogLogging

For the Seq example, you need to be running Seq. e.g. install on Windows, or on Linux open a terminal and run a docker image:

sudo docker run -e ACCEPT_EULA=Y -p 5341:80 datalust/seq:latest

Then in another console, run the Seq example:

dotnet run --project ./examples/SeqLogging

Future

I'll be integrating this into my rolling file logger provider, currently in alpha (being ported from my older System Diagnostics trace listener project), along with supporting things like better support of IReadOnlyList<KeyValuePair<string, object>> scopes.