Sly Gryphon – Page 5 – Software / Wetware

A Guide to W3C Trace Context(10 min read)

Earlier this year the W3C Trace Context Recommendation was finally published. A standard way of passing distributed trace correlation has been needed for a long time, and it is good to see there is finally a standard, and many vendors have already moved to adopt it.

The Recommendation defines what a distributed trace is:

A distributed trace is a set of events, triggered as a result of a single logical operation, consolidated across various components of an application. A distributed trace contains events that cross process, network and security boundaries. A distributed trace may be initiated when someone presses a button to start an action on a website - in this example, the trace will represent calls made between the downstream services that handled the chain of requests initiated by this button being pressed.

What constitutes a single logical operation depends on the system. In the example above it is a single button press on a website, whereas in a batch processing system it might be for each item processed, or in a complex UI it might consist of both a button press and a subsequent confirmation dialog.

The W3C Trace Context Recommendation describes how the correlation information — an identifier for the operation, and the parent-child relationships between components — is passed in service calls, but doesn't cover what to do with that information, apart from how to pass it to the next component.

This is a guide mostly how to use Trace Context for logging, although it also applies to metrics and other telemetry.

3d6 is not less swingy than d20(13 min read)

With tabletop roleplaying game (RPG) systems, sometimes I hear a claim that bell curve dice rolls, e.g. 3d6, are “less swingy” (less variance) than a linear based dice roll such as d20 or d%.

This is, however, incorrect.

While the distribution of dice rolls are different, the distribution of outcomes – success or failure – are the same, and for equivalent circumstances have the same statistical variance / standard deviation.

Although the outcomes have equivalent distributions, the underlying type of dice system is important for analysis of modifiers and skill progression.

Estimation worksheet(3 min read)

This is a worksheet I have had for a while for calculating estimates using not a single value, but three values for best case, most likely case, and worse case. The values are combined using the PERT formula, to calculate a total estimate, statistical ranges, and an overall estimate including contingency.

Download the worksheet here:

estimate_pert_ranges_template.xslx Download

Supply Chain on Blockchain – conference review(11 min read)

On Monday, I attended the Supply Chain on Blockchain conference (https://scobc.net/), usually held in Brisbane, but this year run online.

Here are my impressions of the different sessions I attended. My highlight of the conference was the second keynote from John Wolpert, on the Baseline Protocol project.

Elasticsearch-Logstash-Kibana (ELK) LoggerProvider for .NET Logging(4 min read)

Note: The Elasticsearch logger provider has been moved to the ECS DotNet project.

Find the latest version here: https://github.com/elastic/ecs-dotnet/blob/master/src/Elasticsearch.Extensions.Logging/ReadMe.md

The nuget package is here: https://www.nuget.org/packages/Elasticsearch.Extensions.Logging/1.6.0-alpha1

To add the package to your project:
dotnet add package Elasticsearch.Extensions.Logging --version 1.6.0-alpha1

This ElasticsearchLoggerProvider, for Microsoft.Extensions.Logging, writes direct to Elasticsearch, using the Elasticsearch Common Schema (ECS), and has full semantic logging of structured data from message and scope values.

To use, add the Essential.LoggerProvider.Elasticsearch package to your project:

PS> dotnet add package Essential.LoggerProvider.Elasticsearch

Then add the logger provider to your host builder, and the default configuration will write to a local Elasticsearch service:

using Essential.LoggerProvider;

// ...

    .ConfigureLogging((hostContext, loggingBuilder) =>
    {
        loggingBuilder.AddElasticsearch();
    })

Once you have logged some events, open up Kibana (e.g. http://localhost:5601/) and define an index pattern for dotnet-* with the time filter @timestamp.

You can then discover the log events for the index. Some useful columns to add are log.level, log.logger, event.code, message, tags, and process.thread.id.

Structured message and scope values are logged as labels.* custom key/value pairs, e.g. labels.CustomerId.

Rolling file LoggerProvider for .NET Logging(1 min read)

I have just released version 1.0 of the Rolling File Logger Provider as part of Essential Logging on Github, a port of my .NET diagnostics library across to Microsoft.Extensions.Logging.

To use, add the Essential.LoggerProvider.RollingFile package to your project via Nuget:

dotnet add package Essential.LoggerProvider.RollingFile

Then reference the namespace, and add the logger provider during host construction:

using Essential.LoggerProvider;

// ...

    .ConfigureLogging((hostContext, loggingBuilder) =>
    {
        loggingBuilder.AddRollingFile();
    })

Eth 2.0 state transition(4 min read)

There is a lot of activity going on building the Ethereum 2.0 beacon chain, including the .NET client I am working on, Nethermind.

The beacon chain consists of blocks and a progressive state. Blocks are generated, signed, and transmitted across the network, then applied to transition the state. The following diagram shows the main relationships.

Syslog Structured Data for Microsoft Extensions Logging(4 min read)

The first part of logging I have polished up Microsoft.Extensions.Logging is structured data support with a Syslog Structured Data package that contains a component which will render as syslog RFC 5424 structured data.

To use the Syslog StructuredData component, install the nuget package:

dotnet add package Syslog.StructuredData

You can then use the structured data via BeginScope() on an ILogger:

using (_logger.BeginScope(new StructuredData
{
    Id = "origin", ["ip"] = ipAddress
}))
{
    // ...
}

For default logger providers, that don't understand structured data, the ToString() method on the StructuredData object will render out the data in RFC 5424 format. This format can still be easily parsed by log analyzers, although the surrounding context won't be a syslog message.

Example output: Using the default console logger, with scopes and timestamp

Alpha of Essential Logging RollingFile(1 min read)

The new Microsoft.Extensions.Logging system has some improvements over the previous System.Diagnostics, with built in support for dependency injection and semantic logging (although I tend to think a singleton-type pattern, like TraceSource, is better than cluttering up every constructor with a logger).

Strangely, however, Microsoft did not include a basic file logger; they have App Insights, and even a file logger that works on Azure-only, but no basic logger. I guess they thought between Serilog/NLog/log4net/etc that there were enough third parties.

The only problem with these is that each of then is an entire logging systems, so you end up going through one framework (Microsoft.Extensions.Logging) to get to another framework (e.g. NLog) before you end up at an actual logger (e.g. a file logger). Why two frameworks?

With the old .NET Framework I never understood this either, which is why I wrote a range of TraceListeners, that each independently plugged into System.Diagnostics directly.

And finally I have started to port it across to Microsoft.Extensions.Logging, with an alpha release of Essential.Logging.RollingFile

This won't be another framework, just a bunch of logger providers that plug into the provider system.

It is only alpha; it works -- I mostly just copied it across from Essential.Diagnostics and commented out the invalid parts, but the infrastructure is still in flux while I sort things out.