App Insights trace correlation

Application Insights is the application performance monitoring feature of Azure Monitor, and can be used to monitor deployed applications both in the cloud and on premises. App Insights supports W3C Trace Context standard headers to correlate tracing information across different components.

The features of App Insights, and Azure Monitor, are quite broad, whereas developers may want in some cases to filter down and focus on application-specific logging. Trace correlation is an important part of this, to get and end-to-end overview of operations.

To view logs, connect your App Insights instance to a Log Analytics workspace. Within the workspace, General > Logs will provide access to the query editor — you can either user one of the default Queries pop-up or write your own.

For example, to see all recent traces, and the correlation between them you can use a query like:

union AppTraces, AppDependencies, AppRequests
| where TimeGenerated > ago(30m)
   and Properties.CategoryName !startswith "Microsoft"
| sort by TimeGenerated desc
| project TimeGenerated, Type, OperationId, Id, Properties.SpanId, 
   ParentId, ClientType, Message, Name, SeverityLevel, Properties, 
   Properties.CategoryName, OperationName, SessionId,
   UserId, AppRoleInstance

Example output:

This example shows all the traces from one operation are linked to the same OperationId 029c3..., and the parent-child relationship between two tiers client (Browser) and server (PC) can also be determined:

  1. Client (Browser) AppTraces have a ParentId 7d65e...
  2. The client has a link from this parent to a child AppDependency with Id 73676...
  3. On the server (PC) the dependency is recorded as the parent if the AppRequest Id 15c7e...
  4. Additional traces on the server show the request as the ParentId (and there may be further parent-child links depending on the number of tiers).

There are many other types of records that can be queried, for example developers may often be interested in exceptions and traces that feature a particular keyword:

union AppExceptions, AppTraces
| where TimeGenerated > ago(30m)
| sort by TimeGenerated desc
| search "Password"
Continue reading App Insights trace correlation

Apps on Kubernetes IPv6 – Kubeapps, WordPress

Once you have Kubernetes running on IPv6 only the next step is to install some apps.

This is my first post written on my new WordPress instance, hosted on Kubernetes IPv6 only. If you are reading it, then it is working 🙂

Of course apps have their own issues not being configured by default to work with IPv6, so for each app you need to test and work out what configuration details need to be tweaked (assuming the app supports IPv6 in the first place).

To start off with, I installed Kubeapps, to get an application management dashboard, and then used that to install WordPress.

With WordPress installed, I exported the content from my old blog and then imported it into the new instance, and tweaked a few WordPress settings.

The final step was to configure the Mythic Beasts reverse proxy, to make my blog available for legacy IPv4 users.

Continue reading Apps on Kubernetes IPv6 – Kubeapps, WordPress

Kubernetes on IPv6 only

Kubernetes is an open source platform for managing containerised applications.

IPv6 is the next generation Internet protocol, and running on IPv6 only simplifies configuration and administration, and avoids the performance issues and complexities of IPv4 encapsulation, NAT, and conflicting private address ranges.

The default configuration of Kubernetes is IPv4, and there are few, and scattered, examples and guidance for setting up IPv6 dual stack, let alone single stack.

I have collected instructions from the different sources into a single guide to successfully deploy Kubernetes with IPv6 only.

See the guide for full instructions:

https://github.com/sgryphon/kubernetes-ipv6

The blog post contains some additional background on what I did to gett the deployment working. The deployment was tested on Ubuntu 20.04 running on an IPv6 only virtual server from Mythic Beasts.

Continue reading Kubernetes on IPv6 only

IPv6 only hosting

We ran out of IPv4 addresses a few years ago, and the cost has been steadily increasing, now over USD 25.00 per IPv4 address.

Meanwhile we are increasingly using technologies such as containerised deployments and mass deployment of Internet of Things (IoT) devices, seeing an increase in demand for addresses.

About 30% of the Internet now happily talks IPv6, with several countries having more than 50% IPv6, and for a server hosting environment there are many benefits to going IPv6 only.

IPv6 only hosting is available from several providers such as Mythic Beasts.

Continue reading IPv6 only hosting

Surviving with IPv6 only

Trying to use an IPv6 only machine runs into a roadblock when you need to access a legacy IPv4 resource.

The best solution would be to convince services to move to an IPv6 first solution, incrementally adding reverse proxies for existing services and deploying new services to IPv6 only, with an IPv4 gateway only as needed.

Deploying new services to IPv6 only is much simpler than dual-stack, as you only need to worry about one set of configuration, firewalls, etc. Adding an IPv4 proxy on top, for legacy support, is then a simple, and contained, extension.

However, until that happens you still need a solution.

I found setting up DNS64 + NAT64 for my IPv6 only machines was pretty simple with my OpenWRT router.

Continue reading Surviving with IPv6 only

Ubuntu, Raspberry Pi, IPv6 only

There are very simple instructions for installing Ubuntu on a Raspberry Pi, simply downloading the Imager and then pick the OS (which it will download for you, I used Ubuntu Server 20.04 LTS), and write it to the micro SD card for your Pi. https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi

The image is pre-configured for DHCPv4 using the wired Ethernet connection, with alternative instructions for getting it setup with Wi-Fi, but without mention of IPv6, which is now used by 30-35% of the Internet.

Here are instructions for setting up Ubuntu on your Raspberry Pi up with IPv6 only.

Continue reading Ubuntu, Raspberry Pi, IPv6 only

Open Source Compatible Fonts

One page reference chart of open source fonts that are size-compatible with well known standard commercial fonts.

There are, of course, a large variety of style open source fonts as well, for all manner of uses. But for basic documents, consider using Nimbus Sans or TeX Gyre Termes instead of your system default.

This blog is (currently, November 2020) in Noto Serif, from the worthy Noto project trying to cover the entirety of Unicode.

A Guide to W3C Trace Context

Earlier this year the W3C Trace Context Recommendation was finally published. A standard way of passing distributed trace correlation has been needed for a long time, and it is good to see there is finally a standard, and many vendors have already moved to adopt it.

The Recommendation defines what a distributed trace is:

A distributed trace is a set of events, triggered as a result of a single logical operation, consolidated across various components of an application. A distributed trace contains events that cross process, network and security boundaries. A distributed trace may be initiated when someone presses a button to start an action on a website – in this example, the trace will represent calls made between the downstream services that handled the chain of requests initiated by this button being pressed.

What constitutes a single logical operation depends on the system. In the example above it is a single button press on a website, whereas in a batch processing system it might be for each item processed, or in a complex UI it might consist of both a button press and a subsequent confirmation dialog.

The W3C Trace Context Recommendation describes how the correlation information — an identifier for the operation, and the parent-child relationships between components — is passed in service calls, but doesn’t cover what to do with that information, apart from how to pass it to the next component.

This is a guide mostly how to use Trace Context for logging, although it also applies to metrics and other telemetry.

Continue reading A Guide to W3C Trace Context

Supply Chain on Blockchain – conference review

On Monday, I attended the Supply Chain on Blockchain conference (https://scobc.net/), usually held in Brisbane, but this year run online.

Here are my impressions of the different sessions I attended. My highlight of the conference was the second keynote from John Wolpert, on the Baseline Protocol project.

Continue reading Supply Chain on Blockchain – conference review

Elasticsearch-Logstash-Kibana (ELK) LoggerProvider for .NET Logging

This ElasticsearchLoggerProvider, for Microsoft.Extensions.Logging, writes direct to Elasticsearch, using the Elasticsearch Common Schema (ECS), and has full semantic logging of structured data from message and scope values.

To use, add the Essential.LoggerProvider.Elasticsearch package to your project:

PS> dotnet add package Essential.LoggerProvider.Elasticsearch

Then add the logger provider to your host builder, and the default configuration will write to a local Elasticsearch service:

using Essential.LoggerProvider;

// ...

    .ConfigureLogging((hostContext, loggingBuilder) =>
    {
        loggingBuilder.AddElasticsearch();
    })

Once you have logged some events, open up Kibana (e.g. http://localhost:5601/) and define an index pattern for dotnet-* with the time filter @timestamp.

You can then discover the log events for the index. Some useful columns to add are log.level, log.logger, event.code, message, tags, and process.thread.id.

Structured message and scope values are logged as labels.* custom key/value pairs, e.g. labels.CustomerId.

Example: Elasticsearch via Kibana
Continue reading Elasticsearch-Logstash-Kibana (ELK) LoggerProvider for .NET Logging