Serilog Request Logging + JSON: A Complete Guide

Introduction

Hey guys, let's dive into a super important topic for any .NET application: Serilog request logging with a JSON sink. In this guide, we'll walk through how to set up structured request logging using Serilog, ensuring you capture crucial information like request IDs, user names, paths, status codes, and durations. We'll also explore how to configure a rolling JSON file sink for efficient log storage and rotation. This is not just about logging; it's about gaining valuable insights into your application's performance, identifying bottlenecks, and troubleshooting issues. Proper logging is a cornerstone of any robust application, providing the data needed to monitor, analyze, and improve your system. Think of it as having a detailed record of every interaction your application has, allowing you to understand user behavior and system performance with clarity. Without it, you're flying blind, hoping you can figure out what went wrong if something breaks. With good logging, you're equipped with the information needed to not only fix problems but also to proactively identify potential issues before they even arise. We will be using Serilog, a fantastic structured logging library for .NET. It's flexible, powerful, and easy to integrate. Let's get started!

Setting Up Serilog.AspNetCore Request Logging

First things first, let's get Serilog integrated into your ASP.NET Core application. This involves installing the necessary NuGet packages and configuring Serilog in your Program.cs file. We'll be using Serilog.AspNetCore to capture request-related information. The beauty of structured logging is that you can easily search, filter, and analyze your logs. Instead of just seeing a wall of text, you'll have structured data that's easy to query. This is crucial for any production application. This will allow you to see data in an easy to parse format so you can find problems quickly and efficiently. You will be able to see the request, the path, and how long it took for the server to respond to the request. This is useful information to understand what the performance of the website is and if there are any bottle necks. The first step is to install the required NuGet packages: Serilog.AspNetCore and Serilog.Sinks.RollingFile (or Serilog.Sinks.File if you prefer a simpler file sink). Then, in your Program.cs file, configure Serilog like this. Make sure you using Serilog; at the top of the file. In the ConfigureServices method of your Startup.cs file, make sure you add services.AddControllersWithViews(); and then add .UseSerilogRequestLogging() to the app. Let's get down to the nitty gritty and install the proper libraries. This can be done with a dotnet command. Then, you'll want to add UseSerilogRequestLogging() to your application to allow for request logging.

// Program.cs
using Serilog;

var builder = WebApplication.CreateBuilder(args);

// Add Serilog configuration
Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.File("log.txt", rollingInterval: RollingInterval.Day)
    .CreateLogger();

try
{
    builder.Host.UseSerilog();

    // Add services to the container.
    builder.Services.AddControllersWithViews();

    var app = builder.Build();

    // Configure the HTTP request pipeline.
    if (!app.Environment.IsDevelopment())
    {
        app.UseExceptionHandler("/Home/Error");
        // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
        app.UseHsts();
    }

    app.UseHttpsRedirection();
    app.UseStaticFiles();

    app.UseSerilogRequestLogging();

    app.UseRouting();

    app.UseAuthorization();

    app.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");

    app.Run();
}
catch (Exception ex)
{
    Log.Fatal(ex, "Application terminated unexpectedly");
}
finally
{
    Log.CloseAndFlush();
}

This setup tells Serilog to write logs to the console and to a file named log.txt, rotating the file daily. The UseSerilogRequestLogging() middleware adds the request logging features. Now, every HTTP request your application receives will be logged, including information such as the request path, method, status code, and duration. This is a great start for getting useful data. This ensures you have a centralized place to view your requests.

Configuring the Rolling JSON File Sink

Alright, let's talk about getting your logs stored in a JSON format. Why JSON? Because it's structured, making it easy to parse and analyze your logs. The goal is to store this data in a way that is easy to query and understand. Using JSON will make it easier to read and use this data with tools such as Elastic Search, Splunk, or other logging services. We're going to use the Serilog.Sinks.RollingFile package. The file sink writes to a JSON file, and rollingInterval lets you rotate the logs, preventing your log file from growing indefinitely. To make it JSON, you can configure the sink to use a JSON formatter. This formatter ensures that your logs are written in a structured JSON format, making them easy to parse and query. This makes it easy to filter the logs. Now, let's look at configuring the JSON file sink. We'll modify the Program.cs to use the file sink with a JSON formatter. Here's how you can modify your Program.cs to use the JSON file sink:

// Program.cs
using Serilog;
using Serilog.Events;

var builder = WebApplication.CreateBuilder(args);

// Add Serilog configuration
Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Information()
    .MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
    .Enrich.FromLogContext()
    .WriteTo.Console()
    .WriteTo.File("log.json", rollingInterval: RollingInterval.Day, formatter: new Serilog.Formatting.Json.JsonFormatter())
    .CreateLogger();

try
{
    builder.Host.UseSerilog();

    // Add services to the container.
    builder.Services.AddControllersWithViews();

    var app = builder.Build();

    // Configure the HTTP request pipeline.
    if (!app.Environment.IsDevelopment())
    {
        app.UseExceptionHandler("/Home/Error");
        // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
        app.UseHsts();
    }

    app.UseHttpsRedirection();
    app.UseStaticFiles();

    app.UseSerilogRequestLogging();

    app.UseRouting();

    app.UseAuthorization();

    app.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");

    app.Run();
}
catch (Exception ex)
{
    Log.Fatal(ex, "Application terminated unexpectedly");
}
finally
{
    Log.CloseAndFlush();
}

In this configuration, the WriteTo.File method specifies the file path (log.json), the rolling interval (daily), and importantly, the JsonFormatter. This tells Serilog to format the log events as JSON. Also, notice that we are using the rollingInterval property to control when new logs are made. The rollingInterval setting is important for performance, and storage. This keeps the size of the logs to be manageable. This will write logs to a file that will change daily, and will provide a file for you each day.

Enriching Logs with Additional Properties

Beyond the basic request information, you'll often want to include additional context in your logs. Serilog makes it super easy to add extra properties, like the RequestId (to correlate logs across requests), the UserName of the logged-in user, or any other relevant data. Enriched logs provide more valuable data to your logging. The more information you add, the better the chance of finding the root cause of any issue. This extra information will help you filter and query the data. Enrichers are components that add properties to your log events. For example, you can use an enricher to add the RequestId to each log. Here's how you can add an enricher to include the RequestId:

// Program.cs
using Serilog;
using Serilog.Events;

var builder = WebApplication.CreateBuilder(args);

// Add Serilog configuration
Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Information()
    .MinimumLevel.Override("Microsoft", LogEventLevel.Warning)
    .Enrich.FromLogContext()
    .Enrich.WithMachineName()
    .Enrich.WithProcessId()
    .Enrich.WithThreadId()
    .WriteTo.Console()
    .WriteTo.File("log.json", rollingInterval: RollingInterval.Day, formatter: new Serilog.Formatting.Json.JsonFormatter())
    .CreateLogger();

try
{
    builder.Host.UseSerilog();

    // Add services to the container.
    builder.Services.AddControllersWithViews();

    var app = builder.Build();

    // Configure the HTTP request pipeline.
    if (!app.Environment.IsDevelopment())
    {
        app.UseExceptionHandler("/Home/Error");
        // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
        app.UseHsts();
    }

    app.UseHttpsRedirection();
    app.UseStaticFiles();

    app.UseSerilogRequestLogging();

    app.UseRouting();

    app.UseAuthorization();

    app.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");

    app.Run();
}
catch (Exception ex)
{
    Log.Fatal(ex, "Application terminated unexpectedly");
}
finally
{
    Log.CloseAndFlush();
}

In this case, the Enrich.FromLogContext() enricher is used to enrich log events with properties from the log context. The enrichers are added to the LoggerConfiguration before creating the logger instance. Using enrichers will allow you to enrich your logs so that you can add more data to the logs. There are a lot of different enrichers that you can choose from. If you need to, you can write your own. You can also add Enrich.WithMachineName(), Enrich.WithProcessId(), and Enrich.WithThreadId() to the configuration.

Sample Query in README

To make things even easier, you should include a sample query in your project's README file. This provides a quick reference for how to search and analyze your logs. The format of the query will depend on the tool you're using (e.g., Elasticsearch, Seq, etc.). A sample query makes it easy for you to find information. If someone needs help finding data, you can provide these queries to help them out. The sample query in your README should demonstrate how to search for the specific information you are looking for. This will help people who don't understand how to read the data to understand the information. For example, if you are using Seq, your README might include a query like this:

Find logs with status code 500:

```json
StatusCode = 500

This query will show you all the logs with status code 500, which will help you find all the errors.

Conclusion

Alright, that's a wrap, guys! You now have a solid understanding of how to set up structured request logging in your .NET applications using Serilog and a JSON file sink. This approach provides a powerful and flexible way to capture and analyze request-related data, enabling you to identify and resolve issues, monitor performance, and gain valuable insights into your application's behavior. Remember to document your setup and include sample queries to make it easy for anyone working with your application's logs. Happy logging, and enjoy the benefits of a well-structured, easily searchable logging system. Keep it up and keep learning!