Logging with Json

Posted by jockelarsson on

Kiwi.Json can be used for great logging in sitations where log4net or NLog is opted out.

Personally, such cases arise when I need the logs to both machine and human readable.

Such cases do indeed exist. I often use separate transaction logging to keep a reliable backup of data I put into Sqlite. Logging should be fast, but more important, I want to be able to more or less create a copy of the database by just replaying the log.

Lets reason about a logging API.

public class DatabaseOperationLog{
  public string Operation{get; set;}
  public Dictionary<string,object> Data{ get; set; }
}

public interface IDatabaseLog{
  void Add(DatabaseOperationLog operation);
  IEnumerable<DatabaseOperationLog> GetOperations();
}

Given this interface, we can log as follows:

IDatabaseLog log = ...
log.Add(new DatabaseOperationLog{
  Operation = "update",
  Data = new Dictionary<string,object>{ {"User","John"},{"Age",35} }
});

And, we can print all the entries with

IDatabaseLog log = ...
var allOperations = log.Operations;
foreach(var operation in log.GetOperations())
{
  System.Console.WriteLine("{0}: {1}", operation.Operation, JsonConvert.Write(operation.Data));
}

This far, its easy. The tricky part in practice is the actual layout of the log file. For the log file to be pure json the content should be something like

[
  {Operation:"update", Data:{...}},
  {Operation:"insert", Data:{...}},
  {Operation:"delete", Data:{...}}
]

Notice how logfile content is a wellformed json array of objects. This is easy to parse and understand but a pain to maintain, since appending a new entry must handle the following cases:

  • adding to an empty or missing logfile should write out a full array with the added messages
  • adding to a log with existing entries means parsing from the end to find the closing ‘]’ of the array, append an array delimiter ‘,’, the log entry and then finally the array terminator ‘]’.

A simpler approach is to just blindly append, relaxing the requirement that the whole log file must be wellformed json. The content of the log would then be something like

{Operation:"update", Data:{...}}
{Operation:"insert", Data:{...}}
{Operation:"delete", Data:{...}}

A piece of cake to understand and implement, but this makes reading the log quite difficult with most Json parsers. In their current builds, both Newtonsoft.Net and Servicestack.Text cant handle this problem, since the log in its entirety isn’t wellformed.

Most json implementations have a separate lexical analysis step before parsing, making parsing gready. Kiwi.Json on the other hand will never read additional characters from input once it parsed a valid json fragment.

The code for reading json fragmens with Kiwi.json is as follows:

public IEnumerable<DatabaseOperationLog> GetOperations(string logFile)
{
  var reader = new JsonStringParser(File.ReadAllText(LogFilePath));
  while (!reader.EndOfInput())
  {
    yield return JsonConvert.Parse<DatabaseOperationLog>(reader);
  }
}

This is a post in my series about a practical Json implementation in .NET. The source actual code is hosted on github (https://github.com/jlarsson/Kiwi.Json). A compiled version is avaiable from nuget (http://nuget.org/packages/Kiwi.Json).

Leave a comment