This article presents three object-oriented designs for decoding file formats, in the spirit of the Gang of Four design patterns. Structuring your decoding logic by following one of these designs will increase the maintainability and reusability of your code, turning it into a richer API with minimal additional work.

Most useful programs need to process data from files or network streams. This involves two logical steps: decoding the bytes or characters of the stream according to the file format to extract useful data types, and processing the resulting data according to its purpose for the program. In a simple implementation, those two steps are intertwined, as illustrated by this code for parsing properties in the INI file format, which represents simple key/value pairs organized in named sections:

// Decoding the ini file format
string line = stringReader.ReadLine();
var match = Regex.Match(line, "^(\w+)\s*=\s*(\w+)\s*$"); // name=value
// Processing the data as needed by the program
if (match.Success && section == "settings" && match.Groups[1].Value == "history_length")
  history.Resize(int.Parse(match.Groups[2].Value));

This coupling introduces a few issues:

1. The implementation can become very complex when both decoding and data processing are nontrivial.
2. The decoding logic cannot be reused to process the data in a different way.
3. The decoding and data processing logic cannot be tested in isolation.

To address these issues, these two pieces of logic need to be decoupled. This article presents three designs which achieve this, in the spirit of the Gang of Four design patterns. The basic idea behind each design is simple and commonly found in libraries, but there are various interesting ways to extend and combine them.

The INI file format will be used as an example for its simplicity, but the concepts apply to more complex formats as well. Generalizing, this article will use the terminology "node type" to refer data types defined in a file format, whether hierarchical or not. For example, the INI file format can be described using two node types: sections (e.g. [my_section]) and properties (e.g. my_property=my_value).

Reader design

With the Reader design, every node type becomes a getter on a Reader class. A Read() method and a State property allows the consumer to iterate through the nodes of the file and know which data to retrieve. This is akin to an IEnumerator<T> where T would be a discriminated union of node types. A good example is System.Xml.XmlReader in the .NET base class libraries (XmlReader.NodeType is equivalent to the State property above). Newtonsoft.Json.JsonReader is similar.

class Reader
{
  State State { get; }
  bool Read();
  string GetSectionName();
  string GetPropertyName();
  string GetPropertyValue();
}

enum State { Initial, Section, Property, End }

Data processing logic typically takes the form of a while loop with a nested switch statement to handle the different states. Each consumer will likely have to repeat some amount of this boilerplate.

while (reader.Read())
{
  switch (reader.State)
  {
    case State.Section: Console.WriteLine("New section: " + reader.GetSectionName()); break;
    case State.Property: Console.WriteLine("New property: {0} = {1}", reader.GetPropertyName(), reader.GetPropertyValue()); break;
  }
}

Improvements

A Reader can also expose methods to skip parts of the file of no interest, or even seek to arbitrary locations if the file format supports random access.

bool SkipSection();

Sink design

With the Sink design, every node type in the file becomes a void-returning method on an interface, and a method is provided that takes a stream, decodes the data and "feeds" every node into a sink implementation. A good example is ID2D1SimplifiedGeometrySink.

interface ISink
{
  void BeginSection(string name);
  void AddProperty(string name, string value);
  void End(); // Optional
}

static void ReadToSink(Stream stream, ISink sink) {}

Data processing logic is achieved by creating a class implementing the interface and passing an instance of it to the ReadToSink method. Of note is that ReadToSink will then own the control flow, which prevents scenarios such as reading two streams at once and comparing them nodewise. Each sink implementer will also be required to do some amount of state tracking, for example to associate AddProperty calls with the preceding BeginSection call.

Improvements

If only parts of the file are of interest, the Sink design can be augmented to skip sections by returning a corresponding flag. For example:

bool BeginSection(string name) // Return false to skip (AddProperty will not be called)

For file formats with a more hierarchical structure, multiple specialized sinks can be defined. Some sink methods then become factories for children sinks. This keeps the size of each sink interface smaller and can help implementers structure their code following the hierarchical structure of the data.

interface ISink
{
  ISectionSink? BeginSection(string name); // Return null to skip
  void End();
}

interface ISectionSink
{
  void AddProperty(string name, string value);
  void End();
}

Concrete Sink examples

The Sink abstraction proves to be useful in many ways. Examples of common implementations include:

• Writer: A writer sink encodes the data back into the file format, similar to System.IO.StreamWriter as a subclass of the abstract System.IO.TextWriter class.
• Null: A null sink can be useful as a mock, to unit test the decoding logic.
• Validator: A validation sink can ensure that the sequence of method calls is valid before optionally delegating to another sink implementation, following the Gang of Four decorator pattern.
• Composite: A composite sink can enable processing the data in more than one way at once.
• Builder: A builder sink will accumulate data from its method calls and construct an object model, similar to System.IO.StringWriter as a subclass of the abstract System.IO.TextWriter class (its ToString method returns a string as the object model).

For example, copying a file while validating it could be implemented as:

ReadToSink(inputStream, new ValidatorSink(new WriterSink(outputStream)));

Loader design

With the Loader design, every node type becomes a property in an object model mapping closely to the file format. A method is then provided to decode the data from a stream and produce a corresponding object model. The object model may be mutable or immutable. A good example is XmlDocument.Load.

class IniFile { Section[] Sections; }
class Section { Property[] Properties; }
class Property { string Name; string Value; }
IniFile Load(Stream stream);

Data processing consists in walking the resulting object graph, potentially using recursion if the file format has hierarchical data, and inspecting the exposed properties.

Comparison and summary

Hierarchy of power

These three designs form a hierarchy of power: Reader > Sink > Loader. Less powerful designs can be implemented in terms of more powerful designs, but not the other way around*.

The Sink design can be implemented in terms of the Reader design:

while (reader.Read())
{
  switch (reader.State)
  {
    case State.Section: sink.BeginSection(reader.GetSectionName()); break;
    case State.Property: sink.AddProperty(reader.GetPropertyName(), reader.GetPropertyValue()); break;
  }
}

The Loader design can be implemented in terms of the Sink design by means of a Builder sink:

IniFile Load(Stream stream)
{
  var builder = new IniFileBuilder(); // Implements ISink
  ReadToSink(stream, builder);
  return builder.GetResult();
}

The Loader design can also be directly implemented in terms of the Reader design.

* The Reader and Sink designs could be implemented in terms of the Loader design, but this requires the entire file to be loaded first, whereas Reader and Sink would normally support streaming the data.

Summary

The Reader and Sink designs naturally support streaming, whereas the Loader design requires loading the entire file in memory. The Reader design can most easily support partial file reading, and the Loader design has the most natural representation for hierarchical data.

• Reader: Best for simple formats with few node types. Supports streaming, partial file reading and hierarchical data (though it can be awkward). Leaves the consumer in control of the program flow and is easily testable.
• Sink: Best for complex formats with potentially large file sizes. Supports streaming and hierarchical data. Has limited support for partial file reading. The consumer does not own the control flow but must rather respond to callbacks, which makes testing more difficult.
• Loader: Naturally supports hierarchical data. Has limited support for partial file reading. Does not support streaming. Easily testable.

Recommendation

When possible, follow the hierarchy of power:

• Expose a Reader as the lowest-level design and write decoding tests at that level.
• Expose a Sink as the mid-level design, implemented in terms of the Reader design, to support rich extensibility through sink composition.
• Expose a Loader as the higher-level design, implemented in terms of the Sink design, to support simple use cases where it is convenient to have an object model for the whole file.

Thanks!

These thoughts come from my recent experience writing decoders for several file formats related to music notation. Your feedback on its form and content is very welcome.

Before .NET introduced ReadOnlySpan<T>, a parsing method would typically look like T Parse(string str). Within the body, you would use an int index to keep track of your position as you parse the syntactic parts of the string. For more flexibility, you might expose T Parse(string str, int startIndex, int length) as to not require a new string to be created if its characters existed within a larger string, though this requires more careful handling of various indices and lengths within the implementation.

With ReadOnlySpan<T>, we can use the improved T Parse(ReadOnlySpan<char> str) signature, but the implementation still requires tedious tracking of indices. This article introduces the TryConsume pattern as a less error-prone way to implement parsing methods with ReadOnlySpan<char>.

The idea is to use a series of helper methods of the form: T TryConsume(ref ReadOnlySpan<char> str, ...). These methods attempt to match a prefix in the string. If there is a match, a value is returned indicating the matched value and the ReadOnlySpan<char> is updated to the remainder of the string. If there is no match, false, null or a similar value is returned and the ReadOnlySpan<char> is left unchanged. For example:

static bool TryConsume(ref ReadOnlySpan<char> str, char c);
static bool TryConsume(ref ReadOnlySpan<char> str, string prefix);
static int? TryConsumeInt(ref ReadOnlySpan<char> str);
static bool SkipWhiteSpace(ref ReadOnlySpan<char> str); // Named "Skip" because we don't care what exact string was matched

Usage example: Maui's RowDefinitions microsyntax

The Maui user interface framework supports a short syntax for defining a grid layout with rows of different sizes:

<Grid RowDefinitions="1*, Auto, 25, 14, 20"/>

The value of RowDefinitions is a comma-separated of one of three kinds of sizes: Auto (fit to content), a size in pixels, or a size in stars (used for proportional sizing). For simplicity, assume that the sizes are integers. This format could be parsed using a regular expression, but it would be somewhat awkward. With the TryConsume pattern, it becomes as follows:

static GridLength[] ParseGridLengths(ReadOnlySpan<char> str)
{
  List<GridLength> lengths = new();
  for (;;)
  {
    // Skip any whitespace before a field
    SkipWhiteSpace(ref str);
    if (str.Length == 0) break;

    // Parse one of the supported length types
    if (TryConsumeInt(ref str) is int value)
    {
      var unit = TryConsume(ref str, '*') ? GridUnitType.Star : GridUnitType.Pixel;
      lengths.Add(new GridLength(value, unit));
    }
    else if (TryConsume(ref str, "Auto"))
    {
      lengths.Add(GridLength.Auto);
    }
    else throw new FormatException("Expected a length!");

    // Either we're at the end of the string, or there's a comma and potentially more lengths
    SkipWhiteSpace(ref str);
    if (str.Length == 0) break;

    if (!TryConsume(ref str, ',')) throw new FormatException("Expected a comma!");
  }

  return lengths.ToArray();
}

Considerations

Internationalization

The .NET standard library does not expose type parsing with the TryConsume pattern. If this brings you to implement methods like TryConsumeFloat yourself, you should consider the internationalization implications of supporting different regional formats.

Regular expressions

With simpler formats, regular expressions will often be a simpler alternative to the TryConsume pattern. For more complex formats, however, they may become unwieldly, be unable to express recursive structures, or make it difficult to produce useful error messages. They may also involve additional costs or memory allocations that could be avoided with the TryConsume pattern.

Contiguous memory

Because ReadOnlySpan<char> requires that the underlying chars be in contiguous memory, this approach works best for smaller formats which would be expected to be loaded in a string, such as a date, an email, an url. Think of classes and structs which have both a ToString() and a static Parse() method. For textual data that could potentially be large like a json file, a source code file or a markdown document, it would be preferable to base the parsing logic on TextReader, which supports streaming via StreamReader.

Iterators

When parsing lists, it can be natural to use IEnumerable<T> and yield return statements, known as iterator methods. Unfortunately, this will not work for a method with a ReadOnlySpan<T> as a parameter because its ref struct nature constraints it to living on the stack, and iterators implicitly lift their parameters and locals into the fields of a compiler-generated class. An alternative may be to use ReadOnlyMemory<char> in the method signature and using ReadOnlySpan<char>-based helper methods within the implementation, so long as none of their lifetimes cross a yield return statement.

Reporting the index in error messages

You may want to report the index of any invalid syntax you encounter. The TryConsume pattern updates and shrinks your ReadOnlySpan<char> as you go, so you lose the context of the current substring within the original string. If you stick to the TryConsume pattern and only consume string prefixes, you can store the original ReadOnlySpan<char> before you begin parsing, and find your current index as fullStr.Length - remainingStr.Length.

A sum tree is a data structure which stores items with an associated term (a numerical value) for quick prefix and suffix sum operations. Think of a one-dimensional Tetris game with a stack of linear pieces of different colors, each of which is composed of one or more square blocks:

The data structure supports the following operations in O(log(n)) time:

1. Getting the prefix and suffix sum of an item. This corresponds to counting the total number of blocks in all Tetris pieces left or right of a given piece.
2. Finding the item at a given prefix or suffix sum. This corresponds to finding the Tetris piece at the Nth block, counting from the left or right.
3. Inserting or removing items.

Inspiration

I came up with this data structure to solve a problem in representing music notation, where items are individual notes and rests (generalized as "events"), and terms are duration values (quarter note, dotted eight note, etc.). I needed to be able to get the start time of an event (operation 1), find an event at a given time, and insert/remove events as quickly from the beginning of the score as near the end.

I could not find a reference to this data structure in the literature, though I suspect it exists under a different name.

Construction

A sum tree is implemented as a self-balancing binary tree where every node stores 1. an item with an associated numerical term value, and 2. the sum of the terms of all items in its subtree. Terms may be integers but can be generalized to arbitrary mathematical groups, from abstract algebra. My implementation uses an AVL tree for simplicity, but a red-black tree should work just as well. The properties require nodes to have a parent pointer and O(1) lookup from item to tree node (using a pointer or by embedding the item in the node).

Assuming a well-constructed tree, we can easily show that the properties are honored in O(log(n)) time:

1. To find the prefix sum of an item, add the sums of all left subtrees on the path to the root.
2. To find the item at a given prefix sum, walk down from the root, taking the left subtree if the position is smaller than its sum or subtracting the left subtree size and taking the right subtree otherwise.
3. To insert or remove an item, follow the logic of the underlying self-balancing binary tree and update all ancestors' sum based on the removed term. The tree rotation operations must be updated to maintain the subtree sum invariant, but this is fairly straightforward.

An item's term can also be modified in O(log(n)), by updating all ancestors' sum value.

Variants

Indexable O(log(N)) list

If all items have 1 as their term, the sum tree becomes an indexable list. Each item's prefix sum is its index in the list. The result is a compromise between a linked list and a resizable array (List<T> in .NET):

1. Lookup by index takes O(log(n)) for the sum tree, O(n) for the linked list and O(1) for the resizable array
2. Getting an item's index takes O(log(n)) for the sum tree, O(n) for the linked list and O(n) for the resizable array
3. Insertion and removal take O(log(n)) for the sum tree, O(1) for the linked list and O(n) for the resizable array

Multi-term items

Since the term can be of any mathematical group, a tuple of two terms can serve as a composite term for an item. Going back to the Tetris example, if each piece has a term formed by the tuple (1, block count), then we can both index the tree and look it up by number of blocks from either side.

Prefix sum tree

I first implemented this variant where each node stores the sum of the terms in its left child's subtree rather than the sum of the subtree it defines itself. This provides the same properties as the regular sum tree for all prefix operations, but does not support suffix operations.

Shiftable position tree

A prefix sum tree is best visualized as a sequence of items with different lengths stacked from left to right. Interestingly, it also has a different application where items have a position but no intrinsic length (i.e. they are points on an axis). This is similar to a SortedDictionary<Position, Item>, but has the additional benefit of supporting a shift operation in O(log(n)), which inserts or removes space between two items, shifting all subsequent ones.

In this case, the item's term its distance from the previous item. When inserting a new item B at a position between two existing items A and C, B's term is set to its position minus A's position, and C's position is set to its position minus B's position. This preserves the original sums in the tree, so existing items are not shifted.

In practice

I found surprisingly many uses for the sum tree and its variants in the context of music notation, with integer, rational, fixed point and tuple terms. My implementation uses abstract and generic classes to achieve reuse for different data types:

class SelfBalancingTreeBase<TData>
class SumTree<TSumTerm, TData> : SelfBalancingTreeBase<...>
    where TSumTerm : struct, IAdditive<TSumTerm>
class PrefixSumTree<TSumTerm, TData> : SelfBalancingTreeBase<...>
    where TSumTerm : struct, IAdditive<TSumTerm>
class ShiftablePositionTree<TOffset, TValue> : SelfBalancingTreeBase<...>
TreeList<T> : IList<T> // built on a PrefixSumTree<..., T>

C# 11 is planned to include static abstract members, allowing the use of operators on generic parameters, which would apply nicely to cases like this one.

Your comments on the content or form of this article are most welcome.

People often ask me how I learned 7 languages. Here I'll share the self-guided learning method that I developed over time, and that has worked great for my analytical mind. I find that everyone learn languages differently, so I recommend picking and choosing the tips that work for you.

I consider that classes are not necessary, but can be useful if you have specific objectives (pronunciation, exam preparation), or as a commitment device. If you do take classes, prefer 1-on-1 tutoring to maximize your speaking time time, and look for a teacher who can act as a language coach, helping you find your best way to learn.

I divide the process of learning a language into two phases: study and enjoyment. During the study phase, you're learning what you need to start interacting with the world that the language opens up. In the enjoyment phase, you're integrating using the language into your life and hobbies, such that you continue learning without it feeling like a study session. Each phase has different principles and activities that I describe below.

Phase 1: Study

When you start learning a language, it's unavoidable: you have to study the basics before you can start using it in the real world. This requires some discipline, and it helps if you're interested in the workings of the language and the process of learning it, as there are few extrinsic motivations like enjoying a movie in that language yet. During this phase, keep those two principles in mind:

• Manage your stamina: Build a studying routine that you can sustain. I would often go to the park with my study materials for 30 mins every day. Find a learning buddy if you can. A teacher is not necessary but can help with accountability.
• Favor creating in the language: Make sure your studies involve speaking and writing as these are the skills that will get you to the next phase, and practicing them normally also involves listening and reading.

Listen to an audio course

Audio courses are a great way to break the ice with a language. They start from zero and get you speaking basic sentences very fast, giving you early confidence in your ability. My go-to is the Michel Thomas series, where a recorded teacher leads you through building basic, then progressively more complex sentences. For example, the recording might say "if the door is la puerta and open is abrir, how would you say open the door?", followed by a pause to let you answer before giving and explaining the answer. The downside of audio courses is that they are dissociated from the written language and often explain little to no grammar.

Learn the alphabet and its pronunciation

Languages like Russian or Greek use non-Latin alphabets, so learning those is a prerequisite to any interaction with the written language, but even languages that use a familiar alphabet will have their own pronunciation rules, such as for the z in German or the ll in Spanish. Get familiar with how to pronounce every vowel, consonant, and combinations thereof. It's easy to find charts online and YouTube videos explaining it. If the pronunciation sounds very foreign, this is one time where booking a few classes with an online teacher (e.g. on Italki) pays off.

If you're learning Chinese or Japanese, at this stage you should focus on learning Pinyin and the tones, or Hiragana/Katakana. There are so many Chinese characters and Japanese kanji that learning them requires a more structured, long-term approach.

Study basic grammar and vocabulary

Grammar and vocabulary are the building blocks for expressing your ideas. I suggest building your initial vocabulary through learning basic grammar, especially understanding how to use verbs and construct sentences. I'm a fan of McGraw Hill's Schaum's Outlines of Grammar and Grammar Drills series: they explain one concept at a time and include hundreds of exercises to practice along the way. I often gave myself the challenge of building a single-page cheat sheet that summarized my understanding of the language's grammar — an impossible task, but the exercise helps me mentally organize what I've learned.

I recommend focusing your vocabulary study on the minimum you need for your early conversations (small talk about your hobbies, profession, etc.) by finding learners' podcasts, YouTube channels, or even beginner storybooks about related topics. You can get a lot of mileage out of simple words like good/bad, small/big, and you'll develop more advanced vocabulary naturally from exposure to the language.

Read an easy e-book (stretch)

You can quickly pick up a ton of vocabulary by reading an easy book, but it can be grueling to look up words in your dictionary every sentence, so at this stage I only recommend this if the language you're learning is close to one you're familiar with (e.g. both are romance languages). Use an e-reader and set up a translation dictionary such that touching a word gives you its translation, it's less tedious than constantly switching between a paper book and the dictionary app on your phone.

⚠️ Choose your apps carefully

I don't generally use language learning apps, outside of reference apps like a translator or dictionary. If you use an app, choose one that emphasizes writing or speaking in the language over reading or listening. I recommend avoiding gamified learning apps like Duolingo, which are more incentivized to make you feel good than to make you learn. Everyone has met someone with a 500-day streak who still couldn't say a word. If you must use Duolingo, try these hacks:

1. Use the web app, not the mobile app. A keyboard-centric interface provides better practice than tapping already-written words.
2. Close your eyes before navigating to the next question, listen to the voice and try typing your answer without opening your eyes.
3. Instead of taking the Spanish course in English, take the English course in Spanish. Duolingo often asks you to translate into your spoken language, which is not as useful as translating into the language you're learning.

Phase 2: Enjoyment

This phase begins once you are able to have simple conversations and read basic materials. In the European Framework, you're probably A2, approaching B1. At this point, new learning options open up that are more enjoyable and externally gratifying. The two principles of this phase are:

• Get out there and speak the language: Speaking the language is how you develop your intuition for it; there is no substitute. People whose learning drags on for years don't dare speak it. But you are ready! Get out there and put it to use.
• Combine your learning with hobbies: Whatever you enjoy doing, you can probably do it in the language you're learning. This is how learning stops feeling like studying and just becomes fun.

Do language exchanges

Exchanging languages just means speaking the language you're learning with people who understand that you're practicing, and may be practicing other languages of their own. It's not necessarily an equal time my-language-for-yours situation, but you should be willing to help others with your native language.

Every medium to large city has one or more language exchange meetups, which you can usually find on meetup.com. There are multilingual meetups as well as meetups that focus on a specific language. These events are often free or the cost of a drink. I find that the best meetups impose little structure beyond helping people find others that speak the same language. When there is an active host role, it tends to make all conversations between them and one participant at a time, while everyone else listens.

I recommend attending in-person meetups, but I learned much of my Spanish with online language exchanges. These are interesting because you can make friends with people whose background is very different from yours, and maybe one day visit them. Last I knew Tandem was a popular option, but whichever platform you choose, make sure to have conversations over video calls, not just text messages.

Listen to real podcasts

Podcasts that help you learn a language are useful but still feel like studying. At this point you can start transitioning to podcasts that cover topics you enjoy, while still helping you build your vocabulary. Many languages have podcasts of the sort "News in Slow German", which provide a good middle ground if podcasts for native speakers are still too difficult. I'll often slow podcasts down to 80% or 70% speed if I have trouble following.

Read a book

Reading a book may still be a lot of work, but you can make it more fun by choosing a genre you enjoy, or continuing a series you've started by picking up the next book in the language you're learning. You'll need to use a translation dictionary a lot, and switching to your phone every minute is tiresome, so I recommend getting an e-reader with an integrated translation dictionary (you might have to buy it), such that simply touching a word gives you its translation.

Watch movies and listen to music

I'm neither a cinephile nor do I pay attention to lyrics in music, but I'm weird. Most people will find watching movies in the language they're learning to be a great way to get their ears used to natural uses of the language. Use subtitles if you need them, but first try putting both the audio and subtitles in the language you're learning. If you keep the audio or subtitles in a language you know, make sure it's only for support and you're paying attention to the foreign language content first.

Travel!

Traveling is the ultimate way to put your language learning into practice, but make sure to structure your trip to maximize your opportunities of speaking with locals. Here's what I recommend:

• Choose your travel buddies: Travel solo, with other language learners, or to meet a local friend. If you travel with friends that don't speak the language, you risk creating a language bubble around you and limiting your opportunities to practice.
• Choose your accommodation: Hotels are less likely to provide many speaking opportunities, so better go with bed and breakfasts, AirBnB rooms in a local's home, or hostels. For maximum mingling with locals, consider looking for a couch surfing host. I recommend the Couchers.org platform, but I'm biased as one of its developers. :)
• Plan time for social activities: local sports, dancing or language exchanges events you can find on Meetup.com. If you're brave, visit a Toastmasters club and try delivering an improvised speech.