Feature Toggling from the Trenches

Feature Toggling from the Trenches

The Problem

Imagine you have a software product that is under constant development; that's probably not all that difficult. Imagine now you want to commit your stable but partially-completed feature into your trunk repo, but not make it available until it's ready. Now imagine you are developing a feature that will be a premium feature, only available to a small section of your customers.

NewVoiceMedia had these problems, and more, and solved them - mitigated them at least - by using feature toggling.

A Feature Toggle Mechanism

This story starts about 5 years ago. NewVoiceMedia had a history of implementing naive solutions to the problem of allowing a feature for a specific customer or group of customers. Often the solution consisted of a new control in an existing screen accessed by a NewVoiceMedia administrator, with the setting stored in a bespoke field in the database. Sometimes it came with a whole new admin screen. Sometimes a feature was enabled via a value in a web.config file. These solutions were either costly to implement in terms of time and effort, or costly to maintain so meant that it was not done for every new feature.

The upshot of this is that most code went to production with no ability to shut off a new feature if something unexpected happened - in spite of manual and automated tests, unexpected things do happen. Some of these unexpected things led to rollbacks of production releases, and decreased customer confidence in our product. As we moved towards quicker releases to production, the need to protect the stability of the product became more and more important, together with our desire to check code in to the trunk/main branch as early as possible.

The first implementation of a toggling system was a list of toggle names and default values (on/off) and a database table that stored overrides of that default value for customers. This was backed by a service to read the values to determine if a feature was enabled. The code containing a toggled feature would look something like:

foo();

if(toggles.Enabled(FeatureType.ReticulateSplines, customer))
{
    splines.Reticulate();
}

bar();

This implementation was quick and easy to implement from a developer's perspective but required manual scripts running to set override values in the database. This quickly became tiresome, so the next evolution was a screen to allow administration of overrides.

This is the basic system that's in place today. We've made enhancements and tweaks to the system along the way when the need has arisen, for example:

• Caching of override values
• Tagging of feature toggles, allowing:
• Visual grouping on the admin screen
• Clearer separation between development toggles (to enable swift rollback of code, to protect the platform) and customer toggles (to enable a premium feature for a particular account)
• Introducing different levels of granularity for toggles
• Platform-wide, to turn a feature on or off for all customers
• User-specific, to allow a specific user to have access to a feature

Toggle lifecycle

Introducing this mechanism means that it's very easy to add a developer toggle to every new feature. As you can imagine that leads to a large number of toggles, and their related checks become dotted around the code. When a developer toggle outlives its usefulness, it's in the technical debt pile. This means that each developer needs to be disciplined in removing unused toggles when they're no longer relevant. The lifecycle of a development toggle at NewVoiceMedia generally runs like this:

• Introduce a toggle, default the value "off"
• Develop the feature behind the toggle, including extending appropriate automated tests, to execute code with a toggle on and off
• Enable the toggle for test customers in a test environment. This allows the feature to be tested while it's being developed (for early feedback)
• Turn the toggle on in production for selected clients
• After feedback is gathered, change the default value of the toggle to "on", leaving code and tests in place for toggle on and off
• Remove the unit tests testing the toggle "off" code
• Remove the toggle "off" code
• Remove the toggle itself

Observations

If I was writing a toggle mechanism again from scratch, I would use the patterns that I've learnt along the way with the NewVoiceMedia journey. One thing I would do differently is avoid the choice of a default value for toggles; all toggles would be off unless an override exists. This would simplify logic in calculating the value of a toggle.

I would also consider using a 3rd party library for toggle management. This page lists a few libraries available https://featureflags.io/dotnet-feature-flags/

Read more about feature toggling as a concept on Martin Fowler's blog https://martinfowler.com/articles/feature-toggles.html

Decorator pattern

Why?

I have a colleague who is officially a tester and scrummaster, and he has a growing interest in the code we (the developers) write. He asked a question this morning: "I've heard you talk about decorators, what are they?". I thought the best way to describe it is to write it down.

What?

The decorator pattern allows you to add functionality to a class without changing the underlying class. (See Open/Closed principle)

As an example, there's a class that reticulates splines:

Imagine you wanted to add logging to this class, to store the number of splines reticulated. You could add the logging directly into the class:

Alternatively you could add a decorator (literally "decorate" the class with new behaviour). The idea behind the decorator pattern is that the decorating class implements the same interface as the underlying class, and wraps an instance of the underlying class. This way it can pass through to the underlying instance, and add value along the way.

This gives you the ability of adding several layers of behaviour, and swapping in and out that behaviour as and when required:

For more info, see this

Calling generic methods with runtime Type parameters

In the normal course of events, a call to a generic method needs to have its type resolved at compile time. For example, consider this class and method:

class TypeWriter
{
    public void WriteType<T>(TextWriter target)
    {
        target.WriteLine("T is {0}", typeof(T));
    }
}

This generic method needs to be called with its type parameter resolved at runtime, like this:

var typeWriter = new TypeWriter();
typeWriter.WriteType<Program>(Console.Out);

which gives this output:

But what do you do if you only discover the types at runtime so don't know the types at compile time? This commented code wouldn't compile because the variable assemblyType can't be resolved to a generic parameter:

foreach (var assemblyType in Assembly.GetExecutingAssembly().GetTypes())
{
    // typeWriter.WriteType<assemblyType>(Console.Out);
}

This is where System.Reflection comes in. The MethodInfo class has a method MakeGenericMethod that takes an array of System.Type, substitutes the method's generic parameters for the arguments and returns a new MethodInfo instance. This new MethodInfo instance can then be invoked like any other. Here's how this looks in code:

foreach (var assemblyType in Assembly.GetExecutingAssembly().GetTypes())
{
    // Get a handle on the method to call
    var method = typeof(TypeWriter).GetMethod("WriteType");

    // Pass the type parameter(s)
    var genericMethod = method.MakeGenericMethod(assemblyType);

    // Call the method on the TypeWriter instance, passing parameter(s)
    genericMethod.Invoke(typeWriter, new[] { Console.Out });
}

This gives the output:

KeyedCollection

During a talk at DevWeek 2014 I discovered the KeyedCollection<TKey,TItem> class. The class is in the System.Collections.ObjectModel namespace, and is an alternative to the Dictionary<TKey,TValue> for items where the key is embedded in the item itself. The following code sample shows where it pays dividends to use a KeyedCollection over a Dictionary.

The following program creates a dictionary of Person objects; it uses the Id of the Person as the key, and the person itself as the value:

Bearing in mind that this is a contrived example, there is a problem with this code; having to specify the key manually has lead to a typo (didn't you spot it?):

If the key to use for your collection item can be derived from the value, the KeyedCollection allows you to do that.
The KeyedCollection class is abstract, so you need to inherit from it. Having done so, you can override the GetKeyForItem method; this is the method that the framework calls to obtain the key of the item:

Items can then be added to the collection as you would any collection, either using the Add method, or an object initialiser as I've done here:

When retrieving an item, the KeyedCollection behaves like a Dictionary rather than a List, because you access items by their key rather that their position in the collection:

See more about the KeyedCollection class on MSDN: http://msdn.microsoft.com/en-us/library/ms132438(v=vs.110).aspx

System.BitConverter Class

Every now and again a new framework class comes to my attention. At DevWeek I was attending a talk on Cryptography and an incidental part of the talk featured the BitConverter class. This is a class in the System namespace and has a bunch of static methods that convert to and from a byte array.

The ToString(byte[]) method was the one that I saw being used. It takes an array of bytes and converts each byte to its 2 character hexadecimal equivalent. This is an alternative to Base64 encoding and is arguably more human-friendly.

Other methods on the BitConverter class allow you to convert between a byte array and numerical types:

For more details, see the MSDN documentation at http://msdn.microsoft.com/en-us/library/system.bitconverter(v=vs.110).aspx

Faking AppSettings in tests

If you have some code that uses configuration from an app.config or web.config file, and you need to retrospectively write tests for it, there are several approaches you can take to get fake setting into your tests.

Given this code

using System;
using System.Configuration;
using System.IO;

namespace Demo
{
    class Program
    {
        static void Main(string[] args)
        {
            using (var writer = new StringWriter())
            {
                new SeasonWriter().WriteSeason(writer);
                Console.WriteLine(writer);
                Console.ReadLine();
            }
        }
    }

    public class SeasonWriter
    {
        public void WriteSeason(TextWriter textWriter)
        {
            var month = ConfigurationManager.AppSettings["Month"];
            switch (month)
            {
                case "January":
                case "February":
                case "December":
                    textWriter.Write("Winter");
                    break;
                case "March":
                case "April":
                case "May":
                    textWriter.Write("Spring");
                    break;
                case "June":
                case "July":
                case "August":
                    textWriter.Write("Summer");
                    break;
                case "September":
                case "October":
                case "November":
                    textWriter.Write("Autumn");
                    break;
                default:
                    throw new ConfigurationErrorsException("No setting defined for 'Month'."
                        + Environment.NewLine
                        + "Please add a setting to the .exe.config file under the section AppSettings."
                        + Environment.NewLine
                        + "For more info on the .exe.config file, follow this link "
                        + "http://msdn.microsoft.com/en-us/library/vstudio/1fk1t1t0(v=vs.110).aspx");
            }
        }
    }
}

In order to get tests around the SeasonWriter class, the AppSettings entry "Month" needs to be faked or abstracted. Some options are:

Create an app.config file in the test project

This is the most obvious starting point. Create a test project to contain the SeasonWriterTests file, with an app.config containing a "Month" entry:

<appSettings>
  <add key="Month" value="January"/>
</appSettings>

using NUnit.Framework;
using System.IO;

namespace Demo.Test
{
    [TestFixture]
    public class SeasonWriterTests
    {
        [Test]
        public void WriteSeasonExpectWinter()
        {
            var writer = new StringWriter();
            var target = new SeasonWriter();
            target.WriteSeason(writer);
            Assert.That(writer.ToString(), Is.EqualTo("Winter"));
        }
    }
}

This will allow the test to run, but will limit you to testing a single path through the code. It's also flaky, because your test depends on a magic string in a separate file.

Pass the configuration setting in

This solution is a refactor of the WriteSeason method, so it takes the config setting in as a string. This sounds like a clean fix, with benefits

• the responsibility for getting that value has been handed off to the user of the method
• the code is easier to test

However, there are downsides

• the refactor is an untested change
• there could be many users of this method, and each one would be responsible for getting the month value
• the exception type in the WriteSeason method is no longer correct

Set the config value

The easiest way to fake an AppSettings value is just to set it in your test class. ConfigurationManager.AppSettings is just a NameValueCollection, and items can be written as well as read:

using NUnit.Framework;
using System.IO;
using System.Configuration;

namespace Demo.Test
{
    [TestFixture]
    public class SeasonWriterTests
    {
        [TestCase("January", "Winter")]
        [TestCase("February", "Winter")]
        [TestCase("March", "Spring")]
        [TestCase("April", "Spring")]
        [TestCase("May", "Spring")]
        [TestCase("June", "Summer")]
        [TestCase("July", "Summer")]
        [TestCase("August", "Summer")]
        [TestCase("September", "Autumn")]
        [TestCase("October", "Autumn")]
        [TestCase("November", "Autumn")]
        [TestCase("December", "Winter")]
        public void WriteSeasonExpectCorrectSeasonForMonth(string month, string expectedSeason)
        {

          ConfigurationManager.AppSettings["Month"] = month;

            var writer = new StringWriter();
            var target = new SeasonWriter();
            target.WriteSeason(writer);
            Assert.That(writer.ToString(), Is.EqualTo(expectedSeason));
        }
    }
}

Disposable HtmlHelper Extensions

If you're regularly using a block of HTML elements consisting of opening tags and closing tags, you might want to employ the using pattern used by the framework's FormExtensions:

This is pretty straightforward; you need to create a class that implements IDisposable and render the opening tags in the constructor, and the closing tags in the Dispose method.

As an example I'll implement an HtmlHelper extension that renders a multi-column table with a header row containing the column titles specified, that will be rendered like this:

The first step is to create a static class for the extension method, and the extension method itself. This should be an extension method for the HtmlHelper object, and needs to return an instance of a class that implements IDisposable:

The HtmlHelper exposes a ViewContext, which has a Writer property; this is a TextWriter, and it's this object that stores the HTML that will be sent to the HttpResponse. The writer is the object that we need to pass our HTML string to. For this example, I also need the headers to render, so I've added a params parameter to the extension method:

The implementation of the disposable helper constructor builds and writes the table header, and the dispose method writes the closing tag:

The usage of the helper in the view can be changed to this:

Or this:

And gets rendered like this:

The code for the example helper class is available as a GitGist