Geeks With Blogs


Dylan Smith ALM / Architecture / TFS

There’s been some chatter lately about an old debate between Feature Branches vs Feature Toggles.  I used to be firmly in the Feature Branches camp, but about a year ago (at the ALM Summit) I became convinced that Feature Toggles are a better choice in a lot of cases.

Feature branches are fairly common.  It is the practice of creating separate branches for each major feature, or perhaps choosing a group of features for each “feature branch”.

Feature Branching

Teams usually adopt the practice of feature branching because of the increased flexibility it gives them for doing releases (often fixed date releases vs fixed scope releases).  By creating say 5 feature branches for the 5 major features the team is working on, if only 3 of them are ready for release when the target release date rolls around, then only those 3 feature branches get merged into MAIN, and the other 2 feature branches continue development and will be merged into MAIN when complete for inclusion in some future release.



A team I used to work with would not have fixed date releases, but would let the completed features sort of “pile up” until the powers that be decided there was enough completed features to justify a release (or until some really valuable feature was completed perhaps).  Because we were using feature branches, we had the flexibility to release at any point in time, since the half-finished features were isolated off in their own branches.



The problem with feature branches is there can be a lot of pain related to merges.  Ideally, the developers will frequently merge from MAIN->FeatureBranch (maybe they do this every day if there are any changes to merge in), this way whenever a developer/team completes their feature and merges it into MAIN, all other feature branches will get those changes ASAP.  However, the whole idea behind feature branches, is that the code for each feature is isolated in its own branch until it’s ready to be released, at which point it is merged into MAIN.  This results in one big changeset in MAIN that represents all the code for that feature (possibly weeks of work).  As soon as that feature branch is merged in, all other feature branches will have to merge that potentially massive changeset into their feature branches, resulting in a lot of potential merge pain.



Now, most of the people in the Pro Feature Branch camp will tell you that you deal with this by keeping your feature branches small, ideally a day or two.  But in reality that is going to be difficult for most teams.  While they might like to do that, they are a long way from being mature enough to do that.  And I don’t buy that it’s entirely a team maturity issue, some features are simply large, and the teams don’t want to have to break it down into tiny releasable features.  Take for example the recent Git changes the TFS team implemented.  They have been working on that feature for many months.  Even if they could break it down into many 1-2 day features, I doubt that would have been desirable for them.



Not only that, but there is a subtle backwards incentive system at play when it comes to feature branches.  If I make my feature branch really large and long-lived, it’s not *me* that feels the pain.  It’s everybody else when I finally merge it to MAIN.



So what ends up happening is that feature branches are a mechanism that teams use to explicitly defer integrating/merging their changes together (on purpose!).  This is completely opposite the practice of *continuous* integration that most of would probably say is desirable (even if some of us don’t really understand what it truly means).  Fowler wrote about this in detail some time ago.  By isolating various teams/developers changes off in feature branches, we are explicitly deferring integration.



So what’s the other option?  We said at the start that teams use feature branches to achieve flexibility around releasing.  If we just go back to having everybody working in one branch, sure we will have achieved continuous integration, but we’re back with the release problems that we tried to solve with Feature Branches.



This is where Feature Toggles comes into play.  What if we did all work in the same branch, but we used some mechanism to turn on/off features (say via flags in a config file).  Now when I first heard this suggested, I instinctively told the guy he was a goofball, it sounds like this will result in a giant spaghetti mess of code, with if statements surrounding all kinds of crap, leading to a maintenance nightmare.  However, this doesn’t need to be the case.  For one, the “toggle” is only in place while the feature is under development.  Once the feature is complete all toggle code and any config file entries are removed (in fact, when I break down the User Story into tasks, I’ll make sure the last task on the list is “Remove Toggle”).  It does require some conscious thinking about how to develop the feature in such a way that it can be hidden from the user and implemented in such a way to allow it to be turned on/off.  In my experience, this turns out to be easier than you may think.  The vast majority of features can be hidden behind a feature flag if you give it a little thought; and those few that can’t you can still always create a feature branch if you wish.



Some people worry about the implications of including code for unfinished features in released software (even if hidden behind a toggle). I don’t think this as a big of a deal as some people think. My release process will typically involve creating a Release Candidate branch, and performing pre-release testing on that build of the software to ensure it meets the release quality standards.  If there are any ill-effects resulting from including unfinished features behind a toggle in these Release Candidates they should hopefully be discovered during this testing phase.



This gives you the benefits of *true* continuous integration.  If somebody writes some code that breaks another developers.  The guy that wrote the breaking code will find out immediately, and he will be the one responsible for fixing it.  Unlike feature branches, where if I write some code that breaks another developers feature-under-development, I don’t know because the other dev’s code is off in its own branch.  I merge my code to MAIN, the other dev merges my code into his branch, and his branch becomes broken, only now it’s his job to fix it, not mine; even though I’m the one that wrote the problem code.



There are some other benefits that you get along with feature toggles.  For one, you have a super-easy rollback plan if deploying a new feature goes south, simply turn off the feature toggle (assuming you leave the toggle in place until after the first release).  You can do phased roll outs - how about turning on major new features just for a subset of your users, to get some feedback before turning it on for everybody.  The TFS team at Microsoft uses feature toggles extensively.  They turn on a lot features in advance (for the TF Service cloud offering) for MVP’s to try out and provide feedback on.




To summarize, teams usually adopt feature branches to provide release flexibility.  But they have the undesired effect of deferring integration, and causing lots of merge pain.  By using feature toggles you can have continuous integration, while still retaining the release flexibility of feature branches.  Some care must be taken to implement features in a way to support toggles, and you must be disciplined about removing toggles once features are complete.  You get the added benefits of being able to easily do phased roll-outs on a feature by feature basis too!

Posted on Sunday, February 10, 2013 5:16 PM | Back to top

Comments on this post: Feature Toggles vs Feature Branches – Dylan’s $0.02

# re: Feature Toggles vs Feature Branches – Dylan’s $0.02
Requesting Gravatar...
Using feature branch does not mean that integration is delayed.

That's a misconception.

The integration happens all the time, every dev is responsible for bringing the changes of the master branch and run tests. You discover issues even sooner.

Left by Amir Barylko on Feb 10, 2013 5:34 PM

# re: Feature Toggles vs Feature Branches – Dylan’s $0.02
Requesting Gravatar...
Devs bring MAIN into their branch, but they don't share their code with others. code only gets shared when a feature is completed.
Left by Dylan Smith on Feb 10, 2013 5:39 PM

# re: Feature Toggles vs Feature Branches – Dylan’s $0.02
Requesting Gravatar...
Much of this is a problem of excess inventory / work-in-progress. If you can focus the team to be developing less features at any one time, you'll have less branches / toggles / unused code to deal with in any release. You'll be able to release individual features faster.

Unreleased software is inventory that the development team must manage. This leads to all sorts of wasted effort and wasted distractions.

The only things I use branches for in my team at this point are high-experimental research tasks that end up dying much of the time.
Left by Eric Anderson on Feb 11, 2013 12:24 AM

# re: Feature Toggles vs Feature Branches – Dylan’s $0.02
Requesting Gravatar...
@Eric - I agree with you 100%. The ideal scenario would not be using either feature branches or feature toggles. However, this requires giving up the release flexibility I mentioned.

For a lot of teams this is perfectly acceptable though. If a team is strictly following Scrum, then they finish all WIP and have potentially releasable code at the end of each Sprint. They have no need of either Feature Branches or toggles.

If a team is doing a Kanban/Continuous-Flow type system, then they can choose to drain all WIP from the system when they wish to release. Another approach is to use branching to make this process slightly less awkward. What I like to do is create a new branch when I wish to release. All new WIP goes in the new branch, all existing WIP will be finished on the existing branch. This way it allows you to drain the WIP off the existing branch without having to stop all new WIP in the team.
Left by Dylan Smith on Feb 11, 2013 3:42 AM

# re: Feature Toggles vs Feature Branches – Dylan’s $0.02
Requesting Gravatar...
This is a straw man argument. You do not reflect (or even link to my article which sites the original argument with Fowler) to the proper way branch per feature is done. You integrate relentlessly and share rerere so merging is never an issue - no matter how late the feature is finished.
Left by Adam Dymitruk on Feb 11, 2013 6:00 AM

# re: Feature Toggles vs Feature Branches – Dylan’s $0.02
Requesting Gravatar...
I don't think this is a straw man at all. The way I described FB done is the way 99.9% of teams out there do it today. Your approach with rerere is certainly interesting, but I've had this conversation with you in person before why I think it won't work in the real world. I'd have to write another blog post to describe that in writing. Regardless, as interesting as it is, nobody I know is doing the rerere approach (IIRC you don't even use it on your team).
Left by Dylan Smith on Feb 11, 2013 6:06 AM

# re: Feature Toggles vs Feature Branches – Dylan’s $0.02
Requesting Gravatar...
We do use it. It's up in the air if you share the cache manually or automate it. Your percentage is wrong. Linus would rage quit if this is how merges in bpf on the Linux kernel was done. So you are probably talking about LOB .net stacks. The truth is that with git, there is a much larger world which does not use back merges for very good reason. Your 99% is more like 20% if that in the real world. How many merges in the Linux kernel? Exactly.
Left by Adam Dymitruk on Feb 11, 2013 6:50 AM

# re: Feature Toggles vs Feature Branches – Dylan’s $0.02
Requesting Gravatar...
Has anybody here use Feature Toogles in programming languages with metadata?
Suppose we have an xml or jsp file which is interpreted by an IDE Design Editor.
The only way, if possible at all from the language, seems to be to edit the file only in the source editor.
This means loosing any IDE Design Editor productivity and the risk of loosing the change if accidently open the file with the IDE Design Editor again.
Left by Torsten Kleiber on Mar 18, 2014 8:07 PM

Your comment:
 (will show your gravatar)

Copyright © Dylan Smith | Powered by: