Beware the Agile Smells

For a while now I've been somewhat disturbed by all the methodologies I've so eagerly studied during the last few years. You know, the Agile movement and the Craftsmanship movement that's been coming on strongly. I've been troubled by all the time that is consumed with other activities than coding in my projects.

When I first saw the Programming Motherf**ker manifesto I thought it was ridiculous, but I'm not all that inveighs now. Even though I'd still call it silly it has some kind of point, because lately I've consciously been moving away from safety and coming closer to performing. I'm backing a couple of steps toward where I once was, the pragmatic programmer moving quickly just doing stuff. That was before I was infected by Agile.

Slowing down

Doing agile is a lot about safety where you time box stuff in short iterations with up front estimates, which is good in many ways. But when you spend too much time preparing, planning and estimating it stabs you in the back in the end. Your velocity would be better if you just did something, instead of four of us playing planning poker about if this is a one or two point story, and how you'd best divide it in tasks. I've also had problems with endless retrospective meetings and ad hoc discussions about how we work.

Also TDD and Unit Testing can have the same problem; it can easily be overdone. It's great in so many ways, but you need to keep a good balance between safety and productivity. I'm a big fan of Unit Testing, don't get me wrong, it's just that I don't think you need to test everything. Nor do I think you need to inject every single dependency. More on that later.

Speeding up

I came to notice a course by Dan North called Faster Software Delivery, held in Norway, which sounded really interesting to me. It seems to address exactly this! But since I live in Sweden I asked him on Twitter if there was any book or screencast on the subject, but unfortunately no. A book is in the writing though! However he pointed me to a talk he's been doing on a couple of conferences called Patterns of Effective Delivery. I found a good webcast of it here. You really should watch it when you're done reading this post!

Facebook Poster

Another interesting talk I've found lately is this 10 minute speech from JavaZone by Christin Gorman. She complains about the way of coding taught by Uncle Bob where everything should be broken down in methods of ideally one line of code, and naming that method properly so it's obvious what it does, to make the code more readable. I'm not quite sure who's side I'm on, but I find the discussion really interesting, and the best way is probably somewhere between the extremes.

At Facebook they have a culture inherited from Mark Zuckerberg encouraging to do stuff fast. Mark has this saying, "Move fast and break things", claiming that if things don't break you're not moving fast enough. Just shortly ago we often saw that in production, but now it seems they've either slowed down or put a wall of testers between the fast moving programmers and the production environment as I haven't had much problems with Facebook lately. They also have it written on their walls that "Done is better than perfect" to remind themselves to always keep shipping... that's something I really can relate too and actually put up that poster on a wall next to my desk!

Agile Smells

Talking too much

Since I started doing Agile I've had a lot of more meetings than before. Meeting are good though, making sure we're doing the right thing and keeping every stakeholder up to date with the outcome. But since we have them frequently it's important to keep them short and to have them when needed.

Changing too much

Being open to change is a cornerstone in Agile, and we should be – both in what we do and how we do it. But if we’re too open to change it becomes a problem. It may be adopting a new hot technique without the need for it, merely because there’s a lot of fuzz around it. It may also be not giving the ideas from the previous retrospective a fair chance before changing again or changing back to normal as we can’t manage the work to apply the decided change.

Continuity is needed for any methodology to work, so we can’t expect something to solve the problem in the first week. If something is obviously bad you should of course drop that practice directly but make sure you give your ideas a fair chance before ditching them and trying the next.

Prototyping too much

Prototyping is generally a good practice, where we invest as little as possible to gain as much knowledge as possible. We need to make sure though that we don't invest too much in the prototype though, as it would take away the benefit. Prototypes should be quick to make and shown to the stakeholders as soon as possible, and once the general idea is decided it should be terminated. Optimal efficiency is reached when the prototype is hand drawn with the client present and participating.

In some cases it may also be advisable to use tracer bullets instead of prototypes, a technique from the book The Pragmatic Programmer. Instead of creating a prototype you start writing the code with a first decent idea of how it should be and forge it to perfection through continuous feedback as you go.

Estimating too much

Estimates come to a price, and that price is that it takes time. Hence, by estimating, it will by definition take more time to do something. The more time you put into the estimate the more time will be added to completing the task. Here you need to find the balance between certainty and effectiveness. How much is the estimate worth? And we all know that estimates, however much time was spent making them, are estimates and not promises.

Testing too much

Uncle Bob says he demands 100 % code coverage. What a terrible waste of time for most projects! If you do pacemakers or deal with nuclear devices, fine, but if you do a corporate web site? Hell no! Test logic, test states, but don't test that the text you enter in the CMS show up on the web page - you'd notice if it didn't - and no one would be harmed if it failed to!

Unit tests are great in so many ways, but you need to keep a good balance between safety and productivity. I'm a big fan of Unit Testing, don't get me wrong, it's just that I don't think you need to test everything. For example obvious things. Or tons of nuances. Find all paths through the code and make sure a test case cover them. Would ever a bug occur, add a test for it and learn from it what kind of test you were missing. Certainly a useful one.

Kent's tweet

Nor do I think you need to inject every single dependency. If you're in the top layer of the code, which will only be used in this specific project, it can be OK to call that static method in the core formatting library without making a wrapper that implements an interface that you can inject. And you do not need to inject a wrapper of the CultureInfo class to make sure that your test will still work if you'd suddenly would change culture (unless you have a geographically spread project of course).

Kent Beck sent out an excellent tweet a while ago; "First you learn the value of abstraction, then you learn the cost of abstraction, then you're ready to engineer". In the first 10 minutes it was retweeted by 91 and favorite by 24!

Conclusion

Agile is good in many ways, but agile doesn’t necessarily make you go faster. As a matter of fact it may very well slow you down if you’re not careful! By paying close attention to what you do and what value you get from it you will be able to be more efficient by being less agile. Or rather, being the right kind of agile. The right level is no fixed though, so I can't tell you that. I just urge you to be observant and find the right level for you.

The Best Agile Methodology

I've been developing software for 18 years now, and I've been doing it in different agile ways for the last six years. Starting off with just having a daily stand up and soon after adding more of Scrum until we did the whole package. Later I have tried both XP and Kanban too, but none of them appear to be the perfect fit for me in my daily work. A year ago I read Crystal Clear by Alistair Cockburn and thought that might be what I was looking for... it was just that I wasn't sure how to fit the roles with my organization, so I dropped Alistair an email to get some guidance.

He's response was right on for me. He basically said that Crystal Clear is dead and pointed me to an article of his called "The end of methodology". Instead of trying to buy in on a whole package you should have a toolbox of good practices that you use to build the best methodology for your team in your current project with your current stakeholders. There's no way any off the shelf product will be the best choice, and most agile methodologies aren't even comprehensive enough to be called a methodology - it's just a set of practices that you need to complement with other practices to run a project.

That being said there are a few practices I'll never leave out.

1. First is the story. Be it a user story or a light version or even a use case, but something that explains the functionality we desire in words that make it valuable for the stakeholders. And then make that story 100 % done before going to the next feature. This is where I've had most problems before going Agile. Sitting with a project that is almost done in every single end and then having months of work just tying it all up.
2. Second is priority. That's really the essence of agile: Do what's most important first. If you do it in sprints, if you have burndown charts, if you code test first, if you have continuous integration - its all "processes and tools" and something we should value less. For every new story you start, ask if that's the most important thing to do - if this would be the choice if only thing could be added to what we already have.
3. Third is the retrospective meeting. Without that we won't achieve the continuous improvement that makes the process ever better. This is where problems are brought to surface so we can do something about them and make our team gel. This is also, not to forget, where we can scratch each others backs about all the good things we do, making the team gel even more.
4. Fourth is the demo. This is the developers moment of pride, and the product owners opportunity to make sure the assignment was carried out properly. Two good things make a right! Even a small bug fix may be understood incorrectly by the developer and it's a lot better to find it now than after the release. It's also a thousand times more time efficient to show what's been done and get immediate feedback that can be addressed and discussed than to just send over the software and wait for an email with comments. And if that isn't enough you should also see it as an opportunity to read each others satisfaction level.
5. And finally the fifth, and that's the stand up meeting. If it is a low pace project it might be enough with a weekly stand up, but most projects benefit from a daily stand up. But remember to keep it brief! This is not the time or place to discuss designs or what you did last night. Also make sure everyone shows up, and that they show up on time. And remember, this meeting is for the production team to coordinate today's work and bring up immediate problems, not a status meeting to report progress to the project manager.

That will set you up with a good base that is useful in every project! Then you'll probably add a number of different other practicies, patterns and tools that you should have in your toolbox. You find them in XP, Scrum, Kanban, Crystal Clear and other methodologies and apply them as you see fit. You can also find treasures to add outside the methodologies, like the Pomodoro Technique for example. Then you'll have The Best Agile Methodology - tailor made for you and your context!

Best of luck, and I would love if you took some time to comment my post!

My Top 20 #FailedTechBands

holly woolard ‏@holly_woolard
A Flock of SQLs #FailedTechBands

Dan North ‏@tastapod
JSON Donovan #FailedTechBands

Obinna Egbule ‏@Oegbule
The Black IPs #FailedTechBands

herr beesch ‏@herrBeesch
Run DMZ #FailedTechBands

MrLarry ‏@MrLarry
Rick ASCII #FailedTechBands

Tania ‏@CongoKasongo
Bit.ly Spears "It's Bit.ly b****!" #FailedTechBands

Christine Erickson ‏@christerickson
Linkedin Park #FailedTechBands

Dan North ‏@tastapod
Emerson Lake and PalmPilot #failedtechbands

Vince Speelman ‏@VinSpee
A Dell #FailedTechBands

Sarah ‏@SarahFKessler
Johnny Cache #failedtechbands

Christine Erickson ‏@christerickson
The Google Dolls #FailedTechBands

Mikolas Hämäläinen ‏@mikolas
Red Hat Chili Peppers #FailedTechBands

MissGalore ‏@MissGalore
Samantha Firefox #FailedTechBands

eremy Rosenberg ‏@lexlimo
The Grateful Thread #FailedTechBands

Rich Oglesby ‏@Rich_Oglesby
U2ube #failedtechbands

duckysherwood ‏@duckysherwood
Tori Cmos #failedTechBands

1126 ‏@1126tw
@DerGuteMoritz What about the Dead Lock Chili Peppers?#FailedTechBands

Jørgen Vig Jensen ‏@Kronsj
Iggy Pop3 #FailedTechBands

Scott Kerr ‏@scott_kerr
Dropbox Murphys #FailedTechBands

Alf Jørgen Bråtane ‏@alfjorgen
Perl Jam #FailedTechBands

Jim Halfpenny ‏@jimhalfpenny
System Of A Downtime #FailedTechBands

bgstaal ‏@bgstaal
Nine inch thumbnails #FailedTechBands

Tommy Bryntse ‏@tommycode
System is shut Down #FailedTechBands

Marco Tabini ‏@mtabini
.mobi #FailedTechBands

Dan North ‏@tastapod
The Console Twins #FailedTechBands

Retrieve multiple YouTube videos by ID

It took me quite a while to figure out how to get a list of YouTube clips from their ID's using the C# API. You could get all sorts of feeds, and query in all thinkable ways... except the most obvious thing, the ID.

Finally I found this little link and it was all very easy! Using the Batch method stuck me long before I found this, but it was not very intuitive to use. So this example made all the difference!

http://gdata-sharp.sourcearchive.com/documentation/1.4.0.2/classGoogle_1_1YouTube_1_1YouTubeRequest_f90a3aeef96701490ea5447db0b1501f.html

VirtualPathUtility.GetDirectory doesn't like query parameters

One of my customers have had some trouble with CSS files being cached in the browser for too long after a new version is released, making the site look bad. To resolve that problem I added a query parameter with the current revision to the end of the url, i.e. mystyle.css?rev=123. It worked fine on my computer, but when I sent it to the staging server I got the error "HttpException (0x80004005): '/my/path/mystyle.css?rev=123' is not a valid virtual path.".

After some research it turns out that the home made CSS compressor HttpHandler I'm using calls VirtualPathUtility.GetDirectory to get the directory of the CSS file. On my machine there was a flag set in the registry that allows for the characters ':', '?', '*'. in VirtualPathUtility.GetDirectory(string), on the server it wasn't set.

The flag is called VerificationCompatibility and found in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET. If set to 1 it allows all characters but '\0' in the url, if not it will also disallow the three characters mentioned above. Since it has something to do with security my solution wasn't to set the flag on the production machine, but rather to send AbsolutePath to the method rather than PathAndQuery that was used first.

Since it took me a while to figure this out I thought it would be nice of me to let you know, and hopefully save you some time.

OutputCache on dynamically loaded UserControls

Yesterday I ran into an old bad bug in ASP.NET. When I put two controls with OutputCache set on the same page they rendered exactly the same - even tho they were different. The caching simply took one of them and showed on both places.

First I thought that is was an expected behaviour and that i needed to add some argument to the OutputCache directive, like VaryByControl or VaryByCustom, but I never managed to get that to work... I asked Google for help but couldn't really find any help. I did how ever find that the documentation said that adding two controls to the same page should automatically make them cache separately if not "Shared" was set to true in OutputCache.

That piece of information made me dig in another hole... Maybe it had to do with the fact that these controls were added dynamically, using Page.LoadControl and then added to a placeholder with ControlCollection.Add. I created a very simple test where Default.aspx had a placeholder in the body, and in OnInit I added two controls that only rendered a Guid - and had OutputCache. Adding them one by one didn't cause the bug, but when I added them in a loop it did! Wow!

Now that I knew what was the real problem i managed to get better answers from Google!

The first trace is as old as 2003, including hopes for it to be fixed in .NET 1.1 and .NET 2.0. The workaround suggested back then was to create a wrapping control without OutputCache that was the one that was dynamically loaded, and that had a static reference to the control you want to cache.

I wasn't really happy with that solution though and looked further. I found someone on StackOverflow that had the same problem and a reply leading to another solution, where you make sure that the stack is different on all calls. This would work around the problem, because the core of the problem appears to be that the Control you create with LoadControl is named by some hashkey from the callstack. That explains why it worked when I added the control on two separate lines in my test project!

I made a little test with a switch statement where every case did a LoadControl... but on separate rows. And it worked! Showing it to my collegaue Martin we started thinking about other solutions. First we did a recursion version, where we saved current recursion in Page.Items and made another recursion for every subsequent call. That worked too. Then we came to think of anonymous methods, but that didn't do the trick. Maybe they're named in the same way?

Instead we looked at DynamicMethod, which is the way we ended up using. At first I thought it would be bad for performance, but after implemention I did some profiling - and it was only half a millisecond per call extra! Here's the code if you'd like to solve the problem yourself:

public static class CacheFixer {
    private delegate Control LoadControlDelegate(TemplateControl page, string virtualPath);
    private static readonly Dictionary DelegateStore = new Dictionary();

    public static Control LoadControlWithCachingAllowed(TemplateControl page, string virtualPath, string key) {
        if (!DelegateStore.ContainsKey(key)) {
            DelegateStore[key] = CreateLoadControlDelegate();
        }
        return DelegateStore[key](page, virtualPath);
    }

    private static LoadControlDelegate CreateLoadControlDelegate() {
        var dynamicMethod = CreateDynamicMethod();
        return (LoadControlDelegate) dynamicMethod.CreateDelegate(typeof (LoadControlDelegate));
    }

    private static DynamicMethod CreateDynamicMethod() {
        var dynamicMethod = new DynamicMethod("", typeof(Control), new[] { typeof(TemplateControl), typeof(string) });
        var loadControlMethod = typeof (CacheFixer).GetMethod("LoadControlX");
        var ilGenerator = dynamicMethod.GetILGenerator();
        ilGenerator.Emit(OpCodes.Ldarg_0);
        ilGenerator.Emit(OpCodes.Ldarg_1);
        ilGenerator.Emit(OpCodes.Call, loadControlMethod);
        ilGenerator.Emit(OpCodes.Ret);
        return dynamicMethod;
    }

    public static Control LoadControlX(TemplateControl page, string virtualPath) {
        return page.LoadControl(virtualPath);
    }
}