Object Mentor Blog: Category Tim's Tepid Torrent

VIM as a Diff/Merge Tool

Tim Ottinger — Wed, 30 Apr 2008 21:24:00 -0500

Do you need a really quick, clean, powerful diff/merge tool that you can use in windows, mac, and various Unixen? As a happy and pround vim user, I have been enjoying vimdiff for a number of years.

        > gvim -d file1 file2
        > vimdiff file1 file2

(G)vimdiff is wonderful. You can move changes between two versions of a file with great ease (if you know vim) and amazing speed. It can handle files of any size and does a fine job of syncing up both versions.

Vim can do just about anything. I have learned that no matter how much I use and study Vim I still am just scratching the surface of what the editor can do. I find little tricks and configuration options, new keystrokes, and fun little bits of minutae. I think it’s all new. With my memory, sometimes I just think something is new, and have to go check my past notes to see if I already knew it.

Today I (re)learned about diffopt, which makes gvimdiff more wonderful.

    :set diffopt=filler

By default, diffopt is set to “filler”. Filler doesn’t do much, it just adds vertical spaces to keep the text of the left and right pane aligned. It’s a good setting, and I don’t much like working without it. But there are other settings which are overlooked, even by experts.

    : set diffopt+=iwhite

My second favorite is ‘iwhite’, which stands for “ignore whitespace”. It makes gvimdiff ignore leading, trailing, and embedded spaces so that simple acts like retabbing or deleting vertical spaces won’t obscure actual changes. It really helps you to merge changes in situations where you’ve adopted code that was built to entirely different standards.

    :set diffopt+=icase

iCase ignores capitalization, which is not as useful as the other options, since programmers are normally interested in case, but it may be handy when someone has corrected and reformatted SQL. I don’t expect to use it much.

    :set diffopt+=horizontal

The last interesting bit is the choice between vertical splits (the normal case) and horizontal splits (for very long lines on cinema-aspect screens). I hope to never need this option again, but I have needed to deal with absurdly long lines.

If you can use VIM at all, you will find vim’s diffmode to be a very powerful and useful tool. If you can’t use vim already, then maybe vimdiff will be the reason you will learn vim.

Magic Funnel, Part 3: Covey's Miracle

Tim Ottinger — Thu, 17 Apr 2008 22:15:00 -0500

We had a team of surrogate stakeholders in the room, and all the very most important stories (as far as we could tell) on a table in front of us.

“These are the most important?” Heads nodded. “Then,” I asked, “what makes them important?” Brows furrowed, and well-considered answers poured forth.

I collected their reasons for importance. Some were about keeping promises, some were about providing increased quality (uptime, correct function), and there were other kinds of importance. We wrote them on a 3×5 card. Ultimately, we realized that the company’s reputation is their source of importance. Users needed to know that the company will do the right things, and provide software that worked and kept them working. I thought it was a good criteria.

Next I explained that urgency is different from importance. There may be a hot demand for some cosmetic changes that don’t really impact the company’s reputation one way or another. There may be some changes that in the direction we want to go. These may be hot topics, but may not be truly important.

We could see a difference between the forces of urgency and the force of importance, but there is gray area as a sufficient pent-up demand for otherwise unimportant changes could in fact effect our reputation for service. Still, we could usually see a crisp-enough boundary for any given user story.

I then (re-)introduced the team to Stephen Covey’s four-quadrant system. It has two axes. The vertical is importance (important at the top, unimportant below). The horizontal is urgency with urgent things to the left and non-urgent on the right:

Q1 Important and Urgent	Q2 Important, not Urgent
Q3 Unimportant and Urgent	Q4 Neither important nor urgent

We took our set of cards and the team determined groupings. Was this a quadrant one activity? Quadrant 3? Quadrant 2? There were some real surprises, especially when we realized our “most important” bit of work was not urgent and should wait until quadrant one activities were finished.

Covey’s recommendation is that we tackle first those items that are both important and urgent. When we knock that stack down, we move on to those that are important but not urgent. If we have time left, those non-important-but-urgent items can “fly standby” in our sprint. We reordered the priorities in accordance with Covey’s recommendation.

We had no cards that were both unimportant and non-urgent.

At this point in time, we had what looked very much like a magic funnel. Ideas came in from everywhere, and the items that moved from the planning team to the developers came out in the “magically correct” order. We agreed that the selection and the ordering were the best we could do at the time.

Our next sprint planning meeting was remarkably efficient.

Building Magic Funnels, Part 2: Pragmatic Pedantry

Tim Ottinger — Thu, 17 Apr 2008 22:00:00 -0500

The middle of the funnel we started on needed work. While it is a simple idea that the single, most important thing in the funnel is the first to emerge from the bottom, it is a multi-flavored affair to try to manage for real.

My first strategy was to get the right people in a room and have them fight it out. I think that the prioritization process should be a lot like local government (maybe a school board?) in that people should argue and complain and push and eventually settle on compromises and deals. Ultimately, I believe that people who have a strong interest in the company can make the right decisions. At least they can be right enough for the next 5 days. When you have one-week iterations, the next chance to change the agenda is never far away.

This first strategy didn’t work out, and so we went to a backup plan. A C level manager said he knew what we should do, and so we scheduled an hour or so with him, myself, our priority manager (“funnel guy”) and the senior technical developer.

Our guys used the sticky post-it 3×5 cards and papered the CIO’s windows and whiteboards. They listed the various categories of work from the various stakeholders and pasted them up in priority order.

I asked my first pedantic question: “What is the single most important thing we can work on? If you had only one story that you knew for sure would be finished this week, which would it be?” That led to a nice discussion.

When they placed the card on the table, signifying that it was definitely in the build, I asked again if there was any one card anywhere else in the room more important. I asked if that was really the single most important one.

When the answer was “yes, absolutely” I was ready for my second pedantic question: “Now that this card is off the board, what is the single most important card left for us to do, if only two stories were guaranteed to be done.” Now the pedantry was fully exposed, but the idea had carried. The team collected all the most important stories and placed them in order on the table.

Now I was ready for my next round of pedantry.

Building Magic Funnels, Part 1

Tim Ottinger — Thu, 17 Apr 2008 21:38:00 -0500

The idea of a magic funnel, as you may remember, is that there is some kind of organizational structure where many ideas and proposals and issues go into the top. Through some magic, the item that emerges from the bottom of the funnel (to go to the development team) is the single most important thing they could work on next.

This is all just backlog management and prioritization, of course. But I think it can be simpler than I’ve seen it in the past, and that real people working without magic can approximate it.

Recently, I’ve been working to establish another magic funnel. One of the first things we did was to find the person who formerly handed out work to the developers and made him a single point of contact. In our case this is not the scrum master, but is another trusted line-manager type. He has been given a number of people to work with on the Customer side of the house and also works with technical people.

We have tried to establish story feeds from all the various stakeholders. Developers, operations people, technical writers, customers, sales people, marketing people, inside security consultants, and others have been feeding their ideas in to our point-of-contact man. This part of the funnel is working fairly well.

The next thing we have tried to do is to match the feed of work to the rate at which work can be done. This has created a fair amount of back-pressure on the stakeholders (which I believe to be healthy).

We have also worked on making the bottom of the funnel narrow, meaning that our guy doesn’t scatter new work to the team, but feeds it through the scrum master who protects the team’s productivity and keeps the work flow and tasks visible on the status board. He makes sure that the team is not expected to “absorb” changes, but that adding two points of work to the iteration results in two points of unstarted work coming off. This also creates a healthy back-pressure.

As a pun on “in-flight”, I named another area of the board “flying standby”. This is for stories that aren’t important enough to swap a story off the board (or for stories that are displaced by more important ones). If the developers finish more work than they expected, there are stories that can be picked up even though they’re not a scheduled part of the iteration. Stakeholders are told that there is no guarantee of these stories being picked up at all in this iteration, but there is some small chance that it could happen if the team discovers that it has overestimated some other stories.

The bottom of the funnel is working pretty well.

What’s missing is the “magic” bit.

The Magic Funnel

Tim Ottinger — Wed, 09 Apr 2008 13:18:00 -0500

At a planning meeting, the developers on the team offer up some amount of work they are able to do. This capacity is termed velocity. The customer part of the team then selects enough user stories to consume all of the offered capacity. This is simple negotiation, but that hardly tells the story. What is happening on the customer side is a magic funnel.

All kinds of stories come pouring into the customer group. Some are from actual users in the field who want improvements to existing features. Some suggest entirely new areas of functionality into which the program could expand to good effect. Some are from the operations group, generally describing ways to make the product more reliable, configurable, or manageable. Some are from the developers who realized that some small changes in code could provide large changed in capability or usability. Some are from external regulatory bodies, some are from quality control, but for the most part they’re all good ideas and many have urgency.

When the magic funnel is working, we pour all these stories into the top, and the thing that falls out the bottom is the single most important, urgent task we could possibly do. When it comes time for a new iteration, we fill in our allotted space with whatever comes out the end of the funnel. Whatever we start doing on Monday, it is the best thing we could possibly start doing. We know that it is true because it came out of the funnel that way.

Sadly, magic funnels are hard to come by. We approximate them with human beings in multi-disciplinary small groups. If we leave the funnel purely in the hands of operations, then operational stories are the first to come out. If we leave the funnel in the hands of marketing people, we tend to do things that are more speculative even though the product users don’t really need them right now. If it’s sales, then whatever will sell more product this month will be the first thing out of the funnel. Give control to customer support and we find ourselves constantly appeasing customers and not moving forward on larger issues. If we leave it to developers, it will tend to turn out those stories that improve the experience of developing the application. Of course the problem is about conflicting perspectives (all of them right!) with competing desires being processed in fixed time.

I’m not an expert in creating magic funnels, though it would be a very interesting direction to take at this point in my agile coaching career. I believe that the answer is ultimately to be found in people working together, well-informed people arguing and debating and even trading favors.

I think that part of the answer is Covey-style prioritizing to select the urgent and important (Q1), then the important items that aren’t urgent (Q2), and holding all unimportant work until there is nothing better to do.

Importance in this case relates to how much the change will affect the future of the product and the company. This is not just a matter of closing a contract or soothing an angry customer. It may be that the future is larger and brighter than our immediate concerns. Immediacy is “urgency”, but the leverage to create a more greater future is “importance”. I had to read Covey’s First Things First several times to really understand the difference, as may the kind reader. But I did manage to absorb the fact that until you know what you want your future to be, you don’t know what is important.

But imagine how great it will be when you have established your magic funnel, and the work being assigned to each iteration really is the most important and valuable thing that you could do next. Imagine what it would be like if you really were building the future you would like to live in.

Your Attitude is Affecting Other Departments

Tim Ottinger — Sat, 05 Apr 2008 21:17:00 -0500

The CIO looked into the eyes of his agile development staff last Friday. “Your attitude is affecting other departments” he said.

I’ve heard a lot of department-level speeches start this way, and in relatively small companies it is not unheard-of for a C-level manager to address attitudes of development teams.

The group has been working to stabilize and improve a product that was developed by a tiger team of outsider contractors and handed over to the in-house team post-facto. The developers had to overcome many obstacles to come up to speed on the code, to learn the new programming languages and tools, and to try to keep the feature set moving forward. None of them had been involved in the original design, but it was now their product, and its problems were their problems.

At the time of this meeting, I was one part of a coaching team which had introduced a great many changes. We were trying to help the organization to build a “whole team” mentality that encompassed documentation, security, systems administration, QA, customer representatives, and developers alike. We’d attacked the problem of matrix management. We reorganized the seating floorplan. I think at some point we’d been an inconvenience to just about everybody. The developers were in the middle of their second two-week iteration.

On this occasion, the team was in the midst of a recurring production difficulty, and had been gathered into the CIO’s office to work through a top-20 list of problems to solve.

“Your attitude has been affecting other departments”, he said. “And I want to thank you for it.”

Velocity is Just Capacity

Tim Ottinger — Wed, 02 Apr 2008 19:23:00 -0500

I don’t know why, but somehow velocity as a term really bothers me.

We always want higher velocity, and that’s a good thing. But with the term velocity, we think that we can create velocity with greater pressure (thrust?). Doesn’t it make sense that with a really hard push you can get greater velocity?

I guess I get hung up on words. Essentially, “velocity” is fine. It is the rate at which we’re “burning through” our stories. We use “yesterday’s weather” to set our level, and we are always seeking any sustainable way to increase our velocity. Why would we produce only 12 points when we could produce 20? As motivated developers, we aren’t yearning to accomplish less. But somehow this doesn’t tell the right story.

When our velocity is decreased, it always seems like a tragedy. We have all felt it when our 23-point iteration is followed by a 21-point iteration. We can justify it a lot of different ways, but it always feels tragic. Worse, hitting any kind of plateau seems frustrating.

You increase velocity by pushing harder. If you push harder, and the numbers come up, then you’ve successfully increased the velocity and that seems to be intuitively all there is to it.

I’ve reflected about this for a few years before I realized that I really like the term capacity much more. It is also accurately describing the rate at which we can truly complete stories. Since we use yesterday’s weather, we can even use the term “proven capacity” to describe that we’ve demonstrated the speed. But it also says some other things.

Intuitively it seems that we know that you don’t push your people beyond their capacity to get work done. Instead, you want to increase their capacity. When we talk about longer hours, it is still obvious that working people harder diminishes their capacity. Instead, we want to be working at capacity on the one hand, and increasing our capacity on the other.

Increasing the capacity sounds like the right problem. To improve capacity, you want fresh workers working hard, pairing and communicating, building and/or using better tools (or using tools more capably), using more powerful hardware, learning, and performing the kind of housekeeping that keeps the team productive. Life balance enters into it. Sufficient team size enters into it so you’re not so small you’re short-handed, yet not so plentiful you trip over each other.

I don’t suppose that anyone is going to rename standardized eXtreme Programming terms like “velocity” just because some guy from Indiana says to, but I wonder if maybe we can’t rephrase the problem in such a way that we are intuitively driven toward more reasonable solutions. Maybe we should explain that velocity is just capacity, and then work from that point onward.

Musing over Mutation

Tim Ottinger — Thu, 20 Mar 2008 22:25:00 -0500

I read a mailing list entry in which one fellow (who? I can’t remember!) asked another “Do you want to get better at what you’re doing, or find a better way to get the results you want?”

I’m a sucker for a good one-liner. That one had me thinking, and as I’ve had other conversations about innovation, I keep coming back to that line.

In our Agile practices, we work really hard for a week or two, and then hold a retrospective. The purpose of the retrospective is to find ways to work more effectively for the next two weeks. As we develop better software, we also evolve a better team. We use “tricks” such as tracking our velocity and recording blockages on our ‘waste snake’ to provide data for our decisions, and we use gut feel to evaluate those things that feel like collateral effort to us.

If the practice works, we will see incremental improvement in the team. We will develop ways of avoiding special variations, and we will learn to accept our normal variations. It will make us better at the way we do things now.

XP didn’t come from a series of incremental improvements to the waterfall technology. I wasn’t there when it happened, but it seems that they took on an change in axioms. They didn’t strengthen the contracts between groups, but pulled all the decision-makers onto the same team. They didn’t find more careful ways to examine the code they were changing, but rather decided to lean radically on volumes of tests. They didn’t build practices to improve their anticipatory design, they decided instead not to anticipate at all and simplify their design to allow future change. At the time, this was radical stuff.

I’m sure there have been many other less-successful process mutations, but there is no evolution without mutation.

The man behind the iPod, iPhone, and MacBookPro has had some less-successful product ideas, too. Some exciting high-concept products didn’t make it in the wild. But then some new ideas become category killers.

How do we learn to make the axiomatic changes that lead us to radically better ways to get what we want?

TDD on Three Index Cards

Tim Ottinger — Thu, 06 Mar 2008 19:39:00 -0600

I had the opportunity to talk to a fellow who missed part of a class on TDD. I told him that I could give him a 15-minute overview, and give him all the essentials of TDD on three index cards.

Yes, I know that volumes have been written about TDD and BDD, and that it’s a large topic with many many branches of application, but I didn’t have time for that. I had time for three index cards. I figure that an index card is a token, and it represents a conversation, and that one can always dig deeper later.

They looked more or less like this:

Card 1: Uncle Bob’s Three Laws (Object Mentor)

Write no production code except to pass a failing test.
Write only enough of a test to demonstrate a failure.
Write only enough production code to pass the test.

Card 2: FIRST Principles (Brett and Tim at Object Mentor)

Fast: Mind-numbingly fast, as in hundreds or thousands per second.
Isolated: The test isolates a fault clearly.
Repeatable: I can run it repeatedly and it will pass or fail the same way each time.
Self-verifying: The test is unambiguously pass-fail.
Timely: Produced in lockstep with tiny code changes.

Card 3: Flow (using the famous three-node circle diagram) – origin unknown.

Red: test fails
Green: test passes
Refactor: clean code and tests

Sure there is plenty more, but I didn’t know how I could provide significantly less. As is, I’m pretty happy with the exercise. Now I am wondering if I couldn’t produce most of the important information I wish to convey as a series of index cards. Would that be cool or what?

Clues For Reading New Code

Tim Ottinger — Wed, 05 Mar 2008 19:27:00 -0600

Okay, somebody just handed you a new chunk of code to work on. Your first though on opening the file is “Why, dear God, why?”. How do you get a handle on this masterpiece of clever programming? Let’s look for a few clues.

Where to start?

Documents

Why not look at the documentation? <laughs> Just kidding. You know there’s no documentation, and it’s probably not useful.

People and Interactions

You need a partner who is familiar with the code. Having a guide is better than having a map. Not only will you know where to go, but you’ll know ’’how’’ to go and where not to step. Other people are a wonderful resource. Just don’t settle for someone doing the work in front of you… the goal is to learn as much as to do.

No, really, documents!

OTOH, if you have somethng in the way of a summary or architectural overview, that might help. I’d read the abstract and look at the pictures. We’re after big-picture, so read it only, don’t trust it. Documents are seldom accurate, and seldom for long. It might be good preparation for your partner’s visit.

Use The Tests, Luke

Do you have tests? If you are in a test-driven shop, looking at unit tests is a good idea. If they’re written well, they are specifications (by example) of the system. If they’re written poorly, you know where to start your work at least. Your partner could do a great service by helping clean up the code instead of explaining it.

No tests? Uh, oh. No tests and no doc? Now you’re in trouble.

Scan the File Space

Are the classes well-named? Do you know where to begin working? If you can find code by looking at file names then you’ve got the handle you need. Your partner can help improve file naming if you find the code you’ve been handed is in ill-named files.

So you’re in a file, you have a mission. Either you have tests, or you’re ready to write some.

Sense Of Smell

What if the code is still not obvious? You have one more resource before you have to print out the listing and break out the markers to reverse-engineer your way out of code hell. You have your nose and your refactoring editor. You can use refactorings such as ‘rename’, ‘extract method’, and ‘introduce variable’ to clarify an existing method. You can spot duplication and eliminate it. Maybe you can bring it to a point of clarity, and then you will know

Deep Code Spelunking

It’s time to pour a really big cup of tea, get some food, take a couple of preemptive aspirin, and make sure your printer has paper.

Sometimes, you will have to reverse-engineer some of the code you were handed. Your only hope may be to rebuild it from the inside-out.

You might want to either copy the code to a scratch space to do this. Maybe you want to branch the project in version control. You want to be free to dig in an learn this the hard way, and make it better as you emerge from the depths.

You still can/must use refactoring tricks to capture the knowledge you gain, but you may need to take the code off line. And look for new partners.

Turn Back The Dial

Tim Ottinger — Wed, 05 Mar 2008 19:16:00 -0600

The coolest thing just happened! I broke the glass cover off of my watch. At first I thought it was awful, but then I realized that I could turn the hands.

Imagine my joy as I realized that I could make it 11:30 again, and go enjoy another lunch. Meeting at 3:30? No problem, just turn the hour hand up to 6:00 and go home! I can sleep as long as I want as long as I turn it back to 8:00 when I get to the office. All my work estimates are now “five minutes”, and I complete them every time.

My coworkers have no idea the awesome power that I’ve gained with this one happy accident. They ask “what time is it?” and I say “what time would you like it to be.”

Of course the above is a total fabrication. Pretending it’s 6:00pm when it’s 8:00am isn’t going to do anybody any good at all, and is likely to make a mess of things for me.

But people still try to mandate velocity.

Shoveling Code

Tim Ottinger — Thu, 06 Dec 2007 12:56:00 -0600

I was ill when the first snow fell last weekend, and didn’t get out and scrape the drive and sidewalks. Sadly, neither did I bundle up my kids and send them out with shovels. As a result, the snow melted and refroze. When I left home on sunday morning, clearing the snow was frustrated by the presence of hard, packed ice under the snow.

Now I have to be very careful when moving in the drive or sidewalk because of the black ice in some places and thick lumpy stuff in others. I could choose to leave it that way and just walk carefully and try to avoid the area, but I really don’t want it to be like this all winter.

Instead, I bundled up and spent several painful hours trying to clear not only the six to eight inches of new snow, but the ice under it. Clearing the fresh stuff is tough because the rough and lumpy ice is under it. The ice snags the shovel and hurts my elbows and shoulders. I use the metal edge of the shovel to plane off the ice so that it’s not quite so difficult, and the work gets a little easier.

Smooth, planed ice is not a solution. It is actually going to be more slippery and dangerous when the sun hits it and it melts a little. It will then refreeze to an even smoother and slicker sheen. I can’t ignore the problem, and I can’t smooth it over and leave it at that.

I invested in some complicated salt product to lower the melting point of the ice so I can scrape it away. It wasn’t enough, but now there are some safe areas in my driveway. I continue to scrape it and hope that I can get it all squared away so that my wife and kids have safe passage, though it is tedious and unpleasant work.

Yes, it’s my fault. I should have bundled up my sick self or my children and taken care of this when the first snow fell, back when it was easy.

Then it dawned on me that my driveway is a lot like source code. If I always take care of it when it’s relatively easy and not too polluted, it will remain easier to deal with in the longer term. The more unsafe and ugly it is, the more important it is to clean it up. It won’t do to smooth it out a little or to leave it be. It needs to be safe for me and my team.

And I guess that’s my parable for today.

Naming: A Word To The Wise

Tim Ottinger — Tue, 27 Nov 2007 20:50:00 -0600

Please pronounce your variable names as words, even if they are merely abbreviations. The phonemes indicated by your spelling may correspond to an existing word with an entrenched meaning. It may be one that you do not intend to communicate.

On Being Stupid

Tim Ottinger — Mon, 10 Sep 2007 10:16:00 -0500

This was posted originally to a mailing list, but is reproduced here essentially unchanged by request of a friend.

I frequently see code (written by others) that is completely double-spaced, heavily commented, loaded with many abbreviated or meaningless variable names, and hundreds of lines long. In order to read the function and understand what it’s doing, poor Tim must wear out a mouse, skip comments, and track the variables on paper. A “smarter” programmer could just load it into his head, I suppose, but not the simpleton who writes this email.

I’m not smart enough to just read it from top to bottom and understand it. Sadly, when I read through and understand what in the heck the thxIniFvt variable is doing, I will forget it by the time I figure out the purpose(es) of pszVbt. I can spend all day, or even a few days to figure out a method, and that’s an admission of feeble-mindedness to be sure. I guess I’m not up to the level of some of the rapid hackers. That’s a limitation I face most days.

I find that I can sometimes understand a method like that only if I just delete all the blank lines and comments first, then reformat to break lines, then inline all methods with seven or more parameters, and then start renaming variables, extracting explanatory variables, and extracting explanatory methods. I may have to break the method into classes even. I guess I’m not one of the smart kids.

I used to be one of the smart kids. I once built a module so complex and fragile that nobody but me could figure out what to do with it. It was all tables and indicators, and stunningly clever. I am so ashamed that I wrote it. It was such a mistake that they eventually disabled it rather than field it in such a state. That was years ago, but so memorable to me. Other programmers said it was like the inside of a swiss watch, all delicate and perfectly balanced, and scary to mess with unless you first knew exactly what each part was doing, and why.

I would like to be faster than I am both mentally and in the sense of quickly producing code. I’d like to be a little less intimidated at the start of a project. .But I would not want those things if it meant building crap that people who are not appreciably more talented than myself would trip over every day. Instead, I sometimes wish I could teach the really fast, smart kids how to dumb down the code for the rest of us morons to read.

The funny thing is that dumbing code to my level doesn’t make it harder for the smart kids to use it, and sometimes allows a compiler to do a better job with it. I guess stupid isn’t so stupid after all.

Not A Task, But An Approach

Tim Ottinger — Thu, 02 Aug 2007 22:14:00 -0500

Transitions are tough. It seems that lately I’ve been getting a lot of contact from frustrated people who don’t really have a good handle on the “drive” part of Test Driven Development. A question heard frequently is: “I’ve almost completed the coding, can you help me write the TDD?”

It seems like Test Driven Development is taken backward, that the developers are driven to write tests. The practitioner winces, realizing that he again faces The Great Misunderstanding of TDD.

TDD stands for Test-Driven Development, which is not as clear as TFD (Test-First Development). If the consultant would strive to always say the word “first” in association with testing, most people would more surely grasp the idea. In fact, I’ve begun an experiment in which I will not say the word “test” without the word “first” in close approximation. I’ll let you know how that works out for me.

If the tests are providing nothing more than reassurance on the tail end of a coding phase, then the tests aren’t driving the development. They are like riders instead of drivers. Test-Ridden Development (TRD)[1] would be a better term for such a plan. Even though it is better to have those tail-end tests than to have no automated testing, it misses the point and could not be reasonably be called TDD.

An old mantra for TDD and BDD is “it’s not about testing”. The term BDD was invented largely to get the idea of “testing” out of the way. People tend to associate “test” as a release-preparation activity rather than an active approach to programming. BDD alleviates some of that cognitive dissonance.

In TDD, tests come first. Each unit test is written as it is needed by the programmer. Tests are just-in-time and are active in shaping the code. Acceptance Tests likewise tend to precede programming by some short span of time. [2]

Through months of repetition I have developed the mantra:

TDD isn’t a task. It is not something you do. It is an approach. It is how you write your programs.

I wonder if we shouldn’t resurrect the term Test-First Programming or Test-First Development for simple evocative power. Admittedly there are some who would see that as a phase ordering, but maybe enough people would get the right idea.

Brett Schuchert(with some trivial aid from your humble blogger) has worked up an acronym to help socialize the basic concepts which are somehow being lost in translation to the corporate workplace.

The teaser: Fast, Isolated, Repeatable, Self-validating, and Timely.

As a reader of this blog, you are probably very familiar with all of the terminology and concepts behind TDD. I beg of you, socialize the idea that testing comes first and drives the shape of the code. If we can just get this one simple idea spread into programming dens across our small spheres of influence, then we will have won a very great victory over Test-Ridden Development.

“And there was much rejoicing.”

1 Jeff Langr will refer to this TRD concept as “Test-After-Development”, which he follows with a chuckle and a twinkle, “which is a TAD too late.”

2 Of course, one still needs QC testing as well, however TDD is about driving development, not testing its quality post-facto.

Tuple Madness and STL in C++

Tim Ottinger — Thu, 19 Jul 2007 22:21:00 -0500

I have already complained about Tuple Madness with reference to Python and Ruby, and UncleBob has seen the same problem and noted it on the old blog. I have come to realize that the same problems appear in modern C++ code.

I have always looked at the STL as a way of implementing a class, but never as a replacement for “real” classes with named member values. In some ways, I think that templates have made our C++ code worse.

For instance, I need a two-member structure. Say I am going to store Name and Address. Now, those with C background would immediately think along these lines:

    struct name_and_address { char name[34]; struct address address; };
    struct name_and_address get_name_and_address(void);
    // blah blah blah

    struct name_and_address x = get_name_and_address();
    printf("%s", x.name );
    printf("%s", x.address.line1);
    // ... and so on...
    printf("%05d", x.address.zip);

Got to love those old C programmers. There is a nice, primitive struct, maybe with a nice, primitive struct embedded in it. Look at the clear names of those variables. It is even pretty efficient to copy these things. With a Plain Old Data Object, the compiler knows what you’re doing and can do block copies instead of memberwise copies. That can be very efficient, enough that you can often pass structures around by value instead of by reference. That’s kinda cool. That same efficiency also applies to plain old data classes in C++, but a lot of people don’t know it.

Sadly, a modern C++ developer does what? Yeah, he returns a pair. This is annoying because when I call my function, I have to know that the name is now called “first” and the address is called “second”. But that is only for this pair-returning function. In another pair-returning function, “first” means birthday and “second” is account number. Of course the prototype is not always in view, so I have to have more tiles in my window to check my work as I code.

Pair is named well enough for an arbitrary pair of primitive values, but frankly it makes no sense for x,y coordinates or names and address, or for the rolls of two twenty-sided dice. Wait. For the dice, it works just fine. After all, what would you call them? Dice1 and Dice2? But I digress. The point is that for most two-member structures, a two-member structure with named members is probably a better choice.

Now pair is not alone (oh look, a pun). Even when we are dealing with collections, the interface to a vector, map, or set is unlikely to be the right set of names and semantics for our problem domain objects. Maybe our property should be listed, and not push_back()-ed. Maybe we care about a policy, that:

        policy.covers(theCar)

Maybe we’re not so interested in whether list x will:

        x.find(z.first) != x.end()

The fascination with exposing STL containers (combined with unfortunate naming, as given here for effect) is hurting readability. It also is a very unfortunate form of primitive obsession and a violation of the Dependency Inversion Principle.

The people who work on your code with you should not have to reverse engineer your work in order to add new features or correct bugs. The code should be written so clearly that mistaken assumptions will be more obvious. They should not have to chase down typedefs and keep notes on paper to keep track of your structures. They should especially not have to look for data in obliquely named pairs of pairs (x.second.first means “price”? Or was that x.first.second? Or…).

If you write your code with a pair programming partner, you should get plenty of feedback about the readability of your code. Even if you do not work in pairs, you should read your tests when you get back from lunch or a meeting, and you should see if primitive containers, primitive member functions, and template syntax are obscuring their purpose. If your tests don’t speak clearly, then none of the other users of your class can speak clearly. Get the syntax out of their way by giving them reasonable wrapper functions.

You might think that wrapping your containers will cost you performance by adding dispatch here and there, but you might be surprised to find out that not wrapping it may cost you far more by concealing bugs and obscuring functionality than it ever save you in performance. A better policy is to keep your code obvious and simple, and try a little profiling and optimization where it really matters.

Of course, if one was worried about the cost of one dispatch for a named method v. an exposed container, there is always the option to build an inline member function in the header.

I’m mindful that this is not the time for me to rant about C++ programmers who never profile, and who manage performance by rule of thumb and old wives’ tales. There are more of them than us, my dear reader, and I fear their wrath. Maybe some day later, when I’m feeling brave.

My roots run pretty deep into C++, and I appreciate how hard it is to produce a system. I just want the guys who are now running with the ball right now to realize that it is far more valuable to be clear and obvious than they appreciate, and it is far less expensive to do so than they fear.

I guess the points are:

Don’t be afraid to write small classes and structs.
Don’t be afraid to wrap your containers.
Don’t be afraid to use abstraction.
Everything does not have to be written in templates.
Be afraid to write test code that doesn’t tell a story.
Be afraid to give too much access to your data by exposing containers.

Adjust your values accordingly.

Testing Will Challenge Your Conventions

Tim Ottinger — Tue, 17 Jul 2007 22:28:00 -0500

If you are doing test-first development, you are likely to find your old coding conventions are no longer valid. There are a few changes you will need to make to your coding standards and practices.

1. Interfaces suddenly seem like a really good idea when you have to start introducing test doubles of various flavors (mocks, etc). You will find yourself creating interfaces in places where you probably would not have created them before. You’ve always known that you shouldn’t depend or derive from concrete classes, and now you are feeling the pain of non-trivial concrete classes. You are forced, more or less, to comply with the Dependency Inversion Principle. Abstraction becomes a way of life.

2. Singletons and static methods no longer seem like a great way to do work (esp in C++) because you can’t easily isolate a module that makes use of them. The exception is when a static method is totally self-contained and testable, and does not use resources like a database, file system, etc. Only the very trivial can be static. Test doubles matter that much. Suddenly substitutability is a primary measure of code goodness.

3. Private makes less sense than it used to. You can’t test anything that’s private. You need to have ways to sense that your tests are working as intended, and you have to be able to test any method that is interesting. That means less private and more accessors. Get used to living in a more public world. If you need to hide something from users, don’t include it in the interface or abstract base class. “Implements”/”public inheritance” is the new “private”.

4. You need to be able to pass a class everything it might need at construction time, so that you can pack it a stubbed logger and a faked database access module. Otherwise, you will not have the isolation you need to write good unit tests. Calling a concrete class constructor inside the body of a method now makes you cringe, because you realize you’ve missed an opportunity for abstraction, and limited the isolation of your method. The fat constructor argument list is a concession to the need for isolation and testing with mocks (and fakes, and stubs).

5. Smaller methods are the norm. It is hard to test a two-hundred line method. It’s far harder than testing a many smaller, equivalent methods. Whatever is easier to test is easier to write. Large is bad. This has always been so, since optimizers love simple functions, but now it’s real to you as well.

6. You hate inheriting code without tests, because you don’t want to reverse-engineer what the other programmer was thinking. In fact, you find yourself looking at tests before you look at code, because you know the tests are better than comments. Testing becomes a kind of “code clarity” mechanism. The profound bit is that code is more clear if it is more testable, even if you have to “pollute” the class to make it testable by moving variables and methods out of the private space or by creating interfaces, or by using fat constructor argument lists. Testability is the new legibility.

7. Hard-to-use class interfaces are now hard for you to use, not just hard for other people. You have to write the tests, so you have to use the class. Usability determines how much you will like your own code, not cleverness.

8. Performance management by old wives tales is dead. You will build for testability, and then use measurement tools to improve performance. This is what the wizened greybeards have been telling us for a lot of years, and with TDD you start listening. You stop avoiding trivial issues like copying PODS and virtual dispatches. You pay attention to more important issues. With optimization, YAGNI applies until it doesn’t, and when it doesn’t you need to know why and where.

9. You care deeply how long it takes to run the tests because you need to be able to run all of the tests all the time, after only a few lines of code are written, and after each refactoring. If it’s taking more than 20 or 30 seconds to run all your unit tests, you start looking for ways to speed it up. You must maintain test performance, because it is key to productivity.

10. Dependency hurts. You can’t afford ‘god classes’ and ‘global hubs’. If test setup becomes a chore, you immediately start reaching for the axe. You have to keep your modules isolated better than ever before. Of course, you should have done this all along, but you never felt it this keenly.

11. “Clever” is dead. Clever is hard to refactor. Clever is hard to isolate, hard to internalize, hard to phrase in tests. One point of “obvious” is worth two hundred points of “clever”.

12. Your IDE is good to the extent that it allows you to do quick write/build/test cycles. You need to do several per minute.

For the most part, TDD forces you to start doing the things you always should have done, though it signals a change in values that should be reflected in your coding standards.

Maybe you should get a highlighter and mark each section of your coding standard or architectural document that encourages any practice that in turn discourages testability. The world has changed, values have shifted, and your style guide needs to change if it is to remain relevant.

What I Don't Like About Python

Tim Ottinger — Tue, 17 Jul 2007 21:41:00 -0500

I love python.

I love the simplicity, the nice data types, the strong dynamic typing, the significant indentation, the runtime flexibility, the list comps, the generators, they way that I can get work done in Python. I love the built-in help. I dig the Tao of Python.

There are still a few bits that seem artificial and clumsy. I was told that you don’t really know a language if there are not five things you hate about it. I have more than five, but these will do for now:

Dir is not a very good name for the method that dumps the contents of an object. I don’t like the ruby alternative, where methods and members are separate, but I just don’t like the naming here. A change would be possible. It would have to be grandfathered in, but it could happen.

double-underscores annoy me. I would like to see some other way of denoting operators and other magic functions. I suppose one could grandfather-in a decorator-based trick for defining operators. It would be a real compatibility breaker, but I really feel that __getitem__ is probably not the best naming our guys could come up with.

__name__ has always felt like a hack to me. Try explaining it to a novice programmer, or worse to someone who programs in perl or ruby or java for a living. Comparing the magic variable __name__ to the magic constant ”__main__” feels doubly so. I would rather have a convention such as naming a method “main” or maybe decorating with @script_main now that we have decorators.

Blocks give me of ruby envy or smalltalk envy. I like local functions. Love them. I tolerate lambda. But I really, really would like to see a more rubyesque iterator setup where we pass a callable to the list, and the callable can be defined free-form inline. Python doesn’t really do that and so it’s less of a language lab than I might like.

Properties are unattractive, partly because of blocks being absent. I don’t really want to define a named parameter (with double-underscores, most likely) and then two named functions, and THEN declare a property. That seems like so much work for such a simple situation. It is something I will only do if all other methods fail me, or if all other methods are overriding __setattr__ and __getattr__.

Lack of recognition just kills me. This is a wonderful little language with great libraries and tremendous capability. Google, Eve, Nasa, a great many scientific efforts, a bunch of web sites, and a lot of Linux installers and configuration tools use python. A number of very nice distributed version control tools are written in it. But still people seem to go deaf if anyone mentions python, as if we’d mentioned JCL or something. I don’t get that.

There is a very ugly mutual-inclusion design bug bug. I spent time on it once and was very unhappy.

Self loathing takes over in my functions. I wish that I could refer to member variables without saying “self.” first. Testing would be easier if I could just import all my fixture variables into my local namespace—or better if it were done for me automagically. Typing “self.” doesn’t kill me, but it doesn’t help me. Yes, I have an abbreviation in my editor, but it still bugs me.

Don’t put me down as a complainer. I really, really love working in python. But There’s a lot more love than hate here.

PS: Is it uncommonly sensible that “not” is spelled “not”, “and” is spelled “and” and “or” is spelled “or”? I think so. && I’m ! just kidding || something.

Dependency Broadcaster

Tim Ottinger — Mon, 16 Jul 2007 20:34:00 -0500

I’m not sure if there is already a code smell name for this situation. The idea is rather like “Large Class” or “God Class” but isn’t really related to behaviour. It’s just a matter of dependency.

Michael Feathers refers to “horrible include dependencies”, and that’s the right idea.

So what if you have a class file that includes (or forward-declares) a few hundred other classes, and that class is used by almost every other class in the system?

This is perhaps what comes from writing a very class-rich architectural layer as a single class.

Mainly the idea is that the class takes a lot of ugly dependencies and spreads them evenly over the application. I would call that “broadcasting” although the correct agrarian term is “manure spreader.” I will stick with the less evocative “dependency broadcaster.”

Use VIM to do TUT Unit Testing

Tim Ottinger — Wed, 11 Jul 2007 09:47:00 -0500

I’m not crazy about TUT, but a customer is using it on a project and wanted to get a little assist from VIM. I don’t blame him. It’s a pain to keep track of your test number and there’s always more testing that we’d like in our test frameworks.

So I scribbled up a little VIM script to handle some of the light housekeeping. It’s not a wonderful script, and I’m betting the readers can give me some pointers (I’ve only written a few other vimscripts ever). I’m sure that I could do better if I reflected and refactored, but I haven’t yet.

Maybe you know a good way to unit test vim scripts?.

My script hijacks your F5 key. You can change that easily enough. Maybe I should have mapped it to \nt for “new test” or something. Anyway, enjoy and comment please:

function! NewTutTest() let s:testNumber = 0 let s:newNumber = 0 " Seek a higher number let s:list = getline("1","$") for s:line in s:list if s:line =~ 'test<' let s:newNumber = 0 + matchstr(s:line, '\d\+') if s:newNumber > s:testNumber let s:testNumber = s:newNumber endif endif endfor "Increment (tests are 1..n) let s:testNumber = s:testNumber + 1 "Output the test values let s:line = line(".") let s:indent = repeat(" ", &shiftwidth) let result = append(s:line, s:indent . "template<>") let s:line = s:line + 1 let result = append(s:line, s:indent . "template<>") let s:line = s:line + 1 let result = append(s:line, s:indent . "void object::test<" . s:testNumber . ">()") let s:line = s:line + 1 let result = append(s:line, s:indent . "{" ) let s:line = s:line + 1 let result = append(s:line, repeat(s:indent,2) ) let s:line = s:line + 1 let result = append(s:line, s:indent . "}" ) call cursor(s:line, (&shiftwidth * 2)) endfunction " Paste a version of this line into .vimrc to assign a keystroke (in this case F5) " Comment-out this version if you do " -------------------------------- map <F5> :call NewTutTest()<CR>Aset_test_name(" " Some helpful macros for command mode. Press \ten and it opens a new line " and types ensure(" so you can fill in the string and the parameters. " Overall, not too shabby. map \tn oensure(" map \te oensure_equals(" map \td oensure_distance(" map \tf ofail(" " Abbreviations to help you in insert mode. Type the " two-letter name, followed by a quote. ab tn ensure( ab te ensure_equals( ab td ensure_distance( ab tf fail(

Revisit: The common subgroups

Tim Ottinger — Tue, 03 Jul 2007 10:43:00 -0500

In cleaning up the code, I simplified the algorithm a very little and improved performance considerably. Amazing how that works, how simpler equals faster for so much code. Adding simple data structures, local explanatory functions, and the like often make code much faster.

What I’m hoping is that I will use this in a few different and useful ways.

The first way is to look for interfaces where concrete classes are being used from many other classes. You need to add an interface, but don’t know who needs which part. The goal is to figure out a relatively small number of interfaces that satisfy a number of clients in a module.

The second use would be to look for common clumps of parameters when I’m working in a large code base where the average number of arguments per function call does not remotely approach one. I suspect that there are clumps of similarly-named variables being passed around, and that these are likely “missed classes”. Sometimes these are obvious, but it would be good to see them in a nice list spit out from a nice tool.

So this is a hopeful start on a series of useful tools.

Code follows.

import shelve import sys def find_groups(input): """ Exhaustively searches for grouping of items in a map, such that an input map like this: "first":[1, 2, 3, 4], "second":[1,2,3,5,6], "third":[1,2,5,6] will result in: [1,2,3]: ["first","second"] [1,2]: ["first","second","third"] [5,6]: ["second","third"] Note that the return value dict is a mapping of frozensets to sets, not lists to lists as given above. Also, being a dict, the results are effectively unordered. """ def tupleize(data): "Convert a set or frozenset or list to a tuple with predictable order" return tuple(sorted(set(data))) def append_values(map, key, *values): key=tupleize(key) old_value = map.get(key,[]) new_value = list(old_value) + list(values) new_value = tupleize(new_value) map[key] = new_value return key, new_value result = {} previously_seen = {} for input_identity, signatures in input.iteritems(): input_signatures = set(signatures) for signature_seen, identities_seen in previously_seen.iteritems(): common_signatures = set(signature_seen).intersection(input_signatures) if len(common_signatures) > 1: known_users = list(identities_seen) + [input_identity] append_values(result, common_signatures, *known_users) append_values(previously_seen, signatures, input_identity) return filter(result) def filter(subsets): filtered = {} for key,value in subsets.iteritems(): if (len(key) > 1) and (len(value) > 1): filtered[key] = set(value) return filtered def display_groupings(groupings): "Silly helper function to print groupings" keys = sorted(groupings.keys(), cmp=lambda x,y: cmp(len(x),len(y))) for key in keys: print "\n","-"*40 for item in key: print item for item in sorted(groupings[key]): print " ",item print

Python Subgroup Detection and Optimization

Tim Ottinger — Wed, 27 Jun 2007 23:23:00 -0500

I had a moderately interesting customer problem to work on. I got acquainted with a bit of legacy code that is seriously in need of some interface segregation. It’s an entirely concrete class and used from all over the code base. The question is how to segregate, and that depends on what methods are called from which programs. We ran ‘nm’ to extract the link table from our object files, saving me the trouble of parsing C++ (a scary thought) All that remained was for me to compare the method prototypes used by the object files and find the common sets.

Lacking better ideas, I decided to do this exhaustively in a brute-force kind of way. It is only a few hundred files, so it shouldn’t take too long. I was very wrong. It took a long time and eventually failed.

I had TDD-ed the code, so I had tests of correctness, and I relied on these as I added optimizations, but of course the performance problem occurred only under a real load.

I could have run a profiler on it (and probably should have) but instead I simply monitored my computer. I quickly saw that my time was going into memory allocation, which is also the reason it died after many minutes When I have python performance problems, this is usually the reason, and almost never “interpreter drag”.

My friend Norbert (a gentleman of many programming languages, including ruby) suggested that I wasn’t interning my strings, and of course I was not. I switched to interning strings, and noticed a little improvement which meant the program ran longer before failing from memory problems. Well, the tests still passed, so I knew at least that the logic was still good even if the algorithm was primitive and Occam was spinning in his grave.

Next I realized that I am dealing with a lot of small groups of strings, and in a leap of optimism/faith/stupidity I decided to intern groups of strings. That helped very little, but it did help since my program was now running for the better part of an hour (!!!) and then failing. “More of same” without measurement is hardly a good recipe for optimization.

This is when I realized that I shouldn’t be storing sets or frozensets, which are pretty heavyweight data structures. I had chosen them because I was really working with set intersections, but hadn’t counted the in-memory storage cost. I converted the data structure used by the algorithm and added a local function to make tuples out of the sorted sets.

I was very glad to have my tests to catch me when I had some typing mistake or sloppy conversion. My tests had to be edited, but the changes there were very lightweight, and caused me to abstract out some data comparisons that were (admittedly) repeated. It was all good.

When I ran the full program it completed so quickly that I was sure I’d broken it. It was running with sub-second time, including gathering data from various text files (nm output files). I did a few spot-checks, and determined that it was indeed doing the right thing (as far as I know).

The data is moderately interesting, and I will be able to pick out some useful interfaces. Better yet, I have a program that can pick out all the uses of my big, fat class and recommend interfaces to me. This is all good.
Lessons learned:

The tests gave me peace of mind as I worked. I would so hate to have done this “naked”.

Python’s speed is fine (even startling) if you aren’t doing something wasteful and heavy-handed like storing hundreds or thousands of non-interned strings and heavyweight data structures

I could have been done sooner if I’d measured with hotshot instead of guessing. This is very clear to me, and I won’t think I’m too clever or my problem is too simple to do this ever again.

Keep some ruby friends on hand. They come in handy.

I didn’t really need a cooler algorithm. Brute force is sometimes enough.

I want to build a new wrapper for this code to compare parameter lists, to help find unrecognized classes in these same programs. It shouldn’t be too hard, but it will be a larger data set so I will probably need some more optimization or a cooler algorithm later. I think I see some waste in it now, but I guess I’ll have to fix that after blogging.

Of course, I’m not done learning. I am sure there are a lot of ways to improve the core code. I also believe in other people, that I should make it available to criticism and suggestions.

Here it is:

Tests
(Which, embarassingly, could use more refactoring)
import unittest import clumps def keySet(someMap): return set(someMap.keys()) class ClumpFinding(unittest.TestCase): def testNoGroupsForSingleItem(self): input = { "OneGroup": [1,2] } actual = clumps.find_groups(input) self.assertEquals({}, actual) def testNoOverlapMeansNoGroups(self): input = { "first": [1,3], "second": [2,4] } actual = clumps.find_groups(input) self.assertEquals({}, actual) def testIgnoresSingleMatches(self): input = { "first": [1,3], "second": [1,4] } actual = clumps.find_groups(input) self.assertEquals({}, actual) def testTwoInterfaceMatches(self):because of group = (1,2) names = ("first","second") input = dict( [(name,group) for name in names]) actual = clumps.find_groups(input) self.assertEquals(1, len(actual)) [key] = actual.keys() self.contentMatch(group, key) self.contentMatch(input.keys(), actual[key]) def contentMatch(self, left,right): left,right = map( frozenset, [left,right]) self.assertEquals(left,right) def testFindsThreeGroupsMatchingExactly(self): group = [1,3,8] names = "one","two","three" input = dict( [(name,group) for name in names ] ) actual = clumps.find_groups(input) self.assertEquals(1, len(actual)) [clump_found] = actual.keys() self.contentMatch(group, clump_found) self.contentMatch(names, actual[clump_found]) def testFindsPartialMatchInThreeGroups(self): input = { "a":[1,2,3,4,5], "b":[1,4,5,6,8], "c":[0,1,4,5] } target_group = frozenset([1,4,5]) names = input.keys() actual = clumps.find_groups(input) [key] = actual.keys() self.contentMatch(target_group, key) self.contentMatch(names, actual[key]) def testFindsMultipleMatches(self): input = { "a":[1,2,3,4,5], "b":[1,4,5,6,8], "c":[0,1,4,5], "d":[1,2,3], "e":[1,3] } actual = clumps.find_groups(input) keys = actual.keys() self.assertEqual(3, len(actual)) grouping = (1,2,3) referents = set(["a","d"]) self.assert_(grouping in keys, "expect %s in %s" % (grouping,keys) ) self.assertEqual(referents, actual[grouping]) grouping = (1,4,5) referents = set(["a","b","c"]) self.assert_(grouping in keys) self.assertEqual(referents, actual[grouping]) grouping = (1,3) referents = set(["a","d","e"]) self.assert_(grouping in keys) self.assertEqual(referents, actual[grouping]) if __name__ == "__main__": unittest.main()

The Code

import sys def find_groups(named_groups): """ Exhaustively searches for grouping of items in a map, such that an input map like this: "first":[1, 2, 3, 4], "second":[1,2,3,5,6], "third":[1,2,5,6] will result in: [1,2,3]: ["first","second"] [1,2]: ["first","second","third"] [5,6]: ["second","third"] Note that the return value dict is a mapping of frozensets to sets, not lists to lists as given above. Also, being a dict, the results are effectively unordered. """ def tupleize(data): "Convert a set or frozenset or list to a tuple with predictable order" return tuple(sorted(list(data))) result = {} for name, methods_called in named_groups.iteritems(): methods_group = frozenset(methods_called) methods_tuple = tupleize(methods_group) for stored_interface in result.keys(): key_set = frozenset(stored_interface) common_methods = tupleize(key_set.intersection(methods_group)) if common_methods: entry_as_list = list(result.get(common_methods,[])) entry_as_list.append(name) entry_as_list.extend( result[stored_interface] ) result[common_methods] = tupleize(entry_as_list) full_interface_entry = result.setdefault(methods_tuple, []) if name not in full_interface_entry: full_interface_entry.append(name) return filter(result) def filter(subsets): # Apology: I'm betting I can do this in a functional way. filtered = {} for key,value in subsets.iteritems(): if (len(key) > 1) and (len(value) > 1): filtered[key] = set(value) return filtered def display_groupings(groupings): "Silly helper function to print groupings" keys = sorted(groupings.keys(), cmp=lambda x,y: cmp(len(x),len(y))) for key in keys: print "\n","-"*40 for item in key: print item for item in sorted(groupings[key]): print " ",item print

The Things That Pass For Simple I Can't Understand

Tim Ottinger — Thu, 14 Jun 2007 00:19:00 -0500

(with apologies to Steely Dan for the nearly-lyrical title)

I’ve noticed that “Do the simplest thing that might possibly work” gets universal agreement in principle, and great divergence in practice.

Is a set of variables related by name prefix simpler than a named class containing the same variables? In the “fewest number of the fewest things” sense, I see a group of variables as a lot more to manage than a class. I can’t understand why some people pass the same group of loose variables to a number of methods. Isn’t a seven-argument method “complex”?

Is a custom exception “fancy” or “trivial”? I’ve had that discussion recently. Some feel one way, some the other. Is it more complex to throw a standard exception type and then try to figure out what it means elsewhere?

Is event-handling simpler than polling? I’ve heard this one too. I don’t know how other people see it. I think events have more complex plumbing, but that polling has a greater run-time complexity because of failure modes (periodicity issues, race conditions, etc). I think that it is simpler if you have fewer problems to look out for.

If you need a fixed set of java instances, is an enum simple or fancy? Here you have a newer language feature, but it exists to simplify the management of a fixed set of instances, so it leaves you less to deal with, no?

Is “more primitive” simpler? How about if you use arrays and integer indices into a string rather than a list of strings? Is that simpler or more complex? Is the Bowling Game “simple” because it uses a primitive array and a separate array counter instead of custom objects?

In C++, is it simpler to have a struct that contains two data members or to use the pair<> template? Is it “simpler” that you have to refer to them as “first” an “second”? Or is that just more obscure? I think that “obvious” is more simple.

I am not sure how other people judge simple. I think “simple to use”, “less bookkeepping”, “harder to mess up”, “less setup to call”, “fewer parameters” are simpler. Other people seem to have contrary thoughts.

Clearly simple involves more than “least thought” or “most primitive data components”. It should have something to do with the lack of effort in using something and the ease with which things can be used correctly. Well, clearly to me. And it should have to do with the least difficulty in reading the resulting code, and the least difficulty in changing it.

Unrelated to this, I find some tools that others find simple to be baffling, and some tools that are complex to others seem pretty easy to use to me. I suppose that’s a kind of “personal taste” but is there a more reliable rule to base “simple” on than personal choice? There should be if the rule is for us all to do the simplest thing that works, and to maintain the simplest design we can afford to build.

But I have unique viewpoints sometimes, and maybe simple to you isn’t simple to me. Personally, some of the things that are simple to you may be all but inscrutable to me. I guess that happens.

Collateral Effort Revisited

Tim Ottinger — Mon, 11 Jun 2007 20:36:00 -0500

One of the things I love about TDD is that it takes all the scaffolding and collateral effort for creating a class, all indications of bad coupling, and makes them entirely visible. This is also one thing that makes it very hard to start TDD in legacy system.

When you realize that it’s very hard to test a class out-of-context, the right answer is to start decoupling each class so it can be run out-of-context. When we are successful here, tests run very very fast and are easily self-verifying and isolated. It is a beautiful thing even if it follows many ugly hours of painstaking work.

The wrong answer, I’m convinced, is to build a mega-framework for testing that allows you to test in-context. This type of framework typically creates a totally realistic runtime environment (complete with database, configuration files, directories, etc). This approach allows you to think that you can ignore your dependency nightmare. You really can not Mega-test-frameworks take a such a long time to run that developers stop running all the tests all the time. This is a far worse problem than breaking dependencies because it breaks the whole process. Mega-frameworks don’t solve the problem, they only defer the solution until the problem gets worse.

When you are faced this kind of collateral effort, the answer is to work through it, not to sweep it under a rug. Working through it is work, indeed, but this is about doing the right thing not doing the easiest thing.

The Myth of Learning Compression

Tim Ottinger — Thu, 24 May 2007 13:56:00 -0500

There is an odd idea that learning is compressible, that any training course of any length can be squeezed into a very small time box and delivered to any large number of people without significant loss.

As an educator and as a guy who has a pretty big learning load at any time, I know that this is a myth. I can cover any topic much better if I am working with a handful of individuals over the course of a week or two. If you double the number of attendees, then the interactivity is highly degraded, and discussions become time sinks. If you half the time, then interactivity is likewise degraded, and material that might have been covered must be skipped or skimmed.

I have a joke with my coworkers that the ideal environment for some large corporations would be a three-train roller coaster. The participants would buy a ticket, climb on board, sail past the materials at speeds in excess of 60 miles per hour, and would collect their certificate at the other end. As one train is unloading, another is loading, and another is in progress so that we could cover many hundreds or thousands of students per day. Of course, they would learn next to nothing and would not really participate at all.

But I respect my corporate customers, including the fact that they have much to do, and cannot easily afford a one-week work-stop for each dozen team members, all with consultant/trainer fees. Some large organizations would have us on-site for years in a row before all their developers would be trained in any technology. It’s just not reasonable, and they push for a more feasible schedule.

It’s a rock and a hard place, and people on both sides understand that it is so.

The good news is that compressibility is not a total myth. There are techniques and tools that can make the training more effective, so that less time is necessary in class. There is the idea of a jump-start, where a topic is introduced and discussed in a short time and students are given materials on which they can continue via self-training. The compressibility myth drives us to more dense and useful presentation. It is a valuable spur.

It also has the advantage of the time box. If you have only three days to teach someone how to TDD, you have to choose the three most important days-worth of material to teach. This kind of prioritization is the same thing that we demand of our product owners in a scrum, and is perfectly fair and reasonable.

Even though compressibility is a myth, it is a useful myth.

The only mythological part of the compression myth is the idea that it is lossless. It is lossy compression. But as is the case with audio and visual compression, compressed learning can be “good enough.” If that’s what we’re after, it’s all good.

Unit Tests Coverage: Less Is More

Tim Ottinger — Mon, 07 May 2007 08:06:00 -0500

In TDD a unit test has to be very small to isolate failures. This does funny things to code coverage as a metric. Each test should have a very small area of effect, and so each unit test should have a negligible effect on the overall code coverage statistic. Bear with me here, see where I’m missing out.

Say you have an existing (legacy) system with no coverage at all. Zero percent. If you start doing TDD today, the overall coverage percentage should barely change at all. If only the new code is test-driven, then the old code is not gaining coverage except where tests are necessary to ensure that the new code is being called. The low coverage per test is a good thing because it shows that the unit tests have good isolation. In such a situation, code coverage is really telling you the ratio of new code to old code. Again, this is so obvious and logical to me that I must be missing some cool subtleties.

If all the code was test-driven from the beginning, you should have a very high coverage number, and writing a new unit test before you add code should not impact that number. You only write enough code to pass the test, so again you aren’t getting much in the way of uncovered code. A lowering of the ratio might indicate a problem with test-to-production-code ratio. This seems pretty simple and logical, so I’m sure I’m missing some interesting corner cases.

OTOH, system tests and integration tests paths through many components at once, and should have a more significant effect on coverage in a previously-untested system, though their job is to prove function points, not to raise the metrics. These non-unit tests are the thing that boost your coverage of old code. That is also a good effect, because it is the goal of the system test to ensure that the parts work together.

I’m not saying that code coverage should be low, only that as we move incrementally, the unit tests we write should individually have negligible effect on our overall code coverage numbers… a thought that intrigues me.

CppUnit and Vim

Tim Ottinger — Thu, 03 May 2007 21:54:00 -0500

I’m playing with ways to make CPPUNIT and VIM a bit more agile for my C++ teams.

I’ve been hearing about great lengths people go to, flipping back and forth between .h and cpp for the tests or writing perl scripts to coalesce them together. I’m too simple-minded for that. I figure nobody wants to #include my tests, so I don’t need a .h. I do it all in the .cpp file, all inline. It works fine. This way, everything is in one place. I don’t have to jump up to the header file to add a protoype and then down to the implementation to add the CPPUNIT_TEST line, and then back down to my code to enter it. I have no .h, so no prototype. Inline makes it a bit more like pyunit or junit.

I grokked the CompilerOutputter pretty quick. If you use the quickfix mode (and esp if you use :cw) you will definitely want the CompilerOutputter. I like the way it helps me move from testing error to testing error (err.. if there were ever more than one), just as it does with compile errors (errr. if they happen. They do). I set up my makefiles to run the tests, so there’s no additional step required. That’s handy. I also have the makefile run ctags to make my navigation nice and easy.

Still, adding and removing tests takes some effort. I figure VIM can take up the slack. I have macros to start up a new test file (F2), to add a test class(F11), to register a function(F12), and to delete a function and its CPPUNIT_TEST line (F9) .

I also abbreviate most of the macros that I would otherwise have to type. I am thinking about remapping them maybe to more memorable values, but it helps.

There are probably a lot more elegant ways to do this. I can’t wait to hear about them.

" Add methods to help create CPPUNIT classes function! NewCppUnitFile() call NewCppUnitClass() 0r~/.cppunit_file_template.vim endfunction function! NewCppUnitClass() r~/.cppunit_class_template.vim let name = input("Name your new test class: ") exec "%s/@@@NAMEHERE@@@/" . name . "/g" endfunction nmap <F2> :call NewCppUnitFile()<CR> nmap <F9> ^f(b*ddN0d/{<CR>d% nmap <F11> G:call NewCppUnitClass()<CR> nmap <F12> ebmz"zyw?CPPUNIT_TEST_SUITE_END(<CR>OCPPUNIT_TEST(<C-R>z);<ESC>'z " CPPUnit abbreviations ab cas CPPUNIT_ASSERT ab cam CPPUNIT_ASSERT_MESSAGE ab cfa CPPUNIT_FAIL ab caf CPPUNIT_FAIL ab cae CPPUNIT_ASSERT_EQUAL ab cat CPPUNIT_ASSERT_THROW ab cde CPPUNIT_ASSERT_DOUBLES_EQUAL

The current rhythm is to open a new file and press F2. F2 calls NewCppUnitFile, which sucks in my file header template. It’s pretty bare, but it can be edited to taste. Mostly it helps me remember what to include.

#include <cppunit/TestFixture.h> #include <cppunit/extensions/HelperMacros.h>

Then I give the name of my first test class. I get a starter class. I can then add it to the makefile (note to self: automate the makefile bit). the starter template I have has a dummy test, so I can run it and see if it fails. Failing means it’s being built and run.

class @@@NAMEHERE@@@: public CPPUNIT_NS::TestFixture { CPPUNIT_TEST_SUITE(@@@NAMEHERE@@@); CPPUNIT_TEST(shouldFailAsEvidenceThisMethodIsRegistered); // DELETEME? CPPUNIT_TEST_SUITE_END(); public: void setUp() { } void tearDown() { } protected: void shouldFailAsEvidenceThisMethodIsRegistered() { // DELETEME? CPPUNIT_FAIL("This function is registered and can be deleted"); } }; CPPUNIT_TEST_SUITE_REGISTRATION(@@@NAMEHERE@@@);

Substitution is automagical. the @@NAMEHERE@@ is the magic pattern being replaced with test names.

A nice thing about macros is that they’re atomic. If I hate the name I gave my test, I press ‘u’ for undo, and it vanishes. Back to empty file. :-)

If the ”:mak” works and I get the inevitable failure in the “shouldFail” method, I put my cursor on the “void shouldFail…” line and press F9. In the wink of an eye, the method and its CPPUNIT_TEST line both vanish. Again, macros are atomic, so if I delete the wrong thing, I just press ‘u’ for undo and both the function and its registration reappear.

No flipping between files, no scrolling up and down. We have software to do that now.

I add a method under “protected”. I put the mouse on the method name and press F12 to add the CPPUNIT_TEST for it above. I add some more text. When I get to the part where I need to assert, I use one of the abbreviations. If I’m in insert mode and type “cae(” I get “CPPUNIT_ASSERT_EQUALS(”. I don’t have to remember whether it ends in an S or not. I actually struggle with that, sad thing that I am.

I can press F11 to create more test classes. I like to keep a test class for every circumstance (setup) I am using. I don’t mind at all having multiple tests per .cpp file. I like to group the tests together in files. After all, quickfix knows where the code is. This is actually one of the reasons I like to keep using CppUnit. I’ll have to tell the others someday.

Well, that’s how I do it. You don’t have to, but you could try it. If you find any errors, or glaring omissions, please mail them to me or comment here. In return, I will claim my actual scripts are perfect and blame it on the web master. ;-)

What Cant Who Do?

Tim Ottinger — Tue, 01 May 2007 21:55:00 -0500

I have a fresh idea. When someone is supposed to change their workstyle to do TDD, and they say “we can’t do that”, how about we assume they’re telling the truth.

The first word is “we”. It may well be that the speaker and the people he represents might not be able to do the work. They’re not saying that it can’t be done. They’re only saying that they can’t do it.

The next word is “can’t” and that’s an interesting word, too. It might mean that they lack the ability or knowledge. That’s ideal, because such things can be taught and can be led. The other sense is that they are not allowed. If it’s like that, then there is going to be some political work going on. But at least you know what you’re getting into.

The other tell is the word “that”. Maybe they should describe what it is that they can’t do, and you can tell if the thing they can’t do is the same thing that you are wanting them to do. Sometimes it is not.

Coercive Immediacy

Tim Ottinger — Tue, 01 May 2007 20:13:00 -0500

I have been thinking a lot about the “short reach” idea, and tryng to reach beyond it to capture the thing that the good agile project teams have and several others are missing. At the same time, I’ve been reading up on some agile tools (including one by a guy I know) in my spare time. In addition, I am on a mailing list for multisite agile development. Finally, I have been pondering how to work this agile stuff when the developers cannot be co-located due to shortage of office space or shortage of common areas.

I realized that the whole “short reach” thing creates a kind of coercive immediacy.

The corkboard with all the story cards in their four columns has that kind of immediacy. I can’t walk past it without assessing how far we are in the current iteration. I can certainly ignore a web page, or a display in some application on my computer. I can’t walk into the common area past that board without thinking.

Likewise, I could ignore an app that holds story status and backlog, but the cork board is where I go to show that I’ve completed a task and to get the next task. If my stories were assigned to me (rather than allowing self-organization) then I could neglect the corkboard too. I would not have to look at it or think about it. I could hold “my” story card at “my” desk until I decided I was done. When we are self-organizing, we are always thinking about who could help us, when we can get done, and what is the best way to truly complete this story and move on.

A partner has coercive immediacy. When you are programming in pairs, you don’t answer the phone unless you have to. IM and mail can wait. Interruptions are less welcome. Having two people on the story pushes one to focus. Also, the pair partner is always there to catch the sloppy shortcut, the poor naming, the ill-considered algorithm.

The customer team member also participates, and both feeds and feels the coercive immediacy of the team. Rather than make a guess, the programmer feels the nearness of the customer, and it’s much easier to ask than guess.

I think that TDD shows this most of all. I can’t choose to ignore the red bar. I write a line of code or two, and now my tests pass. The immediacy would be gone with it, except that I know I have some cheesy, minimal-effort code and I can’t leave it that way. I need a better test. When a test fails in some area where I’m not working, I feel the immediacy.

With distribution, multi-site, tools, and the rest I fear that we lose the coercive immediacy of Agile development. If we don’t have a partner in our way/face, then we lose something. If the status board is a tool somewhere, even a really nice tool, but I have to remember to ask for it to see it, then it is not in my way. It doesn’t inspire me.

My thought now is that an agile management tool won’t help unless it is something we have to live with all the time, during pairing, as part of the test/code/refactor cycle. It has to be in our faces, over our heads, and not just tickling the edge of our awareness like a more polite version of “clippy”.

Short Reach

Tim Ottinger — Mon, 23 Apr 2007 15:33:00 -0500

I’m always trying to find newer, better, shorter, more powerful ways to explain what Agile is about. I suppose I’m some kind of obsessive about expressive power and economy.

Finally I decided that Agile, as I understand it today, is about the short reach.

It seems to me that all of the agile practices are about shortening our reach, the distance in time-and-space that one leaves an assumption, decision, or line of code untested and unconfirmed. All the practices seem to follow this one rule.

The customer/analyst is keep in the same room, in the same short reach.

We feed back the iterations to the customer/analyst so that his every decision has a shorter reach.

We do iterations to ensure our planning has short reach.

We keep our teammates very close, in the same room, so that it’s a shorter reach to them.

The test is written first, so that implementation has shorter feedback on correctness.

We compile/test frequently because our code time should have a short reach.

We pair so that our code is instantly vetted through a peer. We don’t pile it up and review it after the tests pass.

Our planning is based on “yesterday’s weather”, data collected a very short time ago.

We don’t plan the team structure and the assignments, we self-organize so that tasks are waiting for the shortest time possible.

We talk face-to-face, not across chat and email and official company documents.

Tell-dont-ask and the Law of Demeter guide us in keeping the reach of our objects very short.

We use unit tests to exercise a class directly, and we isolate with mocks to reduce the reach of our tests through the system.

Shared code ownership means that the guy sitting behind your keyboard has all the permission he needs to do excellent work, even if it impacts existing design.

Test-first development means that the guy who makes a change knows very quickly whether his change is safe or not. He doesn’t have to wait until the week before integration when the “real” tests are run.

Where does Agile run into logistical or operational difficulties? Wherever a long reach is required or imposed. Where an organization chooses to continue in waterfall-style management, where the team is distributed among managers with appointed “point of contact” and “official channels”, and where the developers are not placed in a common work area agility is very difficult.

I’m not saying that agile techniques can’t work for large companies, but that is an area where a lot of experts are trying (maybe succeeding) to extend the agile techniques and where the average “agilist” finds challenges. When it works, it is almost certain it will be because someone has found a way to shorten the reach of the teams so that all they need to know is never more than a few seconds or minutes away.

At least that’s my half-baked observation of the day. Let me know if I’m wrong here. Or if I’m more right than I think I am.

Using VIM

Tim Ottinger — Fri, 20 Apr 2007 01:19:00 -0500

It isn’t done yet, but I figured I would post my work in progress in the meantime for feedback. It’s a little article about the proper use of the VIM editor. I tried to mix in some keystroke quick-reference with the philosophy of the tool in a way that might make it a bit more accessible.

When it’s done (Ha!) I will probably find new places to store it, but for now it’s at my personal blog.

Don’t be afraid to offer corrective assistance, or to recommend sections.

The Two Things Agile

Tim Ottinger — Mon, 16 Apr 2007 22:07:00 -0500

For Agile, maybe the Two Things are:

We can do better/faster this week.

And this way next week will be better yet.

Code is a Liability

Tim Ottinger — Mon, 16 Apr 2007 21:10:00 -0500

I have thought for a long time that code is liability, and that viewing code as a liability has the power to transform our art. I read articles about code-as-a-corporate-asset and most of them are pretty good articles about wringing the value from the code you have, but I still feel that there’s something essentially wrong about it.

Programs are a great means to an end. Don’t get me wrong, I’m in favor of having software and I’m greatly in favor of writing software. I think we would be in pretty poor shape without it. But I can’t call code an asset.

Functionality is clearly an asset. When we can inter-operate with various services (possibly even web services) our businesses produce more business results more quickly. We use software to control machines and to increase public safety in more ways than most of us know (talk to the guys who write the software that controls your car, the over-the-road trucks, forklifts, industrial robots, and aircraft). Software makes our life better and it’s a wonderful thing. I’m all in favor of software functionality and you have to write code to get it.

Our bosses and clients will pay good money to get the functionality they want, and they want it right now! If we could give them what they want without writing a line, it would be a tremendous win. If we could do it with one line or two lines of well-considered code, we would be heroes! Why is doing less so valuable if code is an asset? Clearly less code is better.

Sadly, most companies have to deal with heaping, shaggy mounds of code. Code takes up time and space. It has to be managed. It has to be versioned. It hast to be tracked, and planned. It has to be updated, and packaged, and revised. It needs backup to save us from having to reproduce it by hand. It has to be reviewed (hopefully in an efficient way like pairing). It often drives companies to expand staff and dedicate people to manage it (version control administrators, managers, build czars, consultants, contractors, metric-gathering tool specialists, etc).

Old code gets in the way of new code. Having more code will typically slow development, and will certainly reduce your ability to incorporate new programmers. Of course you’ll need more programmers because you have all this code to deal with. Size has a cost.

The problem doesn’t go away if you artificially reduce the code. Folding a lot of effects into few lines of code makes the code worse. Adding voluminous documentation makes the code worse. Moving it into metadata and models and other forms doesn’t make it any smaller, and often makes it worse. Hand-crafted code is almost always more readable, smaller, more optimal, more focused, more literary in its style than generated code or funky data tables. Since there has to be code, it might as well be the best code we can write. Coding well takes human beings who value minimalism.

It seems that the trick with the functionality/code game is in trying to maintain a positive balance.

We reach a point on some projects where we are producing a positive number of functions with a net loss in lines of code. That’s a wonderful time. We reduce the line count by simplifying the code and eliminating duplication. Sometimes we refactor our code into a more compact form and then only add a very few lines of code to produce a change. If code were an asset, then those moments would be failures instead of triumphs.

Code size reduction is a very good thing. I want the simplest, the smallest, the least code possible, with the least risk of breaking something else in the system. I don’t want big functions that touch everything in sight. I don’t want long complicated blocks of if/else statements. I don’t want monster functions which conglomerate dozens of operations in a single, fat interface. Shallow is good. Short is good. Less code is good. More code is a liability. This isn’t about typing less, it’s about owning less.

This is the point of view that makes test-first (TDD) so important. TDD/BDD has us encode the functionality (the asset) first, and then write minimal code to realize the specified feature. If code is a liability, and function is an asset, this is exactly the right way to do things.

And maybe, just maybe, it might explain why one might open-source less-important useful systems.

It's a Poor Coach Who Slaps The Kids

Tim Ottinger — Mon, 16 Apr 2007 21:01:00 -0500

Today I was remembering a bit of advice I received back in the late 80s when I started trying to work with people more. I’d been a programmer for a while, and mainly survived by reading a lot and practicing what I read. I taught myself most of all the programming I knew (I did have some good mentoring peers). I wasn’t much of a people person.

A fellow named Kevin worked on a project with me in the DC area. One day he had the kindness to take me aside. He explained to me how to work with people. He taught me that when someone had an error in their understanding I shouldn’t tell them they were wrong, but rather tell them they were right except for the part where there was an error. He helped me more than he knows. We only had the one project together, and he went on to a different employer and I went on to become me.

He told me that a good coach doesn’t tear down his kids. He doesn’t tell they they stand like a sissy or throw like a girl. Instead, a good coach will let them know what’s right and what’s valuable about what they’re doing, and then explain what to add to their technique to get better results. And he doesn’t tell them everything all at once.

Now I’m a coach. I teach development methods, including agile methods, including TDD. I now work with people more than technology. I also work with a lot of other coaches from time to time. The best ones seem to intuitively understand when to criticize and how, and when to hold their tongues.

In the warm rush of enthusiasm and passion for the art of development, we coaches have to continually remind ourselves that it’s the poor coach indeed who slaps the kids around. We have to instead meet them where they are and look for ways to help them get the results they want.

Kevin’s lesson continues to teach.