Information Management: Strategies for Gaining a Competitive Advantage with Data (2014)

Chapter 16. Agile Practices for Information Management

There’s a lot of work to do in information management. It must be approached in the most effective manner possible. While full SCRUM is not necessary, here I talk about those aspects that are.

Keywords

agile; SCRUM; methodology

I have been extensively covering information architecture possibilities and the end result that is desired, with only passing mention of the process of how to get there. Now that you are formulating the information architecture for your organization, getting there will be the focus of this chapter.

Naturally, we have been careful to avoid unnecessary latencies in the architecture, making sure that data moves as quickly as it can to where it can be effective. Delivering high performance has practically been the primary driver of the book and, as previously mentioned, is a primary focus in architecture development. Again, we need to do all we can to promote the very important asset of information, removing obstacles even at the cost of some added technology.

The Time-Value Curve (Figure 16.1) is about architectural latencies, but what about delivery latencies? There is another obstacle that keeps many a user away from his data and that has to do with the time it takes to deliver projects on-target to user need. Hopefully by now, as an information manager, you are convinced that you have a lot of work ahead of you. There is no time for the unnecessary, the inefficient, or rework.

FIGURE 16.1 The Time-Value Curve, Richard Hackathorn, Boulder Technology, Inc.

For quite some time I have been breaking up otherwise long projects into smaller projects for delivery. I want my teams to stay as close as possible to delivering to production, which is what it’s all about. I also understand that, despite good requirements gathering, the best feedback comes in the form of reviewing actual work that you are calling somewhat complete.

Those predilections turned out to be a perfect match for the era of agile methodologies. I’m on board! I have seen otherwise good teams become great teams when some components of agile are assimilated. The one I have adopted is SCRUM.¹ Though these methodologies started with software development and not in-house enterprise development, they have serious applicability.

I don’t, however, think you need to become SCRUM-certified, attend weeks of training, adopt the terminology in a strict sense, or have every team member count backwards beginning with their birth year or else be demoted a belt color to be effective. Some of the adoption is counterproductive. I continually see shops going overboard with agile as if it’s all-or-nothing. It is my attempt in this chapter to give you the “best of” SCRUM for your information management projects. After all, it’s all about delivering to production.

Traditional Waterfall Methodology

Traditional waterfall methodologies incur too much cost, rework, and risk to be effective in the dynamic field of information management. ROI tends to be delayed too long. Agile methodologies balance out the risk (and reward) over time. Some of the ways SCRUM does this is through taking “how do we handle this?” questions off the table. To a very high degree, everyone knowing the rules of the game makes for a much better game. But, you say, all methodologies do this. True, good ones do, but let me expose the elements of SCRUM that make it balanced.

We must maintain a balance between rigor and creativity (see Figure 16.2 “Unbalanced Methodologies”). That’s where leadership and judgment come into play and no methodology will ever replace that! Any methodology that claims you just put the people into place and it works without major decision making along the way, is folly. As I discuss agile approaches, and SCRUM in particular, please note I am not discounting the need for leadership and judgment in the process.

FIGURE 16.2 Unbalanced Methodologies.

Agile Approaches

The now somewhat-famous Agile Manifesto has become the foundation of several methodologies, including SCRUM. It simply reads:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

The principles behind the Agile Manifesto include: early and continuous delivery of valuable software, delivery of working software frequently, business people and developers working together,² projects built around motivated individuals, simplicity is essential, self-organizing teams, and regular reflection and tuning.

Your judgment and leadership will have to come into play here to determine to what degree you take it. Many organizations are forcing the issue by adopting SCRUM company-wide, which makes it easy as to which profile you take on:

Company-wide adoption of SCRUM with obvious effects on the business intelligence program

Information technology-wide adoption of SCRUM

Information management-wide adoption of some level of agile/SCRUM principles

I suggest the last bullet is the minimum requirement, so I will delve into some applicable aspects of SCRUM for information management.

SCRUM

SCRUM has forged an immense amount of new terminology. Perhaps the best way to share the methodology is to review it term by term. Please note, in so doing, I am suggesting the right level of SCRUM to adopt. It’s a means to an end. It’s not the gold-rimmed, overdone version, nor is it just picking off some concepts and calling it a methodology.

SCRUM purists will be beside themselves reading this and arguing that I forgot the 23rd principle and that people really need 3 weeks of training and passing tests and whatnot to be agile. If you want to go down that route, it’s available out there. I certainly don’t need to repeat that material here.

As with the technology chapters, implementation will require more in-depth training for the immediate technology team. With SCRUM, a shop may want to consider some web-based or in-person training for a few who will fill key roles, with the understanding that they will do on-the-job just-in-time training for the rest of the participants on their projects. All SCRUM training should be taken with the filter on that I present here.

Likewise, those new to SCRUM may find the terminology awkward and want to lessen the rigor I suggest here. While I can agree it’s ultimately about results, I can only suggest a level based on my vast experience running, consulting on, and auditing information management projects. I may bend some after learning of a unique situation, but far too often people will jump to the “but we’re unique” argument when it is not warranted.

This is a balanced, information management-focused interpretation of SCRUM. It’s a version that tends to get used by successful shops after they start at one of the unbalanced ends.

By the way, while you may exercise this terminology within the team, do not expect the rest of the organization to necessarily engage in it or even care about the methodology. They will care about results.

“It’s better to be roughly right than precisely wrong.” John Maynard Keynes

Product

A product is a long-term deliverable and comprises releases (which are comprised of sprints). A product is what goes into budget cycles and what gets upper management excited, promoted, and fired. A product can take a year or more to complete. A product is SCRUM’s link to waterfall methodologies. Whereas a project comes into being in much the same way in both camps, from that point forward, SCRUM treats a project quite differently.

Release

A release is best thought of as sitting between the product and the sprint. It may be a key milestone in a project. It is something that upper management is interested in as a “project milestone.” It’s a multimonth delivery that will show progress toward the product.

Sprint

OK, here we go. A sprint is 2–4 weeks (2 weeks is best) of activity by the SCRUM team. There are several requirements (user stories) that the team is trying to be accomplish in the sprint. The sprint is characterized by daily semi-structured meetings and detailed tracking of the percentage complete of each story (requirement, explained below). A sprint is also the level that should be communicated to the organization-at-large in terms of what the team is going to accomplish (“failure is not an option!” mentality) in the sprint. If the sprint does not include shippable (to users/systems) components, they need to be “potentially shippable.” You would need to decide what this means in the context of your project, but it should be a high bar, as in going to production and being used.

User Stories

User stories are requirements in the most granular form. Stories are negotiated by the team and the Product Owner in the Sprint Planning meeting at the transition point between sprints. Stories may be themed and longer stories are called epics. All stories will be assigned story points which represent the relative effort required to complete the story. This minimizes long cycles of acute time planning, opting instead for educated speculation.

What Makes a Good User Story

Independent

Negotiable

Valuable

Able to be Estimated

Sized Appropriately

Testable

Impediments

In the daily SCRUM meetings, each person is asked for progress on stories as well as impediments, which are tasks to be done by others (internal or external) or waiting periods to wait out something that is impeding one’s progress against his or her assigned stories.

Product Roadmap

This is the SCRUM version of the iteration plan. It’s a lightly shared document owned by the Product Owner that is a working, directional estimate of what foreseeable future sprints will be comprised of.

Team Capacity

Team Capacity is how much work product the SCRUM team can deliver in a sprint. Obviously, it’s not an assembly line, so there is a lot of judgment involved. Team Capacity is measured by something called story points, described below.

Sprint Backlog

Those stories accepted into a sprint become the sprint backlog. It’s a very manageable list that the SCRUM Team can keep top of mind throughout the sprint. Once stories are accepted into a sprint, they cannot be changed. This is partly punitive and instructive for the process that takes on stories, but primarily it is a relief for the SCRUM team, knowing that their work is stable. For two weeks.

FIGURE 16.3 Many stories per sprint, in different stages, at least one to production.

Product Backlog

This is a place where anybody in the company can add a story. This accumulates over time and only abates as stories are pulled into sprints and delivered.

Daily SCRUM Meetings

Literally daily meetings at a set time in a set room (and/or with set conference/internet call coordinates) for all of the SCRUM team. All team members take turns taking the floor answering the following questions: What did you do yesterday? What are you working on today? Any impediments?

Sprint Burn Down

Stories on the sprint backlog are “burned down” by increasing the percentage completion daily as they are worked on. Everyone can, therefore, have a good sense of progress because the entire sprint will therefore have a math-based “overall percentage completion.” A graph can easily show³ where progress is and where it needs to be on a sprint.

Sprint Planning Meeting

The pre- (or post- since they occur simultaneously) meeting of the SCRUM team and the sprint master to review the product backlog and build the sprint backlog. It’s a participatory process whereby all may help estimate the size of the story. Obviously, only so many stories (up to a predetermined limit) can be accepted into a sprint. Story points are used as surrogates for capacity and work effort. Philosophically, the total story points are what the collective team needs to deliver. The team will win or lose together based on this. If, during the sprint, team members need to pitch in for one another—usually because of slight under- and overestimation in the story points, that is expected. So, there are significant team aspects to the estimation.

Story Points

“Story points” is also a SCRUM concept that may be unfamiliar to someone coming from waterfall approaches. What the heck is 3 story points, anyway? What’s important here is not “how many” points, but consistency of point allocation across the stories and over time. All team members may opt in on estimating EACH story in the sprint—even ones they will not personally touch. It’s “1-2-3-show” and fingers or cards are revealed. Discrepancies are discussed until ONE value per story is reached.

SCRUM Story Point Estimation

Many use the Fibonacci sequence of numbers (1, 2, 3, 5, 8, 13, 21, etc.). This works because it’s easier to estimate the short stories so there’s more granularity possible there (1, 2, 3, 5 points). As the stories get harder and longer, they are naturally more difficult to estimate, so they are estimated at only certain intervals. There should be an upper limit on story points assigned (i.e., 21, 34). Over that, stories should be broken up. The SCRUM guideline for this in terms of hours is 16 hours estimated maximum per story.

Others use an estimate of total team hours on the story, which I am not going to say is as bad to do as some will. After all, it does come down to time. Others use shirt sizes (S, M, L, XL, XXL), etc. or dog breeds (from Chihuahua to Terrier to Great Dane). For fun, I’ve used balls (marble, ping pong, tennis, baseball, softball, volleyball, soccer ball, basketball, beach ball). Obviously, the aggregate levels for the sprint need to be understood as well. That’s why you end up with straight-faced discussions that contain “So, in this sprint, we are committing to three beach balls of stories.” This is only limited by imagination. I’d be interested in how you do it, so let us know in the comments.

While there is clearly a science to the process I spoke of, judgment remains essential for SCRUM success. If you go into sprint planning looking for 29 story points and the points land you at 27 and the smallest story point count you have left is 3, should you add it or not? Judgment. Prerequisites must also be considered. Stories have prerequisites and, while the goal is to deliver a complete “product” of sorts in the sprint, the reality is that there are often multiple threads (i.e., source systems, sets of business rules, sets of reports, etc.) that are being worked on at any point in time.

FIGURE 16.4 Sprint Reciprocal Commitments.

Sprint Retrospective

A meeting of the SCRUM Team immediately following each sprint which, like the other meetings, solicits input from each team member. In this case, there are three questions that each team member responds to:

What worked?

What did not work?

How to improve?

Product Owner

This is the involved person representing the business interests. With the right, but not obligation, to attend the daily meetings, the Product Owner keeps the business interest top of mind. It is possible for one person to be product owner of multiple sprints and products.

SCRUM Master

The SCRUM Master has a high understanding of the SCRUM process and makes sure that it is followed by the team. The SCRUM Master does not have to have anything except a cursory understanding of the information management domain being worked on in the sprint, although the more, the better. The SCRUM Master runs the daily meetings and typically will run the project and report progress to upper management quantitatively and dispassionately.

SCRUM Team

A team of 5–8 who are brought together to work on a sprint, and usually a series (or years) of sprints toward product completion. Being multitalented is a great quality of a SCRUM team member since priorities change from sprint to sprint and the team member continuity is important. For example, many information management professionals can handle data integration and data delivery, so their focus may change from sprint to sprint according to the needs of that sprint.

SCRUM and Methodology Themes

Multitasking

In SCRUM, there is an eschewing of multitasking in favor of focus. Studies of multitasking show a serious drop-off in doing value-added work once the number of tasks exceeds 2. In SCRUM, individuals are focused for the duration of the sprint on “1” task (1 is in quotes because we know how nuanced information management tasks can be).

Scoping Small Enough

Stories must be broken down to fit into sprints. Although many people can contribute to a story, most are fit for one person to deliver, or at least be on point for. Breaking down stories into this fine a grain will be done by the team prior to developing the sprint backlog. The collective team will need a strong sense of the tasks to be done to build information management.

Other Challenges

Many of the things I have encountered in my journey to agile include lack of attendance at the stand-ups at first, programmers estimating with huge buffering (for the worst case scenarios), and a need to hone the stories and tasks in a meaningful way. I can look at one graph and tell how a project is doing now, but this is only after I learned the balance between task size and too many tasks. In short, when the tasks are too large and developers log hours against the tasks day after day, it is difficult to tell if, indeed, the task is nearing completion.

“Letting go as a leader” comes with the “self-managing teams” concept. Often, a Scrum Master that does not have domain knowledge can still be very effective.

The definition of “done” can be challenging to get consensus on. Sprints, and stories and tasks, in the project need clear definitions of exit criteria.

Please note that this is all only referring to project work in terms of using SCRUM. Production work—the arguably more important half of the team’s workload—is still more queue-based work.

Keep a focus on quality, too. Two weeksoften is simply too short a time to deliver complete information management, let alone complete many data integration cycles. You could enlarge the duration of the sprint, but I’ve found that 2 weeks is the best cadence to work scope around. However, too high a focus on delivering a sprint could cause good methodology compromise. If you compromise data quality, metadata, architecture, and other forms of essential value, you may win the sprint battle, but lose the information management war.

To achieve agility for the users, you need not only an agile process, but also tools that are aligned with this approach. It would be a shame to deliver quickly, while users still experience poor performance. Deliver information agility with agile.

Some characteristics of agile tools are:

Intuitive and easy to use

Fast

Interactive

Works with any data

Self-service

Action Plan

Immediately update any waterfall methodology (or no methodology) for information management to be agile—SCRUM preferably—according to the direction in this chapter

Identify a SCRUM Master who will perform the function across information management projects

Identify Product Owners for all discrete information management projects

Identify an upcoming information management project to execute with SCRUM, understanding that there is a learning curve well worth getting past

http://www.mcknightcg.com/bookch16