Bloodletting: Why Controlled Experiments are Important

Bloodletting: Why Controlled Experiments are Important

I want to start this post with the last paragraph of this article:

Next time someone tells you that they are sure their idea will work, consider running a prototype in a controlled experiment, and make a data driven decision!

Now – the article itself is in the context of Microsoft software development, but this is what "lean thinking" is all about.

In the Toyota Production System, processes and the organization are deliberately constructed so that a kaizen experiment in one area is unlikely to affect another – at least not before there is ample warning and opportunity to reverse the change.

Every production cycle itself is conducted as a controlled experiment testing the hypothesis that the process:

  • Can be conducted as specified.
  • Delivers the specified results.

Of course – and here is the key point – it is only an experiment if someone actually examines what really happens. And this is the fundamental difference that is captured in the quote at the top.

A truly lean process has built in checks that check, each and every time, whether the idea works. Any time an idea doesn’t work, there is an expectation that the process will be stopped and understood.

Continuing to blindly produce, without knowing if each and every process is working as planned, without knowing if the result is as planed each and every time is being sure it will work. It is bloodletting.

Don’t bleed your process, or your company, to death with things you "know will work."

One-off and Customization

One of the questions that comes up frequently in “lean” discussions is the issue of non-repetitive work. This is especially an issue with complex information processes such as bid proposals, estimates, etc.

While it is true that breaking down and understanding truly repetitive work is easier, the same principles apply to more complex tasks.

But first, I’d like to challenge some thinking. If you were to tell me that “every time we do this it is totally custom,” my question would be “If it has never been done before, why can’t everyone do it just as well as you?” What makes up your expertise? Why do your customers come to you?

I’ll answer: Because you know how to do it. Obvious, right? But that’s the point. You have experience, because you have done it before. You have a process of some kind. And you carry out that process to produce your customized product.

Further, you have some kind of idea of what a “defect free product” is. You understand what you want to accomplish.

Complex operations are built up from simple ones. The tasks that are done over and over are these simple operations. These simple operations are great sources of waste because (typically) no one ever examines them.

The idea that “lean” only works for repetitive processes is largely the fault of the “lean industry.” It focuses so much on takt time and removing waste that it has not, so far, gotten to the actual heart of what makes continuous improvement work.

In an ideal repetitive process running to takt time, there are a number of key ingredients.

  • There is a highly specified work sequence that calls out the content, timing, sequence and intended outcome of the work. EVERY time that work sequence is carried out, the actual content, sequence, timing and outcome is being compared to what was specified. ANY departure from the specified work (or result) is going to trigger some kind of immediate response to correct the immediate issue, and then start the process of problem solving to find the reason this happened. This is jidoka.
  • Although I mentioned timing above, the importance of time is elevated. Taiichi Ohno is quoted as saying “Time is the shadow of motion.” Ohno was a great student. The idea that it is important to study motion itself rather than focusing on time comes from the pioneer Frank Gilbreth. In application, by specifying how long each normal operation should take, and checking the actual timing, it is possible to detect when unplanned motions creep into the work. This is the purpose of takt time and work balance. It is no more, and no less, than a tool for verifying planned vs. actual so that action can be taken immediately if there is a difference.
  • There is a specified amount of work-in-process. Each piece has a reason to be there. There is some means of detecting any departure from that standard, and any departure triggers an immediate response to understand why.

As you scale up from a work cell to an entire factory, these mechanisms scale as well, but the principle is always the same:

  • Tightly specify what you expect to do, when it should be done, who should do it, where it should happen, and how it should happen.
  • Tightly specify (understand) the intended “defect free” result of each step.
  • Have a way to verify, continuously, that what you are actually doing (and when, who, where and how) matches what you planned.
  • If When you DETECT any departure from what you specified, STOP pretending you are still running to plan; FIX or CORRECT whatever got you off plan and get back on plan; then work to understand and SOLVE THE PROBLEM. This is how the organization learns and acquires what Deming calls “profound knowledge” of itself and its processes.

In a complex one-off situation, you might never carry out that plan again. But plan carefully, applying everything you know… all of your expertise. Understand the elemental tasks that you do all of the time. Understand how long each should take. Understand what result you should get at each step.

As you carry out the plan, if when there are unforeseen real world issues, STOP long enough to understand their impact and start asking questions. First, what must we do to get this back on track?

If someone had to improvise, why? What knocked him off the plan or process?

If someone had to finish up work he got from upstream, why? Was the specification for a “defect free” transfer vague? Does it need clarification?

This can go on, but the bottom line is that most organizations with complex processes expect their people to accommodate and cope with noise in the system. They say “every time is different, we can’t possibly plan that well.”

The ones who are getting better every day say “We should be able to plan this perfectly. Why couldn’t we this time?” In simply asking that question, they get better each time they do it.

So – in the context of the original question. When putting together a complex bid and proposal, there are (I presume) steps you routinely go through. This is so even if you are not aware of what they are. Study those steps. Study the intermediate products. Understand where one step ends and the next begins. Understand what must be present (information, resources, etc.) for that step to execute successfully. If, at any point, those things are not there then STOP and understand WHY. Don’t just ask the people who SHOULD have those things to go find them.

You wouldn’t have an assembler on the shop floor hunt down missing parts. Don’t have a professional engineer hunt down information he should have received.

Check, at each intermediate step, that it was done as you expect.

At the end of the process, check that the bid is what you want it to be. If it needs rework (meaning someone didn’t approve it), understand why, and work on what information the process didn’t have the first time through. Next time, get it right.

The proposal itself is packaging. It is also your first cut at the plan for execution.

If you get the bid, and execute your plan, any departure from what you originally proposed should be understood. Why? What happened? What didn’t you see coming? Some things are truly different, and you are only estimating. But compare your actual vs. your estimate so that next time you can do a better job estimating.

There will always be imperfections. The question is in whether the organization chooses to bury them in waste or learn from them.

Accurate Forecasting

Why can’t we get a more accurate forecast from sales?
Manufacturing managers the world over have the same complaint.

Maybe the word “forecast” is tripping everyone up.

A forecast is a prediction. Maybe it is based on some kind of market analysis, maybe even asking the dealers what they think they will sell. It could be based on a lot of things.

Once a forecast is complete, it is regarded as the best guess for how things will pan out, but those things are (felt to be) largely beyond our influence.

We forecast weather. How many hurricanes will we have this season? Will it rain on the outdoor wedding? Tides are forecast (accurately, but we can’t change them).

A competitor’s sales might be forecast, because we really don’t know their plan.

A sales forecast gets put together, approved, agreed, and entered into the data system.

Then two things happen.

Manufacturing bets the farm on it. They order long-lead parts, establish production plans, set factory capacity. They decide, based on that forecast, how much money is going to be spent, whether anything is actually sold or not. Those decisions often have to be made months in advance.

Meanwhile, all too often, sales has forgotten about the forecast, except perhaps, the top line sales figures. They work hard to sell whatever they can. They push for the big order. They offer the world to prospective customers. They will offer discounts, then push for higher unit volumes to close the dollar targets. Many times they operate on a quarterly (or worse) cycle.. as long as they have a great June, then April and May don’t matter so much.

Meanwhile, back in the factory, when April and May have been dry, they get slammed on June 4th, and end up expediting in parts (at great expense), and working overtime (at great cost), to make product that was sold at a discount.

This is no way to make money.

Let’s get back to what I think is the original issue – the word “forecast” meaning “prediction” (or “educated guess”).

Let’s change one word.
Sales Forecast Plan

That changes the entire meaning.

A “forecast” is a prediction of some event we have little or no control over.

A “plan,” on the other hand, is a set of actions which, if carried out as intended, are predicted to give a specific result. This is a different kind of prediction. This is the kind of prediction that an engineer makes. She analyzes her design, applies her considerable understanding of materials, structure, load transfers, then she predicts at what point that design will fail. If it is a brand new design, it is often tested to destruction (like a new airplane wing). This isn’t to test the design so much as to validate the models used for the prediction.

Sales isn’t engineering, I know that. It involves the most complex thing we know about – human psychology.

The sale planning process goes roughly like this:

  • Financial, margin, volume targets to hit the higher level strategy for profit and growth.
  • What must be sold, when, where to hit those targets. There may be more than one set of options.
  • What must be done to achieve those numbers. This includes consideration for:
    • Unit volumes and mix. (Which are really the only thing the factory cares about.)
    • Total profit targets.
    • The margins that have to be held to hit those profits, at those volume and mixes. (Yes, sales is responsible for margins and profit.. how much money the company can actually keep, not just top line results. “We’ll sell it at a loss and make it up in volume” is not a long-term strategy to stay in business.)
  • Then a process of looking realistically at what must be done, what can be done, deciding on a course of action, and producing a detailed plan to carry it out.

That sales plan then plugs into a production plan. Where there are planned fluctuations, we can apply planned levels of buffer inventory – FIFO inventory, not just make-to-stock inventory, to allow a small time disconnect between when it is made and when it is shipped. This is part of heijunka. (This works in both make-to-order and make-to-stock models, only the mechanics differ.)

Now the entire organization can carry out PDCA.
Are the activities in the sales plan being carried out, as planned, when planned?
If not, why not?
Are they producing the results that were intended predicted? (One-by-one confirmation.) No? OK, what have we learned that we can apply to making a better prediction next time? AND, most critically, What else are we doing to do, because we still have to hit the numbers!

And hit the numbers we must. Not by the end of the quarter. By the end of the month to start. Then in two week increments. Then in one week increments. (And all of this assumes you are making and selling something that doesn’t spoil if it is sitting on the lot for a week.)

The sales plan is the production plan for sales. It is not a guess at how well they will do Just like the manufacturing production plan, it is a firm commitment on how they will support the organization’s overall goals. Yes, reality intrudes and plans rarely get carried off exactly as written. But the thinking that went into making the plan, and the commitment to deliver the results, means the organization, as a whole, is prepared to deal with the unexpected and still stay on track.

Is this idealistic? Absolutely. It is pursuit of perfection. But until the thinking is in place, we will be stuck where we are… waiting for something outside of our control and hoping.

The Value of People

How can some companies not only survive, but thrive when operating in “high cost labor” areas, while others are struggling even as they are busy chasing the lowest possible costs?

I would like to suggest that one key difference is the attitude toward people. On the one hand is the “people as cost” model. This model usually has a couple of built-in assumptions.

  • The number of people required to do a particular task is fixed, often against some kind of earned-hours standard.
  • The cost driver is wages, salaries and benefits.

On the other hand is the (seemingly) rare organization that truly believes that people are their strength, or the well worn out “our greatest asset.”

The assumptions which are required for this belief are:

  1. People’s net productivity can always be improved.
  2. The cost driver is the amount of time wasted coping with all of the small problems that keep things from going perfectly.

The above assumptions, of course, are anchored in a faith-based position that perfection is possible. (see “Chatter as Signal“)

So… what kinds of actions to each of these two models drive?

The first one – people are cost – says to find the cheapest possible labor and hire it. Since factory wages in China are (right now) running about 1/12 or less of those in the USA or Europe, that seems a logical choice. Here’s the rub.

You can outsource the entire job to another company – give the work to the lowest bidder. Now, if you truly believe that the amount of labor is fixed, and that only lower wages can change cost, then this is the obvious choice. You are relying on your superior supply chain management system to ensure you select a supplier that can maintain, and maybe even improve, the quality they deliver, plus hold the line against increases in materials, energy, and their own labor costs. In short, you are looking for a supplier who believes the opposite of what you do. Your ideal supplier knows they can bid aggressively, get your work, and then improve their profit position by applying continuous improvement.

Or you can export the production and set up your own operation in a “low wage area.” You are shifting your core beliefs about people to another culture, and another language. Communication (believe me!!) is a major issue, even if the managers work for you.

AND.. if “labor is cheap” then the solution to problems is to throw people at them. The cost differential you actually get almost NEVER reaches the advantage of a 1:1 substitution. Oh – and you just added 3-4 weeks to your lead / response times.

If, on the other hand, you take the attitude that the most precious resource in your operation is people’s time – no matter what you pay them, and take the attitude that to deliberately waste anyone’s time is to show great disrespect, then I would suggest that even in high-wage areas you can drive levels of improvement in productivity, quality and response to your customers that would be difficult to beat anywhere.

So – before you reflexively outsource or relocate to a “low wage area” please check your attitude about people. What are your expectations, and why is it that you don’t believe your own people are capable of delivering a 10x improvement?

Who are your competitors? What do they do?
What would you do if you had to compete with Toyota? or Komatsu? (to name two that come to mind) They are building product in your back yard, why can’t you?

Afterthought: Some companies end up outsourcing the skills they need to improve their own products and systems. They no longer understand the technology they sell, they no longer know how to make what they sell. I remember a time when I reminded an (arrogant) procurement executive that it was possible to outsource the entire procurement process just as easily. Another team had outsourced all of their direct labor management… they contracted the labor and the first level supervisors into their factory. Where did they really believe this was leading?

Chatter in an ISO Process

I have been in, or encountered, a number of organizations which had (or were working on) ISO-900x quality registrations. While I am fully aware of the intent of the ISO requirements, in the cases I have seen, the effect seems to fall well short of the goal.

On the surface, the types of processes mandated by ISO 9001 seem quite reasonable. They require knowing what your processes are, having documentation for them, and having systems that address problems to root cause.

The requirements are not a lot different from what I mentioned a couple of posts ago in Chatter as Signal. The example of the U.S. Navy’s nuclear propulsion operations would certainly meet the criteria (and then some).

So the question is:
What is fundamentally different about an organization with a “paper only” ISO certification that struggles with chaos every day vs. one which is truly process driven (whether they have an ISO certification or not)?

Chatter as noise.
Chatter as signal.

Chatter as Signal

As I promised, I am going to continue to over-play the afternoon my team spent with Steven Spear.

In his forthcoming book “Chasing the Rabbit” (to be published in the fall), he profiles what is different about those companies which seem to easily be increasing their lead against competitors when there is no apparent external advantage.

One of the core concepts he discussed was the nature of complexity in organizations, processes and products. It is the way this complexity is managed and handled that distinguishes the leaders from the pack of competitors that are fighting and jostling for second place.

In a complex system, there are invariably things people miss. Something is not defined, is ambiguous, or just plain wrong. These little things cause imperfection in the way people do things. They encounter these unexpected issues, and have to resolve them to get the job done.

This is “chatter” in Spear’s words. The sound made when imperfect parts try to mesh together.

Most organizations accept that they cannot possibly think of everything, that some degree of chatter is going to occur, and that people on the spot are paid to deal with it. That is, after all, their job. And the ones that are good at dealing with it are usually the ones who are spotlighted as the star performers.

The underlying assumptions here are:

  • Our processes and systems are complex.
  • We can’t possibly think of and plan for anything that might go wrong.
  • It is not realistic to expect perfection.
  • “Chatter is noise” and an inevitable part of the way things are in our business.

On the other hand, the organizations that are pulling further and further ahead take a different view.
Their underlying assumptions start out the same, then take a significant turn.

  • Our processes and systems are complex.
  • We can’t possibly think of and plan for anything that might go wrong.
  • But we believe perfection is possible.
  • Chatter is signal” and it tells us where we need to address something we missed.

We have all heard about Toyota’s jidoka and andon processes, so let me bring out another example, again, that was used by Spear.

The U.S. Navy has been operating nuclear reactors with a 100% (reactor) safety record for nearly (over?) 50 years. And they operate a lot of nuclear reactors. When they started, they were in totally new and unfamiliar territory – they were doing things that had never been done before. In fact, no one was even sure if it was possible.

They asked the question: How should this nuclear reactor be operated? They answered it with a set of incredibly specific procedures which everyone was expected to follow – exactly, without deviation in any way. These procedures represent the body of experience and knowledge of the U.S. Navy for operating nuclear reactors.

Here is the key point: ANYONE who departs from the procedure, in any way, no matter how trivial or minor, must report “an incident” which rockets up the chain of command. The reasons for the departure are understood. If there was something outside the scope of the procedure, the new procedure covers it. If something was unclear, it is clarified.

This may not be the Toyota Production System at work, but it is a version of something that makes it work: Jidoka.

If the process is not working, can not work, or conditions are not exactly as specified for the process to succeed, then STOP the process, understand the condition, correct it, restore the system to safe, quality operation, and address the reason it was necessary to do this.

Chatter is signal.

So – at a Toyota assembly line in Japan some years ago, I observed a Team Member drop a bolt. He pulled the andon cord and signaled a problem.

Assessing Results vs. Reflection

As we near the end of 2007, most of our respective organizations are looking at what we are going to do in 2008.

Part of that is usually to take a look at this year and look at where we are right now. There are a couple of ways to go about this, and I want to contrast them. This is based only on my own personal experience and, of course, your mileage will vary.

All too often I think this process consists of reviewing results vs. goals. The emphasis is almost solely on targets and actuals. The target was hit or not hit. Top leaders are not interested in “excuses.” I have seen particularly destructive forms of this that included going so far as to re-define success to match what had been achieved. The baseline was re-set at the beginning of the next year, and everything in the past forgotten. Managers took full credit for cost reductions which were “achieved,” not through their own actions, but due to fluctuations in commodity prices of raw materials. Likewise, managers were assigned blame for not hitting targets for the same reason if those prices went up.

There was no review of progress of activities which were predicted to achieve specific results, nor was there a prediction that specific activities would lead to specific results. Instead there was a general high-level target, then a list of actions. Since none of those actions was tied to a verifiable outcome or target, there was no way to know what worked and what didn’t.

Even worse, it really didn’t matter. As long as the targets were achieved, that was what counted. There were great negotiations about exactly how targets would be measured (this company measures everything, and measures nothing). Then, for example, if inventory reductions were to be achieved over the year the actions taken were: (1) Shut down production processes to starve the system. (2) Pull 1Q orders in to 4Q to book the sales. Ships were loaded and sent early because the inventory cleared from the books – even though this was intra-company shipment. They had a LIFO system, so the deeper they could reach into inventory for sales the higher profit they could make since the older the inventory the “lower the cost” associated with it.

All of these games were driven by a “hit the targets and don’t ask about how” mentality. By the way, when 1Q results rolled around things were dismal because they had pulled orders forward PLUS starved the system by shutting down production in 4Q.

This management system is designed intended to deliver results to Wall Street, though it really doesn’t Such is the corrosive nature of trying to manage to “shareholder value” using traditional cost accounting methods. Yes, shareholder value is important, but you can’t manage to it and expect to get the kinds of results that customer and processed focused companies do.

Reflection

Reflection is a learning process. It is designed to incorporate what was learned into shifts in approach for the future. Without it, learning is, at best, an individual action. At worst, the learning is how to survive in the system, not how to do better.

The three key questions are:

  1. What did we intend or plan to accomplish?
  2. What was actually accomplished?
  3. Why the difference?

At a deeper level:

  • Did you accomplished the actions you intended to accomplish? If so, how did that go? What obstacles did you have to overcome? If not, what got in your way that you could not clear?
  • Did each of those actions deliver the expected or planned result? Are you sure? It is just as important to understand why you succeeded as it is to understand why you failed. The commodity price example above is an example of the opposite. They succeeded, but didn’t acknowledge that it wasn’t through anything they did or didn’t do. If an action did not deliver the anticipated result, why not? What did you learn?

Planned? Actual? Please explain.

This is nothing more than the application of PDCA and the Scientific Method. Your plan for the year consisted of a designed experiment. “If we do these things, we expect this results.” Then do that thing, and check that you actually did it. Compare your actual result with the expected result. Explain any difference. Learn.

Adapt, Evolve

I encountered a new level of sophistication in comment spam engines today. This one actually hosts a “blog” of its own. The engine parses quotes from other blogs, posts them as comments in those blogs and links back to itself. On its host site, it looks like a “blog” but, in reality, it is nothing more than a link farm and host for Google ads.

I wrote a note to Google regarding a possible policy violation. In reality, I suppose I wouldn’t mind so much if it just posted things on its own site, but to make me deal with it, and have Google financing it, was a little much.

In an odd ironic twist, the spam filters are the dumb, but automated guardians and the spambots’ algorithms are created by clever people. In this war, people still win pretty much every time.

I suppose I can tie this back to my topic:

In spite of what some would want to believe, the Toyota Production System is not about blind execution of algorithmic standards. It is about continuous evaluation of those standards against a standard of perfection. It is shaped by people, but in ways which are unpredictable except in the macro sense.

As conditions change, the system adapts. As things break, it fixes itself. But all of this only happens if the organization actively works, every day, to ensure that people’s minds are fully engaged doing the right things, the right way.

Training – Critical Questions To Ask

There is lots of “Lean Training” out there, and the quality ranges across the board.

“Lean training” is a megabucks business, and anyone who can assemble a pack of PowerPoint slides and a web site is offering “lean training” out there. It is certainly a case for buyer-beware. So how do you evaluate all of the alternatives, especially if you are just learning and might not be in a position to judge? (Irony: If you are in a position to critically judge these training programs, you probably don’t need them.)

What is being taught and how?
In my experience, most people will readily agree that the tools and artifacts usually associated with the Toyota Production System or Lean Manufacturing are not the system itself. Rather, it is critical for people (and especially leaders at all levels) to understand the thinking behind the tools and artifacts.

The way Toyota teaches the thinking in their new plants is through structured experience. Key leaders are assigned coordinators as mentors. Leaders are taken to established plants to immerse into the system itself. The mentoring continues as the new plant is brought on-line. The process is long, resource intense and expensive. As a result the people who were trained this way are highly sought after in industry. (Another story for the future sometime.) Steven Spear’s article, “Learning to Lead at Toyota” does a great job giving the reader a feel for how this is done. The learning process is entirely experiential.

On the other hand, “talking head lecture” and PowerPoint slides are probably the least effective way to teach this stuff. Even with a couple of simulations with toy trucks or Legos, a classroom-only exercise is only going to get the general concepts across.

If you accept that the real learning comes from guided experience, then it follows to ask if the time spent in the classroom reduces the time required for experiential learning by at least as much. If a week in the classroom (plus the travel time, etc. away from the job) does not return at least two weeks of reduction in the hands-on learning, then it isn’t worth it… no matter how “feel good” it is.

What is the emphasis on direct observation of actual problems? One of the core skills for leaders to learn is how to see problems. If you ask “How much time is spent to watch and understand the work?” the answer you get will tell you a great deal about how well the trainer actually understands the TPS. A high-pressure “kaizen event” especially one which emphasizes just-do-something over first understanding the actual situation – is going to teach exactly the wrong things. Action without understanding results in chaos.

How much of the training involves making actual improvements to actual work? The more the better, but only in the context above.

The classic 5 day kaizen event was originally an educational exercise, and it works very well for this if it is planned and led with learning in mind.

What is the reputation of the teachers? Disregard client testimonials. Ask to speak to some long-term customers. I say long-term because in the initial stages of lean implementation things are pretty easy. A typical medium-sized factory, for example, can get most of the mechanics into place over a few months with aggressive leadership. But if the teachers do not understand (or understand and do not teach) the leadership how to detect, escalate and solve the thousands of problems that will inevitably be flushed to the surface, the implementation cannot sustain for long.

Recognize Reality: The only way to really lean this stuff is to through experience. And not just any experience. Just being told how to implement kanban, fill out the standard work forms, take cycle times, etc. is not learning the things you must know to sustain your gains and build on the initial momentum.

The critical skill – the one that (so far) is only learned through mentored experience – is how to direct actions through guidance and teaching vs. just telling people what to do or how to do it.

What Is Your Takt Time?

If you are the “lean manufacturing expert” you probably know. But what answer do you get if you ask the question in the work area?

Here is a quick diagnostic for you: Go to the shop floor and ask a supervisor, “What is your takt time?”

A reply of “Huh?” is pretty self-explanatory. Either the entire concept hasn’t reached this area yet, or if it has, the day to day variation and disruption renders the concept moot. A couple of follow-up questions can quickly discriminate.

A common reply is the daily output number (e.g. “14 units a day”). Although this shows understanding of a daily production requirement, “14 units a day” does not necessarily translate to “7 units before lunch” much less “a unit every 30 minutes.” This Team Member is still thinking in terms of total output, even if every unit has to be reworked in the last hour of the day. Obviously this is better than reworking every unit in the last week of the month though.

I have also had Team Members do the calculation in their head. They know how to calculate takt time, but don’t use it. This is pretty common when takt time is something that is only a factor during formal kaizen “events” that are run by someone other than the supervisor. I would imagine that standard work is also something that is a “kaizen event thing” rather than daily management as well.

Ideally though, anyone on your shop floor has the takt time embedded in their thinking. If the immediate reply is “28 minutes” then the follow-up question is “how are you doing?” At this point, you can begin exploring how well they use the takt time to manage variation and problems.

What is your takt time?