<aside>
I'm not sure the original post at the start of this thread was the best
set-up for an open exchange of ideas in an environment that is safe for
learning on all sides (after reading a book, someone proposes an idea has
no usefulness and wants to "drive a stake in the heart" of the idea, but is
"interested" in real world experiences).
</aside>

Taking the original request in an optimistic light, I wrote a blog post a
while ago on using control charts for my team [1] (which I copied to this
list) that might provide some information.

Control charts are simply just longitudinal graphs which plot performance
against time. They're just a tool. For example, it occurs to me that many
Agile teams already use a form of them when they graph velocity per
iteration / release. The key, as with any tool, is what to measure and how
to read/use the data.

John Seddon / Vanguard push the use of capability charts [2] for mapping
the end to end time for a service from a customer perspective (covering the
"what to measure" aspect). They value them as a ways of learning about "how
the work works" in terms of what the demand is on a system and how the
system is currently able to respond (covering the "how to read/use the
data" aspect). They have experience applying them in fields such as
housing repairs, where there is a large variability in the type of work and
how long it took (which is one of the challenges in software - the
variation of task complexity and time is high).

I think the problem with the examples presented below of team and developer
effectiveness is that they take an inside-out rather than outside-in
perspective and so don't measure the right things. I think it's better to
view the team as part of the value delivery process and better again to
measure the effectiveness of this process using the customer view.

In my team I'd like to use control charts to show things like how long it
took us to create a new screen for users to price and trade a particular
financial product. Unfortunately the data I have is at the story level
rather than this customer-valued level. However, even with this lower
level of data, the control charts have been useful in the following ways:
* it's demonstrated what we're capable of (e.g. an average story takes six
days from developers starting work until its ready for production, with a
worst case of 15 (upper control limit). We deliver an average of 20
stories (per 10 day iteration with 5 developers) with a max of 50 and a low
of 10).
* it's highlighted the impact of working with other teams, since all of the
points outside our control limits have been around these issues.
* it's shown our sponsors that there is variation in the work that we do
(making it easy to argue against providing single-point in time estimates
of work). It also means that we don't over react when we
* it's shown that the team is broadly under control in terms of it's
process, so there's no point asking us to "be a little better / work a
little harder" (which, sadly, we have heard) without telling us "by what
method". If we want to improve the system then we'll need to make some
changes to our process.

I do think there's a danger of using the control charts to set targets.
Again, the charts are just tools, and in the wrong hands, tools can be
dangerous. John Seddon writes a lot more about the problems with targets,
and command and control thinking in management, in many places, such as
here [3]

1 -
http://benjaminmitchell.blogspot.com/2009/05/control-capability-charts-on-kanban\
.html
2 - http://www.vanguardscotland.co.uk/resources/general/Intro_systems.pdf
3 -
http://www.publicservice.co.uk/pdf/cg/8/CG8%202302%20John%20Seddon%20ATL.pdf

kanbandev@yahoogroups.com wrote on 23/06/2009 18:08:26:

>

> Hi Matt,
>
> This is not a new idea. Watts Humphrey tried to apply the ideas of
> statistical process control to software in the 1990's with his PSP
> and TSP processes, both inspired by Lean/Deming thinking.
>
> I have experience with both of these. But rather then inject my own
> opinion, I would suggest that you take a look at his work and
> related work at the SEI.
>
> Regards,
>
> Paul.
>
> BTW. I have hands-on experience applying swim lanes and other lean
> tools to software development too back in the 19990's. If anyone is
> interested, I'm more than happy to share my experiences.
>
> --- In kanbandev@yahoogroups.com, "chrismatts1968" <chrismatts1968@...>
wrote:
> >
> > Hi David
> >
> > Funnily enough I was discussing this with Nat Pryce at lunch
> yesterday. Nat is the other guy who got the Gordon Pask in London. ;
> -) We are planning something for XP Day. >:-)
> >
> > This is a bit of a ramble as I keep coming back to it during the day.
> >
> > I'm really uncomfortable with this group talking about the whole
> Deming Control Chart / Special Cause / Common Cause thing in
> relationship with Software. I am concerned we are going to send off
> people to engage in pointless and frustrating exercise of creating
> control charts. Once we have some solid evidence that it works, I
> would be happy start promoting it, not yet.
> >
> > First off, I was going to ask Tathaget for more information. How
> did they achieve projects with a tolerance of +/-10% of estimate?
> That is a pretty amazing feat... Todd Little measured actual versus
> estimate and it was a log-normal relationship with a mean of double
> the estimate. So 10% is an amazing feat by comparison. In which
> case, why not carry on doing it.
> >
> > So my problem with using statistical process control in software.
> >
> > Statistical process control is useful to identify cause of
> variation in things that are done again and again. They are used to
> measure variance in things that should be the same.... Like the size
> of widgets which need to be 500 microns +/- 5 microns. Measuring the
> widgets helps us identify why something is bigger or smaller. Is it
> just noise in the process or is there a special cause that needs to
> investigated and resolved.
> >
> > I work in finance and most of our focus is on risk. In finance,
> the worst risk is the one that your do not know about which is why
> we have what is known as a "P&L Explain" process which explains all
> P&L movements caused by known risks. We remove known causes of
> variance in the P&L explain to help us identify other risk. We
> remove the special causes in order to reduce the control limits and
> identify the next level of special cause. I feel I have a handle on
> this process even if I've not worked in manufacturing.
> >
> > So in IT, I can see the use of Control Charts and Common / Special
> cause to analyse batch times / build times and other PRODUCTION IT
> processes that are pretty much ( but not exactly ) the same from day
> to day. We can spot that a release caused a problem etc.
> >
> > Now in manufacturing, the effectiveness of a process is pretty
> much determined by the machine and the process. It is unlikely that
> one machine operator is a hundred times more effective than another.
> The manager can spot from the control chart that one operator is not
> as good as the others and arrange for "training"
> >
> > That is not the case in IT where a developer CAN be 100+ times
> more effective than another. By effectivenees, I mean code quality and
speed.
> >
> > O.K. Now lets look at how we would use control charts and special
> / common cause for software development. Two "obvious" metrics shout
out.....
> >
> > 1. Team effectiveness.
> > 2. Developer effectiveness.
> > 3. I'm happy for others to suggest others.
> >
> > Team effectiveness would be judged based on actual / estimate for
releases.
> >
> > How would we measure and control this?
> >
> > 1. We would have to fix the scope once the estimates are made.
> > 2. We would have to fix the team. Even more silly. No holidays. No
> sick leave. No one leaves, no one joins.
> >
> > So now that we realise releases are too big, we start to make what
> we want to measure smaller and smaller until we have small tasks
> that have no meaning. I believe this is the "problem known to XP
> that Brian mentioned a while ago". So to make this meaningful, we
> now have tasks at the developer level of a day or so.
> >
> > The problem is that we are now ignoring many of the issues that
> affect the effectiveness of the team.... Like holes in the analysis
> where we missed features etc., and wait times between process
> steps... But lets move to developer effectiveness...
> >
> > Developer effectiveness would be judged based on actual / estimate
> for a task.
> >
> > We could track how long it takes for a developer to do a task
> versus the estimate. The task should be fairly well defined by this
point.
> >
> > So what might we learn here. That developer 1 is better than
> developer 2 at estimates, possibly by padding the estimate.
> >
> > What about support? Do the developers do 3rd line support? This
> may be the difference between products and IT depts in businesses.
> Support means a production bug that has to be fixed asap. A silver
> bullet in Kanban COS terms but much faster, A silver laser. Now for
> my figures to be good, I have to record the amount of support time
> being done but worse than that support disrupts flow which also
> reduces effectiveness.
> >
> > ( As an aside, I get developers to indicate when they will finish
> a task on the board Monday to Friday. All tasks are five days or
> less so they wrap. Recently three developers who were good at
> estimating started to "slide of shame". I asked why. It was support.
> We reorganised the way the team handled support. - The team nominate
> a 3rd line support person each day - and they are back hitting
theirtargets. )
> >
> > By now I have a significant management overhead to record and
> clean all this data.... And what will it tell me?
> >
> > Developers should not be allowed to
> >
> > Eat curry at lunch time.
> > Argue with their husbands/wives.
> > Help their colleagues.
> >
> > I now have a really cool Control Chart though that I can use to
> convince management I'm in control, or to use in presentations at
conferences.
> >
> > So I am arguing from ignorance. I have never used control charts
> in software and as far as I know, Tathaget is the only person who has.
> >
> > I remain open to the suggestion that they may be useful. I have
> yet to hear one. Until I have, I will continue to challenge their
> being promoted.
> >
> > Chris
> >
> >
> > --- In kanbandev@yahoogroups.com, David Peterson <david@> wrote:
> > >
> > > Is it just the common cause vs special cause idea that you don'tlike,
for
> > > software, or do you not agree with collecting statistics at all?
> > >
> > > I mean, I kinda agree, it doesn't seem particularly useful to
> > > *control*something that is creative and naturally takes different
> > > amounts of
> > > time. But, maybe the bounds will be wide, but the statistics will
still be
> > > useful?
> > >
> > > I would like to be able to demonstrate, with numbers, how lead-time
has
> > > improved since we started using Kanban or since we made X
> change. How would
> > > you do that?
> > >
> > > I also don't want people to fiddle unnecessarily or jump to
conclusions
> > > about success / failure of changes they make to e.g. WIP limits.How
would
> > > you stop that?
> > >
> > > Thanks
> > > David
> > >
> > >
> > >
> > > 2009/6/18 chrismatts1968 <chrismatts1968@>
> > >
> > > >
> > > >
> > > > Dear All
> > > >
> > > > I have just finished reading Deming's "New Economics". It is a
pretty
> > > > appalling book but that aside, I am now convinced that control
> charts have
> > > > little or no relevance to software development. That said, I am
happy be
> > > > proven wrong.
> > > >
> > > > Has anyone got any experience using control charts? How did
> you use them?
> > > >
> > > > I am not interested in a discussion of control charts, just tohear
from
> > > > people the experience ( positive or negative ) that they have
> had with them.
> > > >
> > > > Depending on the stories, it may be time to drive a stake
> through the heart
> > > > of Control Charts and Common Versus Special Cause of Variance
> in software.
> > > >
> > > > Chris
> > > >
> > > >
> > > >
> > >
> >

>



--------

This communication is confidential, may be privileged and is meant only for the
intended recipient. If you are not the intended recipient, please notify the
sender by reply and delete the message from your system. Any unauthorised
dissemination, distribution or copying hereof is prohibited.

BNP Paribas Trust Corporation UK Limited, BNP Paribas UK Limited, BNP Paribas
Commodity Futures Limited, BNP Paribas Asset Management UK Limited and
Investment Fund Services Limited are authorised and regulated by the Financial
Services Authority.

BNP Paribas London Branch and BNP Paribas Wealth Management London Branch are
authorised by the CECEI and supervised by the Commission Bancaire.

BNP Paribas London Branch is authorised and subject to limited regulation by the
Financial Services Authority. Details about the extent of our authorisation and
regulation by the Financial Services Authority are available from us on request.
BNP Paribas is also a member of the London Stock Exchange.

BNP Paribas Wealth Management London Branch is subject to limited regulation by
the Financial Services Authority. Details about the extent of our authorisation
and regulation by the Financial Services Authority are available from us on
request.

BNP Paribas Securities Services London Branch is authorised by the CECEI and
supervised by the AMF, and subject to limited regulation by the Financial
Services Authority. Details on the extent of our regulation by the Financial
Services Authority are available from us on request. BNP Paribas Securities
Services is also a member of the London Stock Exchange.

BNP Paribas Trust Corporation UK Limited is registered in England and Wales
(registered no. 4042668) at registered office 55 Moorgate, London EC2R 6PA.

BNP Paribas UK Limited is registered in England and Wales (registered no.
1488108) at registered office 10 Harewood Avenue, London NW1 6AA.

BNP Paribas Commodity Futures Limited is registered in England and Wales
(registered no. 2391477) at registered office 10 Harewood Avenue, London NW1
6AA.

BNP Paribas Asset Management UK Limited is registered in England and Wales
(registered no. 2474627) at registered office 10 Harewood Avenue, London NW1
6AA.

Investment Fund Services Limited is registered in England and Wales (registered
no. 6110770) at registered office 55 Moorgate, London EC2R 6PA.

BNP Paribas London Branch is registered in England and Wales (registered no.
FC13447) at registered office 10 Harewood Avenue, London NW1 6AA.

BNP Paribas Wealth Management London Branch is registered in England and Wales
(registered no. FC023926) at registered office 10 Harewood Avenue, London NW1
6AA.

BNP Paribas Securities Services London Branch is registered in England and Wales
(registered no. BR006393) at registered office 55 Moorgate, London, EC2R 6PA.