I’ve just re-read Douglas W Hubbard’s The Failure of Risk Management.  It’s an odd book in that while I agree with most of what’s in it, I’m not particularly convinced by the overall story suggested by the subtitle: Why It’s Broken and How to Fix It.

Hubbard’s theme is that we do not do enough, or good enough, quantitative risk modelling.  We need to throw out the consultant’s snake oil of ERM and build risk management around improved models.  In order to improve the models we need to look harder for data and we need to calibrate the domain experts we use to provide more ‘accurate’ probabilities.

While there is much that is right with this, I think there are two things that are wrong.  Firstly, he doesn’t provide a process whereby the risk model can be created using the calibrated experts.  Secondly, he confuses risk analysis for risk management.  It’s worth saying these criticisms are slightly overstated given that this is a well-written, knowledgeable and witty exposition.  What I particularly welcome is his expose of the many fallacious positions taken and arguments made by people in the risk game.  A good example is the way people use any slight shortcoming in risks analysis (they’re easy to find) as an excuse for wanting to bin the whole lot and go back to examining entrails.  For this alone it should be required reading for anyone in risk.  However, some of his opinions are pretty partial with no real attempt to understand why some of the practices that he abhors have arisen.  This is what leads me to part company with him to the extent I do.  He doesn’t entirely avoid his own fallacies either.

Hubbard splits the risk analysis community four ways: actuaries, with real and relevant data and an underlying model which is well-adapted to probabilistic modelling; war quants who are committed to probabilistic modelling of the relevant phenomena in whatever way is required; economists, who put a normal distribution on everything so they just have to figure out the mean and standard deviation (and also have some funny ideas about risk); and management consultants who purvey the 5×5 matrices and silly scoring and traffic light schemes which have come to be synonymous with much of risk management.  He has hard words for these last, of course, and I can’t fault him for the way he discredits many aspects of this approach.  This is a growing trend and one which I hope will lead to a drastic reassessment of much of ‘best practice’.

However, from the point of view of risk management, and not just risk analysis I think there is value in maintaining lists of risks and what is planned to be done about them, and how we are monitoring the situation.  (For an opposite view, it’s worth reading Matthew Leitch’s gloriously over-the-top rubbishing of risk lists.)

Hubbard joins other writes in blaming the financial crisis on the economists’ love of the bell curve.  While this clearly led to oversimplified models, with drastically understated downsides, I think the real cause can be found less in the flawed assumptions made and more in the nature of complex, tightly-coupled systems.  This is important because we trying to fix things with better models and more regulation.  This already has the undesirable effect of reducing lending.  And if we are trying to prevent future collapses by more regulation we are taking a big risk if this just increases the complexity of the system.  If we want a probabilistic model we need something that emulates this.

This is an example of Hubbard falling into one of his own traps.  We all know the syllogistic fallacy which runs “X didn’t keep a risk register”, “X screwed up”, “Therefore he wouldn’t have if he’d kept a risk register.”  (Read any book on reputational risk management, for example.)  Hubbard thinks the financial crisis would have been avoided if they had developed well-calibrated risk models.  I don’t.

War quants are the heroes.  They are called this because they grew in importance during the second world war and the rise of OR and decision theory.  Here we have the people who are committed to probabilistic modelling and making sure each material issue is properly included.  They recognise interdependence and they don’t just model it with a gormless @RISK correlation function.  They think through how it actually works.  However Hubbard thinks they are less than fastidious about looking for ways to show that their modelling is right and is useful, for example by finding relevant data and using calibrated experts.

I am one war quant for whom these criticisms strike home.  I never bother with calibration.  I think most data is not relevant.  In fact I think I have an alternative approach in which I facilitate a team into reaching agreement on a probabilistic model, taking account of data and eliminating biases as we go along.  This sounds like reaching consensus and Hubbard is very hard on consensus as a desirable property of risk analysis.  Let’s explore this further.

Hubbard’s book is slightly dumbed down from the technical perspective.  However he is clearly in the camp of maximizing expected utility based on a set of subjective probabilities.  A popular, but rigorous, account of this is given by Dennis Lindley in his book Uncertainty.  Lindley takes us through all the elicitation questions you ask to establish the subject’s probability and utility.  I emphasise ‘subject’s’ because Lindley is very careful to say this only works for individuals: every probability is some individual or others personal or subjective probability expressing their degree of belief that an event a sample case will turn out to be realised.

Lindley makes great play of the point that an important outstanding problem for decision science and society is to generalise this to groups of people: a Nobel awaits.  Hubbard’s solution is, I think, for the analyst to decide, for each probability, whose subjective estimate is to be used.  Mine is to sit around discussing with the team and I think that is why I would value consensus more than Hubbard appears to.  To be fair we both agree on the importance of sanity checking the model, not allowing it to become the truth, and recognising the underlying assumptions.  This rationalisation also explains why Hubbard puts a lot of emphasis on calibration to tackle overconfidence bias.  It’s a necessary precaution if you are dependent on individuals for your numbers.  As a further comment, although Hubbard makes great play of the various well-known Tversky/Kahneman biases in estimation, it is only overconfidence he tackles in detail with the calibration.

Hubbard has a lot of calibration exercises aimed both at probability estimation and also the estimation of confidence ranges.  This gave me a lot of food for thought.  For example one of the probability examples is about a TV programme I have never heard of.  Supposedly my answer to a true or false question about it should have 50% confidence as a result.  But to me this expresses concepts that go far beyond any belief I have on the topic.  Another example on confidence intervals relates to the steam train speed record.  As a result of a sad youth I know – with certainty – the answer is 126 mph.  How do I get a 90% confidence interval from that?  I could say, well it must be between 125.5 and 126.5 so the interval is (125.55, 126.45).  Or I could recognise that I could be wrong – it happens! – let’s say the chance being wrong is 5% (the lowest possible non-zero probability) so the interval becomes (125.525, 126.475).  It’s all nonsense and reminds me of my greatest objection to Lindley’s theoretical perfection: what happens if the expert tells him to sling his hook with his effing stupid questions?  It doesn’t work.

I think there are uncertainties which lie beyond degree of belief probabilities.  Black Swans are one example and Taleb’s books are all about how we need to get beyond probabilistic thinking.  Hubbard’s well calibrated expert is Taleb’s lucky fool.  (Incidentally Hubbard takes time out to heap both praise and derision on Taleb.)  Another example is probably climate change.  While the calibration questions are not necessarily good examples, they at least underline the point.  I’ll return to this in other posts no doubt.

Data is a tricky one too.  First of all I have a built in reaction to the idea that risk analysis is about collating data.  It isn’t, you are bound to be thrown back on subjective probabilities which better represent the knowledge and experience of those contributing to the model.  Hubbard doesn’t disagree with this; it’s just a question of the balance.  Secondly, no data is directly relevant: the size is different, the technology is different, the management is different, and so on.  Maybe, like in human reliability, you can formally recognise these things via performance shaping factors.  I’d prefer to have the data on the table during the assessment, discuss what it tells us and get to the probabilities.  I don’t like dividing the number of cock-ups by the number of missions just like that.

What’s more, it’s sad fact that you are often time and resource limited.  Hubbard makes a good case for allocating a lot more resources to risk modelling than most of my clients do.  So until they change their mind I aim to do the best job with what resources are at my disposal.

At this point it’s worth mentioning a very interesting idea of Hubbard’s relating to the value of information.  You work out the expected loss (conditional on there being a loss) of your decision.    For each variable you see how much this is reduced if you knew the exact value of the variable and work out the expectation under the further condition that the decision would not be changed.  This gives you some idea of how much to pay to get information about the variable and is also a possible alternative to the usual importance measures using (ranked) correlation coefficients (aka tornado diagrams!).  Hubbard is actually quite unclear on his method; perhaps it is a trade secret.  He has an online  website for the book with further information and examples.  He promises it contains a worked example of the technique.  Unless I’m missing something it doesn’t, and the account I’ve given here is to some extent my own interpretation.  It’s worth following up on, but it does seem to be quite based on maximising utility decision criteria and knowing the swing value of the variables.

Let’s finish by thinking about risk management more broadly.  Hubbard’s world view is one in which risk analysis lies at the core and the rest is just dressing.  There’s a logic to this if you focus on decision-making.  But I think the real focus of risk management is cultural and managerial.  The big challenges are ensuring you are risk aware as an organisation and ensuring you do what you need to (and decide to) to promote good outcomes.  Hubbard’s treatise is an excellent description of how to improve probabilistic risk modelling.  (And it’s very sound on risk appetite, emphasising it comes down to choice as to where to be on the efficient risk-reward frontier.)  But it is a long way from sorting out organisational risk management.