In an environment of general anxiety, concern about the future, and morbid fascination with charts that show relentlessly rising illness and death, I want to give a view about what we can and can’t take out of the early stages of something that’s largely unknown.
Early Stage Data is Deceptive
In our own small way, we’re very used to seeing early stage data. In these new and usually high growth situations it’s easy to be fooled by the presentation of the data and the early trends.
One example in the current pandemic is the FT’s tracker of total cases and deaths, that shows an exponential growth line, and case numbers going into the tens and hundreds of thousands. Those are log10 scales so they’re going up in steps of orders of magnitude.
This is unmaliciously deceptive, which I’ll illustrate with a few issues that we often see in our world.
1. Some Sources are Unreliable and Many are Inconsistent
The different countries in the chart identify cases with very different levels of rigour and measure them in different ways. So intra country comparisons can at best focus on broad numbers and trends looking at big differences and changes only.
And some countries are less reliable than others. The FT posted on 22 March an implicit criticism of the US regime comparing its totals to China and Iran. The US policy may or may not be poor, but I think it’s a safe general rule of thumb to not quote numbers from autocratic regimes as evidence.
2. Cumulative Numbers Show Accelerated and Relentless Growth
If I wanted to create as much alarm as possible from a set of numbers, I’d plot them cumulatively with exponential trend lines like the FT and many others have done.
Take a look at these charts of (1) total cases and (2) new cases and compare how the UK, Italy and S Korea look in the 2 charts. If you’re like me, you’re a lot less scared by the second chart where the numbers aren’t accumulated from all the previous days, you can see changes a lot more easily, and down is down not up.
We need to measure what’s useful and what gives us insight. I can understand measuring total active cases to plan ICU capacity and scenarios against it. I can understand measuring new cases because we can see what is and isn’t working and whether we’re on an upward or downward trend. I can understand measuring total deaths for context of how critical this all is (more of this later).
But measuring total cases seems like morbid fascination. It’s maybe even psychologically and practically negative as, even when we do things that have an actual effect, the totals by definition just keep going up.
3. Confusing Measurement with Reality
With isolated exceptions, such as cruise ships, the charts we see in the press don’t measure total cases; they measure identified cases. This has a scary effect on growth rates because countries are typically getting better at testing and counting. Early case numbers are very likely understated, and as countries get better at identifying things this accelerates apparent growth.
The flip side of this is that later growth numbers are accurate but the slowdown of growth (and the apparent effectiveness of mitigation measures) will be exaggerated too. So the real growth curve of what’s actually happening will be a lot less curvy than the measured one.
4. The Nature of Lifecycles
Lifecycles of novel things often look exponential early. This is partly because of the measurement we’ve already covered, partly because people with higher propensity succumb sooner, and largely because of the nature of growth from a low base (going from 1 to 2 to 4 in a new space is a lot easier than 32 to 64 to 128 as the space starts getting filled).
How long they stay exponential and the aggressiveness of that exponential is a matter of environment, of action and of time; slowing is inevitable, and decline much more common than not. Let’s have a look at the progression of Italy and South Korea to see this.
For Italy in the left hand chart, I’ve shown the best fit line using data just from the first week, then the first two weeks, first three weeks, and finally the first four weeks, which ended on March 22nd. As you can see, the trend gets continually less aggressive. The type of best fit also changes in week 4 from those scary exponentials to a less frightening polynomial. Looking just at the data, Italy is still trending up but is near or at the brow of the hill.
It’s also worth noticing that Italy’s lock down started in some places but not all on day 9 with an expectation of two weeks for effects. While it has likely been effective, the rate of case growth was already slowing by then.
South Korea, which started exponential, a few days ago was a declining (wide n-shaped top of the hill) logarithmic, but is now a more slowly declining exponential flattening of the other side. You should also notice the axes, where South Korea’s case load is one tenth Italy’s with a fairly similar population but younger demographic.
These stages in the lifecycle illustrate why earlier action to suppress the fast growth exponential makes more difference than later action once you’re up the exponential and into later stages. The South Korea line looks very similar to the Italy one in its early stages before South Korea starting acting heavily. But they also show that exponential growth lasts a limited time before it slows, even for late actors.
5. Numbers are Hard to Understand Without Context
The worst hit country whose numbers we can believe is Italy, where more than 6,000 people’s families have been hit with tragedy as I write this, and where the numbers could easily, easily quadruple.
Here’s some other tragic numbers from Italy to put the COVID-19 deaths in context.
For UK folk, our COVID numbers are less mature but our recent lockdown reflects our government’s desire to learn from Italy and hopefully suffer less. For the UK, I’ve added in a bar of 20,000 deaths, which in the early stages our Chief Medical Officer said would be a good result.
This pandemic is awful, and we’re quite rightly highly sensitised to it because of ICU capacity and lives cut short. But erring on the side of overstatement also has its costs, such as families anxiously over buying because they think they may run out of food, which deprives the very vulnerable people who are most likely to be hit hard by the virus and who need to stay as healthy and condition free as possible.
Early Stage Models are Wrong
As someone whose scientific qualification and very first job was modelling extreme events with 20, 50 and 100 year likelihoods, I’d venture from personal experience that all the early theoretical models will be wrong.
We do still need to model scenarios in early stages, based on conceptual frameworks and best guesses of parameters that use the best historical data we have. We can’t just wait because we need to plan.
However, modelling complex systems from first principles is a fool’s errand, and the more extreme the event and the more totally wrong will be your first model. So we need to square the circle by calibrating against what’s actually happening, changing the parameters or the entire model, and reforecasting. Then we need to recalibrate the next day with the next set of data and keep doing that until we can predict with some workable confidence intervals. In many western countries, we look like we’re still more than a week away from those workable confidence intervals. Of course we need a pessimistic case as well as a central one but our cases need to be continually informed and updated.
So if anyone is criticising Imperial College’s early, frightening estimates of the outturn, then they’re being unduly harsh. If they’re still using them to predict the outturn, then they’re being foolish.
Most of us aren’t advising our government on pandemic management strategy, so what’s on our minds is our satisfaction and degree of compliance with the rules now being imposed on us, how we can plan for our families and livelihoods with an unknown territory in front of us, and how much we should be worried as we see upward moving plots of cases and deaths and pictures of overflowing ICUs.
We’re still in the exponential stage in the UK and US, so each effort we make now to suppress growth counts a lot, much more than the effort we make in a week’s time, and we will be able to relax sooner. The 23 March tightening of the UK lock down seems like bitter medicine to many of us, but will have more effect now than later.
And we should look at the published plots of cumulative exponential case and death growth with a careful and less gullible eye, stop equating higher earlier stage growth automatically to bad management, and keep updating our perception and expectations as the data rapidly emerges.
It’s bad, but it’s far from the end of days, and we’d all be better off replacing the alarmism that comes from believing the early stage models and trends with an updated and better informed view as those trends mature. And when the pubs reopen, I’ll be doing my bit to get the hospitality sector back on its feet.
by Steve Hacking