Wednesday, 19 July 2017

How to talk about climate doomsday scenarios

David Wallace-Wells wrote a 7000 word cover story in New York Magazine on how unchecked climate change may make the Earth uninhabitable. With 2.5 million readers this longread was the most read article in the magazine's history.

That shows the impact a well written article can have. It also point to a change in the mood in America. Since Trump pulled out of the Paris Agreement, happy to risk climate rapture for the quarterly earning of his donors, attention for climate change in America has spiked.

Americans are living through a nightmare, where you see the danger coming, but cannot convince others to stand up to it. Until you wake up bathing in sweat and pick up your phone from the night stand to read tweets from climate "sceptics" mocking you for facing reality. The same people that make a nightmare out of a perfectly solvable problem.

Sixteen climate scientists of Climate Feedback (including me) reviewed the NY Mag article. This number of reviewers may also be a record.

My summary would be that while the dangers of unfettered climate change are real, we found many inaccuracies, which typically exaggerated the problem. Thus the article was rated as having a "low scientific credibility". Both the NY Mag article and the Climate Feedback rating and earlier criticism by Michael Mann have sparked some controversy.

This post will be about how I would prefer the media to report on worse case scenarios. An second post will be about whether our "nitpicking" was the science police striking again?

I have no idea how 2100 looks like. Put yourself in the place of a well-informed citizen of 1900 and what they may have thought today looks like. Meters of horse shit on the streets due to the growth of traffic? Or imagine how an American thought this time would look like 50 years ago. Flying cars and Mars colonies?

Maybe in 2100 humanity has gone extinct, maybe civilization is gone, maybe humans are enslaved by corporations, maybe currently poor countries are also affluent and corporation can no longer repress us, maybe after another century of development we lead wonderful lives, maybe we are building our first intersolar cruiser, maybe no one cares about intersolar cruisers and people impress each other with poetry and four dimensional chess. Very likely they will be painfully embarrassed for me for the options I gave.

I have no idea how they view climate change in 2100. Do they see it as the biggest historical liability put on them? Are they annoyed at the tax burden for the huge necessary geo-engineering program? Do they wonder why people in 2017 thought it was such a big problem, while it was so easy to solve? Are they happy that due to the geo-engineering program they now have weather satellites and it only rains at night in urban areas?

Even in the best case scenario we are taking the climate system out of known territories. There will be many surprises and to be honest those are what worry me the most. The Uncertainty monster is not our friend and that makes it very hard to say which worst case scenarios are unrealistic.

It is custom to accept smaller risks the bigger the stakes are. Cars and smoking kill many people, but one at a time. A risk someone may be willing to take personally will be larger than the risk one takes with a community, a country or our civilization. The risk of dying in a car accident is 1 in 84 (1.2 %), this would be an unacceptable risk for civilization or humanity. Thus we have to look at the tails.

Finally, we expect the impacts of climate change to accelerate: Because some variability is normal, the first degree of warming makes much less problems than the next. Thus the risks of above average warming are expected to contribute much to the total risk. It is thus good that the article explores what surprises may be in store and talks about scenarios that are not likely, but risky.

Four horsemen of the apocalypse

The article reaches the worst case scenarios in four ways:
  1. The worst case for the emissions of greenhouse gases.
  2. The worst case for how sensitive the climate system responds.
  3. The worst case for the impacts and how humanity responds.
  4. The worst case for the scientific assessment of the evidence.
1. The worst case emission scenario was the [[RCP8.5 scenario]] of the IPCC. These scenarios are really just that: scenarios. No probability is assigned to them.

This is the IPCC report from 2013 and the scenarios were created well before that. My impression is that with the [[Paris climate agreement]] and the fast drop in the prices of renewable energy and storage, the RCP8.5 scenario is no longer very realistic. Another optimistic sign is that industrial emissions have stabilized the last 3 years. However, the US mitigation sceptical movement and their president will keep on fighting to make this dystopia a reality. So it is a legitimate question what kind of a world fossil fuel companies and these people want to create.

2. It is completely legitimate to explore the tails of the probability distribution of the climate sensitivity. Even if it had only 30% probability, Trump did get elected. Even if the chance is just one out of six, you sometimes role a six. And let's not start about Russian roulette. Unfortunately the tail of the distribution is not well constrained and very high sensitivities are hard to exclude.

3. The uncertainty of some impacts can be quantified reasonably well. These are the ones with the most physics in them such as heat waves and large-scale increases in precipitation. Then it is legitimate to go into the tail.

Other impacts are not understood well enough yet (Will ice sheets collapse? How much greenhouse gasses will the soil release due to heating?) or will never be fully predictable because of societal and technological influences (Will The Netherlands evacuate before or after Noah's flood? Will plant breeding keep up with climate change? Will societies be able to cope with climate refugees?). In such cases I would like to hear a balanced spectrum of views, including extreme ones.

Because the broad sweeping article discussed many climate change impacts it could not do justice to complexity of individual impacts. Climate change is typically just one stressor of many. When The Netherlands floods, the climate "sceptics" will not suddenly wake up and say "silly me, I was wrong, now I recognize that climate change is a problem, sorry about that". They will say the storm was to blame and it was really bad luck the storm came from the North West and its maximum coincided with high tides, the dikes were not strong enough, the maintenance not good enough and especially the government is to blame.

Looking at history or at the future only from a climate change perspective brings back bad memories of [[climate determinism]]. The Age recently reported on farm workers in Central America suffering and dying from chronic kidney disease. The regions where this new decease happens is well predicted by warming and changes in insolation. Simultaneously the problem is that these people are so poor that they have to work on hot days and also have a strong work ethic that promotes this. They tend not to drink during work and when they do it are often soft drinks because they are perceived as saver. A large part of the problem is funding for preventative care and people die because they cannot afford dialysis.

This example shows two things. First of all, like the dikes breaking in The Netherlands, the problem has many aspects. Secondly, this was a problem because it was new. There will be many surprises due to climate change. The study of (rare) diseases helps in understand how a healthy body works. It shows what is important for healthy functioning. Medicine can study many bodies, we only have one Earth and will very likely be surprised what the Earth did for us without us realising it.

4. Like for non-physical impacts, where I am hesitant to go into the tail is when it comes to the interpretation of the evidence. That quickly ends in cherry picking experts that say what you want to hear. Those are strategies for mitigation sceptics. Even if those experts do not stray from the evidence and only hold a pessimistic view; I feel this is not for serious science reporting. It is fine to explain the ideas of such experts, but they should be balanced with other views.

Concluding, for the objective part of the problem: if you clearly say you are looking at the worst case feel free to go deep into the tail of the probability distribution. Only looking at mean changes understates the risks. The tail is a big part of the risk and thus very important. Do not forget to talk about many further surprises and that the Uncertainty Monster has an ugly bite.

When it comes to the more subjective parts, please balance pessimistic with optimistic voices. Subjective judgements are unavoidable when it comes to worst case scenarios and the far future where the changes will be largest. People can legitimately have different world views and as a science nerd I would like to hear the full range of legitimate views.

An article needs a focus, but please consider that climate change is one stressor of many. Climate change impacts are complicated, do them justice like a great novelist would and do not make a cartoon out of them.

Related reading

The updated New York Magazine piece By David Wallace-Wells: The Uninhabitable Earth - Famine, economic collapse, a sun that cooks us: What climate change could wreak — sooner than you think. (The reviewed original, with annotations)

The Climate Feedback Feedback: Scientists explain what New York Magazine article on “The Uninhabitable Earth” gets wrong.

New York Magazine now also published extended interviews with the scientists interviewed for the piece: James Hansen, Peter Ward, Walley Broker, Michael Mann, Michael Oppenheimer.

Michael E. Mann: The ‘Fat Tail’ of Climate Change Risk

Fans of Judith Curry: the uncertainty monster is not your friend

Introduction to Climate Feedback: Climate scientists are now grading climate journalism

Michael E. Mann, Susan Joy Hassol and Tom Toles in the WP: Doomsday scenarios are as harmful as climate change denial. Good people, but I am not buying it: one negative journalistic story in a full media diet does not make people despair, hopeless and paralysed. Plus reality is what it is.

* Painting of the Four Horsemen by Viktor Vasnetsov -, Public Domain, Link

Monday, 10 July 2017

An ignorant proposal for a BEST project rip-off

It looks like the "Competitive Enterprise Institute" (CEI) just conned their dark money overlords with a stupid report rehashing all the same old claims of the mitigation sceptical movement the BEST project of Richard Muller already studied as a Red Team.

Conservative physics professor Richard Muller claimed that before his BEST project he "did not know whether global warming was real, was completely bogus or may was twice as bad as people said". He was at least open to all sides.

Joe D’Aleo, co-author of the CEI-affiliated report, made the embarrassingly uninformed and wrong claim that “nearly all of the warming they are now showing are in the adjustments.

The report "On the Validity of NOAA, NASA and Hadley CRU Global Average Surface Temperature Data & The Validity of EPA’s CO2 Endangerment Finding - Abridged Research Report" (with annotations) by James P. Wallace III, Joseph S. D’Aleo, and Craig D. Idso provides no evidence for this claim; the graph above shows the opposite is true.

They don't do subtle. But really? You want to claim the Earth is not warming? In 2017?

Glaciers are melting, from the tropical [[Kilimanjaro]] glaciers, to the ones in the Alps and Greenland. Arctic sea ice is shrinking. The growing season in the mid-latitudes has become weeks longer. Trees bud and blossom earlier. Wine can be harvested earlier. Animals migrate earlier. The habitat of plants, animals and insects is shifting poleward and up the mountains. Lakes and rivers freeze later and break-up the ice earlier. The oceans are rising.

Even without looking at any thermometer data, even if we would not have invented the thermometer, physics professor Muller was not sure the Earth is warming? Some corporate lobbyists of the CEI claim the Earth is hardly warming? Really? And the same group of people like to say scientists should get out of the lab more often.

Richard Muller explained in the New York Times the main objections of the mitigation sceptics, which he studied and the CEI wants to study:
We carefully studied issues raised by skeptics: biases from urban heating (we duplicated our results using rural data alone), from data selection (prior groups selected fewer than 20 percent of the available temperature stations; we used virtually 100 percent), from poor station quality (we separately analyzed good stations and poor ones) and from human intervention and data adjustment (our work is completely automated and hands-off). In our papers we demonstrate that none of these potentially troublesome effects unduly biased our conclusions.
In the end Muller and his team found:
Our results turned out to be close to those published by prior groups. We think that means that those groups had truly been very careful in their work, despite their inability to convince some skeptics of that. They managed to avoid bias in their data selection, homogenization and other corrections.

The CEI report carefully avoids any mention of the BEST project. If fact it avoids any mention of previous studies on their "issues". That could be because they are uninformed henchmen, because they want to con their even dumber sponsors or because they want to deceive their friends and keep the public "debate" going on ad nauseam.

If they were real sceptics they would inform themselves and if they do not agree with a claim respond to the arguments. A scientific article thus starts with a description of what is already known and then puts forward new arguments or new evidence. Just repeating ancient accusations, ignoring previous studies, does not lead to a better understanding or a better conversation.

Global mean temperature estimates

Before going over the main mistakes of the report, let me explain how much the Earth is estimated to have warmed, why adjustments need to be made and how these adjustments are made.

The graph below shows the warming since 1880. The red line is the raw data, the blue line the warming estimate after adjustments to account for changes in the way temperature was measured. Directly using raw data, the warming estimate would have been larger. Due to adjustments about 10% of the warming is removed.

This would be a good point to remember that Joe D’Aleo wrong claimed that “nearly all of the warming they are now showing are in the adjustments.”  It is really really hard to be more wrong. Joe D’Aleo gets points for effort.

The main reason why the raw data suggests more warming is how sea surface temperature was historically measured. The ocean surface warming estimates of the UK Hadley centre are shown below. The main adjustment necessary is for the transition of bucket observations to measurement at the engine cooling water inlet, which mostly happened in the decades around the WWII. The war itself is an especially difficult case.

Bucket measurements are made by hauling a bucket of water from the ocean and stirring a thermometer until it has the temperature of the water. The problem is that the water cools due to evaporation between the time it is lifted from the ocean and the time the thermometer is read.

This is not only a large adjustment, but also a large uncertainty. Initially is was estimated that the bucket measurements were about 0.4 °C colder. Nowadays the estimate, depending on the bucket and the period, is about 0.2 °C, but it can be anywhere between 0.4 °C and zero. We studied these biases with experiments on research vessels, in labs and numerical modelling and by comparing measurements made by different nearby ships/platforms.

The graph below shows the warming over land as estimated from weather station data by US NOAA (GHCNv3). Over land the warming was larger than the raw observations suggest. The adjustments are made by comparing every candidate station with its nearby neighbours. Changes in the regional climate will be the same in all stations, any change that only happens at the candidate station is not representative for the region, but likely a change in how temperature was observed.

There are many reasons why stations may not measure the regional climate correctly. The best known is urbanization of the local surrounding of the station. Cities are often warmer than the surrounding region and when cities grow this can produce a warming signal. This is a correct measurement, but not the large-scale warming of interest and should thus be removed. The counterpart of urbanization is that city stations are often moved to the outskirts, which typically produces a cooling jump that also needs to be removed. This can even be important for small villages.

City stations moving to cooler airports can produce an artificial cooling. Also modern equipment generally measures a bit cooler than early instruments.

Where the CEI report gives examples of data before and after adjustment, do you want to guess whether they showed the sea surface temperature or the land surface temperature?

Sea or land? What do you think? I'll wait.

If you guessed the land surface temperature you won the price: a free twitter account to tell the Competitive Enterprise Institute what you think of the quality of their propaganda.

The ocean is 71% of the Earth's surface. Thus if you combine these two temperature signals taking the area of the land and the ocean into account the net effect of the adjustments is a reduction of global warming.

The Daily caller interviewed D'Aleo and calls the report a "peer-reviewed study". Suggesting that it underwent the quality control by scientists with expertise in the field that is typical for scientific publications. There is no evidence that the report is published in the scientific literature and the blog science quality, lack of clarity how the figures were computed and where their data comes from, the lack of evidence for the claims, the lack of references to the scientific literature makes it highly unlikely that this work is peer reviewed, to say it in a friendly way. There is no quality bar they will not limbo underneath; they don't do subtle.


The estimation of the climatic changes at a station using neighbouring stations to remove local artefacts is called statistical homogenisation. The basic idea of comparing a candidate station with its neighbours is easy, but with typically multiple jumps at one station and also jumps in the neighbouring stations it becomes a beautiful statistical problem people can work on for decades.

Naturally scientists also study how well they can remove these artefacts. It is sad this needs to be mentioned, but the more friendly blog posts of the mitigation sceptical movement (implicitly) assume scientists are stupid and don't do any due diligence. Right, that is how we got the moon and produced smart phones.

Such a study was actually how I started with this topic. The homogenisation community needed an outsider to make a blind benchmarking of their methods. So I generated a dataset with homogeneous station data where you need to get the variability of the stations right and the variability (correlations) between the stations. As the name of this blog suggests just the job I like.

To this homogeneous data we added inhomogeneities. For me that was the biggest task, talking with dozens of experts from many different countries how inhomogeneities typically look like. How many (about one per 20 years), how big (about 0.8 °C per jump), how many gradual inhomogeneities and how big (to model urbanization), how often do multiple stations have a simultaneous jump (for example, due to a central change in the time of observation).

I gave this inhomogeneous station dataset to my colleagues, who homogenised it and, after everyone returned the data, we analysed the results. We found that all methods improved the quality of monthly temperature data. More importantly for us was that modern homogenisation packages were clearly better than traditional methods. The work of the last decade had paid off.

The figure to the right is from a similar blind validation study for the homogenisation method NOAA used to homogenise GHCNv3 and shows something important. The four panels are four different assumptions about how inhomogeneities and the climate looks like. This study chose to make some inhomogeneity cases that were really easy and some that were really hard.

On the horizontal axis are three periods. The red crosses are the trends in the inhomogeneous data, the green crosses the ones in the homogeneous data, which the homogenisation algorithms are supposed to recover and the yellow/orange crosses are the trends of the homogenised data.

The important thing is that the yellow cross is always in between: homogenisation improved the trend estimates, but part of the error in the trend remains. In the most difficult case of this study, which I consider unrealistic, the homogenised result was in the middle. Half of the trend error was removed, half remained.

Because real raw station data shows too little warming and statistical homogenisation makes the trend larger, better homogenisation thus also means stronger temperature trends over land. Homogenisation became better because of better homogenisation algorithms and because we have more data due to continual digitisation efforts. With more data, the stations will on average be closer together and thus experience more similar weather. This means that it becomes easier to see homogeneities in their differences.

CEI claims from Daily Caller

Michael Bastasch of the Daily Caller makes several unsupported or wrong claims about the report. Other claims are already wrong in report.
A new study found adjustments made to global surface temperature readings by scientists in recent years “are totally inconsistent with published and credible U.S. and other [New Zealand and upper air] temperature data.”
No shit, Sherlock. Next you will tell me that cassoulet does not taste like a McDonald’s Hamburger, sea food or a cream puff. The warming of different air masses is different? Who would have thought so?

This becomes most Kafkaesk when the authors want to see the high number of 100 Fahrenheit days of the 1930s US [[Dust Bowl]] in the global monthly average temperature and call this a "cyclical pattern". Not sure whether a report aimed at the Tea Party folks should insult American farmers and claim they will mismanage their lands to produce a Dust Bowl in regular cycles.

The report is drenched in conspiratorial thinking:
Basically, “cyclical pattern in the earlier reported data has very nearly been ‘adjusted’ out” of temperature readings taken from weather stations, buoys, ships and other sources.
They do not even critique the methods used or even mention them and do acknowledge that adjustments are necessary, but the pure outcome being inconvenient for their donors is enough to complain.

It also illustrates that the mitigation sceptical movement is preoccupied with the outcome and not with the quality of a study. Whether a new study is praised or criticized on their movement blog Watts Up With That depends on the outcome, on whether it can be spun as making their case against solving climate change stronger or weaker. On science blogs it depends on the quality and the strength of the evidence.

As I already showed above, the adjustments make the estimated warming smaller. The exact opposite is claimed by the Daily Caller:
In fact, almost all the surface temperature warming adjustments cool past temperatures and warm more current records, increasing the warming trend, according to the study's authors.
The study provides no evidence for this. They do not show the warming before and after adjustment for the global temperature, only for the land temperature.

Is it too much to ask to inform yourself before you accuse scientists of wrongdoing? Is it too much to ask if you write a report about the global temperature to read some scientific articles on data problems in the sea surface temperature? Is is too much to ask if you talk about the 1940s to wonder whether the WWII might have influences the measurements?
“Each dataset pushed down the 1940s warming and pushed up the current warming.”
The war increased the percentage of American navy vessels, which make engine intake measurements, and decreased the percentage of merchant ships, which make bucket measurements. That produces a spurious warm peak in the raw data.

Modern data also have a better coverage over the Earth. Locally there is more decadal variability, what they call "cyclical pattern". A better coverage will remove spurious decadal variability from the global average.

I have not clue why they would think this:
“You would think that when you make adjustments you’d sometimes get warming and sometimes get cooling. That’s almost never happened,” said D’Aleo, who co-authored the study with statistician James Wallace and Cato Institute climate scientist Craig Idso.
The transitions in the measurements methods due to technological and economic changes can naturally affect the global average temperature. For example ships in the 19th century used bucket measurements, now most sea surface temperature data comes from buoys.

If you assume inhomogeneities can have no influence on the global mean, like D'Aleo, then why are the mitigation sceptics claiming to be worried about the influence of urbanization on the global mean temperature? If that were the main problem, the adjustments would tend to produce cooling more often than warming to remove this problem. They would not "sometimes get warming and sometimes get cooling".

The report was an embarrassing mixture of the worst of blog science. The Daily Caller post managed to make it worse.

The positive side of Trump claiming that his inauguration was the biggest evah, is that the public now understands were such wild claims come from. Science is harder to check than crowd sizes. Even if you do not know them personally, there are people on this globe willing to deny the existence of global warming without blinking an eye.

Related reading

Quality of climate data

The climate scientists of Climate Feedback had a look at an Breitbart article on the same report. Seven scientists analyzed the article and estimate its overall scientific credibility to be 'very low'. Breitbart article falsely claims that measured global warming has been “fabricated”.

Fact checker of urban legends Snopes judged the Breitbart article to be: False. Surprise. Had Breitbart known it to be true, they would not have published it.

Ars Technica: Thorough, not thoroughly fabricated: The truth about global temperature data. How thermometer and satellite data is adjusted and why it must be done.

John Timmer at Ars Technica is fed up with being served the same story about some upward adjusted stations every year: Temperature data is not “the biggest scientific scandal ever” Do we have to go through this every year?

Steven Mosher, a climate sceptic and member of the BEST project: all the adjustments demanded by the "sceptics".

The astronomer behind And Then There's Physics writes why the removal of non-climatic effects makes sense. In the comments he talks about adjustments made to astronomical data. Probably every numerical observational discipline of science performs data processing to improve the accuracy of their analysis.

Nick Stokes, an Australian scientist, has a beautiful post that explains the small adjustments to the land surface temperature in more detail.

Two posts of mine about some reasons for temperature trend biases: Temperature bias from the village heat island and Changes in screen design leading to temperature trend biases.

You may also be interested in the posts on how homogenization methods work (Statistical homogenisation for dummies) and how they are validated (New article: Benchmarking homogenisation algorithms for monthly data).

Just the facts, homogenization adjustments reduce global warming.

Zeke Hausfather: Major correction to satellite data shows 140% faster warming since 1998.

If you would like to read a peer reviewed scientific article showing the adjustments, the influence of the adjustments on the global mean temperature is also shown in Karl et al. (2015).

NOAA's benchmarking study: Claude N. Williams ,Matthew J.Menne, Peter W. Thorne, 2012: Benchmarking the performance of pairwise homogenization of surface temperatures in the United States. Journal Geophysical Research, doi: 10.1029/2011JD016761.

On my benchmarking study: New article: Benchmarking homogenisation algorithms for monthly data.

Corporate war on science

The Guardian on the CEI report and their attempt to attack the endangerment finding: Conservatives are again denying the very existence of global warming.

Another post on the CEI report: Silly Non-Study Supposedly Strengthens Endangerment Challenge.

My first post on the Red Cheeks Team.

My last post on the Red Team idea: The Trump administration proposes a new scientific method just for climate studies.

Great piece by climate scientist Ken Caldeira: Red team, blue team.

Phil Newell: One Team, Two Team, Red Team, Blue Team.

Why doesn't Big Oil fund alternative climate research? Hopefully a rhetorical question. They would have had a lot to gain if they thought the science were wrong, but they fund PR not science.

Union of Concerned Scientists on the funding of the war by Exxon: ExxonMobil Talks A Good Game, But It’s Still Funding Climate Science Deniers.

The New Republic on several attacks on science by Scott Pruitt: The End Goal of Trump’s War on Science.

Mother Jones: A Jaw-Dropping List of All the Terrible Things Trump Has Done to Mother Earth. Goodbye regulations designed to protect the environment and public health.

Sunday, 2 July 2017

The Trump administration proposes a new scientific method just for climate studies

What could possibly go wrong?

[[Scott Pruitt]] is the former Oklahoma Attorney General who copied and pasted letters for pro-pollution lobbyists onto his letter head. Much of his previous work was devoted to suing the EPA. Now he works for the big money donors as head of the EPA.  This Scott Pruitt is allegedly working on formulating a new scientific method to be used for studying climate change alone. E&E News just reported that this special scientific method will use "red team, blue team" exercises to conduct an "at-length evaluation of U.S. climate science."

Let's ignore that it makes no sense to speak of US climate science when it comes to the results. Climate science is the same in every country. There tends to be only one reality.

Previously [[Rick Perry]], head of the Department of Energy (DOE) who campaigned on closing the DOE before he knew what it does, had joined the group calling to replace the scientific method with a Red Team Blue Team exercise.

A Red Team is supposed to challenge the claims of the Blue Team. It is an idea from hierarchical organisations, like the military and multinationals, where challenging the orthodoxy is normally not appreciated and thus needs to be specially encouraged when management welcomes it.

Poking holes is our daily bread

It could naturally be that the climate "sceptics" do not know that challenging other studies is build in into everything scientists do; they do not give the impression to know science that well. In their Think Tanks and multinational corporations they are probably happy to bend the truth to get ahead. They may think that that is how science works and they may not able to accept that a typical scientist is intrinsically motivated to figure out how reality works.
At every step of a study a scientist is aware that at the end it has to be written up very clearly to be criticised by peer reviewers before publication and by any expert in the field after publication. That people will build on the study and in doing so may find flaws. Scientific claims should be falsifiable, one should be able to show them wrong. The main benefit of this is that it forces scientists to very clearly describe the work and make it vulnerable to attack.

The first time new results are presented is normally in a working group seminar where the members of the Red Team are sitting around the table, ask specific questions during the talk and criticise the main ideas after the talk. These are scientists working with similar methods, but also ones who work on very different problems. All and especially the group leaders have an interest in defending the reputation of the group and making sure no nonsense spoils it.

The results are normally also presented at workshops, conferences and invited talks at other groups. At workshops leading experts will be there working on similar problems, but with a range of different methods and backgrounds. At conferences and invited talks there are in addition also many scientists from adjacent fields in the audience or scientists working with similar methods on other problems. A senior scientist will get blunt questions after the talk if anything is wrong with it. Younger scientists will get nicer questions in public and the blunt ones in private.

An important Red Team consists of your co-authors. Modern science is mostly done in teams. That is more efficient, reduces the chances of rookie errors and very easy due to the internet. The co-authors guarantee with their reputation for the quality of the study, especially for the part where they have expertise.

None of these steps are perfect and journalists should get away from their single-study fetish. But together these steps ensure that the quality of the scientific literature as a whole is high.

(It is actually good that none of these steps are perfect. Science works on the boundary of what is known, scientists that do not make errors are not pushing themselves enough. If peer review would only pass perfect articles that would be highly inefficient and not much would be published, it normally takes several people and studies until something is understood. It is helpful that the scientific literature is high quality, it does not need to be perfect.)

Andrew Revkin should know not to judge the quality of science by single papers or single scientists, that peer review does not need to be perfect and did not exist for most of the scientific era. But being a false balance kind of guy he regrettably uses "Peer review is often not as adversarial as intended" as argument to see merit in a Red Team exercise. While simultaneously acknowledging that "All signs point to political theater"

Red Team science

An optimistic person may think that the Red Team proposal of the Trump administration will follow the scientific method. We already had the BEST project of the conservative physics professor Richard Muller. BEST was a team of outside people have a look at the warming over land estimated from weather station observations. This project was funded in part by the Charles G. Koch Foundation. the Heartland Institute, hard core deniers funded by Koch Brother organisations.

The BEST project found that the previous scientific assessments of the warming were right.

The BEST project is also a reason not to be too optimistic about Pruitt's proposal. Before BEST published their results mitigation sceptics were very enthusiastic about their work and one of their main bloggers, Anthony Watts, claimed that their methods were so good and he would accept the outcome no matter the result. That changed when the result was in.

Judith Curry was part of BEST, but left before she would have had to connect her name with the results. Joseph Majkut of Niskanen Center, who wrote an optimistic Red Team article, claims there were people who changed their minds due to BEST, but did not give any examples yet.

It also looks as if BEST was punished for the result that was inconvenient for the funders. The funders are apparently no longer interest in studying the quality of climate observations. Berkeley Earth now mainly works on air pollution. While BEST did not even look at the largest part of the Earth yet: the oceans. The nice thing of being funded by national science foundations is that they care about the quality of the work, but not the outcomes.

If coal or oil corporations thought there was a minute possibility that climate science was wrong, they would fund their own research. Feel free to call that Red Team research. That they invest in PR instead shows how confident they are that the science is right. Initially Exxon did fund research, when it became clear climate change was a serious risk they switched to PR.

Joseph Majkut thinks that a well-executed Red Team exercise could convince people. In the light of the BEST project, the corporate funding priorities and the behaviour of mitigation sceptics in the climate "debate", I am sceptical. People who did not arrive at their position because of science will not change their position because of science.

Washington Republicans will change their mind when the bribes, aka campaign contributions, of the renewable energy sector are larger than those of the fossil fuel sector. Or when the influence of money is smaller than that of the people, like in the good old days.

Science lives on clarity

As a scientist, I would suggest just wait and see at this time. Let the Trump administration make a clear plan for this new scientific method. I am curious.

Let them tell us how they will select the members of the Red Team. Given that scientists are always critiquing each others work, I am curious how they plan to keep serious scientists out of their Red Team. I would be happy to join, there is still a lot of work to do on the quality of station data. Scientific articles typically end with suggestions for future research. That is the part I like writing the most.

Because the Trump administration is also trying to cut funding for (climate) science, I get the impression that scientists doing science is not what they want. I would love to see how they excuse keeping scientists like me out of the Red Team.

It would also be interesting to see how they will keep the alarmists out. Surely Peter Wadhams would like to defend his position that the Arctic will be ice free this year or the next. Surely Guy McPearson would like to explain why we are doomed and mainstream science, aka science, understates the problem in every imaginable way. I am sure Reddit Collapse of Civilization can suggest many more people with just as much scientific credibility as the people Scott Pruitt would like to invite. I hope they will apply to the Red Team.

That is just one question. Steven Koonin proposes in the Opinion section of the Wall Street Journal that:
A commission would coordinate and moderate the process and then hold hearings to highlight points of agreement and disagreement, as well as steps that might resolve the latter
Does this commission select the topics? Who are these organisers? Who selects them? What are the criteria? After decades of an unproductive blog climate "debate" we already know that there is no evidence that will convince the unreasonable. Will the commission simply write that the Red Team and the Blue Team disagreed about everything? Or will they make an assessment whether it is reasonable for the Red Team to disagree with with evidence?

Clearly Scott Pruitt himself would be the worst possible choice to select the commission. Then the outcome would trivially be: the two teams disagree and Commission Coal Industry declares the Red Team as winner. We already have an NIPCC report with a collection of blog "science". There is no need for a second one.

The then right-wing government of The Netherlands made a similar exercise: Climate Dialogue. They had a somewhat balanced commission and a few interesting debates on, for instance, climate sensitivity, the tropical hotspot, long-term persistence and Arctic sea ice. It was discontinued when it failed to find incriminating evidence. Just like funding for BEST stopped and confirming the general theme of the USA climate "debate": scientists judge studies based on their quality, mitigation "sceptics" based on the outcome.

A somewhat similar initiative in the US was the Climate Change National Forum, where a journalist determined the debating topics by selecting newspaper articles. The homepage is still there, but no longer current. Maybe Pruitt has a few bucks.

"This is yet another example of politicians engaging in unhelpful meddling in things they know nothing about."
Ken Caldeira

How will Pruitt justify not asking the National Academy of Sciences (NAS), whose job these kind of assessments is, to organise the exercise. Surely the donors of Pruitt will not find the NAS acceptable, they already did an assessment and naturally found the answer that does not fit their economic interests. (Like the findings on climate change of every other scientific organisations from all over the world does not fit their corruption-fuelled profits.)

I guess they will also not ask the Science Division of the White House.

Climate scientist Ken Caldeira called on Scott Pruitt to clarify the hypothesis he wants to test. Given the Trumpian overconfidence, the continual Trumpian own-goals, the Trumpian China-hoax extremism, the Trumpian incompetence and Trump's irrational donors wanting to go after the endangerment finding, I would would not be surprised if they go after the question whether the greenhouse effect exists, whether CO2 is a greenhouse gas or whether the world is warming. Pruitt said he wanted a "discussion about CO2 [carbon dioxide]."

That would be a party. There are many real and difficult questions and sources of uncertainties in climate science (regional changes, changes in extremes, the role of clouds, impacts etc.), but these stupid greenhouse-CO2-warming questions that dominate the low-rated US public "debate" are not among them.

The mitigation sceptical groups are not even able to agree with themselves which of these stupid three questions is the actual problem. I would thus suggest that the climate "sceptics" use their new "scientific method" themselves first to make their chaotic mess of incompatible claims into something.

Red Team PR exercise

Donald Trump has already helped climate action in America enormously by cancelling the voluntary Paris climate agreement. Climate change is slow and global. Everyone hopes someone else will solve it some time and attends to more urgent personal problems. When the climate hoaxer president cancelled the Paris agreement the situation became more dangerous and Americans started paying attention. This surge is seen above in the Google searches for climate change in the USA. This surge was noticeable in Reddit where there was a huge demand for reliable information on climate science and climate action.

The Red Team exercise would give undue weight to a small group of fringe scientists. This is a general problem in America, where many Americans have the impression that extremist positions are still under debate because the fossil fuel industries bought many politicians who in turn say stupid things on cable TV and in opinion sections. These industries also place many ads and in return corporate media is happy to put "experts" on TV that represent their positions. Reality is that 97% of scientists and scientific studies agree that climate change is real and caused by us.

On the optimistic side, just like cancelling Paris made Americans discover that Washington is completely isolated on the world stage in their denial that climate change is a risk, the Red Team exercise could also lead to more American learning how broad the support in the scientific community for climate change is and how strong the evidence.

If the rules of the exercise are clearly unfair, scientists will easily be able to explain why they do not join and ask Pruitt why he thinks he needs such unfair rules. While scientists are generally trusted, the opposite is true for Washington and the big corporations behind Pruitt.

The political donors have set up a deception industry with politicians willing to lie for them, media dedicated to spreading misinformation or at least willing to let their politician deceive the public, they have "think tanks" and their own fake version of the IPCC report and a stable of terrible blogs. These usual suspects writing another piece of misinformation for the EPA will hardly add to the load.

The most tricky thing could be to make clear to the public that science is not resolved in debates. The EPA official E&E News talked to was thinking of a "back-and-forth critique" by government-recruited experts. In science that back and forth is done on paper, to make sure it is clearly formulated, with time to check the claims, read the cited articles and crunch the data. If it is just talk, it is easy to make false claims, which cannot be fact checked on the spot. Unfortunately history has shown that the Red Team will likely be willing to make false claims in public.

If the rules of he exercise are somewhat fair, science will win big time; we have the evidence on our side. At this time, where America pays attention to climate change, that could be a really good advertisement for science and the strength of the evidence that climate change is a huge risk that cannot be ignored.

Concluding, I am optimistic. Either they make the rules unfair. It seems likely they will try to make this exercise into political theatre. Then we can ask them in public why they make the rules so unfair. Don't they have confidence in their position that climate change is a hoax?

If they make the rules somewhat fair, science will win big time. Science will win so much, you will be tired of all the winning, you will be begging, please mister scientist no more winning, I cannot take it any more.

Let me close with John Oliver on Coal. Oliver was sued over this informative and funny piece by coal Barron Robert Murray who also stands behind Scott Pruitt and Trump.

Related reading

Are debatable scientific questions debatable?

Why doesn't Big Oil fund alternative climate research?

My previous post on the Red Cheeks Team.

Great piece by climate scientist Ken Caldeira: Red team, blue team.

Josh Voorhees in Slate: EPA Chief Scott Pruitt Wants to Enlist a “Red Team” to Sow Doubts About Climate Change.

Andrew Freedman in Mashable: EPA to actually hold 'red-team' climate debates, and scientists are livid.

Ars Technica: Playing fossil’s advocate — EPA intends to form “red team” to debate climate science. Agency head reported to desire “back-and-forth critique” of published research by Scott K. Johnson.

The pro-climate libertarian Niskanen Center: Can a Red Team Exercise Exorcise the Climate Debate? May I summarise this optimistic post as: if this new "Red Team" scientific method turns out to be the normal scientific method it would be useful.

Talking Points Memo: Pruitt Is Reportedly Starting An EPA Initiative To Challenge Climate Science.

Audobon's letter to Scott Pruitt: "The oil and gas industry manufactures a debate to avoid legal responsibility for their pollution and to eke out a few more years of profit and power."

Rebecca Leber in Mother Jones (May 2017): Leading Global Warming Deniers Just Told Us What They Want Trump to Do.

Scott Pruitt will likely not ask a court of law. Then they would lose again.

The Red Team method would still be a better scientific method than the authoritarian Soviet method proposed by a comment on a large mitigation sceptical blog, WUWT: Does anyone know if the [American Meteorological Society] gets any federal funding like the National Academy of Science does? ... People sometimes can change their tune when their health of their pocketbook is at stake. Do you really want to get your science from authoritarians abusing the power of the state to determine the truth?

Our wise and climate-cynical bunny thinks the Red Team exercise is a Team B exercise, which is the kind of exercise a Red Team should prevent.

Brad Plumer and Coral Davenport in the New York Times: E.P.A. to Give Dissenters a Voice on Climate, No Matter the Consensus.

Steven Koonin in the Opinion section of Rupert Murdoch's Wall Street Journal (April 2017): A ‘Red Team’ Exercise Would Strengthen Climate Science. (pay-walled)

Kelly Levin of the World Resources Institute: Pruitt’s “Red Team-Blue Team” Exercise a Bad Fit for EPA Climate Science.

Statement by Ken Kimmell, President, Union of Concerned Scientists: EPA to Launch Program Critiquing Climate Science

* Photo at the top of Scott Pruitt at CPAC 2017 by Gage Skidmore under a Creative Commons Attribution-ShareAlike 2.0 Generic (CC BY-SA 2.0) license.

Tuesday, 20 June 2017

On the recent warming surge

"Incidentally, when in the journal Science in 2007 we pointed to the exceptionally large warming trend of the preceding 16 years, which was at the upper end of the [climate] model range, nobody cared, because there is no powerful lobby trying to exaggerate global warming."

"And of course in our paper we named natural intrinsic variability as the most likely reason. But when a trend at the lower end of the model range occurs it suddenly became a big issue of public debate, because that was pushed by the fossil fuel climate sceptics’ lobby. There is an interesting double standard there."

Maybe the comment is deleted in shame. At least I cannot find it any more. Someone on Reddit was making exactly the same errors as the fans of the infamous "hiatus" to argue that global warming since 2011 is exploding and we're soon gonna die. Deleted comments can be freely paraphrased.

The data

So let's have a look at what the data actually says. Below are warming curves of two surface temperature datasets and two upper air temperature satellite retrievals. I shortened the warming curve of Berkeley Earth to match the period to the one of GISTEMP. All the other datasets are shown over their full lengths. Taking this step back, looking at the overview, there clearly is no "hiatus" and no "warming surge".

The red dots are the annual temperature anomalies, which are connected by a thin grey line. The long-term trend is plotted by a thick blue line, which is a [[LOESS]] estimate.

If you want to see the "hiatus" you have to think the last two data points away and start in the temperature peak of 1998. Don't worry, I'll wait while you search for it.

A hiatus state of mind

After seeing reality, let's now get into the mindset of a climate "sceptic" claiming to have evidence for a "hiatus", but do this differently by looking at the data since 2011.

Naturally we only plot the recent part of the data, so that context is lost. I am sure the climate "sceptics" do not mind, they also prefer to make their "hiatus" plots start around the huge 1998 El Nino warming peak and not show the fast warming before 1998 for context.

The thick blue lines are quadratic functions fitted to the data. They fit snugly. As you can see both the linear and quadratic coefficients are statistically significant. So clearly the recent warming is very fast, faster than linear and we can soon expect to cross the 2 °C limit, right?

I am sure the climate "sceptics" do not mind what I did. They also cherry picked a period for their "hiatus" and applied a naive statistical test as if they had not cherry picked the period at all. The climate "sceptics" agreeing with Christopher Monckton doing this on WUWT will surely not object to our Redditor doing the same and will conclude with him that the end is nigh.

By the way, I also cherry picked the dataset. The curves of the upper air temperature retrievals are not as smooth and the quadratic terms were not statistically significant. But in case of the "hiatus" debate our "sceptical" friends also only showed data for the datasets showing the least warming. So I am sure they do not object to this now.

If you look at the full period you can see that the variability of the temperature signal is much larger than the variability around the quadratic fit. It is thus clearly a complete coincidence that the curve is so smooth. But, well, the "hiatus" proponents also just look statistically at their cherry picked period and ignore the actual uncertainties (including slowly varying ones.)

Some more knowledgeable catastrophists may worry that it looks as if 2017 may not be another record scorching year. No worries. Also no worries if 2018 is colder again. The Global Warming Policy Foundation thinks it is perfectly acceptable to ignore a few politically inconvenient years and claims that we just have to think 2015 and 2016 away and that thus the death of the "hiatus" has been greatly exaggerated. I kid you not. I have not seen any climate "sceptic" or any "luckwarmer" complaining about this novel statistical analysis method, so surely our Redditor can do the same. Catastrophic warming surge claims are save till at least 2019.

Don't pick cherries

Back to reality. What to do against cherry picking periods? My advice would be: don't cherry pick periods. Not for the global mean temperature, not for the temperature of the Antarctic Peninsula, not for any other climate variable. Just don't do it.

If you have a physical reason to expect a trend change, by all means use that date as start of the period to compute a trend. But for 1998 or 2011 there is no reason to expect a trend change.

Our cheerful Redditor had even a bit more physics in his claim. He said that the Arctic was warming and releasing more carbon dioxide and methane. However, these emissions were not large enough to make the increase in the global greenhouse concentrations speed up. And they count. From our good-natured climate "sceptics" I have never heard a physical explanation why global warming would have stopped in 1998. But maybe I have missed their interest in what happens with the climate system.

If you have no reason to expect a trend change the appropriate test would be for a trend change test at an unknown date. Such a test "knows" that cherry picked periods can have hugely different trends and thus correctly only sees larger trend changes over longer periods as statistically significant. Applied to temperature data the result of such as test does not see any "hiatus" nor any "warming surge".

This test gave the right result during the entire "hiatus" madness period. Don't fool yourself. Use good stats.

Related reading

Cranberry picking short-term temperature trends

Statistically significant trends - Short-term temperature trend are more uncertain than you probably think

How can the pause be both ‘false’ and caused by something?

Atmospheric warming hiatus: The peculiar debate about the 2% of the 2%

Sunday, 11 June 2017

My unsolicited advice for a UK government

After my spectacular success as UK election pollster let my try my luck as UK political adviser. Offering outsider continental perspective from a citizen of a countries where coalition governments are normal.

The electoral results could not have been more complicated. The conservatives lost their majority. Against British custom collaboration is necessary.

Conservative Prime Minister Theresa May has proposed to govern together with the [[Democratic Unionist Party]] (DUP), a small deeply conservative Northern Irish protestant party with ties to terrorists after decades of religious terrorism.

That coalition has a majority of only 2 seats.

Other possible Conservative coalitions would be with the Liberal Democrats, with the Scottish National Party and the taboo option with Labour. Reversely, the only realistic coalition for Labour would be with the conservatives.

A Tory-DUP coalition is highly problematic. The DUP will have to get something in return for their collaboration. Typically a small coalition partner needs to get a lot more than their vote share to make it worthwhile and to survive the next election. Given how extreme they are, a majority of only 2 seats will often mean that more moderate conservatives will not be willing to give them what they bargained for and need.

The UK government is, furthermore, supposed to be neutral in Northern Irish problems. I have the impression that the UK does not fully realise that also in the Brexit negotiations Northern Ireland will be an extremely difficult problem. If the UK really wants a hard border this would cut Northern Ireland off from the rest of Ireland, which will cause economic problems for everyone and will make it harder to visit family and friends (for Catholics).

The DUP would at least make sure this problem is not ignored until it is too late, but their role would be far from neutral.

h/t Simon Noone‏

The two seat majority will likely dissipate quite quickly in future special elections. That could lead to another election in the middle of the Brexit negotiations.

The Brexit clock is ticking and the UK will have to make decades worth of political decisions in about one year time. I do not see that happening with a two-seat majority. It would have been a herculean task with a large majority and a capable prime minister.

More logical coalition parties for the Conservatives would be the Liberal Democrats and the Scottish National Party. However, they both do not want to be add-ons to a dominant Conservative party, especially after the bad experiences the Liberal Democrats had in their recent coalition with the Conservatives. Still given the situation I feel these parties should give this option more thought (after the disposal of May).

For the UK it would be highly unorthodox, but I see a coalition of the Conservatives and labour as the best option. Possibly together with the Liberal Democrats and the Scottish National Party. In Germany and The Netherlands we have such coalitions of conservatives and social democrats more often, although only as coalition of last resort. In Switzerland the government is always made up of all parties governing in partnership.

Not being used to coalition governments the UK is lacking a culture of adult behaviour and professional political debate, especially between the Tories and Labour. Maybe a neutral Prime Minister (of the Liberal Democrats) could help to hold a grand coalition together. I realise that that makes the coalition even more unorthodox. But these are also not normal times.

This would be nothing for a non-communicative prime minister with brittle personality and a penchant for kabuki theatre. Also Boris Johnson would also be an utter catastrophe, I hope this does not need explaining, but he may be a useful tool for regicide.

The kind of personality able to lead a grand coalition would also be the best personality for the Brexit negotiations with the EU. Many in the UK seem to see the EU as the enemy who is now laughing at UK's misery. On Twitter someone asked for a German word for the current predicament. Most offered was "Schadenfreude". I would say "Mitleid" (sympathy), "Verunsicherung" (confusion) and "Beunruhigung" (worry).

The UK gutter press like to cite anonymous antagonistic EU assholes. Anyone on twitter will be able to confirm that anonymous assholes are never short in supply. But that a few percent of humanity does not deserve that title does not change geography: Europe and the UK will always be neighbours. People in England seem to often forget that we are even neighbours with a land border.

The EU is interested in friendly relations with its neighbours. A concept that may be difficult to comprehend for (formerly) fascist rags. We are grateful for Monty Python, British expertise on the weather, and for the liberation from fascist occupation. A Russian liberation would have been much less pleasant. Just like young Brits got to know other Europeans as mostly nice people with shared values, European appreciate the UK. One day these young people may want to join their EU friends again.

No deal or a bad deal is damaging for the UK, but thus also unfavourable for the EU. A no-deal Brexit in no way fits to the nearly 50/50 outcome of the referendum.

Thus negotiations based on a common appreciation of long-term friendly relations would give the best results for all. There are so many topics that need to be negotiated that kabuki theatre on Gibraltar or holding EU citizens in the UK hostage are an enormous distraction. Next to EU negotiations the UK will have to negotiate trade relations with dozens of countries and trading blocks. The UK will have to build up institutions on the EU borders and institutions for the safety of drugs, chemical consumer product, banking products, etc.

Before the Brexit referendum we had ludicrous have-your-cake.and-eat-it propaganda. We are now nine months after the referendum and it is about time to have a public discussion what Brexit means. It is unfortunately only a slight exaggeration that this discussion was limited to Brexit is Brexit.  It is not.

It beats me why the UK also wants to leave [[Euratom]], which is not part of the EU. If necessary that could have been done later. They do not only work on nuclear power, but also nuclear medicine. Does the UK want the European Centre for Medium-Range Weather Forecasts, [[ECMWF]], to stay in Reading? It is not officially part of the EU, but has many of the same member states.

Does the UK want to keep student exchanges with the continent, [[Erasmus]]? Does the UK want to stay part of the European research community? If yes how? There are programmes for applied joint European research, [[Horizon 2020]], for fundamental science, [[ERC]], and for scientific coordination, [[COST]]. The UK could pay into those programs. Switzerland pays their researchers directly if they are part of it and in case of COST even gives them some bonus funding for additional research to stimulate participation.

Those are just the examples from my field, science. Similar hard choices will have to be made on so many topics. Personally I would suggest to change as little as possible and (initially) contribute to EU programmes. Just the minimal Brexit program, rights of EU citizens currently in the UK and UK citizens currently in the EU (residence, work, healthcare, benefits, pensions, voting rights), trade and borders (seas, fishing, exploration, Northern Ireland, Gibraltar), are though to get done in about a year. Loosening the ties on all the other topics can be done later.

That requires a friendly and communicative leader and government. Clearly Theresa may should go and Boris Johnson is not an alternative. A grand coalition may be the best way to get a concrete national debate going and to have fruitful negotiations.

[UPDATE. Oh my. When even a reasonable guy like John Oliver buys into the argument that the EU will be extra hard on the UK to scare others from leaving. Because everyone wants to be member of a union that has no upsides, but will beat you up when you leave and diligently works on making it worse to leave yourself.

The EU exists because it has benefits for its members. The UK may not be able to see this after decades of propaganda, but the other EU states surely do. Leaving the EU will mean losing these benefits. No need for additional punishments. Except if you think it is a malicious punishment not to be in applied research program Horizon 2020 when not paying for it or expecting the discounts of a customer loyalty card without having a card.

Otherwise an excellent video with a guest appearance by Lord Buckethead.


Related reading

Political support grows for cross-party approach to Brexit negotiations. Theresa May under pressure from across political spectrum to build plural coalition – including Jeremy Corbyn – before EU talks.

Der Spiegel looks at the years ahead: A Wave of Anger Crashes over Britain - Brexit Is Dead.

New York Review of Books: Britain: The End of a Fantasy

Desmog UK: Let's Take A Closer Look at the DUP's Climate Science Denial

Tuesday, 6 June 2017

Comrade Trend predicts the UK general election outcome

Poll whisperer Nate Silver himself just predicted that in the UK elections the conservatives would win by 7 %. My prediction is 2 %.

The prediction of Silver is a simple average of the last polls of the ten main polling organisations. He uses the same data as I do, kindly gathered by the volunteers of Wikipedia, who also made the plot below.

The statistics normally used to estimate the outcome of elections assumes that there is a fixed value and the surveys are noise around this value. In this case you have to average (in a smart way) to remove the noise of the surveys and get a better estimate of the fixed value.

Wikipedia uses the same assumption to compute the curve of the running average of all polls and takes the average over the 10 most recent polls. The distance between the curves of the Conservatives and Labour in the Wikipedia graph above is about 8 %.

Comrade Trend

However, if the polls are moving it typically keeps on moving in the same direction. Germans call this Genosse Trend, Comrade Trend. If you are going up in the polls Comrade Trend is a great friend.

This could be because it takes time until "everyone" has heard the news, only small part of the population is a news junky. You may also want to wait whether refuting information comes in some time later, whether a campaign changes its position or you may want to talk about the news with your family and friends.

It takes some time until people have found time to read the manifestos (Conservative | Labour). Hopefully many do, there is a real choice this time. Also the strengths and weaknesses of the campaigns mostly stay the same during the campaign. Each time Theresa May openly refuses to answer basic policy questions she loses voters, at least for people like me.

Comrade Trend seems like a good assumption to me. The polls in the UK move so fast that the difference in assumptions really matters this time. The smoothing method [[LOESS]] estimates the trend at a certain time to make the best estimate at that time. It fits a line to the data over a subperiod. Estimating a fixed value would correspond to assuming the line is horizontal over this subperiod. LOESS thus takes Comrade Trend into account. It gives the figure below.

The predictions for election day, this Thursday, were made by assuming a linear trend for the period since the 1st of May.

Election turnout

An important reason that it is hard for polling to predict the outcome of an election is that it is not known who will actually get off the couch and vote. The figure/tweet below shows for one of the polls how much difference various turnout assumptions makes. The spread in the polls is very large this election. That may well be because turnout is very important this time. Older people tend to favour the conservatives and faithfully show up. The question we will not know until election day is how much young people will show up.

Polling bias

Then there is the question whether polls are not just noisy estimate, but whether they are also biased relative to the election. Nate Silver writes:
Exactly how strong the Conservative tendency to outperform their polls has been depends on where you measure from. Since 1992, Conservatives have beaten their final polling margin over Labour by an average of 4.5 percentage points, and have done so in all but one election. (That was 2010, when both Conservatives and Labour gained ground as Liberal Democrats’ support collapsed, but Labour slightly outperformed its polling margin against the Tories.) Go all the way back to 1945, however, and the average Conservative overperformance is just 1.8 percentage points and is not statistically significant.
I prefer to look at all the data and would say that the bias is not statistically significant. It would be weird for the bias to have become worse, except maybe for really recent elections where parts of the population can no longer be reached with landline telephones. That you can find a period with a higher bias may well be cherry picking. As long as no one can provide a reason for a change in bias since 1992, picking a specific period after looking at the data is statistically suspect.


According to Nate Silver UK polling tends to have a relatively high uncertainty and misses the election outcome typically by about 4 %. Estimating a trend is harder than estimating a mean, thus it could be that my 2 % prediction is even a bit more uncertain. Thus if my best estimate is right and the conservatives are only 2 % ahead the election is a toss up: the difference is less than the uncertainty.

Let's see who turns out.

[UPDATE. If you put in my numbers for the conservatives and Labour and add 8 % for the Liberal Democrats, 4 % for UKIP and 3 % for the Greens (estimated from Wikipedia graph), the Electoral Calculus app computes the following seat distribution:

National Prediction: Conservative majority 4

Party2015 Votes2015 SeatsPred VotesGainsLossesNet ChangePred Seats
CON 37.8%331 40.0%610-4327
LAB 31.2%232 38.1%141+13245
LIB 8.1%8 7.6%05-53
UKIP 12.9%1 3.8%01-10
Green 3.8%1 2.9%00+01
SNP 4.9%56 4.7%01-155
PlaidC 0.6%3 0.5%02-21
Minor 0.8%0 2.5%00+00
N.Ire 18 00+018

[UPDATE. Just before closing of the election let me publish my last updated graph here. Currently Tories are 4.2 % ahead. That is still within the uncertainty and while it is quite likely that the Conservatives will win, there is still a Trump chance that Labour will win. Let's see who shows up to vote.


[POST ELECTION UPDATE. One of the best polls was by YouGov. Good with respect to statistical methodology and accuracy in this election. Interestingly, they did not find a steep trend towards Labour, but had a stable 3% lead for the conservatives since 27 May. Thus Comrade Trend may be more the inertia of the other polling organisations than that of the public.]

Related reading

YouGov, the heavily criticized poll that got it right: UK election: The day after. "But the general picture is clear: the model was a huge success in an election which most politicians, pollsters and commentators got badly wrong."

Nate Silver at 538: Are The U.K. Polls Skewed?

Response to Nate Silver, part one (because it’s early!) This post warns that the averaging method in Nate Silver's post is very basic and not comparable to the advanced methods he uses for US polls.

UK polling report: How the polls have changed since 2015

* The code used to generate my prediction plot is on Github.