Why everything we believe about progress is wrong

A brighter future is not inevitable. It is a choice. And success is far from guaranteed.

Two electrifying stories about the world have shaped our common sense about progress.

The first is the neoliberal story — an inspiring tale of progress propelled by free markets and free individuals. Such a tale is of course encapsulated by the glories of US capitalism. Fittingly it has the thrusting, spartan epicness of a great American novel. This quality makes it both ideologically bewitching and authoritative in a policy setting.

The story goes that civilisation is on a modernising linear trajectory, characterised by ever greater ‘integration’ in the international system. That its end-point is aggressively utopian is disguised by its soothing, reasonable greyness — a world of liberal migration flows, smooth just-in-time supply chains, and homo economicus living in lobotimised contentment.  

The neoliberal story is informed by a satisfyingly simple view of history, which reduces Western progress to the evolutionary growth of markets. Modernity, it is alleged, begins with the bourgeoisie commercial class overthrowing hierarchical systems based on privilege and replacing them with norms and institutions that safeguard economic freedom, from property rights to free trade.

In The Road To Serfdom, Friedrich Hayek traces this movement’s origins through Europe, starting commercial cities of Northern Italy and spreading through Europe to the Low Counties and the British Isles — but for Ayn Rand it reaches its zenith with the rise of the capitalist American dream. 

Of course, some have attacked the neoliberal vision of progress as spiritually sterile, but it is in reality a barely secularised version of a much older Christian ideal, riffing on ideas of individual salvation, and the kind of Calvinist grace that it is alleged can only come from tireless work and reinvestment of accumulated capital.

The second prevailing story — one of an ongoing Manichean liberal battle against the retrograde forces of evil — has similarly religious overtones. The argument goes that society is on a continuous march for a more equitable and inclusive society: as we have grown more civilised, we have successfully purged society of barbarities, from capital punishment to Jim Crow. But, inequities have not been completely stamped out, and more to do, from securing trans rights to battling structural racism. 

Self-congratulatory and humble in equal measure, this narrative psychologically appealing to the Western bourgeoise.

The arc of this second liberal story of progress is particularly contemporary, beginning with the successful fights for women’s rights and civil rights, and continuing to this day through LGBTQ activism etc. As a folk tale it is truly vivid, a collection of symbolic images and sounds reinforced by modern media — the contraceptive pill, the mini-skirt, suffragettes with their ribbons and wide hats, the trembling baritone orations of Martin Luther King.

It is worth pausing to consider just how embedded these liberal stories are in our common sense. Although they are in some ways rivalrous, they both ultimately reinforce our sense that progress is a linear, evolutionary phenomenon, and that it has follows a set — almost predestined — path. Both reify our beliefs that specifically liberal attitudes and policies drive progress, whether it’s free market capitalism or a disposition towards peaceful coexistence and racial tolerance. These ideas are so seared into our psyches that it seems sacrilegious to remotely question them.

And indeed the neoliberal story continues to dominate almost all aspects of policymaking. Economists have dubbed the UK’s stagnant productivity since the financial crash as a ‘puzzle’ because it defies the received liberal wisdom about the inevitable march of progress — a worldview which allows for cycles of boom and bust but cannot compute lengthy periods of stagnation.

And as the UK slides into the worst productivity slump since the industrial revolution, the Treasury can only advocate for what are essentially free market policies — improving the environment for business investment and trade by reforming tax incentives, relaxing migrant labour restrictions and improving wider economic certainty, which, it insinuates, Brexit has undermined. 

This is despite the fact that all the evidence suggests productivity has stagnated not because Britain's business environment or tax environment is unattractive, but due to a technological slowdown which cannot be remedied exclusively through traditional free market remedies. (More on this in my previous blog on the dangerous stagnation facing the West.)

The second story of progress is no less authoritative. So much progressive liberal discourse starts from the assumption that we have achieved a certain amount of social progress on the path towards liberal utopia and it is the job of the enlightened to both fight for greater progress and keep a lid on the xenophobia and bigotry that still lurks in unsavoury corners of society.

This is despite the fact that African-Americans have for quite some time been pointing out that social ‘progress’ is not linear (see Derrick Bell on segregated schools), something brought home by the recent over-turning of Roe vs Wade. And far from being a Manichean battle of good triumphing over evil, so-called progress has produced endless shades of grey. Amid middle-class white flights and self-segregation from both sides, black and white Americans continue to live parallel lives. And while African-American women should in no way be deprived of the right to abortion, the fact remains that the termination rate among this group — accounting for 40pc of all abortions in America — is troublingly high.

Don’t get me wrong, both the liberal stories of progress that I have outlined above have their merits. We should not be totally discarding them. And I have myself pointed out that a liberal attitude to things like intellectual freedom will be fundamental to fostering future innovation.

However, as richer countries stagnate despite a generation of neoliberal economic policies, and the West is plunged into divisive culture wars, I think we have to be willing to think heretically about progress. Put simply, we need to be willing to interrogate the received wisdom about how progress comes about, and what kind of progress we should aim for in the first place.

The first thing to say is that progress is not a continuous, linear phenomenon. In fact it happens in fits and starts. As Thomas Kuhn notably argued, science does not evolve gradually towards truth, but rather stays constant until it cannot explain a phenomenon, which gives rise to a new theory and thus triggers a paradigm shift.

In an earlier substack blog, I wrote in some detail about how the current mysterious computer-era stagnation could be compared to previous lull points before the technological takeoffs spurred by electricity and steam. And perhaps progress can slide back and forth. We are starting to sense this as today’s politics and economic discourse starts to echo the 1970s.

The second heretical thing to consider about progress is that it is not a purely liberal thing — that is to say, it is not a phenomenon spurred by exclusively liberal attitudes and policies. Quite the contrary, the historical evidence is that illiberal trends have been crucial to progress. Uncomfortable as it may be for some of us on the centre-Right to admit, that includes state intervention, both proactive and incidental.

Consider the fact, for example, that most of the technology that makes up our iPhones — from the internet and touch screens to voice-activated Siri — was developed through publicly funded science. Or that the industrial revolution was spurred to life in an environment of state protectionism, regulation, and war ‘capitalism’. In fact, in 19th-century England, excessive taxation actually inadvertently drove innovation as it compelled entrepreneurs to cut costs through the introduction of new machinery.

It may even be that new types of state intervention are required in the fourth industrial revolution - such as more funding for research. This is not least because innovational breakthroughs are getting harder, as as the total amount of knowledge needed to solve problems continuously increases. Each doubling of technological advancement reportedly requires four times as much research effort as the previous redoubling. Matt Clancy has a written a great substack blog on this here.

And yet, we have a tendency to overlook these facts because the somewhat analogue neoliberal view that growth is largely powered by free trade continues to dominate economic thinking. As I argued in my last blog, this overlooks the crucial role of technological innovation — which has undoubtedly benefited as much from state support as it has from government ‘getting out the way’.

The role of illiberal tactics in driving progress is also uncomfortably clear in the social realm. The fact is that violence was just as critical to the success of the 1960s civil rights movement as peaceful protest.

Such a disturbing reality may have been wiped from mainstream mythos, but there are hints that Martin Luther King may have deliberately courted violence as a strategy, inviting bloody clashes by encouraging African-Americans to demonstrate in America’s most racist cities. The overarching aim of his civil rights approach was after all “to dramatize the issue that it can no longer be ignored.”

The horrifying scenes of children being set on by dogs and water hoses in the likes of Birmingham Alabama, and indeed the assassination of Martin Luther King himself, were central – perhaps crucial – to the movement’s triumph. Perhaps one might go as far as to suggest that the civil rights movement triumphed because a queasy Middle America felt compelled to bring an end, not to racial discrimination per se, but to the shameful violence that exploded when black Americans rose up against their oppression. 

Maybe the lesson to draw from such diverse examples is that progress is achieved not by sticking, with romantic dogmatism, to a liberal blueprint based on what we’d like the world to be, but by balancing liberal and illiberal strategies to navigate the world as it truly is.

And that of course means being willing to make sacrifices. One of the great conceits of our time is that there can be progress in the West without having to give anything up. The Left indulge in the idea that civilised society should be able to guarantee a minimum standard of living to all citizens, regardless of ability or effort.

The Right meanwhile has tended to unhelpfully elevate egoism into a benevolent virtue. This is becoming a serious problem as yesterday’s thrusting entrepreneurial baby boomers become today’s intransigent nimbies. The fact is that we will get nowhere unless we are willing to make sacrifices – whether that means stripping back aspects of the welfare state to fund DARPA-inspired research institutions or putting up with noisy and disruptive infrastructural, housing and energy developments in our tidy, tranquil middle-class neighbourhoods. 

The other thing we have to consider about progress is that it isn’t just this one linear path. It is possible to take wrong turnings, and even end up in dead alleys. Perhaps then, first principles are crucial, and where we start from is just as important as where we want to go. In a world of ever-greater specialisation and professionalisation — and an academic environment where too often one can question the parts but not the whole — that is a potentially serious problem.

Just take the current dominating dystopian future vision, which starts from the first principle that one day humans will replaced by machines. If you really think about it, it’s incredible that this idea has been taken so seriously (a tribute, maybe, to the power of Hollywood and Silicon Valley hype).

To state what should be painfully obvious: the brain is not a machine. It does not process information or store memories like a computer. Machines are not — and will most likely never be — capable of thinking the way a human does, through dialectical reasoning. Nor will machines ever be truly creative and future oriented, because they rely on inductive reasoning, and inductive reasoning relies on prior examples. Netflix can only make recommendations on what you should watch based on watch you have already watched. It will never be able to pre-empt your current whims or predilections.

AI is most likely stagnating because it refuses to grasp these first principles. All attempts to develop this technology over the last generation have been based on deductive and inductive reasoning. Absolutely zero progress has been made on getting machines to perform abductive reasoning – ie common sense thinking. 

In his book, The Myth of Artificial Intelligence, Erik Larson has a fascinating theory that today’s AI stagnation might even be traceable back to one simple wrong turn taken by Alan Turing. Larson points out that in his earliest work, Turing made a distinction between ingenuity, which mathematicians use to do work out problems, and insight, which mathematicians deploy outside of the formal mathematical system to decide what part of a problem are interesting or demand further exploration.

Later in his career, however, Turing’s attitude human intelligence became mysteriously reductive, and he adopted the perspective that human intelligence basically boiled down to problem solving. It is this attitude that has informed AI since the emergence of the field — and may well hold it back for decades to come.

Interestingly, one area that we are seeing people actively question first principles of progress is in the culture wars. I have serious concerns about Black Lives Matter, but we should take seriously the fact that here is a movement geared towards fundamentally questioning the first principles of social progress which has been attached to the legacy of Martin Luther King — notably that we should aim for a colour blind society, based on race-neutrality embedded in the law.

Full disclosure, as a mixed-race person, I have always instinctively felt that the “colour-blind utopia” is indeed something we ought to strive for . However, I accept that a lot of people of colour find this perspective harmful and naive.

If it feels like we are going backwards — to a society that feels more conscious of racial distinctions — then that’s because in a way we are. But it’s important to acknowledge that, left to fend for themselves in decaying neighbourhoods long abandoned by the white middle-class, African-Americans are reversing out of a vision of progress that they feel simply isn’t working for them.

Those of us who believe that they may be in danger of throwing out the baby with the bathwater need to be able to engage with this — and be willing to argue robustly based on first principles rather than just rattling off cherry-picked quotes from “I Have A Dream”.

The last thing I want to throw out there is that true progress cannot be conceived of as a process by which enlightened elites, professionals and intellectuals lead — or perhaps drag — the rest of us into a bright new future. Of course, such a view appeals to our middle-class snobberies. And frankly this is indeed how progress operated for thousands of years (including the industrial revolution — the historian Joel Mokyr has written about how the Enlightenment movement which intellectually propelled it was an essentially elite phenomenon).

Today though, democratic societies need to find ways to bring people along. Because without a captivating story and a spine-tingling vision of progress, people start embracing the kind of policies that will only tip us into doom loops of decline – whether that’s nativism and hardline anti-immigration policies or redistributive raids on wealth. But sadly the precise thing that we are missing right now is a fresh story of progress, as awesome and invigorating as those that have dominated until now, but are starting to outlive their usefulness.

The possible shape, smell, feel, colour, sounds and spiritual uplift of such a story is for another substack post.

For now though, I’ll leave you with the bottom line is that progress isn’t inevitable. It’s a choice. And it’s messy. It takes work, thought, commitment and ability to balance both liberal and illiberal strategies. I fear that the West will continue to stagnate and rip itself apart until it confronts this.