Economics profession went astray because economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth. Until the Great Depression, most economists clung to a vision of capitalism as a perfect or nearly perfect system. That vision wasn’t sustainable in the face of mass unemployment, but as memories of the Depression faded, economists fell back in love with the old, idealized vision of an economy in which rational individuals interact in perfect markets, this time gussied up with fancy equations. (..) They turned a blind eye to the limitations of human rationality that often lead to bubbles and busts; to the problems of institutions that run amok; to the imperfections of markets — especially financial markets — that can cause the economy’s operating system to undergo sudden, unpredictable crashes; and to the dangers created when regulators don’t believe in regulation.

Keynes did not, despite what you may have heard, want the government to run the economy. He described his analysis in his 1936 masterwork, “The General Theory of Employment, Interest and Money,” as “moderately conservative in its implications.” He wanted to fix capitalism, not replace it. But he did challenge the notion that free-market economies can function without a minder, expressing particular contempt for financial markets, which he viewed as being dominated by short-term speculation with little regard for fundamentals. And he called for active government intervention — printing more money and, if necessary, spending heavily on public works — to fight unemployment during slumps.

Yet the story of economics over the past half century is, to a large degree, the story of a retreat from Keynesianism and a return to neoclassicism. The neoclassical revival was initially led by Milton Friedman of the University of Chicago, who asserted as early as 1953 that neoclassical economics works well enough as a description of the way the economy actually functions to be “both extremely fruitful and deserving of much confidence.” But what about depressions? (..) Vision of financial markets as a “casino” was replaced by “efficient market” theory, which asserted that financial markets always get asset prices right given the available information. Meanwhile, many macroeconomists completely rejected Keynes’s framework for understanding economic slumps. Some returned to the view of Schumpeter and other apologists for the Great Depression, viewing recessions as a good thing, part of the economy’s adjustment to change. Of course, there were exceptions to these trends: a few economists challenged the assumption of rational behavior, questioned the belief that financial markets can be trusted and pointed to the long history of financial crises that had devastating economic consequences. But they were swimming against the tide, unable to make much headway against a pervasive and, in retrospect, foolish complacency.

In the 1930s, financial markets, for obvious reasons, didn’t get much respect. Keynes compared them to “those newspaper competitions in which the competitors have to pick out the six prettiest faces from a hundred photographs, the prize being awarded to the competitor whose choice most nearly corresponds to the average preferences of the competitors as a whole; so that each competitor has to pick, not those faces which he himself finds prettiest, nor even those that he thinks likeliest to catch the fancy of the other competitors.” And Keynes considered it a very bad idea to let such markets, in which speculators spent their time chasing one another’s tails, dictate important business decisions: “When the capital development of a country becomes a by-product of the activities of a casino, the job is likely to be ill-done.”

By 1970 or so, however, the study of financial markets seemed to have been taken over by Voltaire’s Dr. Pangloss, who insisted that we live in the best of all possible worlds. Discussion of investor irrationality, of bubbles, of destructive speculation had virtually disappeared from academic discourse. The field was dominated by the “efficient-market hypothesis,” promulgated by Eugene Fama of the University of Chicago, which claims that financial markets price assets precisely at their intrinsic worth given all publicly available information. (The price of a company’s stock, for example, always accurately reflects the company’s value given the information available on the company’s earnings, its business prospects and so on.) (..) In other words, finance economists believed that we should put the capital development of the nation in the hands of what Keynes had called a “casino.”

The theoretical model that finance economists developed by assuming that every investor rationally balances risk against reward — the so-called Capital Asset Pricing Model, or CAPM (pronounced cap-em) — is wonderfully elegant. And if you accept its premises it’s also extremely useful. CAPM not only tells you how to choose your portfolio — even more important from the financial industry’s point of view, it tells you how to put a price on financial derivatives, claims on claims. The elegance and apparent usefulness of the new theory led to a string of Nobel prizes for its creators, and many of the theory’s adepts also received more mundane rewards: Armed with their new models and formidable math skills — the more arcane uses of CAPM require physicist-level computations — mild-mannered business-school professors could and did become Wall Street rocket scientists, earning Wall Street paychecks.

Finance theorists continued to believe that their models were essentially right, and so did many people making real-world decisions. Not least among these was Alan Greenspan, who was then the Fed chairman and a long-time supporter of financial deregulation whose rejection of calls to rein in subprime lending or address the ever-inflating housing bubble rested in large part on the belief that modern financial economics had everything under control. (..) By October of last year, however, Greenspan was admitting that he was in a state of “shocked disbelief,” because “the whole intellectual edifice” had “collapsed.” Since this collapse of the intellectual edifice was also a collapse of real-world markets, the result was a severe recession — the worst, by many measures, since the Great Depression. What should policy makers do? Unfortunately, macroeconomics, which should have been providing clear guidance about how to address the slumping economy, was in its own state of disarray.

But there was something else going on: a general belief that bubbles just don’t happen. What’s striking, when you reread Greenspan’s assurances, is that they weren’t based on evidence — they were based on the a priori assertion that there simply can’t be a bubble in housing. And the finance theorists were even more adamant on this point. In a 2007 interview, Eugene Fama, the father of the efficient-market hypothesis, declared that “the word ‘bubble’ drives me nuts,” and went on to explain why we can trust the housing market: “Housing markets are less liquid, but people are very careful when they buy houses. It’s typically the biggest investment they’re going to make, so they look around very carefully and they compare prices. The bidding process is very detailed.” Indeed, home buyers generally do carefully compare prices — that is, they compare the price of their potential purchase with the prices of other houses. But this says nothing about whether the overall price of houses is justified. It’s ketchup economics, again: because a two-quart bottle of ketchup costs twice as much as a one-quart bottle, finance theorists declare that the price of ketchup must be right. In short, the belief in efficient financial markets blinded many if not most economists to the emergence of the biggest financial bubble in history. And efficient-market theory also played a significant role in inflating that bubble in the first place. Indeed, home buyers generally do carefully compare prices — that is, they compare the price of their potential purchase with the prices of other houses. But this says nothing about whether the overall price of houses is justified. It’s ketchup economics, again: because a two-quart bottle of ketchup costs twice as much as a one-quart bottle, finance theorists declare that the price of ketchup must be right. In short, the belief in efficient financial markets blinded many if not most economists to the emergence of the biggest financial bubble in history. And efficient-market theory also played a significant role in inflating that bubble in the first place.

Economics, as a field, got in trouble because economists were seduced by the vision of a perfect, frictionless market system. If the profession is to redeem itself, it will have to reconcile itself to a less alluring vision — that of a market economy that has many virtues but that is also shot through with flaws and frictions. The good news is that we don’t have to start from scratch. Even during the heyday of perfect-market economics, there was a lot of work done on the ways in which the real economy deviated from the theoretical ideal. What’s probably going to happen now — in fact, it’s already happening — is that flaws-and-frictions economics will move from the periphery of economic analysis to its center.

There’s already a fairly well developed example of the kind of economics I have in mind: the school of thought known as behavioral finance. Practitioners of this approach emphasize two things. First, many real-world investors bear little resemblance to the cool calculators of efficient-market theory: they’re all too subject to herd behavior, to bouts of irrational exuberance and unwarranted panic. (..) Behavioral finance, drawing on the broader movement known as behavioral economics, tries to answer that question by relating the apparent irrationality of investors to known biases in human cognition, like the tendency to care more about small losses than small gains or the tendency to extrapolate too readily from small samples (e.g., assuming that because home prices rose in the past few years, they’ll keep on rising).

So here’s what I think economists have to do. First, they have to face up to the inconvenient reality that financial markets fall far short of perfection, that they are subject to extraordinary delusions and the madness of crowds. Second, they have to admit — and this will be very hard for the people who giggled and whispered over Keynes — that Keynesian economics remains the best framework we have for making sense of recessions and depressions. Third, they’ll have to do their best to incorporate the realities of finance into macroeconomics. Many economists will find these changes deeply disturbing. It will be a long time, if ever, before the new, more realistic approaches to finance and macroeconomics offer the same kind of clarity, completeness and sheer beauty that characterizes the full neoclassical approach. To some economists that will be a reason to cling to neoclassicism, despite its utter failure to make sense of the greatest economic crisis in three generations. When it comes to the all-too-human problem of recessions and depressions, economists need to abandon the neat but wrong solution of assuming that everyone is rational and markets work perfectly. The vision that emerges as the profession rethinks its foundations may not be all that clear; it certainly won’t be neat; but we can hope that it will have the virtue of being at least partly right.

— Paul Krugman, How Did Economists Get It So Wrong?