By Ben Casselman
Five Thirty-Eight
Dec 18, 2014

In 2008, I moved to Dallas to cover the oil industry for The Wall Street Journal. Like any reporter on a new beat, I spent months talking to as many experts as I could. They didn’t agree on much. Would oil prices — then over $100 a barrel for the first time — keep rising? Would post-Saddam Iraq ever return to the ranks of the world’s great oil producers? Would China overtake the U.S. as the world’s top consumer? A dozen experts gave me a dozen different answers.

But there was one thing pretty much everyone agreed on: U.S. oil production was in permanent, terminal decline. U.S. oil fields pumped 5 million barrels of crude a day in 2008, half as much as in 1970 and the lowest rate since the 1940s. Experts disagreed about how far and how fast production would decline, but pretty much no mainstream forecaster expected a change in direction.

That consensus turns out to have been totally, hilariously wrong. U.S. oil production has increased by more than 50 percent since 2008 and is now near a three-decade high. The U.S. is on track to surpass Saudi Arabia as the world’s top producer of crude oil; add in ethanol and other liquid fuels, and the U.S.is already on top.

The standard narrative of that stunning turnaround is familiar by now: Even as Big Oil abandoned the U.S. for easier fields abroad, a few risk-taking wildcatters refused to give up on the domestic oil industry. By combining the techniques of hydraulic fracturing (“fracking”) and horizontal drilling, they figured out how to tap previously inaccessible oil reserves locked in shale rock – and in so doing sparked an unexpected energy boom.

That narrative isn’t necessarily wrong. But in my years watching the transformation up close, I took away a lesson: When it comes to energy, and especially shale, the conventional wisdom is almost always wrong.

It isn’t just that experts didn’t see the shale boom coming. It’s that they underestimated its impact at virtually every turn. First, they didn’t think natural gas could be produced from shale (it could). Then they thought production would fall quickly if natural gas prices dropped (they did, and it didn’t). They thought the techniques that worked for gas couldn’t be applied to oil (they could). They thought shale couldn’t reverse the overall decline in U.S. oil production (it did). And they thought rising U.S. oil production wouldn’t be enough to affect global oil prices (it was).

Now, oil prices are cratering, falling below $55 a barrel from more than $100 earlier this year. And so, the usual lineup of experts — the same ones, in many cases, who’ve been wrong so many times in the past — are offering predictions for what plunging prices will mean for the U.S. oil boom. Here’s my prediction: They’ll be wrong this time, too.

To be fair, the drop in oil prices is still too new for the experts to have settled on a clear consensus of what it will mean for U.S. producers. But the range of opinions is narrow, ranging from “production will be keep growing, but more slowly” to “it won’t have much effect at all.”1 Author and analyst Daniel Yergin, long the embodiment of the conventional wisdom on all things energy2, put it this way in a Wall Street Journal op-ed late last month, when oil was trading for just under $70 a barrel:

It is now clear that the new U.S. production is more resilient than anticipated. … True, with prices now near or below $70 a barrel, U.S. companies are looking hard at their investment plans — where and how much to cut or postpone. But it will take time for these decisions to affect supply. U.S. oil output will continue to rise in 2015.

I don’t take issue with anything Yergin is saying here. In fact, it makes sense. But that’s the thing about the conventional wisdom: It always makes sense at the time. It’s only later that we can see all the reasons it was wrong.

I don’t yet know why the conventional wisdom will be wrong this time, but I can guess. Not about what will happen — I’m no better at these predictions than anyone else — but about the sources of error. Here are a few of the most likely candidates:

No one has any idea what oil prices will do: In July 2008, my Journal colleague Neil King asked a wide range of energy journalists, economists and other experts to anonymously predict what the price of oil would be at the end of the year. The nearly two dozen responses ranged from $70 a barrel at the low end to $167.50 at the high end.3 The actual answer: $44.60.4

It isn’t surprising that experts aren’t good at predicting prices. Global oil markets are a function of countless variables — geopolitics, economics, technology, geology — each with its own inherent uncertainty. And even if you get those estimates right, you never know when a war in the Middle East or an oil boom in North Dakota will suddenly turn the whole formula on its head.

But none of that stops television pundits from making confident predictions about where oil prices will head in the coming months, and then using those predictions as the basis for production forecasts. Based on their track record, you should ignore them.

Drilling economics are complicated: In recent weeks, Wall Street analysts have published estimates of “break-even prices” for various U.S. oil fields. According to Goldman Sachs, for example, companies need at least $80 oil to make money in Texas’s Eagle Ford shale but only $70 in North Dakota’s Bakken shale. In theory, that makes it easy to see where companies will keep drilling at a given price and where they’ll pull back.

The reality is far more complicated. Not all parts of an oil field are created equal. Wells drilled in a “sweet spot” can be an order of magnitude better than those in less promising areas. Companies will keep drilling in the best areas long after they’ve pulled the plug on more marginal prospects. Break-even prices also change along with the price of oil. As prices fall and companies drill less, that leaves more rigs and equipment available, pushing down the price of drilling a well and allowing companies to stay profitable even at lower oil prices.

With oil under $60 a barrel, it’s a fair bet that many U.S. wells are now unprofitable. But that doesn’t mean companies will stop drilling them, at least right away. Companies often have contracts for rigs and would rather keep drilling than pay a penalty. They also have contracts for the land where they drill. If they don’t drill within a certain period, they lose the right to the land altogether.

Even when drilling does slow, production won’t necessarily follow. Wells keep producing for decades after they’ve been drilled, although at ever-declining rates. Companies prioritize their most promising projects, so the wells that do get drilled will be the best ones. And technology keeps improving, so companies can coax more oil out of each well. Natural gas provides an instructive example: The U.S. is drilling half as many gas wells today as it was five years ago and producing a third more gas.

Drilling finances are even more complicated: One thing I learned in my years covering the industry is that oil companies, and especially small oil companies, will keep drilling for as long as they can get the money to do so.5 That means the key variable in forecasting oil production isn’t drilling costs or even oil prices; it’s Wall Street.

In recent years, investors have handed energy companies half a trillion dollars in loans. That’s partly because of all the promising new oil fields in North Dakota and Texas, but it’s also because with interest rates near zero, investors are hungry for returns wherever they can find them. Now the Federal Reserve is talking about raising interest rates, which could kill the bond bubble, even as falling oil prices make those loans look riskier than they used to. If Wall Street turns off the money spigot, drilling will slow down no matter what oil prices do.

And then there’s politics: Why are oil prices falling? The short answer is lots of supply (the U.S. oil boom) and not much demand (a weak global economy). The longer answer is all about the Organization of Petroleum Exporting Countries. OPEC usually tries to keep prices high by limiting supply. But right now the cartel — or at least its dominant member, Saudi Arabia — appears content to let prices fall. The Saudis apparently think they can weather the storm of low prices better than companies in the U.S., where oil is much more expensive to produce.

But the policy has created divisions within OPEC, and no one knows when or if the cartel will start pulling back production. Tumbling prices are wreaking havoc on Russia’s economy, and they could easily lead to political unrest in other countries as well.

Oh, right, and geology: It’s easy to forget, but just a few years ago people were fretting about “peak oil,” the idea that global oil production had reached its maximum capacity and was doomed to start falling. The shale boom pushed those fears out of the mainstream, but the underlying questions remain. The shale boom is still young, and it was unclear how long it could last even when prices were higher. The U.S. government’s official production forecasts are subject to an almost comical level of uncertainty, and independent researchers have called even those estimates into question. The government didn’t see the boom coming, after all; there’s no guarantee it will see the end coming, either.