Uncertainty and Forecasting

The following is a compilation of two articles. The articles were published in our October, 2005 and January, 2006 editions. See above for links to the journals in which the original articles appear.

Investors are confronted with a myriad of difficult choices -- for example, about asset allocation policy and the funds that should be used to implement it. In this article, we will review the types and sources of uncertainty that make these decisions so difficult. We will then look at the extent to which forecasting can penetrate this uncertainty, and reach conclusions about what our findings mean for investors.

We will start from the ground and work up, beginning with the sources of our individual uncertainty, proceed on through corporate uncertainty, and end up with financial market uncertainty, and its implications for us as investors.

Let's start with a basic question: what is knowledge? Broadly speaking it consists of a set of theories for categorizing sensory inputs (e.g., “if it walks like a duck, and quacks like a duck, there is a high probability it is a duck”), as well as a set of causal theories to guide action (e.g., “if ___, then ____”). But this definition begs the question, where do these theories come from?

Assume you have just been asked to investigate a car crash. How would you proceed? After arriving on the scene, you would most likely begin to ask questions, or, more formally, collect evidence. Are there skid marks? Is the road wet? Was it raining? Were there witnesses? Was another car involved? Armed with the evidence you collected, you would then develop a number of alternative hypotheses (initial theories) that relate some or all of the evidence you collected to the result of the crash. Formally, this process of going from a result to evidence to possibly hypotheses (i.e., explanations) is known as “abduction.” And right from the outset, at this most fundamental level, you can see a source of uncertainty: the limitations on your ability to identify the full range of possible hypotheses that link the evidence and the result you observe. Some of these limitations are situation specific: you may be tired or stressed. Some have to do with your own experience: if it is limited or narrow, you may not have a wide enough range of analogies to draw on when trying to think creatively about potential hypotheses. But another is more fundamental: in some cases, cause and effect are widely separated in time. To go back to our car crash, if none of the physical evidence you collect is consistent with a hypothesis to explain the crash, you may instead hypothesize it was due to “driver error.” However, while that may, in some sense be the immediate cause, the more important question is the causes of that error. But in all likelihood, those causes are so far removed in time from the accident, and so complex (e.g., the genetic makeup and life history of the driver, and perhaps his family and friends as well) that you will not, if your time is limited, be able to identify them (though in the case of airline crashes, there is an entire “human factors” industry that studies just these issues).

But let's move on. How do you decide which of hypotheses you have generated makes the most sense? One way would be to check an accident database, to see how frequently different pieces of evidence were associated (correlated) with the type of accident you observed. Ideally, this will enable you to disprove some of your tentative hypotheses, or at least reject them for lack of evidence. However, in most cases you will still be left with more than one hypothesis that hasn't been disproved. If forced to choose just one (say, if the press was waiting to talk with you), you would logically weigh the weight and reliability of the evidence supporting the alternatives, and choose one hypothesis as the best possible explanation. Alternatively, you could attach differing degrees of belief to more than one hypothesis (e.g., 20%, 40% and 40%). Formally, this is process of testing hypotheses is know as “induction.” And, as you can see, it is not guaranteed to reduce uncertainty. Evidence can be consistent with more than one hypothesis, and no hypothesis is ever strictly true; the best that can be said is that it has not been disproved.

It is now two weeks later, and you find yourself driving in a rainstorm. This evidence activates one of the hypotheses for which you found support: driving in rain raises the probability of an accident. You conclude that you are facing a situation of increased risk. This triggers another hypothesis: in a situation of increased risk, slowing down your speed will increase your safety. This process is formally known as “deduction.” But, again, there is uncertainty: did you observe all the available evidence (e.g., did you miss that large truck in back of you traveling at high speed too close to your car?) And even if you perceived all the high value information, did your memory activate all the relevant hypotheses (e.g., if you are in a traffic jam, the probability of an accident increases if you slow down too quickly)? In most cases, the answer to both questions is “probably not.” Most of the time, our perception and cognitive processing are not perfect. Formally, another way of saying this is that we are “boundedly rational”, which is a further source of uncertainty.

Many people have thought long and hard about the sources of our cognitive limitations. Some of the reasons seem to be physical: even at the best of time, our brain can process no more than five to seven “chunks” of information (a key difference between novices and experts is how much data is aggregated into each of these chunks). In addition, our perceptual and processing performance worsens when we are tired and stressed. Our ability to perceive and process information is also subject to some well known biases. For example, we tend to be over-optimistic, overconfident about the accuracy of our views, more likely to notice and give more weight to evidence that confirms them, and to change our mind more slowly than is warranted by the available facts.

In short, at the individual level, there are multiple sources of uncertainty that are impossible to fully eliminate from life. Perhaps the most important way people have attempted to reduce these uncertainties is to organize into groups (see, for example, the paper by Hong and Page titled “Groups of Diverse Problem Solvers Can Outperform Groups of High Ability Problem Solvers”). For the purposes of this article, let us consider a familiar group: the corporation. We are all well aware of the pressures on corporate management to deliver superior performance, relative to competitors and relative to the overall equity market index. But consider the uncertainty one confronts when trying to deliver achieve this. Broadly speaking, a corporate strategy can be though of as consisting of three parts. The first is a theory of the environment, including the evolution of customer needs priorities, competitor offerings, technological possibilities and general macroeconomic and regulatory conditions. The second is a theory of competition, involving which customer to target, what to offer them, how to deliver this offering at an acceptable return to shareholders, and how to prevent competitors from copying this business model. The third is a theory of implementation, involving the sequencing and synchronization of actions, communication and coordination requirements, and the collection of critical information that will indicate a need to adapt the original strategy.

There is enormous uncertainty in all of these areas, much of which is grounded in two critical facts of life. The first is heterogeneity - customers, competitors, suppliers, and the firm's own employees all have differing source of information, and abilities to perceive and process it. The second is self-reference, or, as it is sometimes called, “reflexivity” or “recursiveness.” This refers to the fact that in many cases, a person or corporation takes action on the basis of assumptions about the future actions of others, which in turn depend on the very action you are planning to take. Theoretically, there is no limit (apart from exhaustion of resources) on the extent to which one can engage in a cycle of reasoning along the lines of “I will do this, based on the assumption he will do that, because he assumes that I assume…

This means that the fundamental processes generating the returns sought by investors are themselves highly uncertain. This uncertainty is made worse by the fact that heterogeneity and recursiveness often create highly non-linear outcomes that are extremely difficult, and often impossible to accurately predict.

One consequence of this is that firms that consistently deliver superior shareholder returns are exceedingly rare. For example, in their classic paper “The Level and Persistence of Growth Rates”, Chan, Karceski and Lakonishok found that there was no persistence in firm growth rates beyond chance, and that it was extremely hard to predict these growth rates in advance.

The implications of the fundamental uncertainty we have identified are critically important to investors, but too little discussed in what is written about investing.

Let's start with the efficient markets hypothesis (EMH), which is still very much the centerpiece of the investment theory that is taught in schools. In its strongest version, it assumes that all investors have equal access to full information. Moreover, they all use the same model to convert new information into an updated view of the fair price of an investment. Since these models are also assumed to perfectly reflect the underlying return generating process, prices instantly and accurately adjust to the release of new information. This means that the market for the investment in question is continuously in a state of equilibrium between buyers and sellers, in which no investor can earn anything other than the market return (i.e., this is an “all beta, no alpha” world). As we all know, none of this reflects reality. Rather, we confront a world in which information takes time to diffuse to investors who have varying perceptual and cognitive capabilities, who use differing models that all imperfectly describe the return generating process, and who, if they are active managers, earn returns that are above and below the market (i.e., have positive and negative alpha).

At this point, some will say, “Voila”; the case for passive investing is disproved, and we should all pay high fees to active managers. Not so fast. The fact that investors are not all perfectly rational and instantly endowed with perfect information does not automatically mean that index investing doesn't make sense. Fundamentally, to disprove the case for indexing, you have to show how the information and reasoning imperfections we have identified lead to a market that is “inefficient”, in the sense that it creates opportunities for skilled managers to consistently earn risk and tax adjusted returns that are higher than those available on a comparable index fund.

Data on the performance of active fund managers compared to index funds suggest that most financial markets, despite the limitations of their participants, are, if not perfectly efficient, very close to it. The strongest evidence of this is the declining proportion of active managers who outperform index funds as the evaluation period is extended from one to five to ten years or longer. However, active managers sometimes respond with the allegation that this conclusion is specific to the time period chosen for the comparison. How do we respond to this? We return to the subject of uncertainty.

As we have written in the past, there are basically two broad hypotheses that cause a person to buy a stock. Some people buy because, having analyzed the business of the company (using some combination of information and a model), they have a theory that the stock is undervalued. Implicit in this view are two other critical assumptions. The first is that it is possible to accurately determine the fair value of a company. This is no trivial task. It requires forecasts (either explicit or implicit) of future customer needs, competitor actions, technological possibilities, the relative success of a company's offering to customers, its cost structure, and its ability to prevent imitation by competitors. It also requires belief in the accuracy of the asset pricing model being used to translate one's forecast of a company's future cash flows into the fair present value for its stock (as well as confidence in the forecasts - e.g., of interest rates and the equity risk premium - that model requires). As we have seen, the level of uncertainty involved at all levels of the system might make one question the belief that it is possible to accurately value a company. The other (and often implicit) assumption made by these “fundamental value” investors is that a sufficient number of other investors will eventually recognize the undervaluation, and their actions will cause the market price of the stock to increase. As we will soon see, this assumption is also open to question, at least about its timing.

In 1936, John Maynard Keynes began his description of the second group of stock buyers with the following observation: “Investment based on genuine long-term expectation is so difficult today as to be scarcely practicable. [An investor] who attempts it must surely lead much more laborious days and run greater risks than [an investor] who tries to guess better than the crowd how the crowd will behave; and given equal intelligence, he may make more disastrous mistakes…[However] it needs more intelligence to defeat the forces of time and our ignorance of the future than to beat the gun…[Also] human nature desires quick results, there is a popular zest in making money quickly, and remoter gains [from fundamental investing] are discounted by the average man at a very high rate.”

This second group of people buy a stock because they forecast that other people will also be buying it, and this will cause its price to rise. The focus of their information collection and modeling efforts is not on the business of the company and the fair value of the stock, but rather on the expected behavior of other investors.

However, these investors also confront uncertainty, in the form of recursiveness. For example, I decide to buy, because I forecast that other investors will act in a certain way, because I assume they assume I assume, ad infinitum. John Maynard Keynes described this problem as follows: “professional investment may be likened to those newspaper competitions in which the competitors have to pick out the six prettiest faces from a hundred photographs, the prize being awarded to the competitor whose choice most nearly corresponds to the average preferences of the competitors as a whole; so that each competitor whose choice most nearly corresponds to the average preferences of the competitors as a whole: so that each competitor has to pick, not those faces which he himself finds prettiest, but those which he thinks likeliest to catch the fancy of the other competitors, all of whom are looking at the problem from the same point of view. It is not a case of choosing those which, to the best of one's judgment, are really the prettiest, nor even those which average opinion genuinely thinks the prettiest. We have reached the third degree where we devote our intelligences to anticipating what average opinion expects the average opinion to be. And there are some, I believe, who practice the fourth, fifth and higher degrees.” Given the insolubility of this forecasting problem, Keynes theorized that, in the presence of a sufficient number of these type of investors, financial markets would be driven by irrational factors, or what he termed “animal spirits.”

It is also the case that momentum investors are, either explicitly or implicitly making two critical assumptions. The first is that they are sufficiently smarter than other investors (in the sense of having better forecasts) that they will be able to “get out ahead of the crowd”, or, as Keynes said, “beat the gun.” For example, researchers have found that found that the average person forecasts between one and two steps ahead in recursive type situations (see, for example, “A Cognitive Hierarchy of One Shot Games” by Camerer, Ho and Chong). However, the widespread finding that human beings are also habitually overconfident suggests that many people's self-assessment on this point will be wrong (and expensive).

The second critical assumption in momentum investor's forecast is that when he or she decides to sell, there will be sufficient liquidity available in the market to execute this trade at a price that doesn't wipe out his or her profits. Events such as the October, 1987 crash, and the meltdown of Long Term Capital Management in 1998 make it clear that this is not always the case (suggesting another potentially expensive lesson for momentum investors).

Having described the two basic investor types, the next step is to define a stock market as being composed of a heterogenous group of boundedly rational investors, whose strategies, at any point in time, reflect a mix of widely differing approaches to fundamental/value and trend/momentum (e.g., based on the use of different information and models). So what happens over time as these investors interact with each other?

The short answer is, “lots of stuff you couldn't predict in advance.” An early paper on this subject was “Heterogenous Beliefs and Routes to Chaos in a Simple Asset Pricing Model” by Brock and Hommes. Their artificial (simulation based) stock market contains investors who face an incentive of varying intensity to evaluate and possibly change their strategy based on its performance. They find that when that incentive is sufficiently strong (e.g., when many investors face high pressure to deliver good short-term performance versus a benchmark), the pattern of returns becomes non-linear and chaotic, and impossible to predict in advance. This market exhibits “irregular switching between phases [i.e., regimes] during which prices are close to their efficient market fundamental value, phases of optimism with upward trends and phases of pessimism with declining asset prices.” In “Asset Prices and Wealth Dynamics Under Heterogenous Expectations”, Chiarella and He find that introducing just two probabilistic processes (i.e., in which the value for a variable is drawn from a distribution of possible outcomes), one governing the generation of company dividends, which fundamental investors try to forecast, and the other governing the length of time over which other investors try to forecast the future behavior of their peers, is sufficient to produce the familiar non-linear chaotic pattern in prices and returns.

An interesting question raised by these analyses is whether irrational trend chasing investors can survive in the face of trading by fundamental/value investors. Keynes thought they could. In 1986, another paper, “Noise Trader Risk in Financial Markets” by DeLong, Shleifer, Summers, and Waldmann reached the same conclusion. They found that “the unpredictability of noise traders' beliefs creates a risk in the price of the asset that deters rational arbitrageurs [fundamental investors] from aggressively betting against them. As a result, prices can diverge significantly from fundamental values even in the absence of fundamental risk. Moreover, bearing a disproportionate amount of risk that they themselves create [others have suggested this is because of their relatively greater overconfidence] enables noise traders to earn a higher expected return than do rational investors.” Later papers that have used more analytically intensive approaches (e.g. more complicated simulation models, with a larger population of agents and strategies) have repeatedly confirmed the finding that irrational trend chasers will survive over time in financial markets, and some will realize exceptionally high returns.

One of these later papers is “The Price Dynamics of Common Trading Strategies” by Farmer and Joshi. This paper is interesting for many reasons. First, it includes a market maker mechanism, with which all investors place their orders. Theoretically, the existence of a market maker, who will widen spreads as buy/sell imbalances increase, should dampen the price fluctuations in the market. Second, it employs a wide range of fundamental value and trend chasing strategies, to create a reasonably realistic “market ecology.” Third, the authors attempt to adjust the parameters of their model so that key features of the resulting return patterns match those found in real financial markets. These include volatility that varies over time and return distributions that have “fatter tails” (i.e., a greater proportion of extreme returns) than the normal “bell curve.” They are able to do this by balancing the initial numbers of value and trend following investors, as well as tuning some of the parameters of the specific trading strategies they use (e.g., that affect when these strategies become active in the market). The resulting model shows that prices rarely match their fundamental values, with a wide range deviations from them, and lengths of time over which those deviations persist. On the other hand, the authors also find that the market is attracted towards equilibrium, though the specific conditions giving rise to it seem impossible to forecast in practice.

In a more recent paper, “Behavioral Heterogeneity in Stock Prices”, Boswijk, Hommes, and Manzan fit a similar model to annual U.S. stock price data from 1871 to 2003. They find that the market switches the existence of two regimes, one mean reverting, and dominated by fundamental/value investors, and the other mean averting, and dominated by trend followers. As in previous models, the proportion of investors utilizing these different strategies varies over time, in rough alignment with their relative profitability (as some investors are dogmatically committed to one approach or the other, while others have a greater willingness to switch based on their relative performance over some past interval). For example, the authors note that their model “suggests that in the mid 1990s, optimistic, boundedly rational investors, motivated by short-run profitability, reinforced the rise in stock prices triggered by the higher expected cash flows of the internet sector.”

In “Market Mood, Adaptive Beliefs, and Asset Price Dynamics”, Dieci, Farani, Gardini and He find that the market becomes increasingly difficult to predict as the proportion of investors who switch strategies based on recent performance increases. In “Heterogenous Expectations and Speculative Behavior in a Dynamic Multi-Asset Framework”, Chiarella, Dieci and He find that the introduction of diversification across different asset classes does not moderate the finding of chaotic market dynamics in previous analyses. Rather, it adds another source of complexity, to make predictability even more difficult. In another paper, “A Dynamic Analysis of Moving Average Rules”, Chiarella, He and Homees find that even something as simple as a change in the length of the moving average used by trend following investors can destabilize a market, and set off chaotic dynamics, as can a decrease in average investor risk aversion. And in “Lock-in of Extrapolative Expectations in an Asset Pricing Model”, Kevin Lansing from the Federal Reserve Bank of San Francisco shows how a concern with minimizing forecast errors (as might characterize a fund manager who worries about underperforming a benchmark) may inadvertently lead to “lock-in” to a trend chasing strategy.

Another recent paper, “Feedback and the Success of Irrational Investors” by Hirshleifer, Subrahmanyam, and Titman, makes an important point. The authors find that the impact of irrational investors on stock prices and returns feeds back into the real investment decisions of companies, via the inferences they make about their relative cost of capital. In turn, this affects the valuations placed on these companies' stocks by fundamental investors. In short, there is a linkage between the complex adaptive systems that exist in the financial markets and the real economy that makes the behavior of the integrated system even more complex and impossible to predict.

Taken together, all of these studies, and more like them (two good surveys are “Agent-Based Computational Finance” by Blake LeBaron, and “Heterogenous Agent Models in Economics and Finance” by Cars Hommes), lead to the conclusion that the failure of active managers to outperform index funds (particularly over longer periods of time) is not a function of the time period used to compare their respective results. Rather, it reflects the fact that the financial markets are a complex adaptive system, in which predictability will only exist over short periods, and even then will be based on the use of very different types of superior information and models, depending on the proportion of investors using fundamental/value or momentum/trend strategies.

Moreover, successful information sources and models will tend to be self-destructive, as their growing economic success causes asset prices to increase (as buying demand increases relative to supply) and induces other investors to copy them. In sum, while the financial markets may be made up of less-than-perfectly rational investors, who cause prices to deviate from their fundamental values, their fundamental uncertainty causes them to still be highly efficient, in the sense that it remains extremely difficult for an active manager to deliver (particularly over long periods of time) higher risk adjusted after tax returns than those produced by comparable index funds.

Now let us move on to a more in-depth look at the challenges of forecasting. A person's beliefs about the efficacy of forecasting are (or should be) central to his or her approach to investment management. An investor who believes that it is impossible to accurately forecast risk or return (beyond simple luck), even at the asset class level, should logically hold an equally weighted portfolio of asset class index funds. An investor who believes that it is possible to forecast asset class risks and/or returns (either absolutely or relatively) will logically assign different portfolio weights to different asset class index funds. Finally, an investor who believes it is possible to forecast risk and return for individual securities will invest part of his or her portfolio in actively managed products (or pick individual securities herself).

Given the importance of forecasting, or, more specifically, beliefs about its accuracy, this article will summarize recent research in this area. Let's start with the basics. A forecast for a given target (or “dependent”) variable (say, the rate of return next year on an equity market index) contains three elements: (1) “independent” variables that affect the future value of the target; (2) a description of how these variables are related to each other; and (3) estimates of the future values of the independent variables. Together, (1) and (2) are often referred to as the “forecasting model.” This is, in essence, a theory about how a given system works. In contrast, assumptions about the future values of the independent variables are known as “parameter estimates.”

We use forecasting models every day. A few are deliberately created and explicit; most are formulated intuitively, and their terms are implicit. Broadly speaking, people typically draw on three sources when creating a forecasting model. The first is analogy and experience. Most day-to-day decisions we make are based on this approach, because it has the great advantage of conserving our scarce cognitive resources. Experience teaches us to direct our attention to certain cues in certain familiar situations (e.g., look for a traffic light if you round a sharp curve while driving your car, and see an intersection ahead). When those cues are present (e.g., a traffic light is present, and the light is red), they trigger the automatic use of forecasting models that have been used so often that they have become intuitive (e.g., the driver of the car behind me will also see the light and slow down; there is significant probability of getting into an accident or receiving at traffic ticket if I do not stop for the red light; the light will eventually turn to green, etc.).

However, there are times when we don't recognize a situation, either because the situation is unfamiliar, or the cues differ from our expectation. A classic example of this is the first time a newly arrived Canadian or American driver confronts a simultaneous red/yellow traffic light in the United Kingdom. The failure of experience and analogy to quickly provide an appropriate forecasting model typically triggers a quick mental search for a theory that can be used to create one. For example, “red means stop; yellow means caution; therefore I should stop with extra caution.” In this case, our driver will quickly learn that the red/yellow combination actually means “the light is about to change to green.” The red/yellow combination will be consciously added to the driver's mental forecasting model, whose use will again become automatic, in keeping with the principle that human beings try to conserve scarce cognitive resources.

Now consider what would happen if our driver was confronted with a signal containing three lights arranged in a triangle, with the top one flashing blue. Clearly, experience is not helpful here, nor is there likely to be a readily available theory that can be used to quickly construct a forecasting model. In this case, considerable cognitive effort is required to identify the key variables in the situation, develop a theory about how they are related to each other, estimate their most likely future values, predict the future values of the target variables (e.g., how will the car in back of me behave, or the truck coming towards me, or the cars crossing in front of me?), and decide how to act (e.g., stop, slow down, etc.). Clearly, a lot of assumptions are involved here, which differ not only in their potential importance but also in their degree of uncertainty. This triggers yet another mental process, which identifies the critical “linchpin” assumptions that are both highly important and highly uncertain (e.g., the car behind me will also slow down, even though the driver is talking on his mobile), which must be monitored with scarce cognitive resources while carrying out the chosen course of action. It is easy to see how situations like this produce anxiety and mental exhaustion.

In sum, forecasting models and parameter estimates come from three sources, in ascending order of cognitive difficulty: analogy/experience, theory, and analysis of a specific (and usually novel) situation. To move back into the realm of finance, let's now consider the different ways someone could approach this question: what will be the rate of return on domestic equities over the next year?

A professional equities trader at an investment bank might answer on the basis of an intuitive model grounded in her experience. Another investor might use a theory that says the rate of return the equity market is expected to supply is equal to the current dividend yield plus the rate at which dividends are expected to grow in the future.

A third investor might take a much more deliberate approach, and consider not only the fundamental variables that will affect equity values over the next year (e.g., the outlook for economic growth, interest rates, corporate cash flow, and the like), but also those affecting the future actions of other investors (e.g., current momentum and mood, the potential for near-term political crises, changes in the balance of fear versus greed, etc.). This could take the form of either a substantive qualitative analysis (e.g., as is often found in brokers' investment strategy reports), or an elaborately specified quantitative model.

Regardless of the approach used to develop a forecast, the three potential sources of forecast errors remain constant. The first is known as “model error”, which includes getting the independent variables and/or the relationships between them and the target variable wrong. The second is known as “estimation error” which means making an incorrect assumption about the future value of one or more independent variables.

The third source of error is known as “non-stationarity.” This refers to a situation in which a model that accurately explains the past values for the target variable fails to do so in the future because either the relationships between the independent variables or the processes driving their future values have changed in an unanticipated way. In the context of the three approaches to forecasting model formation described above, non-stationarity refers to the use of an approach that has worked in the past (e.g., experience), even when the current situation is so different that it no longer applies, and an alternative approach (e.g., explicitly assessing a situation) would make more sense. In our view, there are two reasons humans seem particularly vulnerable to this source of forecast error. First, given our limited cognitive resource, we have a tendency to err on the side of conserving them, preferring easier approaches to forecasting model development to ones that require more energy. Closely related to this is the so-called “confirmation bias” which causes us to give greater weight to information which confirms our current view, and less weight to information that conflicts with it. Indeed, the confirmation bias is fully consistent with the old saying that “it takes twice as much information to change an opinion as it does to form one.”

Indeed, our susceptibility to non-stationarity error and the confirmation bias may have neurochemical roots. In “Uncertainty, Neuromodulation and Attention,” Yu and Dayan begin by asserting that “making inferences about the state of the world and predictions about the future based on many different kinds of uncertain information sources is one of the most fundamental computational tasks facing the [human] brain.” They then note that Bayesian statistical theory quantifies this problem, and provides a rational approach to updating our views based on the receipt of new information. Yu and Dayan distinguish between “expected uncertainty” and “unexpected uncertainty.” The former “arises from known unreliability of predictive relationships within a familiar environment”, while “unexpected uncertainty is induced by gross changes in the environment that …strongly violate expectations.” The authors go on to show how two different brain chemicals - acetylcholine and norepinephrine - are involved when we confront expected and unexpected uncertainty. This suggests that anything that affects their levels and functioning will affect our susceptibility to non-stationarity error and the confirmation bias.

Outside the world of neurochemistry, other researchers have recently provided us with new insights into the extent and causes of forecasting error. In “Economic Forecasting: Some Lessons from Recent Research”, David Hendry and Michael Clements (two leaders in the field) conclude that the most important sources of forecast error are related to non-stationarity. Another recent paper, “Tactical Asset Allocation and Model Uncertainty”, David Rey uses historical data from the Swiss equity market, and examines the relative contribution to forecast error over time of model error, parameter error, and non-stationarity. He finds that “the relative contributions are highly dependent on the time period under consideration.” We view this finding as consistent with our view that financial markets function as a complex adaptive system, which are characterized by varying periods of high and low average forecast errors.

Bacchetta and van Wincoop provide further evidence of this in their paper “Higher Order Expectations in Asset Pricing.” They start with a view we strongly share: that accurately forecasting future asset prices involves consideration not only of the fundamental factors driving their value (e.g., the current dividend yield, expected dividend growth, current real government bond yield, and equity market risk premium), but also the variables that will affect the future actions of other investors. The authors show how incorrect assumptions about future investor behavior can cause asset prices to substantially diverge from their fundamental value.

An important question in finance theory is whether forecast errors are random or whether some investors make them in a predictable way. In “Predictablility in Financial Markets: What Do Survey Expectations Tell Us?” Bacchetta, Mertens, and van Wincoop analyze survey data on investors expectations in the stock, bond, money and foreign exchange markets. They “find systematic evidence of predictable expectational errors across markets, sample periods and countries.”

This raises an obvious question: what causes these predictable forecast errors? Broadly speaking, there are two schools of thought. The “behavioral school” believes the underlying cause is investors' limited cognitive resources, and less than perfect rationality, as evidenced by the confirmation bias. In “Does Adaptive EPS Forecasting Make Analysts Forecasts Redudant?” Dimitri Kantsyrev provides interesting new evidence on this point. He compares the accuracy of stock analysts' earnings forecasts with ones produced by a statistical forecasting model. In the past, these types of comparison have typically used a time series forecasting model whose terms do not change over time. Unsurprisingly, these studies have found that, because analysts can adapt to new information, their forecasts are more accurate than those produced by unchanging statistical models. Kantsyrev's innovation is the use of an adaptive neural network model. Made possible by modern high-powered computers, neural network models constantly “learn”, in the sense that they automatically identify changing patterns in historical data, use them to specify a forecasting model, examine their own forecasting errors, and then update the forecasting model accordingly. In this manner, they minimize the impact of non-stationarity as a source of forecasting error.

Kantsyrev found that the adaptive neural network model outperformed analyst forecasts for companies with highly volatile earnings and over longer time horizons. The adaptive neural network model was particularly good at predicting downward changes in earnings. In contrast, the “analysts' forecast bias [errors] increased with the volatility of earnings.” In our view, this vividly demonstrates, how the impact of non-stationarity is magnified by the confirmation bias. Kantsyrev draws an even more aggressive conclusion: “financial analysts mainly predict the overall market behavior, and have a lack of ability to predict firm-specific fluctuations.” Not exactly a ringing endorsement of active management (at least by humans!).

The second school of thought sees predictable forecasting errors as caused not by cognitive shortcomings, but rather by a rational process. In “Rational Inattention: A Solution to the Forward Discount Puzzle”, Bacchetta and van Wincoop start with a question that has puzzled many analysts (ourselves included): why does uncovered interest rate parity (UIP) not seem to hold in the short term? For those of you who are scratching your heads, UIP refers to the theoretical relationship between interest rates and exchange rates in two countries. In theory, a difference in interest rates should be offset (less any transaction costs) by an opposite difference in exchange rates, to eliminate the possibility of earning a higher profit (in one currency) by investing in the other country's bonds. For example, if Australian bonds yield 5% more than U.K. bonds, UIP suggests that the Australian dollar should depreciate by 5% against the U.K. pound.

Bacchetta and van Wincoop note “there are significant costs associated with collecting information, processing information, and making decisions based on that information. These costs are added to the usual transaction costs.” Since investors vary in the size of the trades they can make, they also vary in their ability to profit from the collection of information. “This makes it optimal for many investors to only infrequently assess the available information and revise their portfolios. [Many] investors may therefore be 'rationally inattentive', which gives rise to predictable expectational errors” and deviations from uncovered interest rate parity.

A somewhat different line of research has addressed whether or not equity market returns are predictable in advance. Needless to say, there are competing and very strongly held views on this critical question. In “A Comprehensive Look at the Empirical performance of Equity Premium Prediction”, Goyal and Welch conclude that the answer is “no.” They find that none of the forecasting models they examined “would have helped an investor with access only to information [about predictor variables] available [in real time] to time the market.” They conclude that a simple forecast based on historical returns is the best approach. This view is challenged by Campbell and Thompson in “Predicting the Equity Premium Out of Sample: Can Anything Beat the Historical Average?” They conclude that some forecasting variables (e.g., the Price/Earnings ratio) can outperform the historical average, though “their predictive power is small but [still] economically meaningful.” However, the authors also note “a variable is quite likely to have poor [forecasting] performance for an extended period of time even when the variable genuinely predicts returns with a stable coefficient.” They wisely conclude that “the saying 'if you're so smart, why aren't you rich?' applies with great force here, and should lead investors to suspect that highly successful [forecasting models] are spurious.”

In “Reconciling the Return Predictability Evidence”, Lettau and Nieuwerburgh show that taking non-stationarity of the predictor variables into account resolves the apparently contradictory findings of the “predictable returns” versus “unpredictable returns” schools. So far, so good. However, this still leaves the investor with the challenge of forecasting non-stationarity, for which the authors offer some initial suggestions. As you can see, rather than solving the fundamental forecasting problem, this approach simply shifts it to another level.

Finally, no discussion of forecasting error would be complete without mention of Philip Tetlock's outstanding new book, Expert Political Judgment. It is a massive analysis of over twenty years of forecasts produced by a wide variety of experts. Unsurprisingly, it finds that experts are subject to the confirmation bias, find it difficult to learn from their forecasting mistakes, and are outperformed by forecasts made by quantitative models unaffected by emotion or a scarcity of (not always perfectly rational) cognitive resources. In most cases, they perform no better than non-experts.

Tetlock's most intriguing finding is what he calls the contrast between the “hedgehog” and the “fox” styles of forecasting, which are used by experts and non-experts alike. The former tends to apply a single theory to make forecasts under different circumstances. In contrast, rather than relying on a single theory, the fox tries to make sense of situations based on their own logic. Tetlock finds that while hedgehogs are more popular with the media because of the simplicity and certainty of their views, their actual forecasts are outperformed by those made by the foxes. He notes that “the foxes' self-critical, point-counterpoint style of thinking prevented them from building up the sorts of excessive enthusiasm for their predictions that hedgehogs, especially well-informed ones, displayed for theirs. Foxes were more sensitive to how contradictory forces can yield stable equilibria, and, as a result, “overpredicted” fewer departures from the status quo. But foxes did not mindlessly predict the past. They recognized the precariousness of many equilibria, and hedged their bets by rarely ruling out anything as impossible.” On the other hand, Tetlock cautions that foxes can be excessively open minded and prone to confusion caused by “seeing too much merit in too many stories.” On balance, however, Tetlock concludes that “the dominant danger remains hubris, the mostly hedgehog vice of closed-mindedness, of dismissing dissonant possibilities too quickly.”

All of these analyses beg a final question: what can be done to improve our forecasting performance? The key seems to be the ability to adapt one's forecasting model quickly once non-stationarities are discovered. Anticipating in advance these abrupt changes in the structure of the environment seems to be out of the question; the best we can hope to do is quickly react to them. In “Economic Forecasting: some Lessons From Recent Research”, Hendry and Clements make the important point that the use of simple models is not the same as adaptability. To be sure, simple forecasting models facilitate adapability but they are not one in the same. For example, Kantsyrev's earnings forecasting model, while highly adaptive, is anything but simple. Moreover, Tetlock cautions us against the hubris and over-confidence bias (and, perhaps, neurochemical changes!) that simple, successful models often create in their users.

Another technique that has been shown to minimize the risk of non-stationarity errors is the combination of forecasts made using different models. This is the approach we use in our asset allocation models, which combine asset class forecasts made using both historical data and a forward looking asset pricing model. In “Structural Breaks and the Performance of Forecast Combinations”, Aiolfi and Timmerman show why combining forecasts, often using very simple equal weighting schemes, usually works better than relying on a forecasting single model.

Finally, forecast combination does not automatically require the use of quantitative models. In the world of defense and intelligence, “Red Teaming” (also known as “competitive analysis”) is becoming more widely used. In this process, an outside team is used to explicitly challenge a forecast made by an organization. While this can take many forms, two of the most common are (a) assuming a critical uncertain variable has turned out differently than the base case plan assumes, and developing an alternative action plan, and (b) assuming (in hindsight) that the base case plan has failed, and developing a detailed story of why this happened, what could have been done differently, and what warning indicators were missed. In both cases, the end result is a comparison of the base case plan with the alternative one, leading to insights about the implications for key decisions facing the organization (e.g., wait, hedge, go ahead, etc.), and the most important warning indicators to monitor.

So where does this leave us as investors? We began with two questions, whose answers depend on our beliefs about the efficacy of forecasting. Is there any reason to hold something other than an equally weighted portfolio of broadly defined asset class index funds? And is there any reason to pursue active management, either by opportunistically changing asset class weightings, or going long and short individual securities within them?

Our answer to the first question is a qualified “yes.” We start with the assumption, that, because of differing goals and risk preferences, investors will want to hold portfolios with differing risk/return characteristics. This is true even in the absence of differing investor forecasts about different asset classes' and securities' risks and returns. We then make four observations. First, there is evidence that over the long term, investors are compensated with higher returns for holding riskier assets. “The Risk Return Trade Off in the Long Run: 1836 to 2003” by Christian Lundblad is a good example of this research. The second observation, however, is that study after study has found that it is very hard to accurately forecast future asset class returns. On the other hand, the third observation is that the ranking of asset classes by their relative riskiness is quite consistent over time. Triumph of the Optimists by Dimson, Marsh and Staunton is one of the best studies on this point. The fourth observation is another cautionary one, in that the correlation of returns between different asset classes (a key component, along with individual asset class risk, of aggregate portfolio risk) is not stable over time.

These four observations lead us to two conclusions. First, there appears to be a strong case for departing from an equally weighted asset class portfolio, in order to better satisfy investors' differing risk preferences, based on the observations that asset class risk rankings are relatively stable and that higher risk asset classes tend to earn higher returns. We do not believe this argument is undone by changing return correlations over time.

Our second conclusion is less strongly held: that there is also a case for departing from an equally weighted asset class portfolio in order to better satisfy investors' differing return goals, within their specified risk constraints. While we believe that, over time, higher risk is rewarded with higher returns, and while we have taken prudent steps to limit the possibility and potential impact of forecast error (e.g., using asset class return forecast combinations, as well as constraints on the maximum weight for different asset classes), we have no doubts about the inherent difficulty of the task. For that reason, we stress that our asset allocation recommendations are in no sense optimal; rather, our objective is that they are robust enough to achieve, with a minimum probability, a given long-term real return under a wide range of possible future asset class return scenarios.

And what of the second question? Does our review of the latest research about forecasting change the generally unfavorable evaluation of active management we presented in our book Indexing Versus Active Management: The Trial of a Prudent Investor? On the one hand, there is a high probability that in a rapidly changing world economy, non-stationarities are becoming more frequent. At the same time, Philip Tetlock provides ample evidence that human beings' forecasting skills have not similarly improved. On the other hand, Kantsyrev's paper, along with ample anecdotal evidence about the quantitative modeling arms race now underway in the hedge fund world suggest that highly adaptable forecasting models exist. In addition, there is also the possibility that an active manager can make a superior forecast not because he or she has a superior model, but because, due to superior information, he or she can more accurately estimate the value of key model parameters.

However, this gives rise to three more questions: (1) can investors forecast, with any accuracy, which hedge funds that possess accurate forecasting models or information advantages? (2) More important, can investors forecast with any accuracy the probability that these models or information advantages will be able to successfully adapt to future non-stationarities? And, finally, (3), even if an investor answers “yes” to the first two questions, can he or she forecast with any accuracy that the hedge fund's fees will not fully offset the additional returns (above an index fund) the superior forecasting model and/or information will generate? We do not doubt that some investors will answer, “yes” again, and will, through luck or skill turn out to be right. On the other hand, we are highly doubtful that, in the face of financial markets that function as a complex adaptive system, the great majority of investors, and in particular individual investors, can play this active management game for many years and come out ahead.

Why then, do so many actively managed funds continue to exist? We think there are at least three reasons. The simplest, and least likely explanation in our view is that active managers spend much, much more on advertising than index managers. In addition, given the value of active managers' advertising spending, many mainstream publications have a clear incentive not to publicize the advantages of indexing, whatever their claims of separation between their business and editorial operations.

However, we suspect that two other explanations are more important. The first is the interaction of three well-known biases in human thinking: our tendencies toward excessive optimism, overconfidence, and underweighting evidence that conflicts with our most important beliefs (and beliefs backed by one's savings must surely be important!). Too many of us believe that we (or our fund manager) will be the one who beats the market. And indeed, most of us do, from time to time. At some point, many people develop a superior insight that results in a significant investment return, which strengthens their belief in their own (or their manager's) skill. What we forget is the difference between doing this once, and doing it consistently year after year.

However, at a deeper level, it may be that a second explanation is the most important one: our deep emotional reluctance to confront the true degree of uncertainty we face and the weak ability of our forecasting tools to penetrate it. Perhaps it is our need to maintain some illusion of control that leads such a high proportion of investors to prefer active management.

On the bright side, we also note that the obstacles to indexing seem to be eroding, as evidenced by the increasing popularity of exchange traded funds, the growing percentage of institutional money that is passive managed, and the rising interest in separating alpha from beta investing, as more investors realize the inherent difficulty of active management, and allocate more of their risk budgets to a portfolio of index products that is diversified across asset classes. All of these trends make us hopeful that more and more investors are focusing their active management efforts not on their financial investments, but rather on those economic assets where it can produce the highest returns: the productivity of their human capital (i.e., their education and careers) and their investments in residential property.