One of the goals of this blog is to discuss various approaches to forecasting asset returns taken from both the economics and machine learning fields. Before diving into specific models and techniques, however, I begin by discussing the issue of noise in financial markets.

Let’s start by decomposing returns into expected and unexpected returns. We have

where is return in period t, is expected return in period t conditional on the information available at t-1, and is unexpected return.

The first thing to notice is that  is the noise component and cannot be predicted. The expected return is the component that we hope to model, but is unobserved. We could use various models, but a simple choice for modeling  is to use a linear function of a predictor variable . Thus, we have

where I assume a zero intercept for simplicity and is an error term. We want to run a linear regression, but since is unobservable, we must first identify observable proxies. The usual approach, of course, is to use realized returns to proxy for expected returns . Most of the time we follow this procedure without thinking much about it. Why do we bother with pointing out the distinction between expected and realized returns here?

The first reason is that while realized returns are an unbiased estimator of expected returns, the variation in is much larger than the variation in in the data. In other words, the signal-to-noise is very low. The high noise component in realized returns results in imprecise estimates of in [2] when using realized returns as a proxy, making the task of identifying predictive variables very difficult.

The second reason, from Elton (1999), is that some of the return surprises may be so large or that a sequence of surprises so correlated that they do not cancel out over the time period being studied and will significantly influence the mean of the realized returns. Using US data, Elton observes that these periods can be quite long:

There are periods longer than 10 years during which stock market realized returns are on average less than the risk-free rate (1973 to 1984). There are periods longer than 50 years in which risky long-term bonds on average underperform the risk free rate (1927 to 1981). Having a risky asset with an expected return above the riskless rate is an extremely weak condition for realized returns to be an appropriate proxy for expected returns, and 11 and 50 years is an awfully long time for such a weak condition not to be satisfied.

More recently, if we have a data set that covers, say, only the last 10-15 years, the Great Recession will have a very strong influence on most attempts to construct a forecasting model.

What can we do about the situation? One idea is to try to remove the noise ex-post from our sample of historical returns. In places like IEEE conference proceedings, you sometimes see papers that apply smoothing techniques developed in engineering applications to filter noise from returns. Robust M-estimators from the statistics literature can also be used to identify influential observations. Finally, we might try examining the news during periods of unusually large return magnitudes in order to determine whether the returns represent expected or unexpected shocks. Of course, all of these methods are imperfect since it is extremely difficult to determine what was unexpected even with the benefit of hindsight.

Another idea is to obtain a proxy for expected returns that does not rely on realized returns. The leading approach, in the context of stocks, is the implied cost of capital (ICC), which can be thought of as essentially the internal rate of return (IRR) of the dividend discount model. For a given stream of expected future dividends, a lower stock price implies that the ICC is higher. Of course, expected dividends going years into the future are unobservable. Therefore most models use analyst earnings forecasts for the first few years together with some terminal value assumption for the years beyond . Since the ICC should contain much less noise than realized returns, the approach may allow for more precise inferences about which variables can forecast the market. Li, Ng, and Swaminathan (2013) provide a recent example of the ICC approach to predicting stock market returns. A cautionary point is that there are many possible ways to construct and estimate the ICC. The fact that we don’t know the true model introduces additional uncertainty to our estimate.

So while we usually end up using realized returns as our proxy for expected returns, the decomposition in equation [1] is useful keep in the back of our minds to remind us that they are not the same thing and that sometimes we will want to take another approach.

References

Elton, E. 1999. Expected return, realized return, and asset pricing tests. Journal of Finance, 54.4, 1199-1220.

Li, Y., D. Ng, and B. Swaminathan. 2013. Predicting market returns using aggregate implied cost of capital. Journal of Financial Economics, 110.2, 419-436.