Sunday, May 6, 2012

Monte Carlo Simulations vs. Historical Simulations


A second approach to studying retirement withdrawal rates is to use Monte Carlo simulations which are parameterized to the same historical data as used in historical simulations. This can be done either by randomly drawing past returns from the historical data to construct 30-year sequences of returns in a process known as bootstrapping, or by simulating returns from a distribution (usually a normal or lognormal distribution) that matches the historical parameters for asset returns, standard deviations, and correlations. The relevant characteristics of the historical data used in typical Monte Carlo simulation studies are provided in Table 2.2.
Table 2.2
Summary Statistics for U.S. Real Returns Data, 1926 – 2010




Correlation Coefficients

Arithmetic
Means
Geometric Means
Standard Deviations
Stocks
Bonds
Bills
Stocks
8.70%
6.62%
20.39%
1
0.08
0.09
Bonds
2.52%
2.28%
6.84%
0.08
1
0.71
Bills
0.69%
0.61%
3.90%
0.09
0.71
1
Source: Own calculations from Stocks, Bonds, Bills, and Inflation data provided by Morningstar and Ibbotson Associates. The U.S. S&P 500 index represents the stock market, intermediate-term U.S. government bonds represent the bond market, and bills are U.S. 30-day Treasury bills.

A downside of Monte Carlo simulations is that they do not reflect other characteristics of the historical data that are not incorporated into the assumptions, such as the possibility of serial correlation in returns, or the possibility of mean reversion guided by market valuations. Another downside is that the results of Monte Carlo simulations are only as good as the input assumptions, though when thinking about future retirements, historical simulations are likely to be even more disadvantaged by this issue. See "Lower Future Returns and Safe Withdrawal Rates" for more on this. Overall, the advantages of Monte Carlo simulations likely more than make up for any deficiencies with respect to historical simulations.
Monte Carlo simulation has the advantage of allowing for a wider variety of scenarios than the rather limited historical data can provide. Between 1926 and 2010, there are only 56 rolling 30-year periods. And as is about to be discussed, these 56 periods are not independent of one another. Meanwhile, it is not uncommon to see a Monte Carlo simulation study based on 10,000 simulated paths of financial market returns. This provides an opportunity to observe a much wider variety of return sequences that support a deeper perspective about possible retirement planning outcomes than can be provided with the limited historical data.
Another advantage of Monte Carlo simulations, relative to historical simulations, is that because of the way that overlapping periods are formed with historical simulations, the middle part of the historical record plays an overly important role in the analysis. With data since 1926 and for 30-year retirement durations, 1926 appears in one rolling historical simulation, while 1927 appears in two (for the 1926 and 1927 retirees). This pattern continues until 1955, which appears in 30 simulations (the last year for the 1926 retiree through the first year for the 1955 retiree). The years 1955 through 1981 all appear in 30 simulated retirements. Then a decline occurs as 1982 appears in 29 simulations, through 2010 which only appears in one simulation (as the final year of retirement for the 1981 retiree). This overweighted portion (1955-1981) of the data tends to coincide with a severe bear market for bonds. During these years, the real arithmetic return on intermediate-term government bonds was -0.1%, compared to an average of 3.7% for the combined years prior to 1955 and subsequent to 1981. The differences are even more severe for long-term government and corporate bonds. 
On the other hand, Monte Carlo simulations treat each data point equally. The middle years do not play a disproportionate role in determining outcomes. As a result of this discrepancy, a point highlighted particularly well by Dick Purcell (who just started a website on investor education) in the “Trinity Study Authors update their results” discussion thread at the Bogleheads Forum, Monte Carlo simulations of the 4% rule based on the same underlying data as historical simulations tend to show: (1) greater relative success for bond-heavy strategies, (2) less relative success for stock-heavy strategies, and (3) lower optimal stock allocations.
Figure 2.6 provides specific results, comparing the portfolio success rates for varying asset allocations when using a 4% withdrawal rate. When using intermediate-term government bonds, 4% withdrawals did not fail over 30 years in the historical data for stock allocations between 40 and 70 percent. More bond-heavy portfolios experienced much lower success rates, though, with a bonds-only portfolio providing success in 38% of the historical simulations. Such results may scare retirees into holding more stocks than justified by their risk tolerance or by reality. This is especially the case as historical simulations may also induce overconfidence about the potential success of stock-heavy portfolios. With Monte Carlo simulations based on the same historical data, retirees would be encouraged to hold some stocks, but success rates of over 90 percent are possible with stock allocations of only 20 percent. The highest success rates occur in the range between 30 and 50 percent stocks. 

10 comments:

  1. Wade –

    That point you make about Monte Carlo analyses standardly excluding mean reversion is commonly viewed as a weakness. But it may – just may –make Monte Carlo better (or “less worse”) in our attempts to assess probabilities for longer-term stock market investments.

    Back on April 13 you posted about potential of lower future return rates. (http://bit.ly/JI9zYy)

    There and elsewhere, you’ve pointed out enough about our future-assumption uncertainties to wobble our confidence in all this future-probabilities stuff. But if in the stock market as a whole, reversion will continue in the future nearly the same as it has in the past, Monte Carlo’s not including it is a pretty good “uncertainty cushion.”

    In Bogleheads months ago, I think it was Verde who reported that historically, reversion has been strong enough that for 30-year periods, the dispersion of results has been only about half what Monte would show using single-year SD and random walk. Turn that around, that means that for a 30-year investment result, Monte Carlo’s exclusion of reversion doubles the uncertainty spread of the distribution.

    For the 30-year retirement plans you assess, the cushion is not that cushy because some of the money (hopefully not all of it) is taken out before 30 years. Still, it’s cushy . . .

    Dick Purcell

    ReplyDelete
    Replies
    1. Thanks Dick, this is a good point. Monte Carlo would provide more conservative results by not relying on mean reversion after big downturns. It would also provide more overall optimistic results on the upside as well, but we should be focusing more efforts on the downside anyway.

      Delete
  2. Wade,

    OSFI (Canada's financial services regulator) has published their view on the use of mean reversion by insurers in capital models. They conclude that their use is not warranted. I think this generally supports Dick'
    s point. It is worth a read.

    I have been using a hybrid approach for retirement income planning . Instead of drawing from the historical distrubution 1 period at a time (which destroys structure) or using rolling periods (with all the issues you cite and then some), we select a starting point at random from the historical time series and then pull a block of returns of random length (say 1 to n years of monthly returns as an example) and build up the time series through repeated iteration. The blocks could also be defined explicitly and the recombined. It preserves fat tails, some of the correlation structure and (at least partial) market cycles while allowing the generation of multiple different scenarios. It also avoids many of the issues associated with paramater estimation. But in the end there is no escape from model risk.

    ReplyDelete
    Replies
    1. Robert,

      Thanks for the input. And thanks for mentioning about bootstrapping by putting together strings of sequences (such as 3 or 5 periods or more) instead of one at a time. I have played around with that too and I was thinking to mention it, but figured it was esoteric enough to leave out. Did you find that this lessened some of the more extreme outcomes?

      Delete
    2. It depends on how you implement it and what you compare it against. On the margins, monthly draws with replacement from a fatter tailed historical distribution will typically yield more extreme outcomes compared to monthly draws from the normal. As the sampled period size increases you can hit a cross over point, but if you sample with replacement you typically produce results that are at least as extreme... assuming parameterization based on the same historical distribution you are sampling from.

      Extreme outcomes are a double edged sword. For every double-great-depression generated there is an uber-bull market on the other side of the .
      distribution... and there are some deep rooted philosophical problems arising from the frequency approach to probability implicit in the simulation

      Delete
  3. Why is anyone utilizing historical simulations??? "Rolling periods" analysis is not statistics no matter how often Jack Bogle or the professors at Trinity do it. Your mildly stated reservations are too mild Wade. If I recall correctly the DJIA was up about 53% in 1954, how many times is that outlier of a year for stocks being counted? "Overlapping moving averages are not each independent data points or degrees of freedom." Paul Samuelson, Journal of Portfolio Management, Fall 1994.
    I think you are being soft on these folks.

    ReplyDelete
    Replies
    1. Thanks Fred. I see you vote for Monte Carlo.

      At the same time, some people swear by historical simulations. I am still shamed to have not read Jim Otar's books yet, but I looked at them enough to know that he hates Monte Carlo and thinks that historical simulations are better. However, I thought the reason he gave didn't make sense. He said to picture gazelles wandering around randomnly on the savannah until a lion shows up and then everyone runs off in the same direction... Monte Carlo misses that. But I disagree. Well, while Monte Carlo doesn't specifically model the appearance of the lion, some of the simulation outcomes will naturally look just the same as if that lion had appeared.

      Getting back on the subject, I think I do tend now to preferring Monte Carlo. But I can understand why some people like historical simulations better. It is nice to compare the results from each.

      Delete
    2. Wade --

      1. I agree: I also prefer Monte for assessing future probabilities, but we can learn from historical-sequence simulations too. EG, seeing mean reversion in history.

      2. Question: Are the benefits from PE-based strategies that you and Michael Kitces have reported entirely dependent on mean reversion?

      Dick Purcell

      Delete
    3. Dick,

      Let me just answer for myself. In the context of my safe savings rates article, I would say that the answer is yes. Historically, pre-retirement bear markets are followed by post-retirement bull markets and vice versa. Low sustainable withdrawal rates usually follow prolonged bull markets (so you have more wealth at retirement anyway).

      But with the i.i.d. assumptions of Monte Carlo, you can't rely on this. No matter how bad things were pre-retirement, you would still be left using the safe withdrawal rate since the post-retirement period is independent. This means getting very low income.

      Delete
    4. Sorry, but block bootstrapping is perfectly legit statistics. Who doesn't like central periods to be overrepresented can just use circular block bootstrapping instead, or correct the weightings.
      What is not statistics are "outliers", you can't pick only the data that looks nicer and pretend that to be representative. There are no outliers, only bad models; fit a t distribution and so called "outliers" (for a gaussian) will be taken care of nicely.
      Last but not least, historical simulation is just Monte Carlo on the empirical distribution, it's a false dicotomy. You can have all sorts of "hybrids" inbetween by using semiparametric models. Is there a LOT of data? Then go HS. Too little data? Sure, HS is bad, but so is MC too: fitting a normal/whatever not only imposes a false model, but has large estimation error anyway and gives a false sense of security and good coverage.

      Delete