P-Hacking versus Skin in the Game: How can understanding incentive structures help us think about market efficiency?

Since Ioannidis made his argument, academics in various fields have worked to test whether published results could be achieved (or ‘reproduced’) outside of their original study, and finance was always due a similar review. Last month three US authors (Kewei Hou, Chen Xue, and Lu Zhang) published “Replicating Anomalies” which presented their results in attempting to reproduce what they call the ‘entire anomalies literature in finance and accounting.’

This is significant because anomalies are those variables which have been held up to suggest markets are inefficient, and which can form the basis arguments in favour of active management, including the trading rules which can lie behind quantitative strategies.

From the failure to reproduce a majority of previously identified anomalies, the authors draw the conclusion that ‘capital markets are more efficient than previously reported’, and by implication pose a significant challenge to active management and behavioural finance.

Paper 2: Value is not a number

The findings of Hou, Xue and Zhang echo the results of another 2017 paper, titled “Facts about Formulaic Value Investing,” which similarly appears to falsify some pre-existing theories but draws an entirely opposite conclusion as a result.

In this paper, the authors review the returns from backing traditional value metrics (price to book, trailing price to earnings, forward price to earnings) and suggest that there is little evidence that traditional value investing using simple equations results in excess returns. In fact, they contend that though valuations tend to be mean reverting, this is primarily due to shifts in fundamentals (i.e. apparently cheap stocks turn out to be value traps rather than return opportunities) – at least in the US between 2002 and 2014.

Interestingly however, whereas Hou et al. take the apparent falsification of anomalies to argue in favour of market efficiency and against active management/behavioural finance, the authors of the valuation paper take it as evidence that there is a greater need for human insight. In their view, the failure of simplistic metrics suggests that “a capable analyst…should be able to significantly enhance quantitative approaches” by identifying if a value signal is the result of inflated fundamentals or a genuine opportunity.

Paper 3: Academic incentives versus skin in the game

Both ‘Replicating Anomalies’ and ‘Facts about Formulaic Value Investing’ have already seen relatively robust responses. The anomalies paper has been criticised for removing microcaps and using value rather than equal weighting in its sample (it is perhaps no surprise that the returns to strategies such as backing illiquidity are reduced if you strip out some of the least liquid stocks from the sample). As for the value piece, the always worth-reading Wes Gray has written a blow-by-blow assessment.

I won’t presume to engage in these technical debates but there are some broad observations that can be made:

First, it should be no surprise that data mining goes on in the field of academic finance. The incentives to find and publish evidence of quantitative strategies to beat the market are huge, the data relatively limited (the vast majority of studies only look at the US equity market, often using the same ‘Fama-French’ data), and the interest in negative results is virtually zero.

More importantly, it is worth bearing in mind that the incentive structures faced by academics are very different to those of investors in ‘the real world.’ In fact a third 2017 paper, co-authored by the same John Ioannidis who initially sparked off the reproducibility crisis in 2005, seeks possible remedies to academic incentive structures by using lessons from the world of economics.

While it may suffice for academics to have research published and leave it at that, investors ultimately have to test any theories they have ‘out of sample’ if they are to make any money. They have skin in the game. Although ‘false positives’ may linger in the world of academia, in the market they will ultimately be weeded out and successful strategies rewarded.

Unfortunately it is the very competitive nature of the incentive structures that exist in economics and financial markets which add to the complexity involved in empirical studies.

Unlike some areas of science, in finance you cannot identify a single, unalterable truth. Reflexivity means that the very discovery of an anomaly may alter the behaviour of market participants and an evolutionary system of trial and error which magnifies successes and eliminates failures will itself change the environment. For example, if everyone comes to believe in a strategy that generates risk free returns, then that opportunity is more likely (but not guaranteed) to dry up.

Adaptive Markets: A resolution?

This brings us to the last and perhaps most important piece of academic work so far this year, Andrew Lo’s new book “Adaptive Markets.” Lo takes the idea of evolutionary trial and error (survival of the fittest) as a means of reconciling the apparent conflict between efficient markets theories and behavioural finance.

Lo, like Mordecai Kurz before him, suggests that investors can take the same information and come up with different beliefs without that making some investors ‘irrational.’ One set of investors may have a shorter time horizon than others, different groups of investors will have different objectives in terms of return, and a wide range of strategies may be followed.

Importantly, competition between these various groups in a world of ‘survival of the fittest’ will mean that the cohort of investor types making up the market will change over time. Certain groups will drop out of the population, others will grow in significance (think of all those value investors who lost their jobs during the tech bubble). This will change the nature of the opportunities available to investors. In a 2004 paper (sadly not available online for free) Lo highlights some important implications of his hypothesis:

• to the extent that a relationship between risk and reward exists, it is unlikely to be stable over time
  
  
• new opportunities are being created continually as certain species die out, as others are born, and as institutions and business conditions change
  
  
• Investment strategies will wax and wane. Rather than arbitrage opportunities (i.e. anomalies) being constantly competed away for good, these strategies may decline for a time and then return to profitability when environmental conditions become more conducive

I believe Lo’s book could be a clear way of expressing some our own team’s views of how markets work, in particular the ideas that market opportunities are dynamic and that inefficiencies (however defined) do periodically arise for active investors to exploit. Moreover these opportunities seem unlikely to be something that will exist consistently over time or in a way that can be captured by mechanistic approaches and annual rebalancing. Our experience is that active outperformance will be ‘lumpy’ rather than something that can be ground out month after month, year after year.

In a sense these various academic exercises represent the ongoing attempt of economics to compare what works in practice with what works in theory. As the failure to replicate statistic studies show: empiricism without logic is dangerous; we must always seek to have a clear reason for why a relationship may exist. As the weaknesses of simplistic value metrics in certain environments demonstrate: we can never delegate thought to an equation. And as the notions of competition, survival of the fittest, and incentive structures illustrate: the world is always changing and investors will need to be able to change with it.