Efficient Market Frontier

Recently I gave a talk on EMH: https://www.lesswrong.com/posts/3TiEZzw4ikneLGp4J/dissolving-the-is-the-efficient-market-hypothesis-dead

In there I had a bonus slide about what I call “Efficient Market Frontier” (EMF). In this post I want to expand on EMF, since I feel like a lot of disagreements and conversations around EMH and the market in general can be helped with this.

But first I hope you read my previous post: “The Holy Grail” of portfolio management. The importance of uncorrelated strategies will play an important part here.

Also, by request, here’s a quick recap of most technical terms I use in this post:

Market: the set of things you’re trading.
Backtest: see how your strategy would have performed historically by simulating it on relevant data.
Liquidation: losing all your money (but for certain specific reasons not relevant here)
Short(ing): betting on the asset price going down.
Limit order: you commit to buying if the price goes sufficiently low. If the price never goes that low, your order is not filled. (The flip side is selling if the price goes sufficiently high.)

The market is a game

I also want to reiterate an important assumption I’ve made in my talk. It will no doubt color my post and I think it might be somewhat contentious, so better state it upfront.

I think fundamentally the market is a game. You’re in it to make money. At the end of the day you can have all the theory and correct pricing and best backtested strategies, but if the market “mistakenly” decides Zoom Technologies stock should go up, and you short it because “haha the market is so wrong” and get liquidated as more “mistaken” investors pile on… Well, then it seems like you lost that game.

In this view the primary reason to predict a “correct” price of anything is because it will help you make profitable trades. It’s not because you believe that Tesla is worth $350.09B.

How do you play the game?

Your strategy is how you play the game. I’ll define a strategy as an algorithm made of three parts: trigger, positioning and exit.

(You can and should, of course, run multiple strategies. How they interact together was covered my previous post. For now let’s focus on just one strategy.)

Trigger: this is what makes you consider a trade in the first place. If you have an automated algorithm that runs every minute, then the end of a minute bar is the trigger. Or if you’re occasionally paying attention to interesting news then it’s: “I notice an interesting piece of news that makes me consider a trade.”

Positioning: this is how you determine what you’re actually buying / selling, how much, and how you’re adjusting it while in the trade. Do you buy a stock or buy options? Do you enter the trade slowly or all at once? Do you double down when the market moves against you? Do you let it ride or slowly take profit? Or maybe you just play it by ear, in which case that’s the strategy: “I use my brain to determine what do in every case.”

Exit: how do you finally close this position? Do you have a time horizon (like exiting after two years)? Do you have a stop loss? Do you take profit? Is there an external event you’re waiting for? Do you exit when you can’t bear the pain of a losing position? Or do you exit when it feels right?

One trade can be described as one full cycle through trigger, positioning and exit. You haven’t really made money, i.e. the trade isn’t profitable, until you’ve exited your position.

Strategy types

Obviously I can’t cover them all, but the natural categories that stand out to me are: intuitive, algorithmic, formulary and technological. I think these are natural categories because they have specific niches where they excel. (More on that later.)

Intuitive: you just use your intuition. Take in all the information and use your brain to make the best call you can. Your strategy will be as unique as you are.

Example 1: AI development might lead to some companies getting a crazy advantage. Think through what those companies might be and buy their stock.

Example 2: You think you have a pretty good sense for great design, so any time you run across a well designed product you buy the stock of the company that made it.

Example 3: You heard about this cryptocurrency thing. Sounds like a ponzi scheme, but you threw in $100 because why not.

Algorithmic: you code it up. In practice there’s no 100% algorithmic strategy. Usually there’s still some manual discretion, but you can come close.

Example 1: all technical indicators, e.g. MACD.

Example 2: train a RL model and just let it trade.

Example 3: whenever Elon Musk tweets, buy Tesla stock and Dogecoin.

Formulary: this a mix between intuitive and algorithmic. (I came up with the name for this category. Is there a better one?) You define the strategy as rigorously as you can, like you would for an Algorithmic strategy. But the components are hard / impossible to evaluate using code, so you need to use your brain.

Example 1: when a company goes public, I will evaluate the CEO on a rigidly defined 10 point criteria. Each point is somewhat subjective, but I believe I can evaluate it pretty consistently across different CEOs. If they get a score of at least 8, I buy the stock.

Example 2: when a company’s stock goes down because of some PR nightmare, I will evaluate if I think it’s an overreaction. If it is, I will buy the dip.

Example 3: when I feel like a market is panicking—there’s a “blood in the streets” feeling—I wait a day and then start buying into that market a bit every day until it recovers.

Technological: this is based on getting a technological advantage over your opponents. You’re doing what they are doing just better, where “better” usually just means faster.

Example 1: most of high-frequency trading.

Example 2: many forms of arbitrage. This is where you make trades that are “guaranteed” to be profitable. For example, you buy BTC on Bitmex for $10,000 while at the same time selling it on Huobi for $10,050. You need the best infrastructure because you’re competing with other people to make the exact same trade for the exact same reasons.

Example 3: take satellite pictures of Walmart parking lots to estimate their sales for this quarter.

Finally: EMF

You’re at the Efficient Market Frontier when your strategy dominates your opponents’ strategies. The “size” of your edge is roughly how far you’re advancing that frontier.

By the way, it’s not true that if you made a winning trade the person you traded against made a losing trade. Even if your entry was their entry into an opposite position, their positioning and exit could still make it a profitable trade for them (not to speak of how this trade interacts with their other trades).

Now I bet the definition doesn’t sound quite helpful yet, so let’s have a demonstration… in THE ARENA!

In the left corner, the Intuitive strategy, weighing at 3 pounds of pure gray matter. Facing off against the Algorithmic strategy! Weighing at 5 pounds of pure silicon this newcomer is growing fast and itching to prove its merit. Fight!

Intuitive: (mid 2019) Wow, the crypto market sure has been going up a lot. The bulls are back in town! Buy, buy, buy!

Algorithmic: I have defined 12 types of bull markets. For each one I have calculated the distributions of their durations as well as 100 other factors that predict its end. This model has been backtested all the way to the beginning of the crypto market. The expected sharpe ratio is 2.3 and the probability of any given trade being successful is 52%. My current output is to go short.

Point for Algorithmic! Intuitive 0 : 1 Algorithmic

Intuitive: (2016) Hmm, this new cryptocurrency thing seems like a pretty different beast. I think over the long run there’s a good chance it’ll find its niche in the current economy. I think that niche is likely to be at least as important as gold. This means the market is just beginning to grow. I’m going to buy and hold for 5 years.

Algorithmic: I have nothing to say on this. Nobody programmed me to evaluate never-before-seen asset classes. Also there’s no way I’m holding the same position for 5 years, that’s not what I was built to do.

Point for Intuitive! Intuitive 1 : 1 Algorithmic

Intuitive: I decided to buy some bitcoin, but I placed some limit orders so I can get filled when the price goes down a bit. It goes up and down all the time, so I’m sure I can buy it at a discount.

Algorithmic: I have 25 TB of orderbook and trades data for this exchange. I have 35 different algorithms providing liquidity on both sides. I readjust my orders every 10 ms and can sense a mile away when a whale tries to move the market and adjust my limit orders accordingly. If I don’t, I know my orders will get filled as the most inconvenient time: I’ll be buying BTC right as the market is going down.

Point for Algorithmic! Intuitive 1 : 2 Algorithmic

Intuitive: (2020-03-12) The bitcoin price just crashed 50% in less than a day. This is an extremely unusual event the market hasn’t seen in a long time. By my estimate most of the leveraged long positions have been liquidated. Long term I’m still bullish on bitcoin, so I’ll start buying right now until the market recovers.

Algorithmic: The market crash was predictable and I made a lot of money. Now the volatility has piqued and we’ve entered a regime I haven’t seen before. I’m doing my best, but I haven’t been fed the liquidation data, so I’m not sure what’s going on there. However, I have noticed the volume in the market increase and historically that has led to short term momentum moves. I’m going to trade accordingly.

Tie! Intuitive 1 : 2 Algorithmic

Intuitive: Hmm, I just read about this virus that killed a few people in China. Seems pretty serious if true.

Algorithmic: There’s very little historic data on how pandemics affect the market, but it’s probably a good idea to bet on increased volatility. I’m programmed to scan for news about pandemics but so far I haven’t noticed a spike in the news article count.

Point for Intuitive! Intuitive 2 : 2 Algorithmic

I hope you get the idea!

Intuitive strategies excel in novel situations which are hard to evaluate algorithmically. Topical examples are: cryptocurrency, pandemics, and other rare events.

Algorithmic strategies excel in doing trades that could have been performed 1000+ times historically. That way you can backtest these strategies and have a pretty high confidence that they’ll continue working. For most markets this means most of the time there’s little advantage for the Intuitive strategy.

Any time an Intuitive strategy is trying to do something Algorithmic it is at a disadvantage. By the way this includes 99% of trading videos on YouTube. Any time you’re trading using indicators or bar patterns or chart lines, you’re likely going up against an algorithm that’s doing the same thing, except it has the benefit of having been trained on the most optimal parameters over 1000+ such trades.

The difficulty with Intuitive strategies is that they are basically impossible to backtest. You just don’t really know what you would have done back then using the information you would have had at the time. And a lot of the time there’s no clear back then either, because it’s a novel situation.

But it’s even worse than that! You don’t know your trigger. If it’s “anytime I feel like trading” then this will often coincide with other people trading for the same reason: they heard the same news, their portfolio was affected in the same way, or something else.

This also means that one successful intuitive trade is just not that impressive. (Same as winning one roulette roll.) If that’s the strategy you’re going to use, you don’t know in what other situations it’ll fire off and pull you into the market.

This is where you can take your Intuitive strategy and move it towards Formulary. This might give you some ability to backtest, but it’ll still have to be done manually. But even if you check your trade across 10-100 similar cases, that’ll give you some evidence of your strategy’s efficacy. It will also help prevent random events triggering your trades.

With an Intuitive strategy you have no idea if you’re at EMH because you can’t backtest. You can only start measuring it from the time you decide to measure. And your confidence in your strategy will grow over time as you make consistently profitable bets.

With Algorithmic strategies it’s much easier to test if you’re at EMH. You just backtest them. Unless you overfit on the historic data your strategies should have a similar performance going forward. (Until one day they don’t and then you hope that not all of them broke at the same time.)

More edges

To a large extent your strategy type defines the strengths and weaknesses of your strategy. But there are additional things you can do to help or hinder it.

Acting on easily accessible information
- If this information can be easily scraped and analyzed (all price data, news sentiment, etc..) then the Algorithmic strategies will likely have incorporated it already.
- If it takes some intelligence, creative thinking and rationality to fully understand the implications of this information, then it will take a while for the market to react to it, especially if this kind of information is rare.
- Relatedly, if the information is not easily accessible it’s more likely that you’re running a strategy that other people aren’t running.
Using known algorithms
- If you’re doing what other people are doing you need the Technological advantage.
Backtest-able
- If you strategy is backtest-able but you didn’t backtest it, you’re at a disadvantage to those who did.
Code-able
- If your strategy (or a close approximation) can be coded up, but you’re executing it manually, you might be at a speed disadvantage. You’re also likely at a disadvantage because parameter tuning is much easier done with coding assistance.
Time scale
- If your strategy is trading at very short timescale you need to go more Algorithmic. (You need the speed plus shorter time scales mean there’s more data for backtesting.)
- If your strategy is very long term, it’ll benefit more from the Intuitive approach, since there’s essentially less data for Algorithmic approaches to use.
Quick feedback
- If you can get quick feedback on a strategy, you’re at a disadvantage as soon as you start using this strategy to those who have used it before you and used the feedback to adjust.
- With slow feedback strategies it’s a lot more important to get it right the first time. This is where your brain can help you.
Palatable risk
- Some strategies have very hard to stomach risk profiles. For example, betting on mean reversion or volatility often results in months if not years of slowly bleeding money until you make it all back plus some in a few days. Most investors can’t stick to this strategy even if they are using an Algorithmic strategy.
Understandable
- If your strategy makes no sense, it’s less likely that other people have discovered it. That means there’s less competition in trading it.
Explainable
- If you strategy is hard to explain, you might not be able to run it in the environment where you have to explain yourself to higher-ups / investors. This goes double for being able to “explain” the drawdowns. During those critical times you have to make the hard decision of cutting the strategy or letting it ride. If you don’t know why it’s supposed to work, you’ll probably cut it.
Complexity of decision
- Even if your strategy is code-able in principle, if there are a lot of moving parts it’s unlikely someone found it without the corresponding intuitive insight.
Strategy capacity / market size
- This is how much money you can trade using a given strategy while it still remains profitable. Almost all strategies have a cap. For example, if you’re operating in a low volume market, most strategies will have a low capacity. This is great if you’re not trading a lot of money, since you can analyze and fight in this domain. Most big competitors will stay out (it’s not worth it for them to participate) or will treat this market using their standard tools, ignoring potential domain-specific quirks.
Speed of decision
- If you’re trading by yourself, then you can put on a trade as soon as you decide to. If you’re reporting to someone else, it might take more time, especially if your suggestion is unusual. For example, I think this was probably a big factor in the case of COVID. Even if some trading institutions realized that they should react to the virus, its aberrant nature delayed their response time enough so that others who didn’t have to explain their decisions could act first.

Wrap up

Now given many common strategies you can go through the list and see if you can tweak it to give it a bigger edge. If it dominates other strategies that people are running in the market, you’ll make money over time and you can be more and more certain you’re near EMF. In fact, if you’re consistently “beating” the market, you’re the EMF. You’re the reason the market is “efficient”. But as the regimes, strategies and players change, so will the EMF frontier, so you better adjust accordingly.