Alright, going to summarize my notes from this, mostly for my own benefit. If anyone knows anything about this please correct me. Crazy interesting topic.
High Frequency Traders essentially employ computer automated trades to outpace the rest of the market. Having an advantage in execution speed allows them to employ a diverse set of trading strategies, most of which all but guarantee a profit when done correctly. Arbitrage is the most obvious, capitalizing on bid/ask spreads or divergent prices between exchanges and flipping back and forth between positions before the rest of the market can adjust. They also can take up directional strategies, being the first to learn about market relevant news or trends and then beating the rest of the market to certain profitable positions. There are several types of strategies I won't go into, but the basic premise is they trade on speed and thus have a first mover advantage, and can capitalize on any trading strategy that rewards such first movers.
Now here is where it gets really deep. Because HFT can corner a market, and leave little reliable profit for anyone else (even large sophisticated institutions), many financial institutions have set up Dark Pools to better match trades. The premise is pretty simple, no one knows what trades are available in the dark pool, so there really is no informational advantage. Rather than going to an exchange and publicly announcing I'm looking for someone to buy X shares at Y price, that bid goes into a dark pool and remains private, until an appropriate counter-party looking to buy X shares at Y prices is found. The way it was explained to me is like the Priceline bidding app. A hotel will offer to sell you a room at a lower than advertised rate, they simply don't publish it publicly until someone bids at that rate. Large financial institutions need a non-public market like this to match trades, because otherwise high frequency traders would pick up on any public information and bring it to market quicker than any other institution.
Now, many HFT's have gotten wise to dark pools, and will carefully place bids in a number of various dark pools to feel out the waters. They will put in small bids at a variety of prices in a variety of pools, and depending on which trades end up matched, can learn a bit about the composition of the dark pool. Using algorithms, many HFT's can figure out what institutions are selling what shares at what prices in a dark pool, based on a very small amount of feeler bids being put out. To the extent that HFT's can figure out what is being offered in a dark pool, they can incorporate that supposedly non-public knowledge into their trading strategy to make a profit. For example, if a large institutional investor is trying to sell a ton of X share in a certain dark pool, that share price will probably drop shortly in the future. And because dark pools are private, only HFT's who have figured this out will know, giving them a huge market advantage.
However, once multiple HFT's start employing this strategy, dark pools become unregulated markets. Each HFT has their own algorithm and commits a certain amount of capital to making these feeler trades. Some algorithms are better than others, some HFT's make more feeler trades to better figure out what is in the dark pool. This creates information mismatches among the HFT and there effectively becomes a market within the dark pools. One HFT may think a big institutional investors is selling X amount, another may think a big institutional investor is selling Y amount. The difference determines what price the HFT wants to buy or sell at and so you end up with a pretty liquid market, all within these non-public dark pools.
Now institutions have changed up these dark pools to better serve their own interests. They put minimum bid amounts on certain dark pools, minimum complexity requirements, etc essentially to filter out these feeler trades HFT's are using to try to figure out what is in the dark pool. In order to ensure that other hedge funds and investors can participate in the dark pool market, they may also change up the matching tactics for a given dark pool. So rather than the first person to put in a bid at a certain prices getting the shares, they may give the shares to the largest offer of a certain price that comes in a given time frame, or they may split the shares among all the bidders at the same price.
The issue is: does anyone really know what the ENITRE dark pool market looks like. Large HFT's may have a guess, since they play in every pool to a certain extent. But their information is imperfect and based on these feeler bids. The institution running the dark pool may know what is in it's dark pool, but it has no idea what is in anyone else. Goldman can tell you all about the contents of it's Sigma X pool but ask them what's in any other pool on the market and they have no idea. And since this is all non public, no one actor is able to actually see the whole market so we have no way of judging it's overall health or efficiency.
I will add more soon but jesus f**k christ there are some smart people designing and playing this game.