Skip to main content

High-Frequency Trading and AI: Inside Modern Market Microstructure

HFT firms execute roughly 50% of US equity volume. The public narrative focuses on speed. The actual edge is increasingly about ML-based signal extraction and inventory management.

D
Daniel Krause·Quantitative Research Director
·6 min read

HFT firms are not front-running your order. I know this will annoy some readers. But the narrative that high-frequency traders are parasites stealing money from retail investors is wrong, and it has been wrong for over a decade.

The confusion stems from conflating two entirely different activities. Front-running is illegal. It involves a broker trading ahead of a known client order for personal profit. What HFT firms do is market making: continuously quoting bid and ask prices on exchanges, earning the spread between them, and absorbing inventory risk in the process. These are not the same thing.

The vast majority of HFT activity is market making. Citadel Securities handles roughly 25% of all US equity volume. Virtu Financial, the only publicly traded pure-play HFT firm, has been profitable virtually every trading day since its founding. Jane Street, which focuses on ETF and options market making, reported over $22 billion in net trading revenue in 2024. These are not firms skimming pennies through manipulation. They are firms providing a service: liquidity.

The Economics of Spread Capture

The business model is simple to state and brutal to execute. A market maker quotes a bid at $100.00 and an offer at $100.02. If both sides get filled, the firm captures two cents per share. Multiply that by millions of shares per day and the numbers add up. The catch is inventory risk. If you buy 10,000 shares at $100.00 and the price drops to $99.95 before you can sell, you just lost $500 on that position. The entire game is managing the balance between spread capture and adverse selection.

Adverse selection is the core problem. Some of the orders hitting your quotes come from informed traders who know something you don't. Those fills will lose money on average. The market maker's job is to set spreads wide enough to compensate for toxic flow while staying tight enough to win the order. Get the calibration wrong in either direction and you're dead.

This is where machine learning enters the picture.

ML in the HFT Stack

There are three primary areas where ML has transformed high-frequency trading. Each has different constraints and different model architectures.

Signal extraction from order book data. The limit order book is a rich, high-dimensional data source updating thousands of times per second. ML models ingest features like order flow imbalance (the ratio of buy-side to sell-side volume at the best quotes), queue position dynamics, trade arrival rates, and cross-asset correlations. The output is typically a short-horizon price prediction, measured in seconds or fractions of a second.

The models used here are simpler than what you'd find at a longer-horizon quant fund. Linear models, gradient-boosted trees, and shallow neural networks dominate. The reason is latency. If your model takes 10 milliseconds to compute a prediction and your competitor's takes 2 milliseconds, you lose. Inference time is a hard constraint. Deep transformers with attention layers are too slow for tick-level decision making. Every microsecond of compute latency is a microsecond of stale signal.

Inventory management. Once a market maker accumulates a directional position, it needs to unwind that exposure without moving the market against itself. This is fundamentally a sequential decision problem under uncertainty. Reinforcement learning approaches have gained traction here, where the agent learns a policy that balances the cost of holding inventory (market risk) against the cost of aggressive liquidation (market impact).

The state space includes current inventory, recent price trajectory, volatility regime, and time of day. The action space is how aggressively to skew quotes or whether to send active orders to flatten exposure. Firms like Hudson River Trading, which is essentially a technology company that happens to trade financial instruments, invest heavily in this kind of research.

Execution optimization. When a firm needs to execute a large parent order across fragmented venues (there are over 15 lit exchanges and dozens of dark pools in US equities alone), the routing decision is a combinatorial optimization problem. ML models predict fill rates, queue times, and information leakage at each venue, then allocate child orders accordingly. Optiver and Flow Traders, both with European roots, have built sophisticated venue selection models as they've expanded into US and Asian markets.

Firm Profiles

The HFT industry is concentrated among a handful of firms, each with a distinct identity.

Citadel Securities is the largest market maker globally. Founded by Ken Griffin as a separate entity from the Citadel hedge fund, it handles about a quarter of all US equity volume and significant fractions of options and treasury markets. Its scale gives it an information advantage: with more order flow comes better estimates of fair value.

Virtu Financial is the only major HFT firm that's publicly traded. Its IPO filing in 2014 famously disclosed that it had experienced only one losing trading day in five years. This isn't luck. It's the result of extreme diversification across thousands of instruments and geographies. Virtu operates in over 235 venues across 50 countries.

Jane Street started as an ETF market maker and has expanded into one of the most profitable trading firms in the world. Its $22 billion-plus in net trading revenue in 2024 put it in the same conversation as major investment banks. Jane Street's edge is in pricing complex, multi-leg instruments where analytical solutions don't exist and numerical methods are too slow. Their culture is heavily influenced by functional programming and OCaml.

Hudson River Trading is the quietest of the major firms. HRT approaches trading as a pure engineering problem. Its founders came from tower research and built an infrastructure-first organization. The firm recruits almost exclusively from CS and math PhD programs and treats its trading systems as distributed computing problems.

Optiver and Flow Traders both originated in Amsterdam and share a heritage in options market making on European exchanges. Optiver has expanded aggressively into US equities and commodities. Flow Traders has become one of the largest crypto market makers, a natural extension of its ETF expertise into a new asset class with similar market structure challenges.

The Talent War

Starting compensation for top quantitative researchers and developers at these firms exceeds $400,000 per year, and total compensation for experienced traders and researchers can reach seven figures. The talent pool is physics, mathematics, and computer science PhDs. The interview process typically involves timed mental math, probability puzzles, and coding challenges with strict time constraints.

This creates an interesting dynamic. The firms are competing not just with each other but with Big Tech, hedge funds, and AI research labs for the same small pool of people who can reason about stochastic processes, write low-latency C++, and think clearly under pressure. Jane Street and HRT in particular have invested in brand-building through open-source contributions, tech blog posts, and puzzle competitions to attract candidates.

What's Changed Recently

Two shifts are worth noting.

First, crypto market making has become a legitimate business line. The microstructure of crypto exchanges resembles equity markets from 15 years ago: fragmented liquidity, wide spreads, and unsophisticated flow. For firms with battle-tested market making technology, this is a natural expansion. Flow Traders, Jane Street, and several others have built dedicated crypto desks. The challenge is counterparty risk. When FTX collapsed in 2022, several market makers had significant assets trapped on the exchange.

Second, the latency arms race has stabilized. A decade ago, firms were spending tens of millions on microwave towers and co-location to shave microseconds off their execution times. That spending continues, but the marginal returns have diminished. The gap between the fastest and tenth-fastest firm is now measured in nanoseconds rather than milliseconds. This has shifted the competitive frontier away from raw speed and toward signal quality. Having a faster pipe matters less when everyone's pipe is fast. Having a better model matters more.

Where the Edge Actually Is

The durable edge in HFT is not speed. It's not even models. It's the combination of engineering excellence, risk management discipline, and the institutional knowledge to operate across hundreds of instruments simultaneously without blowing up.

Market making is a business where you can be right 51% of the time and make a fortune, but a single risk management failure can wipe out years of profits. The firms that survive and grow are the ones that treat operational reliability as the product. Their systems need to function correctly through flash crashes, exchange outages, and regime changes. The ML is important, but it sits on top of an infrastructure stack that has to be close to flawless.

That's not a sexy answer. But it's the correct one.

D
Daniel Krause

Daniel writes about quantitative finance, trading systems, and the ML models behind them. He previously led a quant research team at a mid-size systematic hedge fund and holds a PhD in applied mathematics from ETH Zurich. His work focuses on where academic ML research meets production trading systems.

quantitative financetrading systemsreinforcement learningportfolio optimization
high-frequency tradingmarket microstructurealgorithmic tradingmarket makinglatencyorder flowelectronic trading

Frequently Asked Questions

Common questions about the topics covered in this article.

Do HFT firms use artificial intelligence?
Yes, but not in the way most people imagine. HFT firms use ML primarily for signal extraction from order flow and market data, inventory risk management, and execution optimization. The models tend to be simpler than what you'd find at a longer-horizon quant fund because latency constraints limit inference time. Feature engineering from order book data matters more than model complexity in this context.
How much of stock market volume comes from high-frequency trading?
HFT accounts for roughly 50% of US equity trading volume, down from an estimated 60%+ peak around 2012. In European equities the figure is approximately 35-40%. Most of this volume is market making, meaning HFT firms are providing liquidity by continuously quoting bid and ask prices. Directional HFT strategies (trying to predict short-term price movements) represent a smaller share of total HFT activity.
Which companies are discussed in this article?
This article covers Virtu Financial, Hudson River Trading, Optiver, Flow Traders, Citadel Securities, Jane Street. Virtu Financial: AI-driven market making firm