Seventeen years of algotrading

I started working in the back office of a forex brokerage in Athens. I don’t know if they’re still around. They ran the entire thing in Java; it crashed and lost trades. We used to call them “ghost trades”, and people would ring up and complain all the time. It was fascinating. I got introduced to forex back then, MT4, and of course started reading all the technical analysis BS like it was gospel. I think that was back in 2009, but I was generally interested in stocks and shares. I was 19 or 20 during the dot-com bubble era, and the joke was that everyone was reading up on the Dow Jones and NASDAQ, even the old ladies in the village.

Dot-com was 26 years ago, and to me it feels like an entire lifetime ago. Anyway, back then I did demo accounts on MT4, and boy did I blow through them fast. “Follow the momentum,” one would say; another would tell me about Elliott waves, Fibonacci sequences, head and shoulders, doji, candlesticks. I ended up buying a ton of books and reading them. I remember I even had a website, market-forecasting.com or something silly like this, running from an old Debian server in my study. I would take screenshots of MT4, paint the patterns and post them on a free website. Eventually, around 2010 or 2011, I opened a real account. I must have put in $300 at first, maybe $500 later, with a couple of widely known bucket shops at the time.

I actually never lost that money. I was lucky, but I also realised I wouldn’t keep my gains. I kept thinking, “there must be a way to automate this, or even better, to actually predict market moves”. Later on, when I was in the army, I met a guy with whom I’d chat about trading and markets. He would laugh at me for believing in TA, and he’d say he only traded fundamentals. At the same time, I was being asked by senior officers to put their money on forex. Of course I didn’t take their money! It reminded me of some customers from the brokerage, who would bet their rent, and then call and ask for it back. Or another one, who would call me and tell me that he was trying to set up an orphanage in Argentina, and we needed to approve his account so he could invest the money. I had started realising that this wasn’t investing, or even trading. It was gambling.

I had started realising that this wasn’t investing, or even trading. It was gambling.

I left the army in 2010. I wasn’t there long, just as the global financial crisis unravelled, and we all know what happened after that. In 2010 I went to do a master’s in robotics and AI; I have three passions in life: robots, AI and the markets. I pursued robotics, but it never really took off as a field, so in 2018 I pivoted entirely to AI. I did my MSc and then found a job developing iOS apps; I couldn’t really find a job in AI. Around that time I got introduced to Bitcoin, but that’s a story for another day. Anyway, long story short, my then boss was a Warwick alumnus, and he convinced me to do a PhD at Warwick, which I did.

I chose to pursue learning by example and imitation learning, because I had found during my MSc that this was the only viable way to teach robots highly complex solutions. Traditional RL wasn’t really up to par with robotics at that point in time. Fast forward a year, and I started working part time for FP7 and then H2020 R&D programmes, which funded my PhD. The irony is two-fold. What I actually wanted to build back then was a machine whose skills and knowledge aren’t simply trained, but taught interactively, and I couldn’t: the technology didn’t exist yet. At the same time, I was focusing on training models on how to construct knowledge graphs from text. This is before transformers, around the time LSTMs were SOTA. Deep learning was still in its infancy.

I remember that just before I started writing up my thesis, CUDA became this new big thing. I spent a good few months trying to build a CUDA-based neural network (the repo is still on GitHub), but failed. Around that time I was focusing on two things: understanding how an AGI may work, and trying to make money by buying NVIDIA stock (I think I bought quite a few back then). I was beginning to make a small but decent salary, so I’d buy stocks myself, and unfortunately I also got introduced to options, warrants and other derivatives.

I made money with options once, and then each time since lost money using them. So I stuck to stocks, and started exploring the idea in my head that news and events were the driving force behind market action. At some point, I don’t exactly remember when, I realised that TA was just voodoo (yes, voodoo), numerology. It couldn’t predict anything. I didn’t have any proof yet, but I could tell: spending hours looking at charts and drawing lines didn’t help one bit. Quite the opposite, it felt ridiculous.

At some point, I don’t exactly remember when, I realised that TA was just voodoo (yes, voodoo), numerology.

After the PhD I went into the startup world. I loved it: the rollercoaster, the hustling, the buzz, the London TechBros; it was amazing. I had begun focusing on ML for vision, not simple OpenCV but CNNs, classifiers and such. They were booming, and ironically it was robotics that introduced me to them. An old colleague and I spun up a startup that used an edge IoT device to track signs of dementia by looking at facial expressions over time. What we hadn’t thought of was that people didn’t like being tracked, or tracking their elderly parents with a device in their homes. And I had a decent job offer, a salary almost twice my previous one, to join a company that wanted to use AI.

This is where things get interesting. The guy who hired me was obsessed with Elon Musk and Steve Jobs. Turns out the only reason he wanted to hire an AI engineer was because he’d read Life 3.0 and thought he could be the company to automate and replace his back office. And at first, I delivered: CNNs that would classify documents very accurately; I even built a SegNet from scratch for region mapping. In the meantime NVIDIA kept going up, but my manual trades were mostly long HODL.

And then it turned out that he kept changing his mind, kept shifting targets, kept cancelling projects just before completion. It was an utter disaster; he couldn’t project-manage his way out of a bathroom. But what happened next was COVID, lockdown and a bored AI engineer with an itch he wanted to scratch. So I started collecting datasets. Any dataset I could find. I had slowly begun making my transition to NLP during my PhD, and with that job I had got into LSTMs, so I was experimenting with news and the market reaction. Crudely, embarrassingly bad, with a naivety that makes me laugh now.

From 2020 onwards I started taking algotrading a lot more seriously. I was trying LSTMs on news, but the dataset was really poor quality. Back then we only had Kaggle, and datasets were a commodity; I remember a quote for historical news running into the thousands of dollars. Even worse, news sources were few and far between, and always delayed. The more I thought about it, the more I started formulating a theory: that it was news that drove some of the market reactions, and as such an NLP approach should work. At least part of the time. I called it a Partially Observable Markov Chain: some news got released, and the market reacted by repricing appropriately. Needless to say, the Reuters dataset and the other free datasets performed extremely poorly on a simple LSTM.

From 2020 to 2022 I was playing with LSTMs and GRUs, and I built a pipeline that was collecting news. I was obsessively collecting news to build a dataset I could reuse later. So I built it, tested it, deployed it, and then hoarded millions and millions of news articles. The database now has 10,665,733 articles, and each day I receive about 3,200 more.

I had abandoned the idea that TA actually works, and realised it was a gimmick, a way to sell courses and pointless material online to the gullible masses. It made no sense: how could a price in the past predict the price in the future? Regardless, I did try it. For about two years, around 2022 to 2024, I tried using denoising Gaussian VAEs, autoencoders and decoder-only models, to see if a model trained on breakout momentum at 1-minute OHLC and volume could indicate how much the momentum would move. I tried random masking, masking the tail and feeding the front; I tried 2 minutes, 3 minutes, 5 minutes, 20 minutes, and so on. The irony? Each time the F1 score would top out at 0.49, 0.495, max 0.50. Basically, the models were telling me that judging how far the momentum would go was a coin flip. The only thing that seemed to have some promise was telling whether the momentum was exhausted or not, aka the duration, and not the magnitude.

Basically, the models were telling me that judging how far the momentum would go was a coin flip.

The hoarding allowed me to focus more seriously. What I was lacking was three major things: high-quality news delivered fast, high-quality price data delivered fast, and a good understanding of how the markets work, which I only recently started mapping to a good degree. In 2024 I was ready to give up, and I had started posting on Reddit that the news drives most prices, and that TA is voodoo. Needless to say, I got attacked a lot, especially by those selling TA bots or courses. However, over 2024 I posted quite a bit about it, and I started getting PMs from random people. Most were pointless, some were asking for insights I didn’t yet have, some were trying to get me to sign up to their platform or service, some were talking about news and TA. It all felt extremely frustrating.

I got a message from a guy on Reddit, and he seemed like the real deal. He told me, “look at biotech stocks and news announcements, check their price reactions to catalysts”. I did, and I was intrigued; I could see massive spikes. This is all pre-vibe-coding and AI code assistants. I think at the time only Copilot existed, and it was poor, so I only used it for autocomplete. He was very secretive and private at first, then, as we kept chatting, we started working together. He coded in Go, I prototyped in Python; I had a lot of news data, and he seemed to understand biotech a lot better. We spent months doing EDA, finding high-quality data, downloading L2 from Databento, running simulations, and so on.

He worked in cybersecurity, but said he had studied biology and understood how biotech worked. We spent months trying to find out if there was an edge; he looked at TradingView charts and showed me that after catalysts, the prices spiked. But there were a few big problems. First, the data I had didn’t use a timezone, and the publication dates were all over the place. Second, even with L2 data, those stocks were tiny, with no liquidity, and it seemed that they often crashed after news. At the time, what we didn’t know was that biotech insiders sold into the news, that good news was often the mark to sell and offload stock, and, more importantly, that biotech news isn’t always positive. We searched for months and months, ran backtests, tried hard, and couldn’t figure it out. And just when he was ready to give up, I said, let’s look at something else, similar but also more common.

As part of my mandatory training in at least three of my previous roles, I’ve had to learn about acquisitions. Don’t trade acquisitions (the ones you hear about at work, before they’re public), basically, and don’t tell friends and family about acquisitions. It really got me curious as to why they’d train us not to do that. And as soon as I started sifting through news that mentioned acquisitions, I understood immediately. It was basically a larger company paying a premium to take all the stock and own another. Yes, there are different acquisitions, and not all are cash; some are shares, some are mergers, and then there are different phases of acquisitions and so on. So I went back to him and I said, “before you give up, let’s look at M&As”, and that was the eureka moment for both of us. It didn’t take long after that: as soon as we had high-quality newswire articles, it was obvious that my theory finally had a home.

“Before you give up, let’s look at M&As”, and that was the eureka moment for both of us.

He stayed until we had an early working prototype, which by today’s standards is a steam engine compared to a rocket ship. We used Python and Hugging Face LLM endpoints, RAG and prompts. It was farcical: the HF endpoints would take anywhere from 2 to 10 seconds to reply, or wouldn’t reply at all. Then we had to keep changing prompts, changing models, and so on. For me, it was fun; prompt engineering is a game I’ve played for a while. But the prototype had serious issues: it would crash, we’d receive news 30 seconds or more after it was released, and we used our broker’s market-maker router, which only got us filled if it was profitable for them. But it was a start, and I got even more obsessed with it. Before all this, it was a hobby, something to do in my free time. Now it was on my mind 24/7, and I went from a few hundred dollars in a live account (because YOLO, who needs paper testing?) to a few thousand, to tens of thousands within a year. Not gains: funds poured into the bot account. He wasn’t so interested; he became fascinated with Pokémon cards. Yep, he saw Pokémon cards as the next gold rush, and just jumped straight into them. After that he disappeared, and I’ve not heard from him ever since.

Yep, he saw Pokémon cards as the next gold rush, and just jumped straight into them.

Eventually I got fed up with Hugging Face’s endpoints. They cost a good chunk of money every month, were slow and unreliable, and I already knew how to train models. So I said to myself, let’s try and build a dataset. We already had a small annotated dataset we used to test against the LLMs, so I added another hundred or so samples and trained a few BERT variants. I made the mistake of trying multiclass classification, where I was training the model to not just say “trade this” or “don’t trade this” but actually “trade this ticker” vs “trade the next ticker”, and that wasn’t a good idea. But it kind of worked. And even more interesting, it only took a few milliseconds to run. From then on, it consumed me. I kept chasing optimisation, smaller and smaller numbers. And in the process, I discovered that it wasn’t just M&As; there were a lot of other types of news that had a similar effect on price movement. At that point I decided that 28-year-old me would have had his mind blown if he’d seen what I had built, all of it centred around news. And for a retail algotrader, the numbers were getting serious. Not just the PnL and Sharpe, but the optimisations, the ML models.

Needless to say, as soon as I discovered Claude Code, the entire thing just exploded. A long time ago I was a C++ developer, and I gave the language up for Python because the entire ML ecosystem moved to Python. In a very ironic twist of fate, there I was, using AI coding tools to code in C++ again. The bots kept getting faster and faster, until I realised that there is a wall.

I’ve made money and lost it from bugs. The acquisitions bot worked well for a year, and then, because of false positives from Chinese pump and dumps, it lost most of its gains. It’s still slightly positive, but only just. Then I made the mistake of trading options (again), iron condors manually, but that’s a story for another day. One of my bots was, until recently, making about $2K profit, so I took some money out of the account. Now I’m back in the negative, but I’ve been battle-tested, and I understand the game a lot better.

The wall? The wall is that once you reach the point where I am now, every further improvement comes at huge costs. We move from being retail into something else. The revelation came to me when I started looking at when the first trades are placed after certain news releases. And the tape doesn’t lie: when you see people entering a trade 9ms after a news event, you know you can’t beat that. Not without a direct feed to newswires, not without colocating in the same building or with a direct connection. Not without a FIX line sending and receiving orders in microseconds. The limit of what can be achieved as a retail trader is now that wall. And as Opus has made very clear, the edge sometimes isn’t in speed, but in the cognitive function of what that news is.

When you see people entering a trade 9ms after a news event, you know you can’t beat that.

That’s what I realised. The high is chasing ultra-low-latency trades, but the numbers there are small, both in execution and in profits. You run trades in milliseconds, make 2%, 3%, maybe 5% if you’re lucky. And it’s stressful; there’s always more work to do, new bugs. Transitioning from Python to C++ is an absolute nightmare: worth it in the end, but really hard. And in all this? The lazy, slow bot, the one that simply consumes news after news like a data-hungry monster, and then at some point opens a position and holds it? That’s the one that has so far made the biggest profits. If I’m not mistaken, before I withdrew a large part of those profits, it was up by double figures within a few months. Will it stay that way? I don’t know. What I’ve learned from previous bots is that sometimes, if something is working, stop breaking it with improvements and optimisations.

If you work on the same problem from the other side of the screen, I would like to compare notes. Contact →