Top News
Tuesday , 17 November 2026
Home Artificial intelligence Looking at the ‘stock price itself’ causes bugs, and looking only at ‘day-over-day changes’ leads to amnesia. How to read charts using ‘fractional differentiation’ from financial machine learning
Artificial intelligence

Looking at the ‘stock price itself’ causes bugs, and looking only at ‘day-over-day changes’ leads to amnesia. How to read charts using ‘fractional differentiation’ from financial machine learning

Share


Nice to meet you, I’m Peko! I usually use machine learning for Nikkei day trading! Recently, I re-read a famous book in the quant world called ‘Advances in Financial Machine Learning’ to revise my own model, and I came across a very interesting statistical dilemma in it.

This story applies perfectly not just to building AI models, but also to why people who look at candlesticks for day trading or swing trading usually lose, so today I’ll break it down.


1. Looking at ‘raw stock prices’ causes both brains and AI to bug out (the trap of non-stationary data)

When you’re trading stocks, don’t you find yourself thinking things like this?

  • ‘A Nikkei average of 61,000 yen is surely too high (even though it was in the 40,000 yen range just a while ago)’

  • ‘This stock was 3,000 yen six months ago but is 6,000 yen now, so it’s overpriced. I’ll short it.’

  • ‘A stock that was 10,000 yen dropped to 5,000 yen, so it’s a half-price sale. I’ll buy it.’

In the world of statistics, the raw stock price numbers themselves are called ‘non-stationary data‘ (data without stationarity).

‘Having stationarity’ means that, like the roll of a die or a human’s normal body temperature, the data fluctuates within a certain range around a generally fixed average value over the long term. In fact, most statistical and machine learning methods are built on the fundamental premise that data is ‘stationary‘.

However, raw stock prices and trading volumes have no ‘average value to return to’ (in statistics, this is called a unit root process, etc.). Due to inflation or corporate growth, it happens all the time that a stock that was once 3,000 yen never returns to 3,000 yen, and 10,000 yen becomes the new normal.

I actually ran my own machine learning model on raw stock price data using a statistical test (a test to check for stationarity called the ADF test), and it failed miserably. If you feed this kind of non-stationary data directly into an AI, the AI will memorize past price levels—like ‘sell if the stock price exceeds 5,000 yen’—and the moment it enters a new level, like the current Nikkei 60,000 yen range, it becomes completely useless.

The reason humans keep getting squeezed by contrarian trading based on a gut feeling of ‘it’s expensive/cheap compared to the past’ is also because their brains are trying to process non-stationary data as-is, causing a bug.


2. Converting to ‘percentage change from the previous day’ leads to amnesia

So, general data analysis textbooks always say this: ‘If raw stock prices are no good, just convert them to the rate of change from the previous day (returns).’

It is true that if you convert them into rates of change like ‘plus 2% today’ or ‘minus 3% today’ (what is called ‘first-order differencing‘ in statistics), they become beautiful ‘stationary data‘ where the average settles around 0%. This way, you can treat them with the same yardstick whether it’s the era of 1,000 yen stocks or 10,000 yen stocks.

It looks like the problem is solved, right? But here, a massive pitfall pointed out by Marcos López de Prado, the author of ‘Advances in Financial Machine Learning,’ emerges.

The moment you convert to the previous day’s change (first-order differencing), instead of becoming stationary, the data ‘loses all memory of the past leading up to that point and suffers from amnesia‘.

This is very easy to understand if you imagine it in a day trading or swing trading scenario. For example, suppose you look at a day-over-day ranking and see two stocks that ‘dropped 3% today’.

  • Stock A: A stock that has been steadily rising to new highs, with almost all holders sitting on unrealized gains, and it just happened to pull back 3% for the first time today.

  • Stock B: A stock that has been slowly sold off from its highs for the past month, with all the margin buyers who bought in at the top suffocating from unrealized losses, and it dug another 3% deeper as a final blow.

If you only look at the day-over-day change (first-order difference), such as ‘down 3% today,’ these two things become exactly the same number. People who lose money by jumping in after looking only at the price decline ranking and thinking, ‘Oh, it’s down a lot today, so it’s a buy-the-dip opportunity!’ do so because they are looking only at stationary numbers and have erased all memory of the trends and accumulated positions over the past few weeks behind them.


3. The financial machine learning solution: ‘Take only a 0.4-order difference’

Summarizing this so far, we are in a very troublesome dilemma.

  • Raw stock price (0-order difference): 100% of past context and memory remains, but because it is ‘non-stationary data’ where the level keeps shifting, bugs in the sense of value occur.

  • Day-over-day return (1st-order difference): It becomes clean ‘stationary data,’ but past memory is erased, leading to ‘amnesia’.

You might be thinking, ‘So what am I supposed to do!’ To mathematically reconcile this, a method called ‘fractional differentiation‘ (FFD for short) was devised.

The idea is very unique: instead of taking the difference 0 times (raw stock price) or 1 time (day-over-day change), you perform a calculation to ‘take only a 0.4-order difference‘ (the image is like a process of subtracting past stock prices from today’s stock price, while gradually reducing the weight of the more distant past).

When you actually apply this ‘0.4-order difference’ to stock price data, something surprising happens. You can extract data that gets the best of both worlds: it is clean ‘stationary data‘ that clears statistical tests (meaning the curse of the level of what the price was in the past is gone), yet it retains ‘over 80% of the memory (correlation with the original waveform)‘ of how it has been sold or bought over the past few weeks.

When I incorporated this into my own model, the sense of stability in unknown market conditions changed drastically.


4. How to apply this to human day trading and swing trading

Of course, you don’t need to calculate ‘0.4-order differences’ in your head when trading at your own discretion. But this statistical principle of ‘forget the absolute price level, but do not erase the memory of the past few weeks‘ becomes the strongest rule when looking at charts.

Specifically, you can do the same thing just by being conscious of the following three points.

  1. Do not judge high or low based on raw numbers like ‘how many yen’ Memories of levels like ‘It was XX yen last year’ or ‘60,000 yen for the Nikkei is definitely high’ are illusions pulled by non-stationary data, so discard them completely.

  2. Do not jump in based on a single day’s rate of change like ‘how much it moved today’ Looking only at daily rankings or a single candle is the same as fighting with data that has amnesia.

  3. Look with stationary memory of ‘current position relative to recent fluctuations and volume accumulation’ For example, instead of ‘how many yen it fell,’ look at ‘how many times more it is being sold relative to the average price movement (volatility) over the past 20 days,’ or whether it is above or below the price range where the most volume has accumulated in the past few weeks (VWAP or moving average). This allows you to make judgments while keeping a yardstick (stationarity) that works even if the stock price level changes, while retaining the ‘memory’ of who is trapped where.


Summary

‘People who get burned because they are convinced it is expensive or cheap compared to old stock prices’ and ‘people who get burned by jumping in after looking only at today’s day-over-day ranking.’ At first glance, these look like opposite mistakes, but from a statistical perspective, both arrive at the exact same root cause: ‘failing to balance data stationarity and memory‘.

When you open a chart, just by taking a step back and asking yourself, ‘Am I being deceived by mere stock price numbers (non-stationary levels)?’ or ‘Am I losing the memory of the past few weeks’ accumulation by being captivated only by today’s movement?’, you should be able to significantly reduce unnecessary losses.

I recently started a YouTube channel, so please check it out if you have some free time! https://www.youtube.com/@pecopekocandy



Source link

Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Articles
Artificial intelligence

Geordie and Scouse accents included in new Google AI update

Google has updated its Gemini AI app to be able to mirror...

Artificial intelligence

Global Mofy Announces Indirect Investment in Anthropic as Part of AI Expansion Strategy

Artificial intelligence in hand ©Shutterstock Global Mofy AI Limited (NASDAQ:GMM) announced that...

Artificial intelligence

What is vibe manufacturing? AI-designed products explained

The next wave of AI projects might just arrive in the mail.As...

Artificial intelligence

AI music used in Somerset Carnival processions ‘disappointing’

The use of artificial intelligence (AI) in music during a series of...