67th [2027 IT Passport] How to ‘Raise’ and ‘Verify’ Machine Learning. You can distinguish between the 3 classifications and overfitting by looking at ‘cooking skills’

Introduction

AI topics are no longer just for engineers.

The sales department introduces ‘AI that predicts customers likely to cancel.’ The store operations department uses ‘AI that groups similar stores.’ At that time, it is people on the business side like us who read the planning documents and reports to make judgments.

That is where the following two perspectives become necessary.

In the exam as well, these two are asked as a set. There are questions where you read a case study and identify ‘which type of learning it is,’ and questions asking what to call the state where ‘it is good in practice but fails in the real world.’

If you understand today’s content, you won’t be confused by either of them.

[What you will learn in this chapter]

• The flow of machine learning (collect -> learn -> verify -> correct)

• Supervised learning: Practicing with recipes that include finished photos

• Unsupervised learning: Sorting ingredients whose names you don’t even know into shelves by similarity

• Reinforcement learning: Continuously adjusting the saltiness while tasting

• Practice identifying business cases (6 questions)

• Overfitting: A chef who can only cook in their own kitchen

• How to verify and improve accuracy

• Positioning in the 2027 fiscal year proposal

1. The flow of machine learning: Collect, learn, verify, correct

[Key point!]

[Very Detailed Explanation]

Please imagine a situation where you are learning to cook.

First, you gather ingredients and recipes. Next, you actually make the dish to practice. Once it is finished, you have someone taste it to verify your skills. If it is not quite right, you adjust how you make it and try again.

Machine learning follows the exact same process.

Instead of humans writing rules one by one, we show the data and let the machine find the rules. This was the starting point of machine learning, right?

And in the 2027 syllabus proposal, machine learning items are divided into two categories.

• Types and characteristics of machine learning … How to raise it (Supervised learning, unsupervised learning, reinforcement learning)

• Improvement and evaluation of model accuracy … How to verify and how to fix it (Overfitting)

Today, we will proceed in this order.

② Supervised learning ~Practicing with a recipe that includes a finished photo~

[Key Point!]

[Very Detailed Explanation]

Imagine someone practicing with a recipe that includes a photo of the finished dish.

Every time they make it, they can compare it to the photo and check their work, noticing things like ‘it’s not browned enough’ or ‘the plating is different.’ Because there is a model, or in other words, a correct answer, they can clearly understand what needs to be fixed.

This is supervised learning.

What we give to the AI is a set of data and correct answers.

• Past weather, day of the week, presence of a campaign (data), and that day’s sales (correct answer)

• Past customer usage status (data) and whether that customer canceled (the correct answer)

By finding rules from many sets such as ‘when these conditions are met, this is likely to happen,’ we try to predict tomorrow’s sales or next month’s customer movements, for which the answers are not yet known.

There are two main types of things to predict.

3. Unsupervised Learning ~Sorting unknown ingredients onto shelves by grouping similar items~

[Key Point!]

[Very Detailed Explanation]

This time, imagine you have bought a mountain of ingredients you have never seen before in an overseas market.

You don’t even know their names. No one tells you, ‘This is a vegetable, this is a spice.’ Even so, if you compare their colors, shapes, smells, and textures, you can sort them onto shelves by saying, ‘This and this are similar,’ right?

This is unsupervised learning.

You only give the AI data. There are no correct answers.

• Give only customer purchase history to group customers with similar buying habits (customer segmentation)

• Give only store location and customer demographic data to group similar stores (store clustering)

The point is that the AI does not name the groups.

Names like ‘the group that buys in bulk on weekends’ or ‘the group that buys a little bit every weekday night’ are assigned by humans after looking at the resulting groups. What the AI does is strictly limited to ‘collecting similar items’.

4. Reinforcement Learning ~Continuing to adjust the saltiness while tasting~

[Key Points!]

[Very Detailed Explanation]

Finally, we have a situation where you decide the taste of a soup without a recipe or a finished photo.

Add a little salt and taste it. It’s bland. Add a little more. Now it’s too salty. Dilute it with a little water. If your family says, ‘It’s delicious!’, you remember that balance.

This repetition is reinforcement learning.

Unlike supervised learning, no one tells you, ‘This is the correct answer.’ Instead, you receive a score of ‘good’ or ‘bad’ based on the results of your attempt. This score is called a ‘reward’.

AI finds the method that maximizes the reward through trial and error.

• Vehicle dispatch routes … Give a reward when waiting time is shortened, and find a better way to make rounds

• Building vibration control … Give a reward when the vibration is reduced, and find a better way to suppress it

5. Distinguishing by business case studies ~Test your skills with 6 questions~

[Key Points!]

[Very Detailed Explanation]

In the exam, questions that ask you to read a business case and choose ‘which type of learning’ are common. Let’s practice here.

First, try to think about it without looking at the answers.

1. Find customers likely to cancel next month from data on customers who canceled in the past and those who did not

2. Group members with similar purchasing habits into several groups based on their purchase history

3. In taxi dispatch, give a score when waiting time is shortened, and have it find a better way to make rounds

4. Predict tomorrow’s sales based on the past three years of weather, day of the week, and actual sales performance

5. Group stores nationwide into clusters with similar locations and customer demographics

6. In a high-rise building, have the system find a way to move to suppress swaying by giving it points when the swaying decreases

How was it? Here are the answers.

⑥ Overfitting: A chef who can only cook in their own kitchen

[Key Point!]

[Very detailed explanation]

Imagine a chef like this.

They can make a perfect omelet every time if they use their own kitchen, their usual stove, their usual frying pan, and ingredients from their usual supermarket.

However, when they made it on a friend’s induction stove, it burned. If the egg size was different, it didn’t turn out fluffy.

What this person learned wasn’t ‘how to make an omelet,’ but ‘how to make an omelet in their own kitchen.’

This is overfitting.

The same thing happens in machine learning.

It is a state where the AI has memorized not only the ‘truly important trends’ in the data but also the minor quirks that happened to be present only in that specific data.

The reason this is a problem is that we want to use AI for ‘future data for which the answer is not yet known.’

Tomorrow’s sales and next month’s customer behavior are both data you are seeing for the first time. If it is perfect only with practice data but fails in the crucial real-world scenario, it cannot be used in business.

⑦ How to verify accuracy: Tasting it yourself doesn’t tell you your skill level

[Key Point!]

[Very detailed explanation]

Returning to the ‘95% accuracy’ topic from the beginning.

When checking cooking skills, if you make it yourself and taste it yourself, saying ‘it’s delicious’ isn’t very reliable, is it? That’s natural, since you made it to suit your own tastes.

Your true skill level is only revealed when you serve it to a customer for the first time.

In machine learning, too, we first divide the available data into two parts.

• Training data … used for practice

• Verification data … never used for practice, used only for the final test of skill

Then, we consider the accuracy measured with the verification data as the AI’s true capability.

Therefore, when you hear ‘95% accuracy,’ the correct way to verify it is as follows.

Just being able to say this one sentence will make you stand out in AI implementation meetings.

Results generally fall into three patterns

What is asked on the exam is almost always ‘overfitting’ in B.

C is generally called ‘underfitting’ or ‘insufficient learning.’ Since this is a state where there hasn’t been enough practice, the causes and countermeasures are the opposite of overfitting. Let’s be careful not to confuse them.

⑧ How to improve accuracy: 3 ways to prevent overfitting

[Key Points!]

[Very Detailed Explanation]

If you end up with overfitting, how should you fix it? If you replace it with a chef, the answer comes out naturally.

Method 1: Practice in various kitchens

Because you only practiced in your own kitchen, you developed skills specific to your own kitchen. In that case, you should practice with gas, IH, large eggs, and small eggs.

In machine learning as well, if you increase the amount of data or mix in data with various conditions, it becomes harder to be swayed by accidental habits.

Method 2: Do not get hung up on overly detailed tricks

‘Since the back right of our frying pan is hot, move it to the left for just 3 seconds.’ If you memorize tricks like this, they will get in the way in other kitchens.

AI also, if you make the rules too detailed and complex, will memorize the habits of the data entirely. Moderately simple rules are often stronger with new data.

Method 3: Stop practicing at just the right time

If you make the same omelet 500 times in the same kitchen, you will become perfect only in that kitchen.

In machine learning, there is also a way of thinking where you proceed with learning while checking the accuracy with verification data, and stop when the accuracy for verification stops improving. This is because if you continue learning while only chasing the grades for practice, overfitting will progress.

Extra Edition: If the model is wrong, there is nothing you can do

There is another important pitfall in supervised learning.

If the finished photo is wrong, no matter how much you practice, it won’t become the correct dish, right?

Supervised learning is the same; if there are many errors in the correct answer data, the AI will memorize the mistakes instead. The work of attaching correct answers to data is called ‘annotation,’ and it is a term that appears in the field of data utilization in both the current syllabus and the 2027 fiscal year proposal.

AI accuracy changes significantly not only with the cleverness of the algorithm, but also with the amount, variety, and correctness of the data. This is a way of thinking that is also very effective in practical work.

⑨ Positioning in the 2027 fiscal year proposal ~Technology and usage are in different places~

[Key Points!]

[Very Detailed Explanation]

I will convey this accurately. The 2027 syllabus is still at the ‘proposal’ (Ver. 0.1) stage. Please read this with the understanding that it may change in the future.

Contents of 18-1 ‘Machine Learning’

There are only these four examples of terms. And all four are also listed in the current syllabus. They are not ‘terms newly added from fiscal year 2027’.

Examples of use cases differ slightly between the current version and the proposal

Examples of use cases are not listed under the technology section, but under Medium Classification 10 ‘AI Utilization’, Small Classification 2 ‘Identification AI Utilization’ on the business side. Comparing them with the current syllabus examples, it looks like this.

The fact that the example of reinforcement learning has changed from ‘control’ to ‘support (optimization of vehicle dispatch routes)’ and that ‘disease prediction’ is not found in the examples of supervised learning stands out.

Even for the same ‘supervised learning’, the technology side (18-1) asks ‘what kind of mechanism it is’, and the business side (10-2) asks ‘where and for what it is used’. Today’s practice in ⑤ will be effective for both as they are.

[Challenge Past Questions!]

From here on are practice questions.

Practice Question 1 (Original Question)

A telecommunications company wants to create an AI that predicts customers who are likely to cancel their contracts in the future, using data on the usage status of customers who have canceled their contracts in the past and those who have not. Which of the following is the most appropriate machine learning method to use in this case?

a. Reinforcement learning
b. Supervised learning
c. Unsupervised learning
d. Overfitting

Correct answer: b

Why b is the correct answer

The deciding factor is the part about ‘customers who have canceled and customers who have not canceled’.

Each individual’s data is labeled with a correct answer, such as ‘canceled’ or ‘did not cancel.’ Learning from data with correct answers to predict future customers—this is supervised learning.

This is exactly what the syllabus refers to as ‘churn prediction.’ It is like practicing with a recipe that includes a photo of the finished dish.

Let’s look at all the other options as well

A: Reinforcement learning is a learning method that provides ‘rewards’ based on the results of trial and error to find the action that yields the highest reward. Neither rewards nor trial and error appear in this scenario. It is not about deciding the amount of salt while tasting, so this is incorrect.

C: Unsupervised learning is a learning method that finds clusters of similar items from data without correct answers. This scenario does not apply because it has correct answers, such as ‘whether or not they canceled.’ Even with the same customer data, if you were to ‘divide them into groups with similar purchasing habits’ without correct answers, it would be unsupervised learning. This is the tricky part.

D: Overfitting is not a learning method. It is a ‘state’ where the model is too closely matched to the data used for learning, causing accuracy to drop with new data. Since the question asks for a method, this is incorrect.

Practice Problem 2 (Original Problem)

When a machine learning model was evaluated, high accuracy was obtained for the data used in learning, but accuracy dropped significantly for new data not used in learning. Which of the following is the most appropriate combination of the term describing this state and its countermeasure?

Correct Answer: A

Why is A the correct answer?

‘High accuracy with data used for learning’ and ‘accuracy drops significantly with new data.’ This is exactly the definition of overfitting. You were a perfect cook only in your own kitchen.

The countermeasure ‘increasing the amount or variety of data’ is equivalent to practicing in various kitchens. It makes the model less likely to be swayed by coincidental quirks, so it is an appropriate way to prevent overfitting.

Let’s look at all the other options as well

B: The term ‘overfitting’ is correct, but the countermeasure is reversed. If you evaluate only using the data used for learning, you will not be able to see whether overfitting is occurring. It is the same as making something yourself and tasting it yourself. Accuracy must be verified using data not used for learning.

C: Unsupervised learning is the name of a learning method that groups similar items from data without correct answers; it is not a term that describes the state where ‘it is good in practice but accuracy drops in the real thing.’ The countermeasure ‘removing correct answer data’ also does not lead to a solution for this problem.

D: Reinforcement learning is the name of a learning method that uses trial and error based on rewards. It is not a term that describes a state. Even if you increase the reward, it does not solve the problem of being too closely matched to the learning data, so it is incorrect.

Summary

I will repeat the important parts one more time.

• The way to distinguish between the 3 classifications is just these two things: ‘whether there are correct answers attached’ and ‘what you want it to do’

• If you see ‘reward’ and ‘trial and error’, it’s reinforcement learning

• Overfitting is a ‘state’, not a method. Perfect in practice, fails in the real thing

• Accuracy is measured using data not used for training. ‘What did you use to measure that number?’ is the watchword

• All 4 terms are also in the current syllabus. The 2027 fiscal year proposal is still a ‘proposal’ and subject to change

Just one last thing.

The way of thinking for verifying AI accuracy can actually be used exactly as is for our exam studies.

Once you have gone through past exam questions many times and can get a perfect score, try solving questions from a year you haven’t tackled yet. The score you get there is your true ability.

Let’s acquire the skills to cook delicious meals not just in our own kitchen, but in any kitchen.