[General Affairs AI Learning Record #3] Multimodal AI, I was actually already using it

Last time, I wrote about how Google provides various generative AI tools and technologies for individuals, organizations (Google Workspace), and developers (Google Cloud), and that Gemini Notebook seems useful for general affairs. (I will write about Gemini Notebook again in another article!)

This time, I will break down [Multimodal Generative AI].

What is Multimodal Generative AI?

In the course, it is explained as follows.

In English, this means, “Generative AI can process and combine information from various formats (modals) such as text, images, videos, audio, and PDFs. This is called multimodal generative AI, and it can process a single format or combine multiple formats simultaneously.”

Just reading this explanation, I felt like I understood it, but also didn’t… As usual, I threw it at Gemini this time too!

That is very easy to understand! In other words, just as we summarize and understand ‘information we see,’ ‘information we hear,’ and ‘information we read’ in our heads, multimodal AI can handle various formats of information at once.

ChatGPT, too, was just for typing text instructions when it was first released, but now you can attach an image and ask, ‘What is this?’ This is also multimodal.

By the way, if you look closely at Gemini’s answer, it says ‘five senses (sight, hearing, language, etc.),’ but language is not one of the five senses. Even in an easy-to-understand explanation, such small mistakes can creep in. As I wrote last time, I realized once again that it is important to check with your own eyes at the end.

Actually, we are already using it in our daily lives

We don’t hear the word ‘multimodal’ very often, but we take its benefits for granted in our daily lives. For example, camera translation. You can just take a picture of an English menu and it will immediately translate it into Japanese. The transcription feature that turns audio into text is also one of them.

As a general affairs worker, I was actually already using it

Since my workplace uses Google Workspace, I use ‘Gemini,’ which is a multimodal generative AI, for my work.

For example, when I am building an app with AppSheet. If I paste a screenshot of the design screen into Gemini and ask a question, it will pinpoint the cause of an error or advise me on a layout that is easier to see. Previously, I would explain the error situation at length in text, but now I can convey it with a single screenshot. Looking back, this was exactly how to use multimodal AI.

In business, there are even bigger ways to use it

The course also introduced more full-scale examples of business applications. All of them are examples of using a single AI to perform tasks that were previously done by separate tools or by human hands.

• Marketing (text + images): Creating social media posts and images as a set to match the brand image, and even suggesting the best times to post for visibility.

• Legal (text + PDF): Reading and summarizing thick contracts, answering questions, and pointing out potential trouble spots.

• Market Research (text + video + data): Combining videos of respondents’ facial expressions, what they said, and survey results to analyze their true intentions and emotions.

When you try to incorporate AI into your work, just knowing this concept of “multimodal” will significantly improve your resolution of “what AI can do.” Things you thought you had to explain in text might actually be able to be passed as images or audio. For General Affairs, this includes things like paper application forms, photos of receipts, and recordings of meetings.

Be careful with “settings” when using AI at work

There is always a concern with AI that “the information entered will be used for training, leading to information leaks.” Therefore, I think companies with strong security often place restrictions on how AI is used.

With free personal services, there are settings where the content you enter is used for training. On the other hand, many enterprise-grade services, such as Google Workspace, state that they do not use the data you enter for training.

However, just because it is set not to be trained on doesn’t mean you can put anything into it. As a rule, you should not enter confidential information or personal information in the first place.

Also, company accounts may have restrictions on features that are available on personal accounts. Settings vary by company, but in my company’s case, it’s like this.

• Cannot reference past conversations: With a personal account, the AI remembers past conversations and gives advice like, “You said this earlier, right?” On the other hand, company accounts cannot read the content of other conversations, so you have to explain everything from scratch every time. I hear the phrase “training the AI” (personalizing it to your own thinking by having repeated conversations), but you can’t do that with a company account.

• Free version API is prohibited: The free version is prohibited because the content entered could potentially be used to improve the service.

General Affairs is a department that handles a lot of confidential company information and employee personal information. To protect both the company and ourselves, we want to acquire the correct knowledge about AI.