Generative AI Frontlines

How to Build AI Lipsync, Part 5 (Second Half): Implementing Transformers and Making Practical Technology Choices

Why can't the Transformer that powers GPT simply be dropped into lipsync? We unpack three challenges — data, computation, and overfitting — how to choose between LSTMs and Transformers in practice, and transfer learning as a third option, in a dialogue between Professor Quo and Manabu.

Qualiteg Product Development Team

Oct 23, 2025 — 18 min read

Hello!

Our exhibit at CEATEC 2025 wrapped up successfully, and several weeks have already flown by since those whirlwind four days.

Makuhari Messe was as energetic as ever, but before we knew it autumn has deepened, and the mornings and evenings have turned noticeably cooler.

As we sort through the valuable feedback we received from so many visitors, we are reminded once again of just how deep this technology goes. At our booth, visitors watched our characters speak naturally and delighted us with reactions like "Is it really moving in sync with the audio?" and "How can it look this natural?"

This series is the behind-the-scenes story of the technology that made it possible.

In the first half, we took a detailed look at the Transformer's groundbreaking Multi-Head Self-Attention mechanism. We saw how eight "experts" analyze the relationships between phonemes from different perspectives, and how the clever division of roles among Query, Key, and Value lets every phoneme interact with every other phoneme simultaneously. In theory, this appears to completely solve the LSTM's long-range dependency problem.

In practice, however, applying Transformers to the lipsync task runs into unexpected walls. Why is it that the Transformer, so successful in large language models like GPT and BERT, cannot simply be dropped into lipsync?

In this second half, we first explain the role of the Feed-Forward layer and clarify the structural differences between LSTMs and Transformers. We then dig into three serious challenges that Transformers face. Why do Transformers require enormous amounts of training data when LSTMs can learn from orders of magnitude fewer samples? How serious is the O(n²) problem, where computation explodes as inputs grow longer? And what is the overfitting trap that comes with so much modeling freedom?

With these challenges understood, we introduce the hybrid approach as a realistic solution. By combining the efficiency of LSTMs with the expressive power of Transformers, we explore how to achieve high-quality lipsync even with limited resources. We also touch on the next challenge — extending the system to emotional expression — and offer concrete guidance for technology choices in real product development.

Let's dig into the practical insights that bridge the gap between theory and reality.

"How to Build AI Lipsync" series
Previous articles

Part 5, Second Half

1.2 The Feed-Forward Layer: Feature Transformation

Yes, we are jumping straight into Section 1.2 — this article picks up exactly where the first half left off, so please bear with us ^^;

After the Self-Attention layer comes the Feed-Forward Network (FFN). This is a two-layer fully connected network that applies the same transformation at every position: it expands from 512 to 2,048 dimensions, applies a ReLU activation, and then compresses back down to 512 dimensions.

The role of this layer is to transform the contextual information gathered by Self-Attention in a more expressive space. The expand-and-compress process lets the model learn complex nonlinear transformations.

1.3 Structural Differences from the LSTM Architecture

The most fundamental difference between Transformers and LSTMs lies in how information flows. In an LSTM, information flows sequentially in one direction (or two, if bidirectional), whereas in a Transformer every position is directly connected to every other position.

Professor Quo: Let's illustrate this difference with a school phone tree.

Manabu: A phone tree again? You used that analogy in Part 4 too, didn't you, Professor?

Professor Quo: Yes — phone trees make for very handy analogies. To recap: in the old telephone chain, the teacher would call the class representative (or student number one), who would call the next student, and so on, passing the message along in sequence. That is the LSTM approach.

Manabu: It takes time, and the message can get distorted along the way.

Professor Quo: Exactly. These days, though, you can message the whole class at once in a group chat. That is the Transformer approach. Everyone receives the same information at the same time and can talk directly with anyone as needed.

Manabu: I see! That's why the information travels quickly and accurately.

Professor Quo: Precisely. But the fact that everyone can talk to everyone also means far more communication. In a class of 30 students, there are 30 × 30 = 900 possible exchanges. This is why the Transformer's computational cost is O(n²).

These structural differences also affect how the models learn. Because the LSTM carries a strong inductive bias — "time series should be processed in order" — it tends to learn well even from small datasets. The Transformer, with fewer built-in constraints, can learn more flexibly, but it needs correspondingly more data.

Transformers also have the advantage of stacking deep layers more easily. Because every layer can see the entire sequence, even deep layers retain global information, which makes it possible to learn more complex relationships between phonemes.

2. The Transformer's Pitfalls: The Price of Freedom

2.1 Why So Much Data Is Needed

The Transformer's biggest challenge is that its very flexibility demands enormous amounts of training data. The root cause of the Transformer's data hunger lies precisely in how flexible it is.

The LSTM comes with a strong built-in assumption (inductive bias): time-series data is processed in order. It is as if the model starts out already facing the right direction. The Transformer has almost no such assumptions — it must learn entirely from data which positions should be related to which.

Professor Quo: Let's compare the two using map navigation.

Manabu: Map navigation?

Professor Quo: Yes. The LSTM is like a navigation system constrained to follow the roads. With limited options, it can learn the correct route from relatively little experience.

Manabu: True — it's never going to fly off the road.

Professor Quo: The Transformer, on the other hand, is like a drone that can fly anywhere directly. It has far more freedom, but it must learn everything on its own: avoiding buildings, finding the shortest path, accounting for the wind.

Manabu: I see! Because it's so free, it takes a lot of practice before it discovers the right way to fly.

2.2 The Explosion in Computation (the O(n²) Problem)

The Transformer's other major challenge is that its computation grows with the square of the input length. This stems from the essential nature of Self-Attention: it computes the relationship between every pair of positions.

For example, to process an utterance of 50 phonemes, an LSTM needs 50 sequential steps, while a Transformer needs 50 × 50 = 2,500 attention computations. At 100 phonemes that becomes 10,000; at 200 phonemes, 40,000 — the cost climbs steeply.

Professor Quo: Let's put this computational cost in terms of meeting time.

Manabu: Meeting time?

Professor Quo: Yes. If ten people hold a meeting and each speaks in turn for three minutes, the meeting ends in 30 minutes. That is the LSTM approach.

Manabu: Simple, and easy to predict how long it will take.

Professor Quo: But what if everyone talks one-on-one with everyone else for three minutes each? With ten people that makes 45 pairs, so 135 minutes.

Manabu: Over two hours! And if there were 20 participants...

Professor Quo: That's 190 pairs — 570 minutes, or nine and a half hours! This is why the Transformer's cost grows with n². You get much deeper discussion, but the time and resources balloon.

In real applications, this computational cost translates directly into memory usage. The memory needed to hold the attention matrix also grows with the square of the input length, so processing long audio requires chunked processing or memory-efficient implementations.

2.3 Why Overfitting Comes So Easily

The Transformer's expressive power is also a double-edged sword: it raises the risk of "overfitting" — fitting the training data too closely. The problem is especially serious when training on limited data.

When overfitting occurs, the model performs perfectly on the training data but fails on new data. For example, it may over-learn the quirks of one particular speaker and produce unnatural mouth movements for everyone else.

Professor Quo: Let's compare overfitting to an actor's rehearsals.

Manabu: An actor's rehearsals?

Professor Quo: Yes. Suppose an actor rehearses one script — and only that one — hundreds of times. They will come to perform that script flawlessly, but...

Manabu: They'd get so locked into that one role that they couldn't play any other.

Professor Quo: Exactly. Worse still, they may memorize the script down to its tiniest details and pick up unnecessary habits, like "always raise the right hand after this line."

Manabu: And then their acting would look strange in any other play.

Professor Quo: That is precisely the Transformer's overfitting. With so much freedom, it memorizes even trivial features of the data and loses sight of the essential patterns.

To prevent overfitting*, regularization techniques such as Dropout, weight decay, and early stopping must be used appropriately. Data augmentation also matters: generating diverse variations from limited data improves the model's ability to generalize.

*Reference: one metric for measuring overfitting and generalization performance is the mean squared error (MSE). The article below, written by one of our team members, explains it clearly — we recommend it as further reading.

The fundamental solution, however, is still to prepare a sufficient amount of training data. As the success of large language models shows, Transformers deliver astonishing performance when given enough data. The question is whether building such a large-scale dataset is realistic for the lipsync task. For a venture or startup like ours in particular, building such datasets while securing all the necessary permissions is, frankly, a high hurdle.

Given these challenges, it becomes clear that while the Transformer is certainly powerful, it is not the optimal choice in every situation.

In the next chapter, we therefore look at a more practical approach that combines the strengths of LSTMs and Transformers.

3. Practical Choices: When to Use LSTMs and When to Use Transformers

3.1 Understanding Each Model's Strengths

As we have seen, LSTMs and Transformers each have clear strengths and weaknesses. What matters is not which one is superior, but understanding which to choose in which situation.

Professor Quo: Let's compare choosing between an LSTM and a Transformer to choosing a mode of transportation.

Manabu: A mode of transportation?

Professor Quo: Yes. Which do you think is better: a bicycle or a bullet train?

Manabu: Well... it depends on where you're going. For the corner store, a bicycle; from Tokyo to Osaka, the bullet train.

Professor Quo: An excellent answer! The LSTM is like a bicycle: ready to use right away, nimble, and cheap to maintain. The Transformer is like a bullet train: overwhelmingly strong for large-scale travel, but it needs infrastructure — in this case, massive amounts of data.

3.2 Choosing Based on Data Volume

In real development settings, the amount of available data is often the first deciding factor.

With small datasets, the LSTM has an overwhelming advantage.

Because the LSTM carries the strong inductive bias that time-series data should be processed in order, it can learn fundamental patterns efficiently even from little data.

It is as if the correct direction is built in from the start, so the model reaches its destination with less trial and error. Its simple structure also keeps the risk of overfitting low. When you need stable performance from limited data, the LSTM is a dependable choice.

When large amounts of data are available, on the other hand, the Transformer shows its true worth. Given sufficient data, it learns the complex relationships between phonemes and produces more natural, more varied expression.

3.3 Choosing Based on Real-Time Requirements

Next, let's consider LSTMs and Transformers from the standpoint of real-time performance.

Where real-time processing is required, the LSTM's characteristics become a major advantage. The LSTM supports streaming: it can generate mouth shapes instantly as audio arrives. It can achieve extremely low latency — a few tens of milliseconds — making it ideal for live streaming, video conferencing, real-time avatars, and other applications that demand immediacy. Users get natural interaction without any sense of lag.

By contrast, when batch processing is possible, the Transformer's strengths can be used to the fullest. In video production or animation work that processes pre-recorded audio, quality can take priority over processing time. Because the Transformer surveys the entire audio at once, it generates more natural, more expressive lipsync that fully accounts for surrounding context. In workflows where pre-rendering is possible, this difference in quality can decisively shape the final product.

Professor Quo: Let's think about real-time processing in terms of how food is served.

Manabu: How food is served — like a course meal, that kind of thing?

Professor Quo: Good instincts. The LSTM is like conveyor-belt sushi: each piece can be made and served on the spot, one after another. The Transformer is like a course meal: every dish must be ready before anything is served.

Manabu: I see! For live streaming or conference calls, the sushi has to keep popping out or you'd never keep up.

Professor Quo: Exactly. In situations where even 100 milliseconds of delay feels wrong, the LSTM's sequential processing is overwhelmingly advantageous.

3.4 Compute Resources and Cost

Development and operating costs are another important criterion that cannot be ignored when choosing a technology.

In resource-constrained environments, the LSTM is often the realistic choice. Training typically finishes in a few hours to a day, and the model is relatively compact. The LSTM does have an important limitation, however: because of its sequential nature, batch-parallel processing on GPUs is difficult. Even when trying to process multiple audio streams at once, each time step must be processed in order, so the GPU's parallel computing power cannot be fully exploited.

Professor Quo: Let's compare the relationship between LSTMs and GPUs to a factory line.

Manabu: Wait — was there a particular relationship between those two?

Professor Quo: Yes. The LSTM is like a workshop where artisans craft each item by hand. Even with ten artisans, the steps for a single product must proceed in order, so there is a limit to parallelization.

Manabu: But you could still make several products at the same time, right?

Professor Quo: Good observation! Indeed, you can run multiple LSTM instances at once. The latest GPUs even offer technology to split one physical GPU into several virtual GPUs. But each instance runs independently, which drives up memory usage and management complexity. And the GPUs that can do this are limited to a handful of very expensive models.

For example, the latest enterprise GPUs such as the H100 offer MIG (Multi-Instance GPU) technology, which allows a single physical GPU to be treated as up to seven independent GPU instances.

This makes it possible to run multiple LSTM models in a quasi-parallel fashion, and with a well-designed inference server you can handle multiple requests efficiently. But this kind of approach is quite complex: you need mechanisms to manage resource allocation across instances, route requests appropriately, and use memory efficiently.

Moreover, since each LSTM instance can ultimately handle only a single stream, it is hard to say the GPU's compute is being fully utilized compared with the Transformer's true parallel batch processing. And if you have the budget to buy H100-class GPUs by the rack, one has to wonder whether you would be better off putting that muscle behind a data-hungry, heavily trained Transformer in the first place.

As for the Transformer — at the risk of repeating Section 3.3 — it is designed from the ground up for parallelism, so a single model instance can efficiently batch-process multiple inputs. The Self-Attention mechanism processes every sample in the batch simultaneously, making full use of the GPU's tensor cores. The efficiency gap is especially pronounced at large batch sizes.

3.5 Making the Call in Practice

That said, as a practical matter, how should you choose between an LSTM and a Transformer?

[Data volume]
The first thing to check is how much data you have available (the size of your dataset). As noted earlier, without sufficient data it is difficult to get good results from a Transformer, no matter how favorable the other conditions are.

[Real-time requirements]
Next, consider your real-time requirements. Where millisecond-level latency matters, the LSTM may be the only realistic option. (We noted earlier that LSTMs struggle with concurrency, but in certain applications — say, a guide avatar at a theme park or public facility, where concurrency demands are modest but real-time response is essential — they are genuinely useful, and LSTM-based models are in fact deployed in exactly these settings.)

[Compute resources]
Then factor in your compute constraints. If GPUs are not always available, or operating costs are tightly capped, choosing the LSTM is the sensible move.

[Development timeline and budget]
Finally, weigh the project's goals and development timeline as a whole. Only when you are pursuing the highest quality and have ample development time should you consider adopting a Transformer.

3.6 Choosing with the Future in Mind

Professor Quo: So far I have simplified this into a binary choice — LSTM or Transformer — but there is one important perspective I want to add. Technology choices should consider not just the present but the future.

Manabu: Right, we've been comparing which is better from several angles. But what do you mean by considering the future?

Professor Quo: Even if you have little data today, it will grow as your service grows. A staged approach — start with an LSTM, then migrate to a Transformer once data has accumulated — can be very effective.

Manabu: I see! Like a startup that begins in a small office and moves into a bigger building as it grows.

Professor Quo: A perfect analogy! What matters is shipping something that works, seeing how the market responds, and keeping the flexibility to upgrade your technology as needed.

Manabu: Praise for my analogy from Professor Quo, the analogy connoisseur himself!

Many success stories follow a staged evolution path: start with an LSTM to build a minimum viable product and bring it to market.

Then collect user feedback, accumulate data, and strengthen the technical foundation as the service grows. By considering a migration to Transformers — or a hybrid approach that skillfully combines the two — as the need arises, you can raise quality step by step while keeping risk to a minimum.

This kind of flexible approach lets you pursue sustainable development while balancing technical ideals against real-world constraints.

3.7 A Third Option: Transfer Learning as a Breakthrough

We have now walked through the trade-offs between LSTMs and Transformers, but there is

in fact one more important option.

It is called transfer learning.

Professor Quo: There is a third path that solves the problem of scarce data: a technique called transfer learning.

Manabu: Really? Transfer learning? What is that?

Professor Quo: Let's use a chef's training as an analogy. Suppose a top French chef decides to open a Japanese restaurant. They do need to learn Japanese cuisine from scratch — but knife skills, judging the heat, a sense of flavor balance: the fundamentals of cooking are already there.

Manabu: I see! The foundation is already in place, so they only need to learn what is unique to Japanese cuisine.

Professor Quo: Exactly. In lipsync, we can likewise repurpose the "ability to understand sound" cultivated in speech recognition and apply it to the "ability to generate mouth movements."

With transfer learning, you can directly leverage the knowledge held by large pre-trained models.

In the field of speech recognition, there are models trained on millions of hours of audio. These models already possess a deep understanding of speech: phoneme discrimination, speaker characteristics, intonation patterns, and more. By transferring this knowledge to lipsync, we can obtain high-quality results even from small datasets.

What is especially wonderful is the abundance of commercially usable open-source base models. State-of-the-art speech recognition models are published under permissive licenses. These models were developed by research institutions and large companies at enormous cost, yet anyone can use them. Being able to build on these research achievements is something we are deeply grateful for.

Manabu: But Professor, a speech recognition model knows how to turn sound into text — it doesn't know anything about mouth shapes, right?

Professor Quo: An excellent observation! It's true that the mapping from phonemes to mouth shapes must be learned anew. But there is a big advantage here as well.

Manabu: What do you mean?

Professor Quo: The hardest part — identifying the phonemes — is already solved, so all that remains is teaching the correspondence: "for this phoneme, this mouth shape." In map terms, the road network is already drawn; we just need to install traffic lights at the intersections.

Learning the mapping between phonemes and mouth shapes does indeed require motion-capture data (of mouth shapes).

Here too, though, the benefits of transfer learning are substantial: the amount of data required drops dramatically. Compared with training from scratch, practical performance can be reached with orders of magnitude less data — drawing, of course, on every technique we have covered in this series. Advances in motion-capture technology are also helping. The latest systems can record precise mouth movements in a short time; what once required markers attached around the mouth can now be done markerless.

Moreover, with proper planning, collecting data that covers the basic phoneme set can be done very efficiently. Working with professional voice actors and narrators makes it possible to gather high-quality data in a short period. Building the dataset does, of course, require contracts and the necessary permissions from these collaborators — but reducing the scale from tens of thousands of people to a few hundred puts this kind of effort within realistic reach even for a venture or startup.

Furthermore, data augmentation techniques let you generate diverse variations from limited recordings. By combining methods such as varying speech rate, adjusting pitch, and adding environmental noise, you can substantially increase your effective data volume. It is like the craft of a creative chef who turns a few ingredients into a rich and varied menu.

Professor Quo: Another advantage of transfer learning is that it allows incremental improvement.

Manabu: Incremental improvement?

Professor Quo: Yes. Start with just the basic phonemes, identify the problem areas while running the service, and collect additional data only where it is needed. It is like a restaurant that refines its menu based on customer reactions while staying open for business.

Manabu: I see! Rather than preparing endlessly in pursuit of perfection, you can launch first and improve step by step.

Transfer learning frees you from the either-or choice between the LSTM's efficiency and the Transformer's expressive power. You can use a powerful Transformer-based model even with little data, and the implementation is relatively simple: in many cases, existing frameworks and libraries let you implement transfer learning in just a few lines of code.

That said, transfer learning is no panacea. There is much to consider: the fit between the base model and the target task, the fine-tuning method, the quality of the data, and more. That, in fact, is the "secret sauce" — but used well, these ingredients become a powerful weapon for building a competitive lipsync system even with limited resources.

In this way, transfer learning offers a realistic and flexible third option that goes beyond the simple dichotomy of "little data, so LSTM" versus "plenty of data, so Transformer."

4. Summary and a Look Ahead

This Time: From Theory to Practice

In this two-part article, we began in the first half with the Transformer's concrete network design and its implementation challenges, and in this second half examined in detail how to choose between LSTMs and Transformers in practice.

The Transformer's groundbreaking Self-Attention mechanism made the long-range dependency problem theoretically solvable. But it also became clear that the price is a heavy demand for data and compute. In real product development, these trade-offs must be weighed carefully.

We also introduced a third option: transfer learning. By leveraging commercially usable open-source models, you can build powerful Transformer-based models even with little data. The path is open to repurposing knowledge cultivated in speech recognition and building a practical lipsync system with minimal motion-capture data.

In real product development, a staged approach works well: prototype quickly with an LSTM, accumulate data while watching how the market responds, and consider moving to Transformers or transfer learning as needed. With this kind of flexible strategy, we believe you can grow a product while balancing technical ideals against real-world constraints.

Professor Quo: Now, over these five installments we have covered all the fundamental mechanics of lipsync. But there is still one important story left to tell.

Manabu: Really? I thought we had learned everything about generating mouth movements from audio...

Professor Quo: We can indeed "generate" them now. But when you actually use the system, a problem reliably appears in one particular situation. Can you guess what it is?

Manabu: Hmm... one particular situation?

Next Time: The Overlooked Challenge of the "Final Moment"

Professor Quo: Here is a hint: it's about "endings." When a person finishes speaking, what happens to their mouth?

Manabu: Normally it closes. But... ah! It might differ depending on what they said and the situation.

Professor Quo: Exactly! In fact, no matter how accurate the model, the mouth movement at the very end of an utterance somehow always comes out unnatural — snapping shut abruptly, trembling slightly, or mistiming the close...

Manabu: That would definitely bother me. Everything could look natural right up to the end and then fall apart at the last moment. And if the mouth just hangs open (　ﾟдﾟ), it looks pretty silly.

Professor Quo: That is precisely the problem. This "end-of-utterance problem" is not a simple technical bug — it is a fascinating challenge that touches on the fundamental limits of machine learning.

Next time, we will explain in detail why this unnatural behavior occurs at the end of speech and the technical mechanisms behind it. Model uncertainty, ambiguity in training data, information loss from discretization — fundamental challenges of AI technology are all concentrated in this one small "final moment."

We will also introduce practical solutions to the problem, comparing a range of approaches from simple post-processing to fundamental techniques built on the latest AI methods.

Now that you have the theory, let's explore together the real-world challenges of product development and how to solve them. It may look like a small problem, but tackling it is what leads to more natural, more convincing virtual characters.

See you next time!