AI Mathematics

How to Build AI Lipsync, Part 5 (First Half): Transformer Implementation and Practical Technology Choices

Qualiteg Research Team, Qualiteg Consulting

Sep 1, 2025 — 17 min read

Hello! Our lipsync technology series is finally entering its home stretch.

Last time (Part 4), we took a detailed look at the LSTM training process and its limits. While we saw the LSTM's strength of learning effectively even from limited data, it also became clear that it struggles with long-range dependencies. We then introduced the Transformer's Self-Attention mechanism — an innovative approach that can reference information at every position simultaneously — as the way to solve this problem.

In this fifth installment,
we start with the concrete Transformer network design and lay out its implementation challenges. (First half*)
We then introduce a hybrid approach that combines the strengths of LSTM and Transformer, offer guidance for technology choices in real product development, and finally touch on the next challenge: extending to emotional expression. (Second half*)

*Because the Transformer's inner workings are intricate, Part 5 is split into a first half and a second half.

1. Transformer-Based Network Design

1.1 Overall Architecture

Let's dive right into the structure of a Transformer-based lipsync conversion network.

The ★ marks in the diagram indicate trainable components (parameters that get updated). For example, the Encoder/Decoder/linear projection are trained, while **the audio feature extractor (wav2vec) is frozen or fine-tuned (chosen per deployment)**

Compared with the LSTM-based network, the input (wav2vec features) and the output of 26 mouth-shape parameters* are the same, but the processing in between is fundamentally different.

*The output is a numeric vector called "mouth-shape parameters." We use 26 dimensions here as an example, but that is only one illustration — the actual dimensionality depends on the character model and rig you use. This article sticks with 26 dimensions for clarity, but in practice the number varies with the design of the rig or morph model.

The biggest difference is that while the LSTM processes the sequence step by step, the Transformer processes all time steps in parallel*. This makes it possible to directly learn relationships between distant positions, such as the beginning and end of a sentence.

*Note that inference and online processing may use a causal mask (blocking future information), so you may need to switch between bidirectional and causal attention depending on the use case.

1.2 The Role of Each Layer

Positional encoding: embedding order information

The Transformer's first distinctive element is positional encoding. Self-Attention treats every element the same way regardless of position, so on its own it cannot tell "こんにちは" (konnichiwa) apart from "はちにんこ" (the same characters in scrambled order). We therefore need to add position information unique to each position.

Positional encoding uses periodic functions built from sine and cosine waves. For a position p and dimension i, it is computed by

PE(p,2i) = sin(p / 10000^(2i / d_model))
PE(p,2i+1) = cos(p / 10000^(2i / d_model))

the formulas above. This approach gives each position a unique representation while also preserving the relative relationships between positions.

Professor Quo: Let me explain positional encoding with seat numbers as an analogy.

Manabu: Seat numbers?

Professor Quo: Yes. In a concert hall, if every seat looked identical, you would be in trouble — you would have no idea where to sit.

Manabu: True — seat numbers are how I find my own seat.

Professor Quo: The Transformer works the same way. By giving the phonemes こ (ko), ん (n), に (ni), ち (chi), and は (wa) seat numbers 1 through 5, the model can tell their order. But with plain numbers it is hard to see relationships like "seats 1 and 2 are adjacent" and "seats 1 and 5 are far apart," so we use sin/cos functions — a clever numbering scheme that also expresses positional relationships.

About the positional-encoding formulas

PE(p,2i) = sin(p / 10000^(2i / d_model))
PE(p,2i+1) = cos(p / 10000^(2i / d_model))

The formulas alone may look intimidating, but the key point is that positions are distinguished by wave patterns.

Why combine sine and cosine waves
By overlaying waves with different periods, both short-distance and long-distance differences can be expressed.
For example, positions 2 and 3 are easy to tell apart using the high-frequency waves, while the difference between positions 10 and 100 shows up in the low-frequency waves.
Why use 10000
It is the base value used to scale the waves exponentially.
Because the frequencies spread out on a logarithmic scale, positions can be represented uniquely for sequences of any length.
Relative relationships are preserved too
Thanks to the periodic nature of sin/cos, the difference between p and q is embedded as a phase difference, so the model can naturally learn information such as "these two are three steps apart."

The Multi-Head Self-Attention layer: a mechanism for seeing the whole picture

Now, this is unquestionably the core.

Focusing on the Multi-Head Self-Attention layer

The Multi-Head Self-Attention layer — the heart of the Transformer — consists of eight independent attention heads. Each head learns the relationships between phonemes from a different angle.

For example, one head might focus on relationships between adjacent phonemes, while another focuses on phonemes that share the same place of articulation (such as た (ta) and だ (da)). Yet another head might focus on the relationship between the beginning and end of a sentence, capturing the overall tone.

The outputs of the eight heads* are ultimately merged, enabling a multifaceted understanding of the relationships between phonemes. It is like eight experts each analyzing from their own viewpoint and then combining their findings into a single judgment.

*We described the eight heads as "adjacency," "vowels," "prosody," and so on for convenience, but in actual training the heads do not necessarily specialize into roles humans can interpret. Also, d_model is designed to be divisible by num_heads.

That said, this can be hard to picture, so let's take a closer look.

How does Multi-Head Attention actually work?

As shown in the figure above, our Transformer for lipsync training has eight independent attention heads*, each of which looks at the relationships between phonemes from a different perspective.

In other words, the model learns the relationship between sounds and mouth shapes from eight "different perspectives."

So what are these eight perspectives? Let's look at a concrete example.

Head 1: adjacencyEmphasizes the connections between neighboring phonemes, such as こ (ko) → ん (n)Learns smooth transitions of mouth movement
Head 2: same place of articulationた (ta) and だ (da) (same tongue position)ま (ma) and ば (ba) (both use the lips)Learns relationships between phonemes with similar mouth shapes
Head 3: vowel relationshipsFocuses on the vowel segments あ, い, う, え, お (a, i, u, e, o)Learns patterns of mouth opening
Head 4: sentence beginning and endThe relationship between sentence start and end (intonation)Captures the overall tone of the utterance
Head 5: accent positionDetects emphasized phonemesIdentifies where mouth movements become larger
Head 6: silence and pausesPositions of punctuation and breathsLearns the timing for closing the mouth
Head 7: consonant clustersConsonant sequences such as "str" and "spl"Complex combinations of mouth movements
Head 8: long-range prosodyThe rising intonation of questionsRhythm patterns across the whole sentence

(*The eight perspectives above are just one example. In actual training, a wide variety of perspectives emerges through trial and error.)

Why do we need as many as eight perspectives?

As noted above, it is because doing

"something like eight experts analyzing from their own viewpoints and combining their results into a judgment"

promises a more sophisticated level of lipsync.

In short:

A single expert (one head) can only look from one perspective
Eight experts (eight heads) can analyze from multiple perspectives at once
The eight analysis results are ultimately combined to produce a more accurate mouth shape

That is what it comes down to — and it enables natural lipsync that simultaneously accounts for complex factors such as context, intonation, and rhythm, not just the bare sequence of phonemes.

Professor Quo: Today, let's study the core of the Transformer: Multi-Head Self-Attention.

Manabu: The Transformer! Even the name sounds cool. Wasn't there a movie with the same name?

Professor Quo: Indeed. I was quite a fan of the "Transformers" anime back in the day.

Manabu: Professor, Transformers is a CG-heavy movie franchise, not an anime, isn't it?

Professor Quo: No, an anime. When I was little, a TV anime called "Transformers" was on the air. It was so cool that I even bought the Famicom (NES) game.

Manabu: Never heard of it. Though it's nice to know you had a childhood like that, Professor. I wonder what the Famicom Transformers game was like. For the record, I love the movie version — all that CG and those spectacular transformation scenes.

Professor Quo: Generations, I suppose. Now, back to the AI kind of Transformer. Manabu, when you view a painting in a museum, how do you look at it?

Manabu: Well... I look at the whole, then at the details...

Professor Quo: An excellent observation. The Multi-Head Self-Attention layer works in much the same way. It is like eight art critics analyzing the same painting, each from a different angle.

Manabu: Do we really need eight critics? Wouldn't one be enough?

Professor Quo: Good question. A single critic might only look at color. With eight, they can analyze composition, brushwork, light and shade, historical context, and more. The same holds in the world of phonemes.

Manabu: I see! So what specialty does each critic... I mean, each head have?

Professor Quo: Let me introduce each one as a specialist.
Expert 1, "the adjacency appraiser"
Watches how neighboring sounds connect, like the transition from こ (ko) to ん (n). Rather like a nursery teacher observing children walking hand in hand.

Expert 2, "the twin-spotter of sounds"
Excels at finding sounds with the same tongue position, like た (ta) and だ (da). A bit like a maternity-ward nurse who can tell twins apart.

Expert 3, "the vowel tuner"
Focuses on the vowels あいうえお (a, i, u, e, o). Adjusts how far the mouth opens, the way a piano tuner adjusts pitch.

Expert 4, "the story editor"
Looks at the relationship between the beginning and end of a sentence. Like an editor weighing a novel's opening against its ending.

Manabu: What fun analogies! What about the other four?

Professor Quo: Here are the rest.
Expert 5, "the conductor of dynamics"
Detects accent positions. Decides where the mouth opens wide, the way a conductor shapes loud and soft.

Expert 6, "the master of pauses"
Finds punctuation and breaths. A specialist who understands the aesthetics of silence, like the timing of a rakugo storyteller.

Expert 7, "the tongue-twister analyst"
Handles consonant clusters like "str" and "spl." Something like a coach who trains announcers' diction.

Expert 8, "the intonation musician"
Senses the melody of the whole sentence, such as the rising tone of a question. Grasps the flow of sound the way a composer writes a melody.

Manabu: So when the eight experts work together, we get perfect lipsync!

Professor Quo: Exactly. Like the harmony played by eight instruments, the perspectives are integrated to produce natural mouth movements. Fine details that a single expert would miss, eight together can reliably capture.

What is the attention matrix?

The attention matrix is a table that quantifies how much each element attends to (references) every other element.Each row (per Query) is normalized with Softmax into a probability distribution that sums to 1 (in general the matrix is not symmetric).

In the context of lipsync, it expresses how much each phoneme consults the other phonemes when deciding its mouth shape.

For example, with the five phonemes* of "こんにちは" (konnichiwa), it expresses the strength of association as a number for every one of the 5 × 5 = 25 combinations.

*The actual training input is not "phonemes" but continuous feature frames such as wav2vec.

For example, the matrix below represents the following:

A 5×5 grid: the relationships among the phonemes こ (ko), ん (n), に (ni), ち (chi), and は (wa)
Red circles (dark): self-attention on the diagonal (each phoneme attending to itself)
Orange circles (medium): attention between adjacent phonemes
Pale orange circles: attention between distant phonemes, such as the beginning and end of the sentence

That is what each mark stands for.

Manabu: Professor, does the attention matrix have anything to do with matrices in math?

Professor Quo: A sharp observation. Actually, think of it as something like a "friendship map" of a school class.

Manabu: A friendship map? What do you mean?

Professor Quo: Imagine the five sounds of "こんにちは" (konnichiwa) as five classmates. A table of who pays how much attention to whom — that is the attention matrix.

Manabu: I see! With five people, that makes 5 × 5 = 25 relationships.

Professor Quo: Excellent arithmetic! And the interesting part is that the strength of each relationship is shown by the depth of the color. A bit like deciding the proportions of seasonings in a recipe.

Manabu: A cooking recipe! Now that I can picture.

Professor Quo: Right. The "self and self" relationship on the diagonal is the leading role — the soy sauce — in deep red. Neighbors like ん (n) and に (ni) are the well-matched supporting cast — the mirin — in orange. Distant pairs like こ (ko) and は (wa) are like a hidden pinch of salt and pepper, drawn in pale colors.

Manabu: So it's a "recipe chart" of how much the phonemes consult one another!

Professor Quo: Precisely! And looking at this recipe chart, each phoneme decides, "This time I'll take 70% from my neighbor, 20% from the sound two back, and 10% from the last one." Like a chef tasting as they adjust the balance of seasonings, it crafts the optimal mouth shape.

Manabu: A master chef of phonemes!

Professor Quo: Well put! And because the eight chefs (the eight heads) each hold a different recipe chart, we get a richer flavor... that is, more natural mouth movements.

The cast of actual Transformer training: Q, K, and V

Now, let's focus on this part.

What are the Q, K, and V shown here?

Self-AttentionHere, the input data is transformed into three distinct roles.

Q (Query)
The role that asks, "Who am I most closely related to?"
= "who is looking" (the questioner)
K (Key)
The role that responds, "These are my characteristics."
= "who is being looked at" (the matching target)
V (Value)
The role that offers, "This is the actual information I hold."
= "what gets received" (the actual information)

All three are produced from the same input, but different weight matrices (Linear layers) transform each into a different representation.

In implementation terms, with sequence length T and per-head dimensions d_k and d_v, Q, K ∈ ℝ^T×d_k and V ∈ ℝ^T×d_v. They are generated by multiplying the input vectors X ∈ ℝ^T×d_model by learnable projection matrices W_Q, W_K, and W_V; in the multi-head case, they are further split per head for processing.

Let's think it through with the "こんにちは" (konnichiwa) example. As introduced earlier, head 1 is the one that learns the adjacency relationships between sounds.

Here, Q (Query) and K (Key) can be thought of as follows:

Query axis (horizontal, blue): which phoneme is doing the attending
Key axis (vertical, red): which phoneme is being attended to

"Who" is looking at "whom" — that is the relationship being expressed, where

Query: the one looking (the subject)
Key: the one being looked at (the object)

. For example, in the figure above, the part where Query "ん" (n) attends to Key "に" (ni) looks like the following.

Let's walk through this Query–Key attention computation as a concrete process.

STEP 1. Compute attention scores from Query and Key

How much does ん (n) look at に (ni)? → attention 70%
How much does ん (n) look at は (wa)? → attention 10%

STEP 2. Next, use those attention scores to retrieve the Values with weights

Receive the information (Value) of に (ni) with a weight of 70%
Receive the information (Value) of は (wa) with a weight of 10%

STEP 3. Combine the weighted information

The result becomes the new representation of ん (n)

In other words, it looks something like this:

Query ん (n): "Whose information should I consult to decide the next mouth shape?"
Key: "Check the relevance"

に (ni) → the next sound, so it's important! (70%)
こ (ko) → the previous sound, so somewhat important (20%)
は (wa) → far away, so not very important (10%)

Value: "the actual mouth-shape information"

mouth-shape info of に (ni) × 70%
mouth-shape info of こ (ko) × 20%
mouth-shape info of は (wa) × 10%
↓
Combine these to decide the optimal mouth shape for ん (n)

Now, why do Key and Value need to be separate in the first place?

Because if Key and Value were the same, the "relevance judgment" and the "information retrieved" would be identical (which is obvious when you think about it).

In other words, by keeping them separate, the model can learn

Key: features used for judging relevance
Value: the rich information actually being conveyed

independently of each other.
This is what makes the attention mechanism more flexible and expressive.

Q, K, V: The Three Elements of Postal Delivery

Manabu: Q, K, V — they sound like some kind of secret code.

Professor Quo: They do seem cryptic. But compare them to a postal delivery system, and they become quite clear.

Manabu: Postal delivery? How so?

Professor Quo: Think of it this way.
Q (Query) is "the person looking for an addressee." It asks, "I want to deliver this letter — who will receive it?"

K (Key) is "the address plate." It announces, "This is where I live!"

V (Value) is "the actual package" — the precious contents you want delivered.

Manabu: I see! But how does this play out in the world of phonemes?

Professor Quo: Suppose the phoneme ん (n) is the mail carrier. Query ん asks around: "To decide the next mouth shape, who should I get information from?"

Manabu: And that's where the Keys answer!

Professor Quo: Exactly! Key に (ni) raises a hand: "I'm the next sound, so I have very important information — relevance 70%!" Key こ (ko) responds modestly: "I'm the previous sound, but my influence still lingers. Relevance 20%."

Manabu: Then what role does Value play?

Professor Quo: Value is the treasure! Think of Values as library books. Read 70% of に's book, 20% of こ's book, and 10% of は's book, then integrate that knowledge to craft the perfect mouth shape for ん.

Manabu: But Professor, why do Key and Value need to be separate? Couldn't they be the same...

Professor Quo: A wonderful question! Take a restaurant. The menu (Key) may say "today's special," but the actual dish (Value) is something else entirely. The menu is information for choosing; the dish is what you actually taste. That difference is what matters.

Manabu: I see! Separating "information for choosing" from "information actually received" makes everything more flexible!

Professor Quo: Exactly right. The phonetic similarity (Key) between ん (n) and に (ni) may be low, yet their sequential importance is high — and the mouth-shape information actually received (Value) has characteristics of its own. This three-way division of labor is the secret of natural lipsync.

What the actual attention matrix expresses

This part is genuinely hard to grasp, so at the risk of repeating ourselves, let's go back to the attention matrix from earlier.

So what is this matrix, in terms of the actual computation? It is a visualization of the dot product of Q and K — that is, of the attention scores computed from it.

Attention Score = Q × K^T / √d_k

(d_k: the Key dimension of each head)

In this matrix,

Horizontal axis (Query): how much each phoneme, as the "questioner," looks at the others
Vertical axis (Key): how much each phoneme, as the "target," is looked at by the others
Each cell value: the strength of association (attention) for that phoneme pair

That is how it reads.

This time, take the relationship between ん (n) and に (ni): the cell at coordinates (1, 2) — ん on the horizontal axis × に on the vertical axis —

the Query of "ん" and the Key of "に" — the cell expresses their similarity
A larger value = the information of に (ni) matters more when processing ん (n)
In proportion to this attention score, the Value of "に" (the actual information) is retrieved

At the risk of belaboring the point, let me explain once more why splitting into K, Q, and V is necessary.

If only a single representation were used, the "relevance judgment" and the "information conveyed" would be one and the same (as noted earlier).

By separating them:

Q–K pairs: can learn which elements relate to which (the attention patterns)
V: can hold the rich information to actually convey, kept separately

For example, even if the phonetic similarity (K) between ん (n) and に (ni) is low, their sequential importance (the learned Q–K result) can be high, and the mouth-shape information received from に (V) can carry yet other characteristics. Complex relationships like this become expressible.

From the matrix to the actual processing

Reviewing the network diagram below once more:

Attention computation: build the matrix with Q × K^T
Normalization: Softmax makes each row sum to 1 (a probability distribution)
Information retrieval: take a weighted average of V using the attention as weights
Result: a new, context-aware representation of each phoneme

In short, this matrix is the blueprint for "which phoneme references how much of which phoneme's information," and the actual information (the Values) is combined according to this blueprint.

Eight different perspectives and their attention matrices

So far we have looked at the example of head 1. To close this section, let's visualize the matrices of all eight perspectives.

Characteristics of each head

The heads are as follows:

Head 1 (adjacency): attends to the diagonal and to adjacent phonemes
Head 2 (vowel tracking): attends to vowel segments and matching vowels (the "i" sound in に (ni) and ち (chi))
Head 3 (consonant features): attends to identical consonants (ん and ん) and similar consonants
Head 4 (sentence start/end): emphasizes the relationship between the beginning and the end
Head 5 (accent): attends around the emphasized sound (に)
Head 6 (speech rate): patterns of even timing adjustment
Head 7 (long-range dependencies): relationships between distant phonemes
Head 8 (rhythm): wave-like rhythm patterns

In each matrix, color and size express the degree of attention:

Large, dark circles = strong attention
Small, pale circles = weak attention
Different colors = different kinds of attention

These are ultimately integrated to generate natural lipsync.

Computing the Actual Attention Matrix: Completing the Recipe

Manabu: Professor, the actual computation sounds hard... will there be formulas?

Professor Quo: No need to worry. Picture a cooking show. The formula "Attention Score = Q × K^T / √d_k" is like the golden ratio of a recipe. (*d_k is the Key dimension of each head.)

Manabu: A cooking show! That sounds approachable.

Professor Quo: Right. The horizontal axis is "who does the tasting (Query)," and the vertical axis is "whose dish gets tasted (Key)." The number in each cell is the score for "how much to take as reference."

Manabu: What about the relationship between ん (n) and に (ni), for example?

Professor Quo: Picture a chessboard. The square at coordinates (1, 2) — where the row of ん meets the column of に — shows how much chef ん consults chef に's dish. A high score means, "This is important — let me take notes."

Manabu: It's like the scoring sheet of a cooking contest!

Professor Quo: A good analogy! And dividing by √d keeps the scores from inflating — like choosing whether to grade out of 100 points or out of 10.

Integrating the Eight Heads: An Orchestral Performance

Manabu: How do the results of the eight heads come together into one at the end?

Professor Quo: Picture an orchestra. Eight instrument sections each play a different part, yet in the end it becomes one beautiful piece of music.

Manabu: I see! But concretely, how does it work?

Professor Quo: First, each head computes attention scores with Q × K^T. That is the stage where each instrument practices its own part. Next, a bit of magic called Softmax makes each row sum to 1 — like balancing the volume levels.

Manabu: Volume balancing! That makes sense.

Professor Quo: Then the Values are averaged, weighted by those attention scores. It is like mixing each instrument's sound at just the right volume to produce the final performance. The result is a new representation for each phoneme.

Manabu: So eight conductors each lead their own section, and in the end it becomes a single symphony!

Professor Quo: A superb understanding! Exactly so. Just as the violins carry the melody, the cellos the bass, and the winds the color, each head captures different features. When they harmonize, a plain sequence of sounds turns into vivid, natural mouth movements.

Manabu: Music, math, and AI fused together — a true work of art!

Professor Quo: Indeed. Behind the technology hides this beautiful mechanism. The harmony woven by eight perspectives is what produces the natural lipsync we see.

Coming up next

In the second half, we will clarify the role of the feed-forward layer and the structural differences from LSTM, and then focus on the implementation challenges the Transformer faces.

See you next time!