Hello!

Today I would like to give a comprehensive overview of lipsync technology, which we actually use in our MotionVox product.

Lipsync technology, which generates natural mouth movements synchronized to speech, is an important technology for AI avatars and 3D animation production.

In this article, we take a technical deep dive into lipsync learning with modern deep learning—from the fundamentals through to implementation.

1. Fundamentals of Lipsync Learning

1.1 Problem Setup

Lipsync learning can be framed as a regression problem: predicting the corresponding mouth movements from audio data

f: audio features(t) → mouth movement parameters(t)

The core of this problem is

learning the correspondence between phonemes (sound characteristics) and visemes (visual mouth shapes)

.

1.2 The Complexity of Phoneme-Viseme Mapping

There is a catch, though!
The relationship between sounds and mouth shapes in human speech is not a simple one-to-one mapping.

The same sound changes with context

Even for the same vowel "あ" (a):
- "あ" after "か" (ka) → the mouth opens from a slightly narrowed position
- "あ" after "ん" (n) → the mouth opens wide from a closed position

Coarticulation

The "ん" (n) in "かんぱい" (kanpai, "cheers"):
- "ん" on its own → the mouth closes completely
- "ん" before "ぱ" (pa) → the mouth stays half open, preparing for the next articulation

This complexity is what makes realistic lipsync so rewarding—and it is the technical challenge at the heart of the learning problem.

2. Facial Expression Parameter Representation

2.1 Specification of a Hypothetical FaceModel

In this article, we will assume a hypothetical facial expression model (we call it QualitegFaceModel, or QFM) like the following

The full face has 120 expression parameters, of which 26 are used for the mouth area—that is, for lipsync control.

# FaceModel parameter structure
total_params = 120  # total number of expression parameters
mouth_params = 26   # number of mouth-area parameters (indices: 94-119)

# Parameter groups
expression_groups = {
    "eyebrow": [0, 1, 2, 3, 4, 5, 6, 7],          # eyebrow movement (8)
    "eyelid": [8, 9, 10, 11, 12, 13, 14, 15],     # eyelid movement (8)
    "eye_gaze": [16, 17, 18, 19],                 # gaze direction (4)
    "cheek": [20, 21, 22, 23, 24, 25, 26, 27],    # cheek movement (8)
    "nose": [28, 29, 30, 31, 32, 33],             # nose movement (6)
    "general": [34, 35, 36, 37, 38, 39, ..., 93], # other overall expression (60)
    "mouth": [94, 95, 96, 97, 98, 99, ..., 119]   # mouth area (26)
}

2.2 Detailed Breakdown of the Mouth-Area Parameters

The 26 mouth-area parameters, the most important ones for lipsync, are as follows

mouth_parameter_details = {
    # Basic mouth opening/shape (8)
    "jaw_open": 94,           # jaw opening
    "mouth_open": 95,         # mouth opening
    "mouth_wide": 96,         # mouth width
    "mouth_narrow": 97,       # mouth narrowing
    "lip_upper_up": 98,       # upper lip raise
    "lip_lower_down": 99,     # lower lip lowering
    "mouth_round": 100,       # mouth rounding (for the "う" (u) sound)
    "mouth_stretch": 101,     # horizontal stretch (for the "い" (i) sound)
    
    # Left-right asymmetric movement (6)
    "mouth_left": 102,        # mouth shift left
    "mouth_right": 103,       # mouth shift right
    "lip_corner_left_up": 104,    # left mouth corner raise
    "lip_corner_right_up": 105,   # right mouth corner raise
    "lip_corner_left_down": 106,  # left mouth corner lowering
    "lip_corner_right_down": 107, # right mouth corner lowering
    
    # Fine lip movement (8)
    "lip_upper_left": 108,    # upper lip, left part
    "lip_upper_right": 109,   # upper lip, right part
    "lip_lower_left": 110,    # lower lip, left part  
    "lip_lower_right": 111,   # lower lip, right part
    "lip_pucker": 112,        # lip protrusion
    "lip_press": 113,         # lip press
    "lip_bite_upper": 114,    # upper lip bite
    "lip_bite_lower": 115,    # lower lip bite
    
    # Special articulation movements (4)
    "tongue_tip_up": 116,     # tongue tip raise (for the "ら" (ra) row) ※1
    "tongue_back_up": 117,    # back-of-tongue raise (for the "か" (ka) row) ※1
    "lip_trill": 118,         # lip trill (for the "ぶ" (bu) sound)
    "mouth_dimple": 119       # dimple (when smiling)
}

※1 About the tongue-movement parameters
In real face-capture and avatar systems, tongue movement is hard to observe from the outside, so in practice it is handled by a separate system. In this example, however, we include it in the data to improve lipsync accuracy.

2.3 Example Correspondence with Phonemes

Now let's look at the concrete correspondence between Japanese phonemes and mouth-area parameters for an actual utterance of "こんにちは" (konnichiwa, "hello").

This is a simplified example, but expressing each phoneme of "こんにちは" in mouth-area parameters looks like the following. (Reality is more complex.)

In essence, what we need to learn is the correspondence between the sound "こ" (ko) and the mouth shape—the control parameters describing the mouth—produced when "こ" is pronounced.

So as a learning problem, the theme itself is simple, and the correspondence is intuitive.

phoneme_mapping = {
    "こ": {  # the "こ" (ko) sound
        "jaw_open": 0.5,      # opens moderately
        "mouth_open": 0.6,    # mouth opens
        "mouth_round": 0.4,   # slightly rounded
        "tongue_back_up": 0.7 # back of tongue raised (a "か"-row trait)
    },
    "ん": {  # the "ん" (n) sound
        "jaw_open": 0.0,      # closed
        "mouth_open": 0.0,    # mouth closes
        "lip_press": 0.9,     # lips pressed together
        "mouth_round": 0.0    # no rounding
    },
    "に": {  # the "に" (ni) sound
        "jaw_open": 0.3,      # opens slightly
        "mouth_stretch": 0.6, # stretched slightly sideways
        "tongue_tip_up": 0.8, # tongue tip raised
        "lip_corner_left_up": 0.3,
        "lip_corner_right_up": 0.3,
    }
}

That said, simple does not mean easy.

3. Why wav2vec Is Needed: Features That Let Computers Understand Sound

3.1 What Are Audio "Features"?

Computers cannot understand raw sound

If you play the spoken phrase "こんにちは" to a computer, all the computer sees is a plain sequence of numbers—nothing more

# Raw audio data for "こんにちは" (16kHz, 1 second)
raw_audio = [0.001, -0.002, 0.003, -0.001, 0.004, -0.002, ...]
# ↑ just 16,000 fine-grained numbers in a row

# From the computer's point of view:
"What does any of this mean? Which part is 'こ' and which part is 'ん'?"

A human instantly understands "ah, that's a greeting," but a computer has no grasp of what the sound means.

Features = converting data into "meaningful information"

Features are raw data converted into "a form that is easy for a computer to work with and that carries meaning".

# Raw audio → feature conversion
raw_audio = [0.001, -0.002, 0.003, ...]  # a meaningless string of numbers
↓ feature extraction
meaningful_features = [0.8, -0.2, 0.5, ...]  # a meaningful representation

# Each feature value carries meaning:
# 0.8 → strength of "vowel-ness"
# -0.2 → scarcity of high-frequency components  
# 0.5 → degree of "back-of-tongue" articulation

3.2 Why Features Are Needed: Understanding Through the Human Example

How humans understand "こんにちは"

The human brain, in fact, also performs feature extraction:

[How humans understand speech]

1. The ear catches the sound wave
   ↓
2. Features such as "low pitch," "high pitch," and duration are extracted
   ↓  
3. Sounds are classified: "this is a vowel," "this is a consonant"
   ↓
4. The phonemes "こ-ん-に-ち-は" (ko-n-ni-chi-wa) are recognized
   ↓
5. The meaning is understood: "it's a greeting"

Computers want to do the same thing, but since they lack human-style intuitive understanding, they need mathematically defined features—that is the idea.

3.3 Problems with the Traditional Approach (MFCC)

While we are at it, let's deepen our understanding with a brief look at the history of audio features.

The fundamental limits of MFCC

Traditionally, speech recognition and lipsync relied heavily on Mel-frequency cepstral coefficients, or MFCC (Mel-frequency Cepstral Coefficients). MFCC, however, came with a number of inconveniences.

Although we call it classic, anyone who studied audio signal processing up until a few years ago knows it as a very mainstream feature: in speech signal processing and speech recognition, terms like "mel spectrum" and "Mel-frequency cepstral coefficients (MFCC)" should ring a bell.

Everyone used to say, "Raw waveforms won't tell you anything about speech—analyze it with the mel spectrum!" Yet looking back now, the mel spectrum didn't really tell us all that much either. Such is the remarkable pace of technological progress.

Old war stories aside, MFCC follows the processing flow below.

[MFCC processing flow]
Raw audio → Fourier transform → mel filter bank → log transform → DCT → MFCC coefficients

Problems with MFCC

MFCC had the following problems. (Of course, at the time it was all we had, so they were hardly recognized as problems—like the egg of Columbus, that is simply how technology and science work.)

1. Hand-crafted features
   ⇒ A fixed feature extraction scheme designed arbitrarily by humans, based on human hearing and perception
2. No understanding of phonemic meaning
   ⇒ Simple frequency analysis
3. Lack of contextual information
   ⇒ Independent frame-by-frame processing
4. Poor handling of individual variation
   ⇒ A tendency to be speaker-dependent

Concrete examples

Before deep learning, and in its early days, when speech recognition was done with these older techniques, things like the following would happen

(1) Similarity in place of articulation could not be captured

"か" (ka) and "が" (ga):
- Physically the same place of articulation (soft palate)
- Nearly identical mouth shapes
- With MFCC: completely different feature vectors
- As a result... the model cannot learn the similarity between "か" and "が"

(2) Context-dependent phonemic variation was ignored

Pronouncing "ん" (n):
- The "ん" in "かん" (kan) → the mouth closes completely
- The "ん" in "かんぱい" (kanpai) → half open, preparing for the following "ぱ" (pa)
- With MFCC: both are processed as the same "ん"
- As a result... unnatural mouth movements that ignore context

(3) Fragility across speakers

Even for the same "あ" (a):
- Male speaker (low fundamental frequency): MFCC = [1.2, -0.5, 0.8, ...]
- Female speaker (high fundamental frequency): MFCC = [2.1, -1.2, 1.5, ...]
- As a result... completely different mouth movements are predicted when the speaker's gender changes

And so, making its long-awaited entrance, came wav2vec .

As the name suggests, it is a tool for turning a wav (audio signal) into a vec (vector).

Roughly speaking, the idea is similar to the famous word2vec.

3.4 wav2vec's Innovative Approach

Concretely, wav2vec is a neural-network-based audio feature extraction model※2 developed by Meta AI (formerly Facebook AI).

Phonemic understanding through self-supervised learning

So what makes wav2vec special?

It is that
wav2vec "understands the meaning of phonemes".

Technically, wav2vec uses a CNN encoder to extract local acoustic patterns from raw audio, then a Transformer encoder to understand the contextual relationships between phonemes, finally outputting a 768-dimensional※3 feature vector. Through this processing it acquires a rich representation carrying not just acoustic information but the semantic information of phonemes, becoming "a feature extractor that understands the meaning of sound".

※ Today, wav2vec 2.0 is the mainstream version.

• ※3 768 dimensions for the Base model, 1024 dimensions for the Large model

Data-driven feature learning

wav2vec's greatest innovation is that, instead of a fixed human-designed transform like MFCC, a neural network learns the optimal feature extraction method from data. The CNN encoder learns the best way to extract acoustic features from raw audio, and the Transformer encoder learns the best way to integrate context by understanding the relationships between phonemes

This learning process makes it possible to capture complex, non-linear phonemic relationships that MFCC could never represent. Moreover, by changing the training data or training objective, you can obtain feature representations optimized for specific tasks

Acquiring phonemic similarity

Through self-supervised learning on large volumes of speech data, wav2vec comes to understand the semantic relationships between phonemes. Using a technique similar to masked language modeling—masking part of the audio and predicting it—it automatically discovers similarities and relationships between phonemes. As a result, linguistically similar phonemes end up close together in the feature space: phonemes with the same place of articulation, such as "か" (ka) and "が" (ga), have high cosine similarity, while completely different phonemes, such as "か" (ka) and "あ" (a), have low similarity. Acquiring this phonemic similarity greatly improves learning efficiency and also improves generalization to unseen phoneme combinations. In this respect, wav2vec resembles word2vec.

Learning speaker-invariant representations

By training on speech from a diverse range of speakers, wav2vec acquires phonemic representations that do not depend on speaker-specific characteristics (fundamental frequency, formant frequencies, and so on). The training objective is to map the same phoneme from different speakers to the same feature vector, which is what produces speaker invariance.

As a result, whether the speaker is male, female, a child, or elderly, wav2vec outputs similar feature vectors for the same phoneme. This speaker robustness makes it possible to build lipsync systems that handle a wide variety of speakers even with limited training data.

4. Optimizing the wav2vec Feature Extractor with Language-Specific Training

The base wav2vec model can handle multiple languages, but for tasks demanding high precision, such as lipsync, building a feature extractor specialized for a language's specific phonological system can push accuracy even further.

Why language-specific training matters

Every language has its own phonological system. Japanese, for example, is a simple system built on five vowels (あ, い, う, え, お), whereas English has more than 15 vowels plus complex consonant clusters (such as the "str" in "street").

A phenomenon unique to Japanese is that the pronunciation of "ん" (n) changes dramatically depending on the sound that follows. The "ん" in "さんぽ" (sanpo) closes the lips, while the "ん" in "さんか" (sanka) leaves the mouth open. Special phonemes such as "っ" (the geminate stop) and "ー" (the long vowel mark) are also unique to Japanese.

A general-purpose wav2vec model learns the characteristics of all these languages "on average," so it may miss the fine-grained traits of any particular language. It is rather like the difference between "a chef who can cook every cuisine in the world" and "an itamae who specializes in Japanese cuisine." For work as delicate as lipsync, you need a specialist in that language.

The concrete benefits of language-specific training

Training a wav2vec model specialized for Japanese yields remarkable accuracy gains, and these gains translate directly into lipsync quality. A Japanese-specialized model accurately captures the subtle differences in mouth shape between the "ん" in "こんにちは" (konnichiwa) and the "ん" in "こんばんは" (konbanwa), producing more natural mouth movements.

An English-specialized model, in turn, can accurately predict tongue positions for phonemes that do not exist in Japanese, such as the "th" in "think," and the complex lip movements of consonant clusters. A Chinese-specialized model can even distinguish the subtle mouth-shape differences that tones impose on the same sound.

The actual training process

Language-specific wav2vec training requires a large-scale speech dataset in that language. For Japanese, it is important to use hundreds to thousands of hours of speech and to include speakers of various ages, genders, and dialects. The data should span diverse genres, balanced so that every phoneme appears evenly.

Summary

wav2vec is not merely "a new feature extraction method"—it deserves to be viewed as "an AI that understands the meaning of phonemes."

Where MFCC deals only in frequency distribution patterns based on human-designed methods of sound analysis, wav2vec deeply understands the semantic relationships and context of phonemes through a way of understanding sound that the AI discovered for itself.

This technical breakthrough promises high-precision, highly realistic lipsync that seemed out of reach for conventional MFCC-based systems.

Furthermore, by using a wav2vec feature extractor trained per language, you can

1. accurately capture the phonological phenomena unique to that language,
2. distinguish the fine sound variations that context produces, and
3. as a result achieve more natural, more accurate lipsync.

Especially for a language like Japanese, rich in context-driven phonemic variation, using a language-specialized wav2vec model can make or break lipsync quality.

In upcoming installments, we plan to cover lipsync drift control, the procedure for learning the mapping between speech and lipsync from wav2vec features, and more.

See you next time!