Natively Multimodal Model

Natively multimodal models are models trained from the start on more than one modality, so that text, images, audio, and sometimes video are represented in a single token stream and processed by one shared set of weights. The contrast is with the stitched approach that dominated 2022 to 2024: take a finished language model, take a separately pretrained vision or audio encoder, connect them with a small trained adapter, and fine-tune. That works, but the language model never actually learned to see or hear during pretraining; it learned to read a compressed summary that an encoder handed it. A native model closes that gap by making every modality a first-class citizen of pretraining. Google described Gemini as “natively multimodal” and “pre-trained from the start on different modalities”; OpenAI described GPT-4o as “a single new model end to end across text, vision, and audio,” where “all inputs and outputs are processed by the same neural network.” Both phrasings point at the same architectural commitment.

Native Versus Stitched (Bolt-On) Multimodality

The stitched pattern, often called a late-fusion design, keeps modality-specific components and merges their outputs deep in the stack:

graph TD
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    subgraph LATE[Late fusion: stitched / bolt-on]
      IMG1([Image]):::data --> VE[Frozen vision encoder]:::process
      VE --> PROJ[Trained projector]:::process
      TXT1([Text]):::data --> TOK1[Text embeddings]:::process
      PROJ --> LLM1[Pretrained LLM<br/>weights largely frozen]:::process
      TOK1 --> LLM1
      LLM1 --> OUT1[Text out]:::output
    end

    subgraph EARLY[Early fusion: natively multimodal]
      IMG2([Image]):::data --> PATCH[Patch / codec tokens]:::process
      TXT2([Text]):::data --> TOK2[Text tokens]:::process
      AUD2([Audio]):::data --> FRAME[Audio frame tokens]:::process
      PATCH --> SEQ[One interleaved sequence]:::process
      TOK2 --> SEQ
      FRAME --> SEQ
      SEQ --> XF[Single transformer<br/>trained from scratch on all modalities]:::process
      XF --> OUT2[Text / image / audio out]:::output
    end

The practical differences:

Tokenizing Every Modality

Native models need every modality to become tokens the transformer can attend over alongside text. There are two common routes:

• Continuous patch embeddings. An image is cut into patches, each linearly projected to an embedding, and those embeddings are spliced into the sequence next to text-token embeddings. Nothing is discretized. This is how many vision-language natives handle input.
• Discrete modality tokens. A vector-quantized codec turns an image (or an audio clip) into a sequence of integer tokens drawn from a learned codebook, which are appended to the text vocabulary. Meta’s Chameleon does this for images, giving a single softmax over a joint text-plus-image vocabulary, which is what lets one model generate interleaved image and text rather than only consume images.

# Scenario: early fusion by continuous patches. Image patch embeddings are
# concatenated with text token embeddings into one sequence; the transformer
# never sees a "modality" flag, only positions in a stream.
def build_sequence(text_ids, image, patch_embed, tok_embed):
    text_vecs  = tok_embed(text_ids)                 # (T, d)
    patches    = to_patches(image, size=14)          # (P, 14*14*3)
    patch_vecs = patch_embed(patches)               # (P, d)  same width as text
    return concat([patch_vecs, text_vecs], axis=0)   # (P + T, d) -> transformer

# Scenario: discrete mixed-modal tokens (Chameleon style). Images are
# quantized into codebook ids offset above the text vocab, so one model
# predicts the next token whether it is a word or an image code.
IMG_OFFSET = 32000  # text vocab size

def encode_document(text: str, image, vqgan, tokenizer):
    seq = tokenizer.encode(text)
    img_codes = [IMG_OFFSET + c for c in vqgan.encode(image)]   # ints in one vocab
    return seq + [BOI] + img_codes + [EOI]                      # single stream

Early Fusion Versus Late Fusion: the Scaling-Law Evidence

For years the assumption was that a strong pretrained vision encoder was worth its parameter cost, making late fusion the safe default. Apple and Sorbonne’s Scaling Laws for Native Multimodal Models (ICCV 2025), a study spanning 457 trained models, pushed back. Its headline findings:

• No inherent advantage to late fusion. At a matched compute budget (FLOPs), early-fusion and late-fusion models reach similar validation loss.
• Early fusion wins on efficiency. It gets there with fewer parameters and more training tokens, is cheaper to train, needs no image encoder, and is simpler to deploy. At smaller scales it is outright stronger.
• Different optimal mix. Early and late fusion sit at different points on the parameters-versus-data trade-off for the same compute, early fusion tilted toward data.
• Mixture of Experts helps a lot. Adding Mixture of Experts, so the model can learn modality-specific weights implicitly through routing rather than through a hard-wired separate encoder, significantly improves native multimodal performance.

The widget models that last-mile trade-off: at a fixed compute budget, where should the FLOPs go, into more parameters or more tokens, and how does that optimum shift between early and late fusion.

Selected Natively Multimodal Models (2024-2026)

Numbers and capabilities move quickly; treat this as directional.

Where It Falls Short

• Cost of entry. You cannot assemble a native model from parts. It requires a full pretraining run over interleaved data, which is why most teams still fine-tune a stitched model.
• Data scarcity. High-quality interleaved multimodal corpora are far smaller than text corpora, so native pretraining leans on synthetic and web-scraped mixed-modal data with its own noise.
• Uneven modality strength. A native model’s text ability can lag a same-compute text-only model, because some capacity and data budget went to vision and audio. The scaling-law work quantifies this trade rather than denying it.
• Debuggability. With no intermediate caption or transcript, there is no text artifact to inspect when the model misreads an image or mishears audio, the same auditability gap seen in audio-native models.
• Serving complexity. Mixed-modal token streams complicate batching, caching, and context-length accounting compared with plain text.

What’s New (2025-2026)

• Scaling laws settled the architecture debate. Apple and Sorbonne’s 457-model study showed early fusion matches late fusion at equal compute while being smaller, cheaper to train, and encoder-free, removing the main reason teams defaulted to stitched designs.
• MoE became the standard native backbone. Learning modality-specific weights through expert routing rather than a hard-coded encoder is now common, visible in Llama 4 and in the scaling-law paper’s own best configurations.
• Native audio generation went mainstream. GPT-4o, Gemini, and Qwen-Omni ship speech output from the same model that reads text and images, collapsing the old text-to-speech stage.
• Native multimodal embeddings. Google’s Gemini Embedding 2 extends the idea to representation learning, embedding text, image, audio, and video into one shared space from a shared transformer rather than aligning separately trained encoders after the fact.
• Any-to-any generation. Research models increasingly output interleaved image-and-text documents from one joint vocabulary, the direction Chameleon pointed, rather than only consuming non-text input.

Practical Guidance

“Native” is an architectural claim about how a model was built, not a marketing tier. It says the modalities were in the room during pretraining, sharing weights and a token stream from the first step, and the 2025 scaling-law work is why that is now the default way to build a new multimodal model rather than a premium option.

# Scenario: a maintenance assistant that takes a photo of a broken part,
# a voice note describing the symptom, and a text question, then answers
# in text. A natively multimodal model handles all three in one request,
# with no separate OCR, caption, or transcription step.
from google import genai

client = genai.Client()

response = client.models.generate_content(
    model="gemini-3-pro",
    contents=[
        "Here is the part and a description of the fault.",
        genai.types.Part.from_bytes(open("pump.jpg", "rb").read(), "image/jpeg"),
        genai.types.Part.from_bytes(open("symptom.wav", "rb").read(), "audio/wav"),
        "Which component is failing, and what is the likely cause?",
    ],
)

print(response.text)
# The image patches, audio frames, and text tokens are embedded into a
# single sequence and attended over together, not summarised separately
# and stapled into a prompt.