arxiv:2609.16591
Published on Sep 15
· Submitted by
Guangyu Sun on Sep 16
· Meta
Upvote
16
Authors:
,
,
,
,
,
,
,
,
Abstract
Traditional multimodal representation learning and generation are two stages: a contrastive or self-supervised visual encoder is trained first, followed by a separate downstream generative model. This setup bottlenecks generative performance behind frozen embeddings. To bridge this gap, we revisit joint multimodal representation learning and generation to produce linearly interpolatable embeddings that are directly consumable by generative decoders. We present FLAT (Flexible-Length Aligned Transmodal representations), a representation pre-training framework that jointly optimizes a shared multimodal encoder alongside downstream text-to-image (T2I) and image-to-text (I2T) decoders. By combining contrastive alignment with bidirectional cross-modal generative objectives, FLAT ensures its representations function as both discriminative semantic descriptors and generative conditions. Architecturally, FLAT maps visual and textual inputs into a unified continuous 1D sequence space, applying nested dropout over prefix-K tokens to enable dynamic output lengths. A single pre-training stage allows FLAT to perform cross-modal retrieval and generation across variable prefix K, achieving a T2I GenEval score of 71.1. Task-specific fine-tuning aligns model performance with state-of-the-art baselines: 83.1 GenEval on T2I generation; 40.5 BLEU-4 and 138.6 CIDEr on MS-COCO image captioning; and Recall@5 scores of 86.8 (I2T) / 75.8 (T2I) on MS-COCO alongside 98.3 (I2T) / 93.6 (T2I) on Flickr30K. Finally, qualitative evaluations demonstrate that FLAT representations natively support linear interpolation, latent space arithmetic, and zero-shot composed retrieval.
View arXiv page View PDF Project page Add to collection
Community
Paper author Paper submitter 6 days ago
Excited to share FLAT, a unified framework for multimodal representation learning and generation. FLAT resamples images and text into flexible-length, aligned 1D token sequences that support retrieval, image captioning, and text-to-image generation within a shared representation space. Its representations also enable interpolation, latent arithmetic, and zero-shot composed retrieval, while allowing adjustable token budgets through variable prefix lengths.
We’d love to hear your feedback!
6 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
UniSpace: Unified Visual Representation and Scalable Multimodal Modeling (2026)
-
GALA: Generation-Aware Cross-Modal Alignment for Text-to-Time-Series Synthesis (2026)
-
Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation (2026)
-
Twins: Learn to Predict Unified Representations with Focal Loss (2026)
-
MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations (2026)
-
DIFFCZSL: Compositional Zero-Shot Learning Regularized by Diffusion Representations (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
16
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.16591 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.16591 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.16591 in a Space README.md to link it from this page.