Accepted at ACL 2026 · Main Conference, Oral

MULTI-SCORE Zero-Shot Multimodal Retrieval with Multi‑Scale Contextual Representations

Sourajit Saha and Tejas Gokhale

University of Maryland, Baltimore County  · 

Paper ACL Anthology Code BibTeX
Pyramid Rank, running illustrative run, 48 candidates
query: a dog in green grass
Candidates left
48
Embedding size
32 / 1024
Similarity ops
0 / 49,152
image video audio kept for Stage 2
12retrieval tasks
32datasets
5.7Mdatabase items
250Kqueries
0training or fine-tuning
6×faster than prior SOTA

Abstract

ACL 2026, Main Conference (Oral)

In multimodal information retrieval (MMIR), candidates relevant to an input query need to be retrieved from a database, where the query and database items span different modalities. As real-world databases evolve, repeatedly annotating and indexing data and re-optimizing domain-specific models across modalities is impractical.

We present MULTI-SCORE, a fine-tuning-free, two-stage MMIR approach that couples efficient candidate filtering with fine-grained multimodal re-ranking. Stage-1 adopts Matryoshka representations to efficiently filter out low-relevance candidates without expensive similarity computations on full-scale representations for the entire database. Stage-2 re-ranks the filtered candidates by computing their fine-grained multimodal contextual representations with two scoring functions for semantic alignment using chain-of-thought prompting and question-answering.

Experiments demonstrate state-of-the-art zero-shot retrieval on 12 MMIR tasks across 32 datasets while outperforming supervised methods on 23 datasets.

  • A zero-shot MMIR system that is efficient and aligns text, image, video and audio in unimodal, cross-modal and composite query-candidate combinations.
  • Pyramid Rank, a filtering algorithm with an admissible similarity upper bound over multi-scale representations.
  • Two re-ranking scores, Bidirectional-CoT embedding and QA relevance, for fine-grained multimodal alignment.
  • Experiments at scale, on 12 tasks and 32 datasets, plus a joint pool of 5.7M items retrieved in one universal search space.
Overview of MULTI-SCORE: a multimodal query is matched against a large database, with examples of text to video, text to audio and image plus text to image retrieval, and a radar chart comparing MULTI-SCORE to prior zero-shot and supervised methods across 12 tasks.
One pipeline, every modality. MULTI-SCORE handles unimodal, cross-modal and composite query-candidate combinations without any task-specific training. Multi-scale Matryoshka embeddings do the coarse retrieval; multimodal question answering, token-wise contextual aggregation and in-context learning do the re-ranking.

How it works

Coarse filtering, then fine-grained alignment

Rank a database without ever touching full-scale vectors

Matryoshka embeddings are nested: the first 32 numbers of a 1024-dimensional vector are themselves a usable 32-dimensional embedding. Pyramid Rank exploits that. It starts at the smallest scale, computes a cheap upper bound on what the full-scale similarity could be, and discards every candidate whose ceiling already falls below the threshold.

sim(xqL, xcL)  ≤  Uq,cℓ Uq,cℓ = sim(zqℓ, zcℓ) + √( (1−‖zqℓ‖²)(1−‖zcℓ‖²) )

Both terms on the right are known at level ℓ, so the bound costs nothing beyond the small-vector dot product. Survivors move up one scale, the threshold tightens by bisection, and the loop repeats until only top-K remain.

  • Admissiblea pruned candidate can never have been relevant
  • Convergent⌈log₂(w₀/ε)⌉ steps, independent of database size
  • Scales32, 64, 128, 256, 512, 1024 (Qwen3-MRL)
Pyramid Rank diagram: at each Matryoshka level the embedding grows and more database images are cut, compared to naive full-scale search at the bottom.
Low-similarity candidates are cut while vectors are still short, so the expensive full-scale comparisons are only ever made on a handful of survivors.

Results

Zero-shot throughout, compared against supervised systems

+37%over the best zero-shot method on InfoSeek
+36%over the best zero-shot method on WebQA
23datasets where it beats supervised methods
55.7R@1 on MSRVTT-1kA, text to video
best overall MULTI-SCORE, zero-shot Metric per column as in the paper

Scroll the table sideways to see all 16 M-BEIR columns.

Qualitative comparison of top-3 retrieved results for a composed image-plus-text query and a text-to-video query, comparing MULTI-SCORE, naive MRL and LamRA, with retrieval times below each column.
Right answer, first rank, less time. For the composed query, LamRA misreads the scene context and returns a cafe with the wrong wall colour; naive MRL returns visually close but imprecise videos at three times the latency. MULTI-SCORE grounds the textual edit inside the visual context and returns the target at rank 1 in 0.11 s.

What we learned

Use the arrows, or the left and right keys

Cite this work

ACL Anthology 2026.acl-long.930, pages 20304 to 20324

BibTeX 
@inproceedings{saha2026zero,
  title={Zero-Shot Multimodal Retrieval with Multi-Scale Contextual Representations},
  author={Saha, Sourajit and Gokhale, Tejas},
  booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={20304--20324},
  year={2026}
}