Accepted at ACL 2026 · Main Conference, Oral

MULTI-SCORE Zero-Shot Multimodal Retrieval with Multi‑Scale Contextual Representations

Sourajit Saha and Tejas Gokhale

University of Maryland, Baltimore County  ·  {ssaha2, gokhale}@umbc.edu

Paper ACL Anthology Code BibTeX
Pyramid Rank, running illustrative run, 48 candidates
query: a dog in green grass
Candidates left
48
Embedding size
32 / 1024
Similarity ops
0 / 49,152
image video audio kept for Stage 2
12retrieval tasks
32datasets
5.7Mdatabase items
250Kqueries
0training or fine-tuning
6×faster than prior SOTA

Abstract

ACL 2026, Main Conference (Oral)

In multimodal information retrieval (MMIR), candidates relevant to an input query need to be retrieved from a database, where the query and database items span different modalities. As real-world databases evolve, repeatedly annotating and indexing data and re-optimizing domain-specific models across modalities is impractical.

We present MULTI-SCORE, a fine-tuning-free, two-stage MMIR approach that couples efficient candidate filtering with fine-grained multimodal re-ranking. Stage-1 adopts Matryoshka representations to efficiently filter out low-relevance candidates without expensive similarity computations on full-scale representations for the entire database. Stage-2 re-ranks the filtered candidates by computing their fine-grained multimodal contextual representations with two scoring functions for semantic alignment using chain-of-thought prompting and question-answering.

Experiments demonstrate state-of-the-art zero-shot retrieval on 12 MMIR tasks across 32 datasets while outperforming supervised methods on 23 datasets.

  • A zero-shot MMIR system that is efficient and aligns text, image, video and audio in unimodal, cross-modal and composite query-candidate combinations.
  • Pyramid Rank, a filtering algorithm with an admissible similarity upper bound over multi-scale representations.
  • Two re-ranking scores, Bidirectional-CoT embedding and QA relevance, for fine-grained multimodal alignment.
  • Experiments at scale, on 12 tasks and 32 datasets, plus a joint pool of 5.7M items retrieved in one universal search space.
Overview of MULTI-SCORE: a multimodal query is matched against a large database, with examples of text to video, text to audio and image plus text to image retrieval, and a radar chart comparing MULTI-SCORE to prior zero-shot and supervised methods across 12 tasks.
One pipeline, every modality. MULTI-SCORE handles unimodal, cross-modal and composite query-candidate combinations without any task-specific training. Multi-scale Matryoshka embeddings do the coarse retrieval; multimodal question answering, token-wise contextual aggregation and in-context learning do the re-ranking.

How it works

Coarse filtering, then fine-grained alignment

Rank a database without ever touching full-scale vectors

Matryoshka embeddings are nested: the first 32 numbers of a 1024-dimensional vector are themselves a usable 32-dimensional embedding. Pyramid Rank exploits that. It starts at the smallest scale, computes a cheap upper bound on what the full-scale similarity could be, and discards every candidate whose ceiling already falls below the threshold.

sim(xqL, xcL)  ≤  Uq,cℓ Uq,cℓ = sim(zqℓ, zcℓ) + √( (1−‖zqℓ‖²)(1−‖zcℓ‖²) )

Both terms on the right are known at level ℓ, so the bound costs nothing beyond the small-vector dot product. Survivors move up one scale, the threshold tightens by bisection, and the loop repeats until only top-K remain.

  • Admissiblea pruned candidate can never have been relevant
  • Convergent⌈log₂(w₀/ε)⌉ steps, independent of database size
  • Scales32, 64, 128, 256, 512, 1024 (Qwen3-MRL)
Pyramid Rank diagram: at each Matryoshka level the embedding grows and more database images are cut, compared to naive full-scale search at the bottom.
Low-similarity candidates are cut while vectors are still short, so the expensive full-scale comparisons are only ever made on a handful of survivors.

Results

Zero-shot throughout, compared against supervised systems

+37%over the best zero-shot method on InfoSeek
+36%over the best zero-shot method on WebQA
23datasets where it beats supervised methods
55.7R@1 on MSRVTT-1kA, text to video
best overall MULTI-SCORE, zero-shot Metric per column as in the paper

Scroll the table sideways to see all 16 M-BEIR columns.

Qualitative comparison of top-3 retrieved results for a composed image-plus-text query and a text-to-video query, comparing MULTI-SCORE, naive MRL and LamRA, with retrieval times below each column.
Right answer, first rank, less time. For the composed query, LamRA misreads the scene context and returns a cafe with the wrong wall colour; naive MRL returns visually close but imprecise videos at three times the latency. MULTI-SCORE grounds the textual edit inside the visual context and returns the target at rank 1 in 0.11 s.

What we learned

Use the arrows, or the left and right keys

Cite this work

ACL Anthology 2026.acl-long.930, pages 20304 to 20324

BibTeX 
@inproceedings{saha2026zero,
  title={Zero-Shot Multimodal Retrieval with Multi-Scale Contextual Representations},
  author={Saha, Sourajit and Gokhale, Tejas},
  booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  pages={20304--20324},
  year={2026}
}

Acknowledgments

This work was funded in part by the Defense Advanced Research Projects Agency's (DARPA) SciFy program under agreement number HR00112520301. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views, opinions, and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either express or implied, of employers, funding agencies, or governments. We acknowledge high performance computing support from UMBC HPCF and a Lambda Inc. award to SS. We thank Reno Kriz for initial discussions on training-free retrieval and Frank Ferraro for feedback on the manuscript.