CM² · HiCM²: Memory-Augmented Dense Video Captioning

From a flat memory to a hierarchy

Dense video captioning has to find every event in an untrimmed video and describe each one. Visual features alone carry little semantic content for that. Both papers take a cue from how people describe what they see: by recalling something similar they have seen before.

CVPR 2024

CM²flat memory

Every training caption becomes a memory entry. Each segment of the video retrieves its nearest sentences, which guide both where events are and how to describe them.

One entry per caption; a segment pulls its nearest ones.

Read Chapter I

AAAI 2025

HiCM²hierarchical memory

Captions are clustered level by level and each cluster is summarized by an LLM into one sentence. Recall starts from the abstract top and narrows down.

YouCook2 memory, entries per level (bar length on a log scale).

Read Chapter II

Chapter I · CVPR 2024

CM²

Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval

📄 Paper 💻 Code

Overview

CM² stores the training captions as an external memory. The input video is split into segment-level temporal anchors, and each anchor retrieves its most similar sentences through video-to-text matching. A versatile encoder processes the visual features and the retrieved text with shared weights, and the decoder reads them through separate visual and textual cross-attention to predict event boundaries and captions.

Without pretraining on extra videos, CM² reaches 33.01 CIDEr on ActivityNet Captions, above Vid2Seq, which is pretrained on 15M videos.

CM² concept: like a person recalling relevant memories, the model searches an external memory bank for relevant sentences to describe and localize events in an untrimmed video. — **Recall, then describe.** CM² searches an external memory bank for clues relevant to each part of the video, and uses them to describe and localize events.

See it in action

Method

CM² overall architecture: a frozen CLIP image encoder, memory retrieval with temporal anchors, a versatile transformer encoder, and a decoder with visual and textual cross-attention that outputs start time, end time, and caption. — **Overall architecture.** Video features query the memory; retrieved text and visual features go through a shared encoder, and the decoder attends to both before predicting each event.

Memory read module: temporal anchors are compared with the text features in memory, and the top-k retrieved features are aggregated per anchor. — **Memory read.** Segment-level anchors retrieve and aggregate their nearest text features.

Versatile encoder-decoder: one weight-shared encoder for visual and text features, and a decoder with separate visual and textual cross-attention. — **Versatile encoder-decoder.** One weight-shared encoder for both modalities; separate cross-attention for each.

Results

ActivityNet Captions · CIDEr

Without → with memory retrieval31.24 →33.01+1.77

YouCook2 · CIDEr

Without → with memory retrieval23.67 →31.66+7.99

Event captioning, ActivityNet Captions

Method	Backbone	#PT videos	CIDEr	METEOR	BLEU4	SODA_c
Vid2Seq	CLIP	15M	30.10	8.50	–	5.80
MT	TSN	–	6.10	3.20	0.30	–
ECHR	C3D	–	14.70	7.20	1.82	3.20
UEDVC	C3D	–	–	–	–	5.5
PDVC^†	CLIP	–	29.97	8.06	2.21	5.92
CM² (ours)	CLIP	–	33.01	8.55	2.38	6.18

Event captioning, YouCook2

Method	Backbone	#PT videos	CIDEr	METEOR	BLEU4	SODA_c
Vid2Seq	CLIP	1M	47.10	9.30	–	7.90
MT	TSN	–	9.30	5.00	1.15	–
ECHR	C3D	–	–	3.82	–	–
E2ESG	C3D	–	25.00	3.50	–	–
PDVC^†	CLIP	–	29.69	5.56	1.40	4.92
CM² (ours)	CLIP	–	31.66	6.08	1.63	5.34

Event localization

Method	PT	ActivityNet Captions			YouCook2
Method	PT	F1	Recall	Precision	F1	Recall	Precision
Vid2Seq	✓	53.29	52.70	53.90	27.84	27.90	27.80
PDVC^†	✗	54.78	53.27	56.38	26.81	22.89	32.37
CM² (ours)	✗	55.21	53.71	56.81	28.43	24.76	33.38

Bold and underline mark the best and second best, as in the paper. ^†Reproduced from the official implementation. #PT videos: extra videos used for pretraining. All methods in the localization table use CLIP features.

How much a good memory could add

If retrieval were perfect (querying the memory with the ground-truth captions and placing the results on ground-truth segments), YouCook2 CIDEr would reach 183.95. CM²'s real retrieval is a first step toward that ceiling.

YouCook2 CIDEr. Oracle rows use the retrieved captions directly as output, without the model.

Next chapter II HiCM² From a flat memory to a hierarchy of compact summaries.

Chapter II · AAAI 2025

HiCM²

HiCM²: Hierarchical Compact Memory Modeling for Dense Video Captioning

📄 Paper 💻 Code

Overview

People rarely recall a memory flatly. We sense the gist first, then the kind of thing it was, then the specific episode. HiCM² builds the captioning memory the same way. Training captions are clustered bottom-up into levels, and an LLM summarizes each cluster into one compact sentence, so higher levels hold more abstract knowledge. At inference, retrieval runs top-down: it picks relevant abstract entries first and follows them to finer ones.

On YouCook2 the memory has four levels with 1,758, 313, 68, and 8 entries, built from 10,337 captions. HiCM² reaches 71.84 CIDEr on YouCook2 and 51.29 on ViTT, the best among methods that use pretraining.

HiCM² concept: humans recall from memory at different levels of abstraction; the model retrieves from a hierarchical compact memory in the same way. — **Memory in layers.** Like human recall, the model reaches abstract concepts first and specific episodes last.

See it in action

Method

HiCM² overall architecture: video-to-text retrieval at each hierarchical level, a temporal encoder, speech and time tokens, and a text decoder that outputs start time, end time, and caption. — **Overall architecture.** Retrieval runs at every level of the memory; the retrieved sentences, visual features, and speech are fed to a text decoder that writes each event as start time, end time, and caption.

Hierarchical memory construction: captions are clustered at low, middle, and high levels; an LLM summarizes each cluster; a text encoder stores the summaries as memory. — **Building the memory, bottom-up.** Cluster, summarize each cluster with an LLM, encode, repeat.

Hierarchical memory read: top-k selection at the high level, then hierarchy tracking down to the middle and low levels. — **Reading it, top-down.** Select at the top, then follow the hierarchy down.

Results

YouCook2 · CIDEr

Vid2Seq^† → HiCM²66.29 →71.84+5.55

ViTT · CIDEr

Vid2Seq^† → HiCM²48.84 →51.29+2.45

Event captioning

Method	PT	YouCook2 (val)				ViTT (test)
Method	PT	CIDEr	METEOR	SODA_c	BLEU4	CIDEr	METEOR	SODA_c	BLEU4
PDVC	✗	29.69	5.56	4.92	1.40	–	–	–	–
CM²	✗	31.66	6.08	5.34	1.63	–	–	–	–
Streaming V2S	✓	32.90	7.10	6.00	–	25.2	5.80	10.00	–
DIBS	✓	44.44	7.51	6.39	–	–	–	–	–
Vid2Seq^†	✓	66.29	12.41	9.87	5.64	48.84	9.51	14.99	0.71
HiCM² (ours)	✓	71.84	12.80	10.73	6.11	51.29	9.66	15.07	0.86

Event localization

Method	PT	YouCook2 (val)			ViTT (test)
Method	PT	F1	Recall	Precision	F1	Recall	Precision
PDVC	✗	26.81	22.89	32.37	–	–	–
CM²	✗	28.43	24.76	33.38	–	–	–
Streaming V2S	✓	24.10	–	–	35.40	–	–
DIBS	✓	31.43	26.24	39.81	–	–	–
Vid2Seq^†	✓	31.08	30.38	31.81	46.21	45.89	46.53
HiCM² (ours)	✓	32.51	32.51	32.51	45.98	45.00	47.00

Bold and underline mark the best and second best, as in the paper. PT: pretraining on additional video datasets. ^†Reproduced from the official implementation. All methods use a CLIP backbone.

Summaries beat raw captions

A flat memory of all training captions helps a little. Clustering them into a hierarchy is about the same. Writing each cluster as an LLM summary is what lifts CIDEr to 71.84.

Memory	Hierarchy	CIDEr	METEOR	SODA_c	F1
No memory	✗	66.29	12.41	9.87	31.08
All training captions	✗	67.90	12.49	10.38	32.31
Clustering	✓	67.15	12.97	10.17	32.30
Clustering + LLM	✓	71.84	12.80	10.73	32.51

YouCook2.

Every level pulls its weight

Any single level helps over no memory, and using all three gives the best captions (71.84 CIDEr).

High	Mid	Low	CIDEr	METEOR	SODA_c	F1
			66.29	12.41	9.87	31.08
		✓	67.75	12.33	10.35	32.27
	✓		67.59	12.45	10.37	32.23
✓			67.21	12.21	10.28	31.98
	✓	✓	68.61	12.35	10.51	32.07
✓		✓	69.81	12.63	10.54	33.00
✓	✓		68.05	12.33	10.41	32.87
✓	✓	✓	71.84	12.80	10.73	32.51

YouCook2. High is level 4, Mid covers levels 2 and 3, Low is level 1.

BibTeX

CM² · CVPR 2024

@inproceedings{kim2024you,
  title     = {Do You Remember? Dense Video Captioning with
               Cross-Modal Memory Retrieval},
  author    = {Kim, Minkuk and Kim, Hyeon Bae and Moon, Jinyoung
               and Choi, Jinwoo and Kim, Seong Tae},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer
               Vision and Pattern Recognition (CVPR)},
  pages     = {13894--13904},
  year      = {2024}
}

HiCM² · AAAI 2025

@inproceedings{kim2025hicm2,
  title     = {HiCM$^2$: Hierarchical Compact Memory Modeling for
               Dense Video Captioning},
  author    = {Kim, Minkuk and Kim, Hyeon Bae and Moon, Jinyoung
               and Choi, Jinwoo and Kim, Seong Tae},
  booktitle = {Proceedings of the AAAI Conference on Artificial
               Intelligence (AAAI)},
  year      = {2025}
}