Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control

Zhida Zhang¹, Jie Ma², Zhan Peng³, Yang Han², Haoxue Wu², Jun Liang², Jie Cao¹, Jing Li²

¹ New Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences    ² Youku Moku-Lab    ³ Huazhong University of Science and Technology

       

teaser.mp4

📝 Abstract

The narrative quality of a video fundamentally determines its perceptual value. Although existing video generation methods can produce visually appealing content, they predominantly rely on sparse conditioning signals such as text prompts or first/last frames, which limits precise control over narrative structure and temporal pacing. In this paper, we propose SmartDirector, a framework that enhances the narrative capacity of video generation models through multiple keyframes. SmartDirector supports flexible generation scenarios including single-shot generation, multi-shot narrative synthesis, and video extension. The framework operates in two stages: Director-Gen generates a low-resolution video conditioned on the provided keyframes, and Director-SR refines the output by exploiting high-resolution keyframes as semantic anchors to recover fine-grained details. To enable robust multi-keyframe training, we construct a data pipeline that curates single-shot and multi-shot sequences from movies. Extensive experiments demonstrate that SmartDirector substantially outperforms existing state-of-the-art approaches.

🖼️ Framework

🔥 Updates

• [2026.05.28]: Release the project page.
• [2026.05.28]: Paper released on arXiv.

📅 Open Source Plan

The following content of this project will be open-sourced at the appropriate time:

⏳ Coming Soon

• 💾 Code & Dataset: The release of this code and dataset are subject to a corporate compliance and security review.

💭 ETHICAL CONSIDERATIONS

The insertion condition images and videos used in these examples are sourced from publicly available channels or generated by models, and are intended solely to demonstrate the capabilities of this research. If there are any concerns, please contact us (wuhaoxue.whx@alibaba-inc.com) and we will remove the relevant examples in time.

🌟 Citation

Please leave us a star 🌟 and cite our repo if you find our work helpful.

@article{zhang2026smartdirector,
  title     = {SmartDirector: Keyframe-Conditioned Cinematic Video
               Generation with Narrative Pacing Control},
  author    = {Zhida, Zhang and Jie, Ma and Zhan, Peng and Haoxue, Wu
               and Yang, Han and Jun, Liang and Jie, Cao and Jing, Li},
  journal   = {arXiv preprint arXiv:2605.27891},
  year      = {2026}
}