Multimodal intelligence / Research
I study how multimodal agents can see, listen, and interact with people in real time.
Explore selected research
I am currently with ByteDance Seed and a Ph.D. candidate at the Institute of Automation, Chinese Academy of Sciences (CASIA) and the University of Chinese Academy of Sciences (UCAS), advised by Prof. Tieniu Tan.
My research focuses on multimodal intelligence. I believe natural, real-time interaction is an important next step for multimodal models. I am particularly interested in voice agents and streaming audio-visual systems that continuously perceive their surroundings and interact with people through full-duplex conversations. Questions that interest me include when an agent should respond, how it should handle interruptions, and how memory and tool use can support sustained interaction.
My previous work spans multimodal model training and evaluation, post-training and reward modeling, and agentic multimodal systems, with 15+ papers as first, co-first, or corresponding author at leading venues. These experiences shape my current interest in building interactive agents that understand people and their environments, respond with appropriate timing, and act reliably.
Citation snapshot: . Latest figures on Google Scholar.
Previously, I have been fortunate to work with Prof. Jingdong Wang at Microsoft Research Asia and Prof. Rong Jin at Alibaba DAMO Academy. I have also interned at ByteDance, Kuaishou, Skywork, and Squirrel AI.
Current focus
Voice agents, streaming audio-visual perception, and full-duplex dialogue. I am interested in how an agent follows a changing environment and handles turn-taking, pauses, and interruptions with appropriate timing.
Tool use, active exploration, and memory for sustained, context-aware interaction. My work includes Thyme, Skywork-R1V4, and Omni-DeepSearch. I am interested in how perception and memory inform what an agent does next.
Learning from preferences and feedback to improve multimodal behavior, through MM-RLHF, R1-Reward, and BaseReward. I am also interested in rubric-based rewards and self-evolving reward systems.
Developing models such as SliME, Keye-VL, and VITA, alongside benchmarks such as MME-RealWorld, MME-Unify, and MME-VideoOCR. I have also contributed to VLMEvalKit and the MME-Survey. I aim to build evaluations that reflect real-world needs and human preferences.
Earlier interests include continual learning, out-of-distribution generalization, time-series forecasting, AI for education, and content moderation.
I also maintain reading notes on multimodal models, LLMs, and diffusion models.
Selected work grouped by research direction. Full publication list on Google Scholar. * Equal contribution. † Corresponding author.
Selected collaborations and first-author work that connect to my current interests.
Preprint, 2026
Personalized concept understanding and evaluation over continuous video streams.
NeurIPS 2025 Spotlight
Real-time vision and speech interaction in an open-source multimodal model.
Preprint, 2025
Concurrent multimodal perception, speech, and action for embodied interaction.
Preprint, 2026
Starting from audio to search for evidence across text, images, and video.
ICLR 2026 First Author
Image processing and computation through executable code for multimodal reasoning.
Technical Report, 2025 First Author
Interleaved visual reasoning and search in agentic multimodal systems.
Preprint, 2024
ICLR 2026 First Author
ICLR 2026 First Author
ICML 2025 First Author
ACM MM 2025 First Author
ICLR 2025 First Author
IEEE TPAMI 2025 First Author
Technical Report, 2025 Main Contributor
ICLR 2026 Co-first Author
CVPR 2026 Corresponding Author
For earlier work on domain generalization, adaptation, and time-series forecasting, see Google Scholar, AdaNPC, and OneNet.
Current affiliation
Research interests in real-time multimodal interaction and agents.
Multimodal Large Language Models
Agentic Multimodal Systems
Research Intern
LLMs for Education
Advised by Prof. Rong Jin
Advised by Prof. Jingdong Wang
ML/AI: ICML (2022–2026), NeurIPS (2022–2026), ICLR (2023–2026), AISTATS (2025), AAAI (2023–2024)
Vision: CVPR (2022–2024), ICCV (2023, 2025), ECCV (2022, 2024)
NLP: ACL (2025), EMNLP (2023–2024), NAACL (2024)
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
IEEE Transactions on Image Processing (TIP)
International Journal of Computer Vision (IJCV)
Transactions on Machine Learning Research (TMLR)
IEEE Transactions on Information Forensics & Security (T-IFS)
PC Member for MILETS@PAKDD’23, DMLR@ICML’23