In the age of artificial intelligence (AI), complex photo editing can be done with the click of a button. Whether you want to remove an object from the background, change the color of your shirt or make your face appear flawless, AI can transform your pictures with simple text-to-image commands.
Editing videos with those same AI commands isn’t quite as simple, however, UCF researchers in the College of Engineering and Computer Science aim to change that.
Professor Nazanin Rahnavard, Associate Professor Chen Chen, and UCF alumni Nazmul Karim ’20MS ’23PhD and Umar Khalid ’20MS ’23PhD have developed novel text-to-video AI technology that can dramatically change videos in minutes.
“Our system takes the ‘brain’ of an AI that’s already skilled at generating images from text and adapts it for video, without losing the creative power that makes it effective in the first place,” Rahnavard says. “Our breakthrough came from recognizing a fundamental inefficiency in existing text-to-video editing approaches. Current systems either require massive text-to-video datasets for training or rely on computationally expensive, per-video adaptations of text-to-image models. We believed there had to be a more efficient and elegant solution.”
With Rahnavard’s background in electrical engineering and Chen’s background in computer science, the team used linear algebra techniques to examine the numerical parameters of an AI model that are optimized and adjusted while learning a new task. They realized that instead of fine-tuning the entire parameter set, they could update only the singular values, preserving the AI model’s ability to generalize while speeding up its adaptation time.
“The key was learning which parts of the AI’s ‘memory’ to adjust, and which to preserve,” Rahnavard says. “By focusing only on the most essential elements and leaving the rest untouched, we created a method that adapts much faster and more efficiently while still producing high-quality, expressive results.”
The AI model works best on existing video clips and can edit them in minutes. It can change the colors of clothing, swap a cat for a dog or transform the clip into a cartoon. The more complex the commands, the longer the editing time. But Rahnavard says the process can still be completed in minutes, not hours.
The university was recently awarded a patent for the technology, which movie studios and social media companies could use.
“This technology has the potential to revolutionize video editing across a wide range of industries,” Rahnavard says. “Movie studios could use it for rapid scene modifications without the need for costly reshoots, while social media platforms could offer their users instant, highly sophisticated video filters far beyond what’s available today.”
