Meta’s SAM 2.1 brings complex video editing to Instagram creators

Meta’s Segment Anything Model (SAM) 2.1 has rapidly transitioned from research project to practical application, now powering the innovative Cutouts feature in Instagram’s new Edits app. This technology enables creators to perform sophisticated video editing tasks previously reserved for desktop applications, demonstrating how advanced AI research can evolve into consumer-facing features that empower digital creativity.

The big picture: Meta has successfully deployed its open-source segmentation model SAM 2.1 into Instagram’s Edits app, allowing mobile creators to perform complex video editing through the Cutouts feature.

• The feature was used hundreds of thousands of times within 24 hours of the app’s launch, showing strong user adoption.
• This implementation represents a rapid transition from research to practical application, with less than a year between SAM 2’s research demo and its integration into a consumer product.

How it works: Cutouts uses an object detection pipeline that can automatically suggest objects in video frames or allow manual selection through interactive clicking.

• Once an object is selected, SAM 2.1 predicts a high-quality mask defining the object’s boundary in the selected frame.
• Users can track the object throughout the video, with SAM 2.1 automatically generating consistent masks across all frames.

Key improvements: The engineering team made significant performance enhancements to make the technology viable for mobile use.

• Model throughput was increased by 1.8x, making the feature more responsive.
• End-to-end first frame preview latency was reduced by 3x on NVIDIA H100 GPUs, creating a smoother user experience.

What’s next: Meta is already developing SAM 3, which will expand capabilities to automatically detect, segment, and track objects in both images and videos.

• The next-generation model will introduce open vocabulary text prompts alongside click prompts, making the technology more intuitive to use.
• This advancement could further democratize sophisticated video editing capabilities for mobile creators.