Pittsburgh Review

Researchers at Carnegie Mellon University's Robotics Institute (RI) have developed a robotic system that interactively co-paints with people. The Collaborative FRIDA (CoFRIDA) can work with users of any artistic ability, inviting collaboration to create art in the real world.

"It's like the drawing equivalent of a writing prompt," said Jim McCann, an associate RI professor who runs the RI's Textiles Lab. "If you're stuck and you don't know what to do, it can put something on the page for you. It can break the barrier of an empty page. It's a really interesting way of enhancing human creativity."

CoFRIDA builds on past work with FRIDA, a multilab collaboration in the School of Computer Science. Named after the artist Frida Kahlo, FRIDA (Framework and Robotics Initiative for Developing Arts) can use a paintbrush or a Sharpie to create a painting from a human user's text prompts or image examples. The project was founded by Jean Oh, an associate research professor in the RI and head of the Bot Intelligence Group (BIG), jointly with McCann and Ph.D. student Peter Schaldenbrand.

To support a more collaborative artistic creation experience, RI Ph.D. student Gaurav Parmar and Assistant Professor Jun-Yan Zhu joined the FRIDA team to develop CoFRIDA. The new system allows users to provide text inputs to describe what they want to paint. They can also participate in the creation process, taking turns painting directly on the canvas with the robot until they've realized their artistic vision.

"CoFRIDA requires a higher level of intelligence than the original FRIDA, which creates artwork alone from start to completion," Oh said. "Co-painting is analogous to working with another person, constantly needing to guess what they want. CoFRIDA has to understand the human user's high-level goals to make that user's strokes meaningful toward the goal."

Developing data that trains a robot to collaborate is difficult and time-consuming. To address this challenge, CoFRIDA uses self-supervised training data based on FRIDA's stroke simulator and planner. The researchers created a self-supervised fine-tuning dataset by having FRIDA simulate paintings consisting of sequences of brush strokes, from which some strokes could be removed to produce examples of partial paintings.

"We tried to simulate different states of the drawing process," Zhu said. "It's easy to get to the final sketch, but it's quite hard to imagine the intermediate stage of this process."

Using this dataset of partial and complete paintings, researchers fine-tuned a text-to-image model called InstructPix2Pix that enabled CoFRIDA to add brush strokes and work with existing content on the canvas.

Outside the lab, researchers hope CoFRIDA can teach people about robotics and expand creativity, encouraging those who may doubt their artistic abilities.

"If you start from a very simple sketch, CoFRIDA takes the artwork in vastly different directions. If you ask for six different drawings, you'll get six very different options," Schaldenbrand said. "It's nice to be able to make decisions at a high level because it makes me feel like an art director."

Researchers hope further work can integrate personalization into CoFRIDA, giving users even more control over the style of their finished product.

The team's paper titled "CoFRIDA: Self-Supervised Fine-Tuning for Human-Robot Co-Painting" won Best Paper Award on Human Robot Interaction at the 2024 IEEE International Conference on Robotics and Automation (ICRA) in Yokohama, Japan. An accompanying demonstration was also a finalist for Best Demo at ICRA EXPO.

___