Pittsburgh Review

In a bid to address the growing deficit of skilled welders in the United States, researchers at Carnegie Mellon University have developed a groundbreaking training system that integrates machine learning and extended reality (XR) technologies to revolutionize the way welders are trained.

The project, funded by the Manufacturing Futures Institute (MFI), involved the collaboration of Dina El-Zanfaly, an assistant professor in the School of Design, and Daragh Byrne, an associate teaching professor in the School of Architecture, among others. The team's innovative approach aims to help welders acquire the necessary skills through immersive experiences provided by XR technology.

Sandra DeVincent Wolf, the executive director of MFI, highlighted the significance of the project, stating that it aligns with the institute's mission to advance manufacturing technology, contribute to workforce development, and engage the local community.

The XR welding system developed by the researchers combines virtual reality, augmented reality, and mixed reality to create an immersive training environment for welders. By integrating a welding helmet with a Meta Quest headset and a machine learning model, the system provides real-time feedback and guidance to users during welding sessions.

One of the key features of the system is its ability to use auditory cues to assess welding quality in real time. El-Zanfaly explained that experienced welders can evaluate welds based on sound, noting that "a good welding speed should sound like sizzling bacon, not popcorn."

To enhance the training experience further, the researchers employed TinyML technology for sound detection to recognize errors such as incorrect settings and gun tip distance. This enabled the system to provide visual feedback on errors detected by sounds, improving the overall learning process for welders.

Moreover, the XR system includes features to promote mindfulness practices among trainees, such as breathing exercises to induce relaxation and enhance focus. By tracking breathing patterns and providing prompts for regulating breath, the system helps students improve their task performance in the challenging welding environment.

The success of the project has already been recognized with awards at prestigious conferences, indicating the impact of the research in the field of interactive technologies. The team plans to continue refining the system through further studies and deployments, with the aim of enhancing the overall training experience and skill development for novice welders.

The innovative approach taken by the Carnegie Mellon researchers not only addresses the current shortage of skilled welders but also paves the way for new possibilities in XR-based training for various crafts and skills beyond welding.