Traditional academic silos mean little to Phillip Baldwin, an associate professor in the Department of Theatre Arts in the College of Arts & Sciences who guides his students toward innovative thinking that merges science, technology, engineering and mathematics with music, film, theater and art.
“What I see in a lot of STEM students is this thirst to get beyond sheer memorization because the human mind loves a healthy diet of innovation, novelty, curiosity and things that aren’t quite in the scientific method,” Baldwin said.
Some call it “STEM to STEAM” – adding an “A” for art to the STEM curriculum – but Baldwin sees no need to label the process.
In his classes, students are learning coding and virtual reality coding in conjunction with the latest high-tech gadgets. These include XboxKinect, which interprets in real time players’ motions and speech and translates them into movement expressions and movement therapy that can manipulate music to speed up, slow down, rise or drop in volume with both the video and the sound.
Other devices are HTC Vive, which enables individuals to experience room scale — where you can move around as opposed to the ‘bar stool’ version of merely turning — virtual reality, and Leap, which can track movement of an individual’s hands and fingers and manipulate a 3-D object on display.
“Professor Baldwin is giving me a real chance to not only work on coding, but virtual reality coding, which is on the path to becoming the future in technology and job opportunities,” said computer science major Richelle Chen ’19.
A professor of design and digital media who specializes in telecommunications technology and pedagogy with institutional architectural and urban projects, Baldwin also maintains a design practice in New York City, where he has worked on a slew of theater, interactive installations, opera, music videos, film, interior and graphic design, television projects and off-Broadway productions.
“I’m a trained designer like my father. I have been doing it since the age of three,” Baldwin said.
In addition, Baldwin has spent considerable time during the past 10 years in Korea designing interactive and immersive environments for festivals and construction companies. “I got a $700,000 grant from the Minister of Korea as a co-principal investigator to increase creativity in the school curriculum. The current system of STEM [in Korea] can crush the creative life out of young students,” Baldwin said.
At a recent exhibit at the Charles B. Wang Center, Baldwin and his students presented some of their class projects.
In one project, students made a slide show of family photos and clips of music videos using A-Frame, a coding structure to project those photos into virtual reality spaces.
During the exhibit, Rachelle connected two live subjects to a brain computer interface to detect muscle and emotional activity. Every time a subject’s emotion changed or a muscle moved, a wave with a color corresponding to the color of the sine wave of the corresponding emotion responded.
One practical real-world application of these technologies is to make theater more immersive, such as in virtual reality, where a user feels as if he or she is right in the middle of a play, story line, and dance or music performance.
“Players can participate in live theater groups outside of the expressions, drama and the games, but to actually manipulate the plots and have a fully immersive experience in their own homes is the main allure of virtual reality technologies,” said Richelle. “The idea is to live the lives of the characters and change the plots so people can participate with a combination of the virtual reality and the EEG sensor.”
By using a neurosensor and attaching nodes to a student’s ears and forehead, facilitators can read the emotions of someone watching a play or listening to music. The neurosensor can measure meditative and attention levels and even pick up eye movements.
“We can gauge their attention and learn if they are really enjoying the play or want to recommend it to their friends,” says theater major Hao Lu ‘17.
Applications also hold promise outside the realm of the arts. Baldwin recently invited a yoga and tai chi instructor from Stony Brook to test a program that analyzes movements. The goal: to create a program for the blind that uses sound to alert the user if yoga poses are executed correctly.
“A lot of pre-med students have asked me to teach a biomechanical engineering lab because we use these things,” Baldwin said, holding up a pair of neurosensors. “We’re pushing paper, reading books and checking boxes in regular classes. What if we deal with STEM from a designer’s point of view and make a hands-on point of sharpening our STEM students’ intuition?”
— Glenn Jochum
Add comment