Shimon and the Shimis (which is a great band name, by the way) are showcased in the above video, in which they jam along with one of their creators, PhD student Mason Bretan. He gave them an arrangement of what he'd be playing and recorded some tracks and cues for them, but in between (when you hear funky electronic noises) they're doing their own thing based on his chord progressions. And the mallet solo in the middle is completely robot-improvised.
When Bretan joined the lab five years ago, Shimon was being taught how to compose jazz music on the fly based on music theory. "I jumped right in," Bretan said. "And with Shimi — which don't just generate music, we call them 'robotic musical companions' because you can talk to them and use them to interact with your playlist — with Shimi I've been there from the start."
"I'm always trying something new with the robots, and sometimes they surprise me with something that's sort of out there or pretty cool," he added.
His dissertation, which he hopes to turn in by the end of 2015, centers around teaching the robots to understand their physical constraints and abilities.
"So the goal is that if you gave the same input to a robot with 20 arms, it would perform differently than an eight-armed robot because it would be optimizing its performance," he said."Combined with the new algorithm we have for jazz music improvisation, these skills really allow them to more optimally achieve musical goals."
And while he certainly doesn't want to replace human musicians like himself with robots, he's excited about the mechanical abilities they have that we don't.
"I mean, Shimon already has four arms and can hold eight mallets," he said, "So it can already do something a person can't."