Controlling a Servo with Voice Commands and Xbox Controller

Using the Google Voice AIY kit and an xbox controller to control a servo motor. At first the board listens for custom voice commands and will not respond to xbox input. After saying "Xbox" it will only listen to xbox input, and pressing B on the controller will revert it to only listening to voice commands.

The servo motor is shaking because I am powering it through the pi rather than supply it it's own power.

Platform: Raspberry Pi 3
Programming Language: C
Other: Google Voice AIY Kit
Xbox 360 Controller and bluetooth adapter
Servo motor