Play Rhythm Games using any MIDI Instrument

Play Rhythm Games using any MIDI Instrument

Egyokeo.com Home

Check it out here in this video, I'll get into the details below...

direct link: http://www.youtube.com/watch?v=AR_fdzNrVP8

The Components

Since I injured my middle left finger playing Guitar Hero 2 way too much when it came out, I've been dying to get back to playing it. But my finger hasn't healed. I was holding the neck too tightly on the X-plorer guitar controller and it hurts to bend it anymore. I've never had an injury playing the drums, so I thought "wouldn't it be great to be able to play Guitar Hero on the drums?" So I thought about how that might be accomplished... researched, implemented, borrowed, and here I outline the finished product.

Here's the whole chain of what's going on:

Me banging on my drumKat MIDI drum pads
drumKat MIDI Out to MIDI/USB adapter to PC
PC running my own custom MIDI Hero software
MIDI Hero calls into XIM which sends input to the Xbox 360 console

1: Me banging on my drumKat MIDI drum pads

I needed to solve the basic problem: decide which pads should simulate the 5 fret buttons in the game. Also, how am I going to play 3-note chords with only two hands?

The answers... The drumKat has 10 pads total (plus it is expandable to up to 9 additional drum triggers). The way the pads are laid out, there are 6 lower pads and 4 upper pads. I found a reasonable mapping of the 6 lower pads to the 5 fret buttons. The middle/yellow fret button got two pads assigned to it, which allowed me to have some freedom in my sticking patterns. Then I mapped the 4 possible combinations of "adjacent 2-note chords" to each of the 4 remaining upper pads. Now I can play any 3-note chord with 2 sticks, except for the G-Y-O chord, which I've never seen in any Guitar Hero song, so it's not really needed. UPDATE: G-Y-O is all over the place in Guitar Hero 3, so I accomplish that by playing G with my left stick and Y-O with fingers on my right hand. Not all 4-note chords are possible with this approach either, but again, they are rare. And given that I can have additional pads as trigger inputs, I could get creative with extra chord pads if I ever wanted to.

I actually do use one of my 9 trigger inputs for a bass drum pedal to deploy Star Power. I also use a variable-depth hi-hat pedal for the Whammy Bar and that has its own input on the drumKat.

Here's a graphical layout of the pads/frets: (click to enlarge)

2: drumKat MIDI Out to MIDI/USB adapter to PC

My drumKat has MIDI Out. Some PC sound cards have a way to accept MIDI In. But it was just simplest to go buy the Turtle Beach MIDI/USB adapter. Works perfectly and easily.

3: PC running my own custom MIDI Hero software

I needed a PC in the mix so that I could write the MIDI Hero software to do some special things to make playing Guitar Hero possible on drums:

Auto-hold: It's not a good feeling to a drummer to have to mash a drumstick into a drum and hold it there. So I had to come up with a way to get a good feeling note hold feature. I thought about using a sustain pedal, but a drummer is already used to having all their notes held. This is because when you hit a drum, it goes "doooommmmm". Hit a cymbal, it goes "tsshhhhhhh". So, I implemented this same feel when playing on drum pads: every note is auto-held by the MIDI Hero software, and auto-released just before you hit the next note. There is a buffer in my software that handles the timing.
Chording: I don't really need this, because Guitar Hero has some play in when it recognizes similarly-timed fret button hits to do a chord. But since I had the buffer for auto-hold, I used it to also sync up closely hit notes between my two sticks into a single chord. There is a very tweaked timing used here. I can play a single stroke drum roll at about one hit per 60-70ms and this is also just about the fastest note-run speed in any song in the game. So I'm using 40ms as my chord-detection timing.
Double-taps: When playing the drumKat with hands/fingers, sometimes a double tap can occur when you really only meant to hit the pad once. I use the chording delay to filter those out. It's not necessary when using sticks like I do, but it's in there anyway.
I delay sending all of my hits by 40ms, a tweaked value. It must be constant while playing a timing game like Guitar Hero, and even though all of the above bullet points could be done without a buffer, a buffer is necessary for consistency due to the 35ms per input that the XFPS component of XIM requires (see the XIM section below). I had to use the Guitar Hero Calibration screen to adjust the music by a certain amount so that my drum hits would match precisely with what I hear, not what I see. You might have noticed this in the video -- the notes are visibly activated well past the time they pass the row of fret circles at the bottom of the screen, yet they are timed perfectly to the music. This is what you want in order to feel like you're playing the music: calibration by ear, not by eye. More on this below.

4: MIDI Hero calls into XIM which sends input to the Xbox 360 console

What is XIM? XIM is what makes this all possible! It stands for Xbox Input Machine. This was going to be the most challenging part of my project to get this all working. So I went looking online to see if anybody had done anything remotely like what I needed, which was to be able to have a USB device that simulates Xbox 360 controller input to the Xbox 360 console itself. Then I found OBsIV. He had already done the "hard part" while working on one of his own projects to allow him to play Halo 2 and Halo 3 with Wii Remote and Nunchuk. All I needed to do was ask and hope that he'd share the technology he designed and implemented. As luck would have it, he was planning on releasing a package for people just like me, to make their own XIMs!

I really can't thank OBsIV enough for doing this. Check out this amazing technology here.

Calibration

From my experiments using various controllers with Guitar Hero 2 on the Calibration screen in the game, I noticed that when you calibrate "by ear" (i.e. you strum when you hear the "beep", not when you see the "blink"), that even a wired controller had 70ms lag. Add the 35ms lag from the XFPS component of XIM, plus my MIDI Hero 40ms buffer, and you've got 145ms delay from the time you hit a drum pad to the time that the console recognizes that you hit a note! This is a long time! But as long as it's consistent, always 145ms on every note hit, you just calibrate Guitar Hero to be 145ms and then every note you hit matches precisely with what you hear. Take a listen to the drum pad hits vs. the sound of the music track in the video above. I'm playing right in time with the music. The drawback is that this feature in Guitar Hero 2 was designed to solve HDTV lag, not controller lag. What this means is the notes you see are shifted too far into the future. It takes a little getting used to, and sight-reading a new song becomes a bit more challenging, but for proper feel, it's more important to play in time with what you hear, not what you see.

Update: 2008.04.17

I've updated MIDI Hero to allow playing with a MIDI Keyboard. Theoretically, a MIDI Guitar would also work, but I don't have one to test with. Keep an eye out here in the coming months for a download if you would like to try it. Reminder: you will need to buy or build yourself a XIM, and this only works on Xbox 360.

Here's a video showing it using a MIDI Keyboard...

direct link: http://www.youtube.com/watch?v=HHoayGGByNY

Update: 2008.09.30

I've created my own XIM-like technology called "Hero On Board". I had been waiting for XIM2 to come out, but I missed the initial shipment batch and they're sold out. So I decided to try and implement what I needed in software/firmware myself. It was my first microprocessor programming project, but the ToolStick microprocessor turned out to be relatively easy to program.

Hero On Board replaces my need for XIM and makes using MIDI Hero possible for games that do not support a standard controller. With XIM the Xbox 360 console sees the input as a standard controller. So games like Rock Band, Rock Band 2 and Guitar Hero: World Tour will not let you play guitar, bass or drum parts using XIM. With Hero On Board, you connect the ToolStick microprocessor, flashed with Hero On Board firmware, directly to the board inside your Rock Band guitar or drum controller, and it simulates the button pushes electronically. So the console sees the input as a guitar or drum controller and has no way to tell if it's being played by a human being directly, or through MIDI Hero.

Here's a video showing it with all Rock Band instruments...

direct link: http://www.youtube.com/watch?v=RqaOmaYuHnQ

MIDI Hero Features

I'm keeping this section current with the latest version of MIDI Hero. Here's a screenshot of MIDI Hero (click to enlarge).

Here's a screenshot of the Mode settings window (click to enlarge). You have to configure it depending on which game you're playing, which technology you're using (XIM vs my new Hero On Board firmware), which controller you're simulating, and which kind of MIDI instrument you're playing with. Different configs interpret and simulate Xbox 360 button pushes differently to get the desired effect in-game.

Downloads

Below are the available MIDI Hero packages. Use at your own risk! You agree that I am not responsible for what you do with it or what happens if you use it. MIDI Hero is free for non-commercial use. You may not redistribute any part of MIDI Hero. All copies of MIDI Hero must be downloaded from this website.

The packages only include software and firmware. You'll need some specific hardware before MIDI Hero will do anything for you. If you're going to be using XIM, you'll need to build or buy a XIM and get a copy of the XIMcore.dll from www.xim360.com. See the README.txt file in the downloaded package for details on exactly which version you need. If you're going to use Hero On Board, you can get the ToolStick equipment from www.mouser.com. You'll need a Base Adapter and a C8051F362 Daughter Card.

If there is enough interest in MIDI Hero, I will take the time to write up detailed documentation. But for now, this is only for the most hard-core mod'ers out there who don't mind getting their hands dirty. Message me on my YouTube channel if you are interested in attempting this mod, and I'll help you.

UPDATE: I've gotten some messages from interested mod'ers, but I'm still not convinced! When the first person who shows me proof that they've ordered the ToolStick parts (partial receipt copy/paste, or uploaded photo of received ToolStick, etc) then I'll take the time to document the details. I'm sorry, it's just very time-consuming and I'm not going to do it unless someone is really (really) going to try it.

Thanks

Thanks to "Dave" from the xim360.com forums for helping me test pre-release versions of MIDI Hero to get it ready for public consumption.
Thanks to "OBsIV" for creating and helping me with XIM, and for introducing me to ToolStick microprocessors.
Thanks to my YouTube subscribers, who give me a reason to show off stuff I work on.
Thanks to "Jason", my electronics sensei since high school, for always answering my electronics questions!