Use a smartphone as a virtual game controller for any computer game!

This was a project I did during my junior and part of my senior year of high school. I got the idea from a website I found called airconsole.com, which has some multiplayer games that you control using your phone. I figured that I could make my own version of that using emulated keyboard keys and gamepad input, so that it could be used for any game I wanted.
The goal of the project was to make a convenient way to play games with separate game controllers, meaning without having to bring a physical controller with you everywhere, you can simply use your phone and play multiplayer games (no phone apps needed!).

This project was written with Python using its Flask framework, HTML and CSS for the UI webpage (mostly through flask macros), and JS to keep track of the UI and make requests to Flask.

The gamepad emulation uses this project to emulate usb HID devices on mac (on windows I would use vjoy).
The keyboard events are generated with the quartz framework for mac.
The touch screen joystick on the webpage UI uses this JS project virtualjoystick.js.

Note: not shown in the video, is a feature later added to generates a qr code, so that you users can quickly scan that rather than manually type in the ip address.

Also, the gamepad emulation also supports multiple players.

Code Snippets:

Python:

Translating an array of button values to a single number:

The HID emulation requires a struct of values (x, y, z, rx axes and a button value), where the 16 button states of the gamepad are represented as a single short (C type).
Meaning that if button 1 was pressed, the value would be 1; if 2 was pressed, the value is 2; if 1 and 2 are pressed, the value is 3; and if button 3 is pressed, the value is 4, etc. I hadn’t learned about bit strings and how numbers were represented at the time, so I created my own solution. Fresh out of my precalculus class, I thought I could handle this by using mathematical combinations:

def button_translate(buttons):
    """ buttons is a sorted array of button values, representing
    the buttons that are in the pressed state """
    but_sum = 0 # this int represents the button states (the short value)
    for i in range(1, len(buttons) + 1):
        num = buttons[-i] # go through the buttons in descending order of button numbers
        # ncr(n, r) is a function defined earlier in program for combinations
        # get the number of combinations for the next highest number
        base = sum([ncr(num-1, x) for x in range(1, num)] + [1])
        but_sum += base
    return but_sum

Generating qr codes and running flask app:

(flask, socket, qrcode, and PIL are imported from earlier in the program)

if __name__ == '__main__':
    # get the local ip address that the app will be hosted on
    ip = "http://" + socket.gethostbyname(socket.gethostname()) + ":8000"
    # for each player, generate a qr code
    for ind, player in enumerate(players):
        # players is an array of the name of each fooHID device
        qr = qrcode.make(ip + "/{}".format(ind+1))
        width, height = qr.size
        # create a bigger image with the qr code, so that you can
        # number the images by player number
        img = Image.new('RGBA', (width, height + 100), 'white')
        img.paste(qr, (0,0))
        fontsFolder = '/Library/Fonts'
        # use size 32 Arial font
        arialFont = ImageFont.truetype(os.path.join(fontsFolder, 'arial.ttf'), 32)
        draw = ImageDraw.Draw(img)
        draw.text((width/2 - 7, height + 20), str(ind+1), fill='black', font=arialFont)
        img.show()
    app.run(debug=True, port=8000, host='0.0.0.0')

Flask Macros:

Representing a touch joystick with a macro:

Creating a 4 button grid (for A, B, X, Y buttons), using an HTML table:

JS:

Functions for representing button and joystick interactions:

var player = 
pressed = [];
function press(key) {
    if (key != "*") {
        if (!pressed.includes(key)) {
            pressed.push(key);
        }
    }
}

function release(key) {
    if (key != "*") {
        pressed = pressed.filter(k => k != key)
    }
}
function clamp(val, min, max) {
    return val > max ? max : val < min ? min : val;
}
function joystickCheck(joystick) {
    return [clamp(joystick.deltaX(), -63, 63), clamp(joystick.deltaY(), -63, 63)];
}

Sending values to Flask with an HTTP request:

// function to send values
function joysend(buttons, dx, dy, z, rx, player) {
    var reQ = new XMLHttpRequest();
    var data = JSON.stringify({"buttons": buttons, "dx": dx, "dy": dy, "dz": z, "drx": rx, "player": player});
    reQ.open("POST", "/p/", true);
    reQ.setRequestHeader("Content-Type", "application/json");
    reQ.send(data);
}
// check the values 20 times a second, and send them if they're different
var old_pos = [0, 0, 0, 0];
var old_but = [];
setInterval(function() {
    var position = [].concat(joystickCheck(joystick_ljoy), joystickCheck(joystick_rjoy));
    joypadCheck(joystick_dpad, [13, 15, 16, 14]);
    old_but.sort();
    pressed.sort();
    // only sends if the values are different from last time
    if ( ( JSON.stringify(old_pos) != JSON.stringify(position) ) ||
            ( JSON.stringify(old_but) != JSON.stringify(pressed) )
        ) {
        // copy values into the comparison arrays
        old_but = pressed.slice(0);
        old_pos = position.slice(0);
        joysend(pressed, position[0], position[1], position[2], position[3], player);
    }
}, 1 / 20 * 1000);