Gemini Librarian Project, Part 1

Building a better UI for organizing video game soundtrack files.

As a geek who grew up in the 80s and 90s, I played a lot of video games. Probably too much! But it became a part of who I am today.

When I’m focusing on work, like most people I enjoy having some music playing in the background. It helps me establish a positive energy and greatly enhances my productivity having some good tunes to jam to. While I listen to plenty of traditional music genres, mainly classic and alternative rock and occasionally some hip-hop of the mid 90s, I frequently find myself looking for something to just have on the background to just experience the miracle of composed music.

Example of one of my favorite themes from the 8-bit console generation, composed by Yasuaki Fujita for release in 1990.

Before the turn of the century, when recorded music such as MP3s became the norm for video game soundtracks, sequenced music was how they managed to add so much variety and flavor to classic games. The cartridge media of the 1980s had music by directly interfacing with a sound chip. The composers would instruct your game console specifically what it wanted the square-wave channel to sound like and for how long, and similar instructions to triangle-wave channels and noise channels and so on.

Working in such a restrictive palette really forced the artists of the time to understand the platform and what it was capable of. While more contemporary games do have some amazing orchestral music, it just isn’t the same as what was made in my childhood and adolescence. There was a sense of intimacy between the composer and the hardware that came across to the player in the finished product.

Flash forward to much later and, like many other things, it turns out I’m not alone! There are entire communities dedicated to recreating these soundtracks accurately and developing tools to archive them. Rather than storing them as waveforms with either lossy compression or extraordinary file sizes, several standards have been created to more accurately emulate the sound chips of old video game hardware.

Unfortunately in the earlier days of the internet, this effort wasn’t quite as organized as in a more modern standard. There are multiple file formats, some specific to different platforms, that aren’t compatible with one another. Getting the files to play in a media player such as WinAmp or Foobar2000 isn’t particularly difficult. There are plugins available that handle decoding the input and recreating it on your speakers. Metadata is also a bit scattered, there are multiple tags attached to these files depending on the platform of origin.

Foobar2000 displaying a library of Super Nintendo soundtrack files stored in RSN archives of the SPC format.

These more generalized media players get the job done, but they’re a bit unfocused for my purposes. They “get the job done” but I’ve never been able to quickly and efficiently organize my library the way I want it. Getting the software just the way you want it requires too much investment from the listener. When I want to hear the soundtrack to Super Turrican, I just want to find it and hit play! While this is possible to do with extensible software like Foobar2000, you have to learn how to organize the layout panels and configure them precisely to your liking.

Initial layout prototype for my software: Gemini Librarian

My vision is to allow a user to point the Librarian to the directory containing their soundtracks. Using the various metadata tags embedded in them, it will sort them by console of origin and game title. Such features are possible in Foobar2000 by creating multiple playlists, but ultimately require too much work from my perspective.

Some opportunity that I will expand into once I have the base functionality in order are more generalized audio player features. The listener should be able to quickly create playlists such as “High Energy Workout” or “Relaxing Wind Down” or “Music for Studying.” A simple 5-star rating system will let them easily shuffle through a giant library and tag songs they’re encountering for the first time that they want to revisit when they like them.

I’m excited to work on this project for not only my own use, but also the use of others I’ve talked to who have similar frustrations with existing solutions.

Masterson, the librarian at Oxford in Gemini Sector, the setting for 1993 classic Wing Commander Privateer, published by Origin System. The namesake for my project.

Development of this software can be followed and reviewed on my Github account at https://github.com/mcgallag/Gemini-Librarian

Expanding the Wing Commander Memory Reader – VGA Capture

Demystifying how VGA programming worked and working with its binary data.

A follow up to the previous post.

Weapon status (left) and Navigation data (right) displayed on the in-game VDUs of a Broadsword fighter-bomber.

Another idea was proposed in the CIC Discord after I developed the prototype. An ongoing feature of the Wing Commander series are in-cockpit VDUs (Visual Display Units.) Essentially, they are small computer monitors inside your in-game cockpit that are used to give vital information to help you navigate the game and quickly access options while in combat. They might contain a systems damage readout, information on an enemy target, navigational data, or a short animation of your wing man’s communication to you in the form of a talking head.

Your wing man in the first mission of WC2, Shadow, tells you that enemy fighters have been sighted.

Obviously such a thing would be a very cool feature for my previously mentioned external cockpit! I knew that the Arduino I used in my previous blog entry was capable of driving a color LCD so I began exploring how to extract such information from the game. The games run in DOSBox, which emulates the VGA graphics standard that was common in MS-DOS games of the era of their release. Though I didn’t have much experience with this exact kind of programming, I knew the basics of how it worked from a high level: by writing pixel color data to a reserved portion of memory, the graphics card would read this exact data back to display to the user’s screen.

Things were a lot simpler back then. The CPU would do all the work of rendering a bitmap and then write it to this reserved memory. Though different graphics cards would offer various features, the basics were the same throughout. I knew that DOSBox must provide a similar buffer of memory for its emulation. After all, the games themselves expect there to be a buffer of memory to write to, so it must be there somewhere!

I began by doing some research on how VGA worked so I could find the data that I needed to do this. I found a wonderful website maintained by David Brackeen called 256-Color VGA Programming in C that explained so many things in a wonderful mix of technical and abstract explanation! I genuinely would not have been able to complete this step of my project without his information.

Viewing the VGA buffer at memory address 0xA000 in DOSBox’s debugger.

I utilized DOSBox’s debugger to view the VGA buffer (known to be at the emulated memory address of 0xA000) on an in-game screen that I knew would remain static and not change. I then was able to return to this screen in the more common non-debugger version of DOSBox and use Cheat Engine to locate the binary data in the (non-emulated) memory of DOSBox itself!

I was able to narrow this down to a consistent memory location after a few consecutive tests.

The next step was to determine the custom palette that Wing Commander 2 uses. Again, David Brackeen’s site gave me a lot of information on how to program this. Unfortunately, I wasn’t able to easily determine where in memory DOSBox stores this information (although I’m sure it’s there somewhere.. It is a puzzle I am determined to solve!) I did come up with an alternate solution to the problem. I took a screen capture in DOSBox to a PNG file. Because PNG stores its palette information in the file itself, I could easily process this file using some convenient .NET classes to generate a reusable palette.

Armed with the memory address of the VGA frame buffer and the palette the game generated, I could now capture video directly from memory! I spent some time building a GUI in Visual C# to not only display these video captures, but also create a more comfortable user interface for connecting to the Arduino.

Live capture of game data and VGA frame buffer from DOSBox.

Obviously the next step would be to connect a color LCD to the Arduino and see how it works. However I haven’t yet acquired one to test. In theory, it should be able to work, though perhaps a bit slowly. The Arduino communicates with the PC over a serial connection, which is fairly slow compared to more modern technology. After doing some Googling, I was able to find that the Arduino should be capable of at least 1 Mbps, or approxmiately 125 kilobytes of data per second.

Each VDU ranges in size, depending on the ship you’re flying, but are at most 75×75 pixels, or 5.625 kilobytes each. If you have two VDUs, we’re looking at 11.25 kilobytes. The frame rate at 1 Mbps would therefore be around 10 frames per second for a live feed in a perfect world. In my research, I saw claims of up to 2 Mbps for the Arduino, but I’d have to get my hands on some hardware and do some more research to find out the capabilities.

I can’t thank David Brackeen enough for keeping this information available on the internet. MS-DOS programming is a bit of a historical artifact at this point but it’s great knowing that there are people out there keeping it alive for the rest of us.

The current source code for the Memory Reader GUI is available on my Github profile. As of this writing, I haven’t yet added the Arduino functionality to it but it will be a quick add via a commit within a day or two.

Weekend Project: Wing Commander External Display

Cobbling together a numerical display and hooking it up to a classic PC game.

In the Wing Commander CIC Discord server the other day, we were having a conversation about those elaborate flight simulator home cockpits that extreme enthusiasts of that genre put together. Things like toggle switches, fuel gauges, LCD panels etc. are not uncommon. Why has nobody ever attempted such a thing in the Wing Commander community before?

Something like this! A bit elaborate for my taste but I love the commitment to the craft!

Wing Commander, a space combat simulator from the early 1990s, would really lend itself to this kind of treatment. The game’s interface takes place in a cockpit of a space fighter in the year 2654. Obviously one reason something like this hasn’t been attempted is the year of release. Though there’s a thriving community, early 90s MS-DOS software isn’t necessarily the easiest thing to pick apart and wrap in modern hardware.

Example of an in-game cockpit. Numerical gauges are highlighted in yellow, Boolean on/off indicators are highlighted in magenta, and graduated gauges are highlighted in red.

Some of the community members expressed interest in designing custom controls and hardware, so I took it upon myself to figure out what exactly it would entail. There are a ton of peripherals that were made (mostly by Logitech/Saitek) specifically for use with MS Flight Simulator. These great enclosed panels with things like analog fuel gauges, toggle switches, control knobs, numerical readouts and so on. Unfortunately they are both expensive and not well-supported by the manufacturer. No developer information or APIs could be found at their website and preliminary research suggested they can only be used with Flight Simulator without extensive software modding.

Inspired by some efforts done by players in the Kerbal Space Program community, I decided to use an Arduino as a controller. It allows simple serial communication and is inexpensive and very extensible. I also happened to have an Arduino project starter kit that had been sitting on a shelf untouched for months waiting for the day to come.

Inspiration in the form of a control panel for Kerbal Space Program designed by KSP community member “Sputnix”

I needed to learn a bunch about how Arduino works. I’ve never used one before this project and the platform was largely unknown to me. The sketches are written in what I believe is C/C++ and uploaded via USB to the controller which runs independently on the board. I also did a fair amount of tutorials and research on basic circuit theory and components, a field I’ve touched on but is largely outside of my wheelhouse.

Eventually I got a sketch written that reads a single value from the serial connection to my desktop PC via USB and displays the number on a 4-digit LED display.

Functional prototype of the Arduino displaying a value sent over USB from host PC.

I had previous experience pulling data from Wing Commander in a project that I will write a blog about some day so the host PC’s side of the project was relatively easy to breeze through. I utilized Cheat Engine, a program that streamlines the process of searching a process’s memory for data and isolating the memory addresses for repeated use.

Some parts of this didn’t go as well as I would have liked. I wanted to be able to pull both the “set” speed of the player’s ship as well as the “actual” speed. Wing Commander is a 3-dimensional simulator with a realistic physics simulator. The player is able to set their ship’s throttle to a desired speed, say 400kps.

Unconfirmed speculation follows: the game likely applies this by scaling a normalized 3D velocity vector to the desired speed. So as the player rotates the ship’s orientation with the joystick, it generates a new acceleration vector for the x/y/z velocity. As it attempts to correct the velocity to compensate for the movement, the actual velocity of the player decreases as the rotation stabilizes and the velocity pulls into a straight line in one direction.

For a much better and more thorough explanation of this type of system, try Daniel Shiffman’s series on autonomous agents and steering behavior on YouTube. Regardless, it isn’t as simple as just pulling a single byte from memory and requires more information about the game’s architecture. Perhaps some persistence in the future or help from the community could help mine the data further.

However, the set player speed is stored as a single byte in the game’s memory and is very easy to pull once you know where it is. I was able to throw together a command line interface that continuously polls the game’s memory for velocity and send the data to the Arduino for display!

A quick demo I recorded of me playing a bit of the game. The webcam in the bottom left corner shows the display being updated in real time.

Overall I’m very proud of this little thing! I’m already considering some more options to explore. Plans for the future include:

  • Use an integrated circuit to drive the display to free up pins on the Arduino.
  • Implement some of the other elements of the in-game cockpit. How cool would it be to have a display that shows your afterburner fuel level, or a warning light that flashes when you should eject and avoid a game over, or indicators of your shield and armor levels? Some of these are more easily doable than others, but I feel like it’s possible!
  • Are inputs possible? Could I press a button on a housing and engage the Autopilot in the game? Maybe!

The source code for the Arduino sketch as well as the CLI program are available on my Github account. Please feel free to contact me via email or on the CIC’s Discord if you have any ideas, suggestions, or want to collaborate!

Case Study in Binary Data: The Vocabulary of Daventry

I wax nostalgic about old adventure games and then start ripping them apart byte by byte.

Rumpelstiltskin. Rumplestiltskin? Nikstlitslepmur?

That guy!

The AGI engine was developed by Sierra On-Line in 1982 for the initial release of King’s Quest for the IBM PCjr. In a move that was brilliantly innovative for the time, Sierra didn’t just write the game for IBM’s new platform, but rather wrote an engine that compiled the game into a form that a generic interpreter could then play back for the end user.

This design allowed the company to easily release their games to multiple platforms. By developing a reusable engine and porting the interpreter to the majority of personal computing platforms, they were able to focus on narrative and game design within their engine and easily release the games to their fans. Ultimately 14 different games were released for eight competing platforms between 1985 and 1989 before technology demanded a more fulfilling platform for contemporary hardware.

These games were a fundamental part of my childhood and it turns out I was not alone! Flash forward about 30 years and the internet has long since reverse engineered the platform and torn it apart. As an exercise for myself, I started reading how the thing works to see how it was implemented. The assets for the games were compiled into data that is processed by the interpreter to present to the player and respond to their input.

The graphics were a vector graphics format, with bytecodes explaining how to draw the primitives and the interpreter would then render the display by writing to the video buffer. Sounds varied wildly depending on the capabilities of the end platform, from bytecodes explaining the frequency and attenuation for voice channels on the IBM PCjr to straight up MIDI wrappers as the market became more capable. A proprietary scripting language called LOGIC was compiled to bytecode for puzzles and character behavior and other internal game logic.

I decided to start by exploring the games’ vocabulary data, a snack-sized puzzle to solve one Saturday morning. The player UI was almost entirely text based with keyboard arrows for character, a minor step up from the text adventures that paved the way for the graphical adventure genre. Rather than straight ASCII data, the game’s vocabulary is converted through a very simple encryption and a form of compression that is almost useless for actually saving file size. I speculate that it was used more to obfuscate the vocabulary in an effort to prevent the player from cheating.

The first 26 2-byte words of the file are essentially indexing the offsets at which you can seek to, finding the vocabulary words that start with a particular letter. The entire file’s vocabulary is in alphabetical order, so this must have been a timesaving measure for the slower I/O speeds of the era.

Once at a particular letter’s offset, there is some very odd encoding of the ASCII data. Each byte is interpreted as follows:

  • The first byte of every vocabulary word is actually an unsigned integer, telling the interpreter how many of the previous vocabulary word’s letters are repeated in the first word. (That is, if the word is “battle” and the previous word is “ball” it starts with the byte 2 for repetition of the first two letters of the previous word.)
  • Every following letter of each vocabulary word is initially the ASCII encoding XOR by 0x7F.
  • The last letter of each word is offset by 0x80, marking the end of each word.
  • Lastly there is a two-byte integer that is an internal “word number” that is used by the game’s logic scripting to translate words to logical in-game actions. Words that share the same word number are considered synonyms that refer to the same thing.

Armed with this knowledge, I threw together a short Python script that implements the algorithm to build the in-game vocabulary. The code for the script is located on a public repository on my Github account along with the vocabularies from Sierra On-Line classics King’s Quest 1, Space Quest 1, and Police Quest 1.

That troublesome gnome’s name by the way?

#videogames

Some General Advice for Learning

Recently I’ve been looking at Stack Overflow a lot. It’s a website that focuses on answering questions related to programming. I’ve always bumped into this website without fully engaging it. As you encounter those small problems when you’re programming and the immediate explanation isn’t always obvious, the natural instinct for a 21st century being is “Google it!”

Chances are, you’re not the first person to have this problem so Stack Overflow very frequently pops up in your results. You skim the listed answers, find one that fits into your brain to help you grok it, implement the solution or workaround and move on. It’s always been a passive thing for me that I’m entirely grateful for.

But people need to write those answers! What better way to help out someone who is struggling with the puzzles in programming? The site doesn’t discriminate, with filters and categories for questions both big and small.

So where do they get their answers from? Experience is most likely, but so is copying another answer they got from someone else, or a copy-paste when they ran into the problem initially, or documentation, or a YouTube video or a Udemy course. One of those should stand out: documentation.

When a new product is released or a new public API is developed or a new protocol is established, documentation holds the keys to understanding. The internet, despite all of its shortcomings, houses one of the most important and remarkable things in modern society: freely accessible and open information. What better way to learn how a thing works than by reading the specifications written by the people that made it! (Aside from exploring the source code, but that requires a different headspace.)

For an engineer, answering a question isn’t just about the answer, it’s understanding how to arrive at that answer. Reading isn’t always easy or interesting and takes longer than a ready-to-eat solution, but it’s okay to reinvent the wheel when you want to learn how the wheel works.

Some helpful references for developers: