Final Year Project Blog

If you follow developments in the AI field closely, you've probably already seen the Dota 2 match between Open AI bot and Dendi. In case you haven't, check it out here.

Well, if I was going to pull off anything even close to this crazy, I'd need to find out what I'm up against first, so I started going through some of the literature on the topic, this is pretty much the very least you should be familiar with if you want to do something like this yourself.

Now to the deep end, literally. Q-learning is fairly intutive, I was able to pick it up pretty fast and so should you. The basic concept is, we give our "agent" some reward for certain special states and ask it to evaluate a "policy" to maximize its score. Now I know this all sounds super techincal but it's really not. It's literally just the Markov equation evaluated with a max function from any standard library. This kind of agent can solve a simple maze of a few hundred to few thousand grids easily. Quick sidenote, if you're wondering what Dynammic Programming has to do with any of this or why I linked you to that particular paper, it's because DP is father of a technique called memoization. When the agent is working it's way through the maze, you'll to memoize and update the Q values in a table the size of the maze.

Deep Q-Learning is where the challenge really begins. For this you're gonna need to know what Artificial and Convolutional Neural Networks are. At this point I'd link a ton of resources to learn about them for your reference but I'm sure you don't need my help finding tutorials on the biggest buzzword of decade. Here's just one quick 4 part video series that I thought was brilliant.

Thanks to the environments, all I need is a program that can make decisions on which action to play given some video input. These actions are typically indexed by numbers, for example, 0 might be to press the W key and move forward and so on. Of course, analysing video feed is no simple task for a computer, in fact, until recently this component of the project alone was considered worthy of international awards. For this task, we must make use of a convolutional neural network to process the images (video feed is just images played at several frames per second). Since we’ll be working with multiple games, our architecture must be ready to accommodate multiple convolutional nets that perform image segmentation and recognition, one for each game to be precise.

We’ll also need an LSTM layer or some other recurrent architecture to accommodate time delayed rewards from the system. Now, if you’re wondering what that means, take a look at this paper on Asynchronous Methods for Deep Reinforcement Learning by Volodymyr Mnih (2016)

So, our plan of action is build an deep reinforcement learning system that receives as input the positions of various items of interest in the scene as a vector and outputs the Q values for each action. We then select an action either based on a softmax or an adaptive epsilon greedy strategy. We have a separate convolutional neural network for each game so it’s easier to tag items that look different in different games.

Finally got assigned a computer at the CS lab in Mong Man Wai Building which is great because I can officially start development in a dedicated space. Of course, I’d have to reinstall all the software and libraries onto my new system. I decided to install Ubuntu, a popular Linux distribution on this PC so I could test my code on multiple platforms. Let’s just hope this doesn’t come back to bite me in back. So, I got to work flashing the OS, connecting to the new network and installing the dependencies on my new machine. Most of the process remains the same as detailed above, however, since I’m working on Linux, I can create virtual environments from the command line and Python comes preinstalled, so I didn’t have to install Anaconda.

For some further research, I also read the following papers and recommend strongly that you do the same.
Playing FPS Games With Deep Reinforcement Learning
Playing Atari With Deep Reinforcement Learning

I started to tinker with the ViZDoom engine on Linux, just to get a feel of the environment. I started off simple with a script that just loaded the environment and played by selecting a random action at every step. Below is a screenshot of the clip.

I also had my presentation due on Friday the 19th so I worked on writing my report and my presentation. Preparing the presentation and report helped me reflect on my progress thus far and bolstered my ability to plan my work for the weeks to come.

After writing the scripts to train and test my model, as well as working out kinks with the display to record videos of the training process, I finally came up with a model that performed incredibly well on the basic scenario. Below are videos of the model playing the basic scenario of ZDoom before and after 20 epochs of training.

As is fairly evident by the above videos, the performance of the model increased drastically from epoch 1 to epoch 20.

This week I added two convoluted layers (with max pooling) and one fully connected layer and let it have a go at the defend the centre scenario in Doom. After achieving less than ideal results, I increased the number of epochs and decreased the learning rate, giving the program more time to learn and adapt to the delta movements involved in this new situation. Below is the result of two days of training and tuning. It averaged 9 kills over it’s 10 best episodes.

Week 13 was a busy week so I chose to focus my attention on the presentation and final report rather than the code. As any good software developer will tell you, never cram features in at the last minute. My final report can be found here and includes details on the methodology of my study and implementation of the programs so far.

Just as planned, I’ve written some scripts to capture my screen and to press different keys. You can find the script to capture your desktop screen in real time at about 20 fps here and a script to press the required keys to play CS1.6 here.

The first logical step is to ensure this works with a simple script that presses random keys to make sure the actions are selected appropriately. After this, we should build up the solution reach the same level as the current Doom agent by first using a simple DQN and then a more complicated version.

This was quite a productive week. Below is a simple sketch of the architecture I went with for the system as a whole.

Quickly summarizing, we have 3 separate networks, one convolutional for the input images, be it in Doom or CS1.6 (Doom for now) and two fully connected layers for the measurements and goals. This permits a system of dynamic goals which is especially important because it’s the basis of the logic we humans play with. For example, if you notice your health is low, your goal shifts from just getting kills to increasing your health by collecting health packs. Thus, our priorities shift over time and an agent that is only trying to optimize one goal will never succeed as we want it to in such a complex environment.

I’m writing all the code for the network with Tensorflow (despite the models shown in the papers being tested in PyTorch) because I feel confident with Tensorflow after working with it on a few other projects.

I’m finally done building the main model architecture. I’ve isolated it and its various helper functions into a single file and class so as to keep the structure modular and easy to use. Here’s the file, just to explain some of it in greater detail, I use the tf.concat function to concatenate the outputs of the different networks, the conv2d function creates a convolutional neural network based on the method parameters and the fc_net function does the same for fully connected layers.

The reason I’ve opted with this structure rather than just creates the nets in the agent class is so I don’t have to re-write the code to create the multiple different fully connected networks I intend to create (projects generally just use one network so there’s no need to create a function for this). The lrelu function implements the leaky relu activation function required for introducing some non-linearities. The code is commented profusely so you should be able to understand the rest by going through it.

This was a very busy couple for weeks for me personally, so I wasn’t able to get a lot done. I began the long training process which took two nights. Finally, I had the weights ready in a neat 220-megabyte file (sarcasm).

I tested it out a lot on different scenarios using the two-script setup I developed last week and it seemed to work really well. Much better than expected actually. It averaged about four kills a game against me while I managed nine. Not bad, considering I’ve been playing this game for quite some time now. It also seemed quite robust in the sense that it wasn’t repeatedly doing the same things and (thankfully) wasn’t mindlessly running into walls. Here is a video of it playing against me. I’m going to retrain it with some different parameters because I think this can be even better.

Very busy week with every course’s final project due but I managed to link the model to the CS1.6 convolutional network and test it against the easy bots on aim_headshot. Unfortunately, the multiplayer for CS1.6 is not going to be as straightforward due to the lack of a platform meant for learning.

Since we’re simply grabbing the screen buffer images and processing the video stream from there, we obviously can’t run a bot on the same computer as the host (because the bot has to see the view of the character it controls and the player has to be able to see his character’s view). Thus, I have to make this work over LAN or the Internet but I don’t think I have enough time to do this right away so I’m going to have to leave this for later.