Fighting Conventional Wisdom - "Moneyballing" StarCraft II
Fighting Conventional Wisdom - "Moneyballing" StarCraft II
In StarCraft II, conventional wisdom is everywhere. Assumptions that we made early on while initially exploring the game have overstayed their welcome and survived largely uncontested. But before we get into that: Baseball.
I cannot claim to understand baseball that well, or at all really. The "World Series" hasn't quite caught on in Europe yet. The one and only baseball-related memory I have is from a trip to Florida many years ago. I ate Domino's pizza on my hotel bed while watching baseball on TV. I have never felt more American - it was glorious. But regardless of the novelty value that it holds for us non-Americans, the sport went through a rather fascinating transformation when new ways to statistically evaluate players took hold. Methods under the collective name "Moneyball". If you haven't yet seen the movie on the topic, allow me to summarize the concept:
Throw away everything you think you know about Baseball. Eliminate every observation, opinion and bias. "Tabula Rasa" all that. Then proceed to break the game down using cold and raw numbers. It may seem to be a simple idea, but it is really difficult to do accurately. Once done however, it can have huge ramifications on every aspect of the game. When building the game back up using nothing but numbers, it showed that certain player abilities were dramatically undervalued and that certain strategies to win games were largely unused. For example, traditional statistical tools such as “Batting Average” proved to be “relics of a 19th century view of the game and the statistics that were available at the time.” New statistical tools such as on-base percentage and slugging percentage were more accurate indicators of offensive success. Decades of scouting methods and conventional wisdom went right out the window.
The 2002 Oakland Athletics threw caution to the wind and adopted new strategies based on calculations instead of the subjective and often flawed opinions of industry insiders. They bought players that the market undervalued, and sold players that the market overvalued. Through a superior understanding of what leads to winning ball-games, the Athletics managed to field a highly competitive team while only having one third of the player salary budget that many of its competitors had.
Malcolm Gladwell, a distinguished writer who I particularly admire, wrote about all this for The New Yorker back in 2006. In his piece titled "Game Theory", he goes deeper into the problems of evaluating individual performances, especially in team games such as basketball. But he ends with a quote from the book "The Wages of Wins" that goes: "One can both play and watch basketball for a thousand years. But if you do not systematically track what the players do, and then uncover the statistical relationship between these actions and wins, you will never know why teams win and why they lose". In essence, we can continue to draw fancy strategies on a whiteboard, but until we actually do the dirty work of crunching the numbers underneath them we will always get sub-optimal results. Don't get me wrong, good coaches will still develop successful strategies, but they will never be optimal; they will always be clouded by inaccurate conventional wisdom.
What the Athletics did in Baseball not only showed on paper that there were advantages to be gained, but managed to translate those numbers into actual wins on the field. The math checked out. Theory and reality matched. Now imagine if we could do the same in StarCraft II. If we could dispel some commonly held beliefs regarding how the game should be played, if we could establish a better understanding of what precise actions result in winning games, just like what the Athletics did. We would have to begin by throwing all accepted truths under the bus, and then back that bus up for another go just to make sure we got them all.
For example: conventional wisdom in StarCraft II states that Zerg is the reactionary race. Drone as hard as you can until you see the enemy moving out, at which point you build a suitable army to deal with that specific threat. Most people play Zerg with the aspiration of being like water and be absorbing the enemy as fluidly as possible, devouring the enemy aggression and expanding behind it. But does that really have to be the case? Some French guy named Stephano came along in the late 2011 and played Zerg as if it was a sledgehammer instead, using Roaches to blunt-force-trauma his way to victory. Stephano also defied another widely accepted Zerg truth: the need for excessive amounts of workers. His Drone count is seemingly unimportant to him in comparison with more conventional Zergs such as Liquid'Ret, whose high Drone numbers have become a distinguishing feature of his play. Stephano didn't like the way conventional wisdom told him to play Zerg, so he ignored it and has found immense success going in another direction.
Now, there's only one problem with the above description of Stephano's play: it has no numbers. It sounds accurate though, just like it sounds accurate that Ret uses a lot of workers -- it’s common knowledge that he does so. That's what our commentators say almost every time he plays a tournament game. But is it true? In Stephano's case, I don't have any numbers to back up my claim that he plays Zerg like a sledgehammer. It’s merely an observation, a feeling I have regarding his play style. It might be true, there might be numbers available to support that theory, but I don't have them, and until I do, the value of my analysis is sub-optimal. If we truly want to understand how to play StarCraft II and know exactly why Stephano has had such widespread success as of late, then we need to break it down with numbers. The Athletics did away with 150 years of Baseball "knowledge"; they refined existing concepts and threw others out entirely without looking back. I'm betting we could easily do the same with over ten years of StarCraft knowledge.
So, how do we get this rolling? Well, for starters let us take a look at the SC2 replay file. It's really small in size, but it contains every mouse click, every key press, and every camera pan. It is the equivalent of having full motion-capture technology on all the players on the Baseball field. The potential wealth of information hiding within replay files is astounding. Find a way to extract the data hiding within (like programs such as SC2Gears already does) and we are well on our way. Imagine being able to cross-reference a few hundred tournament matches from the previous season and poke around for trends in the data. Not only would we be able to see game trends come and go, but we would be able to study the evolution of a specific player and how his play style has changed overtime. However, the really valuable elements are not player specific; they involve how the game itself is played.
Wouldn't it be interesting to ask questions like: "How relevant is creep-spread in the win-rate of Zergs vs Protoss?"
With the data of a tournament season behind us we could begin to calculate an answer to that question. We might find that Zergs who place 25 or more creep tumors in a game are twice as likely to win than a Zerg who only places 15 or less. If this truly is the case, then comes the difficult part of proving that the spread of creep is a real contributing factor and that the higher win rate is not a byproduct of game length or anything else affecting the match. That will be hard to prove, but if nobody attempts it we will be stuck with pretty drawings on a whiteboard and strategies based on conventional wisdom. The truth will elude us until we work for it. So get to it. Community contributors like Godulous have already begun digging. And remember that when the math gets tricky, we can always turn to our own math major-in-chief, Day[ 9 ].
Written by Joel “Offsajdh” Hakalax
Extra Credit Reading:
- Quantify decision making using BER (Beneficial Engagement Ratio).
I submit to you that APM is an ancient relic from a late ‘90s understanding of StarCraft. Much like “Batting Average” for Baseball. But strangely enough, there are very few other statistical tools available to the StarCraft community to evaluate players. Sure, we have race specific win rates, but that is utterly non-descriptive of how the player plays. So what if player A has a 60% win rate versus Zerg? All we know from that is that he’s slightly better than a coin flip.
How about a way to determine how sharp the decision making of a player is in regards to combat? Let’s define an “engagement” as any battle resulting in the elimination of at least 1000 minerals and/or 300 gas in total. The player who inflicts the most material loss is deemed to be the winner of that engagement. If a player has 4 engagements, 3 of which are in his favor, he is deemed to have a 3:1 Beneficial Engagement Ratio.
Obviously this tool is far from perfect, for example, it’s only taking into account battles that are above a certain threshold but does not distinguish between really large battles and medium ones. The player with a 3:1 BER could still have lost the match after having won 3 medium sized battles and then lost the final really large one. Perhaps there would be a way to factor in total resources lost somehow? I don’t know, but I do feel like we need to start to invent better statistical models of describing the play styles and abilities of our professional StarCraft players.
Do you want to have your voice heard in the esports community? WellPlayed is looking for skilled and motivated SC2 and LoL aficionados to join its staff of volunteer writers! If interested, please send a personal statement (including ideas for content) and a writing sample to [email protected].