How to fix [Call of Duty's] matchmaking?

Matchmaking is a perennial source of negative player sentiment for Call of Duty, but does it have to be?

It might sound crazy, but there is some debate about whether the negative sentiment is even a bad thing for the franchise, given its financial success. But I'm going to make the bold assumption that it's generally bad for the brand when players say the same bad things about it year over year — even if many of the bad things are inaccurate or contradictory.

So, as someone who wrote COD's MMR algorithm, this is my perspective on how the matchmaking experience could realistically be improved.

Acronyms

• SBMM = Skill-based matchmaking

• MMR = matchmaking-rating; the number that indicates a player's skill for matchmaking purposes

TL;DR: Realistic ways to improve SBMM:

1. Fix MMR whiplash (low-hanging fruit)

2. Hard-Level Labeling in Casual

3. Transparent MMR in Ranked

4. Fix team-balancing / Optimize matchmaking

Is SBMM too perfect?

Two years ago, with Modern Warfare III in the SBMM dunk tank, Halo 3's lead designer tweeted that the problem with modern SBMM was essentially that it was too perfect; that it lacked variety:

The same sentiment has been echoed by players and media as the obvious problem with COD's SBMM. But is that diagnosis correct?

For Halo, probably. In Call of Duty's case, definitely not.

MMR matters

The variability Max was referring to was actually already baked into the behavior of COD's MMR system:

Fig. 1: Simulated Player MMR histories in a system similar to Call of Duty. Note the volatility.

From a Halo 2 & 3 perspective, it's logical to suggest that Matchmaking might be too tight, because, by contrast, Halo's MMR (TrueSkill) was fairly static. Halo needed to inject "variability" at the matchmaker level because the MMR numbers being fed into the matchmaker lacked volatility:

Fig. 2: Simulated Player MMR histories in a system similar to Halo 2 & 3. Note the decreasing volatility over time.

Tight SBMM in Halo would indeed have been a bad experience for players. Not because every match would feel the same per se, but because players would be punished for going off-META or relaxing — because TrueSkill reacts too slowly to recent performance.

(The reasons behind that, the first-order fixes, and the second-order problems with TrueSkill are out of scope for this article. Just know there's a lot more that could be said about it.)

TrueSkill's limited responsiveness seems like an issue the Halo team was aware of, based on an older tweet from Max:

For example, a game using TrueSkill would have to implement an explicit streak-breaking system to give players an easier match after a string of bad matches.

That responsive behavior though (often mistaken for "Engagement-Optimized Matchmaking") was also already an implicit feature of COD's MMR. It adapts quickly to good or bad performances so teams can be immediately rebalanced within persistent lobbies, or so matchmaking can find a better match in the case of disbanding lobbies. Either way, "streak-breaking" doesn't require matchmaker intervention as long as the MMR system is responsive enough.

On paper then, Modern Warfare III already had the features that ideal SBMM should have: for most players, every match is not the same. So what's actually wrong with it?

Spoiler: COD-matchmaking's biggest weakness is not that it's too perfect.

First real matchmaking problem: Whiplash

The first real problem is that the MMR system reacts too much for average players, giving them a feeling best described as whiplash. A system like COD's has too much volatility in the middle, but not enough at the extremes (See Fig. 1).

The whiplash problem actually has an easy solution though: fix the MMR. Good MMR should still adapt, but with controlled responsiveness and uniform volatility across the entire range of skill. The math is hard, but once you have that, it's a drop-in fix.

Once the MMR is appropriately responsive, then any additional variability can be tuned (and AB tested) as controlled noise in the matchmaker. (IMHO, additional noise is probably not required, but it's still worth testing.)

An MMR system like IVK Skill solves the whiplash problem, while preserving the advantages of COD's MMR system:

Simulated Player MMR histories using IVK Skill, configured for Casual matchmaking.

It's also easy to A/B test something like IVK Skill to find the optimal MMR tuning for any matchmaking environment.

So, one problem at least has a straightforward solution.

Second real matchmaking problem: Sweatiness

Sorry to disappoint everyone on Twitter and Reddit, but "sweatiness" cannot be solved with loose-or-no SBMM. There simply isn't a satisfactory one-dimensional solution to this problem, because different groups want mutually-exclusive things. A real solution requires coherent, concerted design effort.

The complexity is because the perception of "sweatiness" depends on the demographic:

• For most players (about 90%), looser SBMM feels sweaty, because it makes them less likely to be the top performer in the match. This is universally counterintuitive to many players, who will blame SBMM for sweaty matches, even though the cause is the opposite. E.g. during low/off-peak CCU, average players will assume SBMM is "cranked up".

• For elite players (top 10%), which includes content creators and influencers, tighter SBMM feels sweaty. In this case, the accusations are correct: any amount of SBMM reduces their chances of being the top performer in a match, and COD's design gives them nothing to compensate. It's logical that these players hate SBMM, yet developers have mostly ignored them.

Loosening SBMM to appease the top 10% will increase churn pressure on the bottom 90%. Even worse, a lot of those 90% will perversely blame SBMM for the increased sweatiness, which pressures designers into a doom spiral of looser SBMM.

Moderate-to-tight SBMM is ultimately best for short-term retention and revenue, but it ignores and burns out the highest-skilled players. IMO it's not indefinitely sustainable to take the most passionate and committed players for granted, even when it's not reflected in shorter-term KPI.

In summary, there is no good solution to SBMM with one-dimensional (tighter/looser) thinking.

How about variety?

Players and designers can all agree on one thing though: they ostensibly want more variety in matchmaking.

Unfortunately, "variety" has opposing definitions, depending on the demographic:

• Most players want more variety of outcomes, so they get a chance to stomp once in a while, which mathematically requires less variety of skill within each match.

• Elite players want more variety of skill within each match, because it gives them more consistent outcomes, so they can pretty much always stomp (even if their team loses due to worse team-balance — which matters much less than personal performance).

So, more "variety" or "a more varied experience" sounds like a win/win at first, but once the definition is clarified, it's just the same intractable sweatiness problem.

The solutions for matchmaking

Looser SBMM by itself is a dead end, but there are two additions that can make it viable:

Reduce the real skill gap

More RNG. More "cheap" ways to get kills. More corner-camping. OP shotguns. Faster TTK. Comeback mechanics. Last stand. Etc.

This is a terrible solution; DON’T actually do this.

Party games with low skill gaps need very little SBMM for everyone to have a good chance at performing well, but that's not really Call of Duty.

Bring in the bots

A lot of players won't like it, but bots are the only real option with looser matchmaking.

If the goal is to feed more fodder to high skill players without increasing churn pressure on the fodder (asymmetrically, disproportionately), then matchmaking has to offset with bots at the low end.

The question is whether below-average players would find this irredeemably offensive, or if it could be done in a way that COD's audience doesn't despise.

Better matchmaking solutions

Hard-level labeling in Casual

It's not flashy, but Candy Crush received profound results from giving players information about the difficulty of their levels:

Hard Level Labeling: One Change, $1b in Revenue.

There isn't an exact analogue in PvP, but the underlying idea is that when players know something is difficult, they're more likely to embrace the challenge.

Getting placed into difficult lobbies should be a source of pride, not frustration. SBMM is necessary, but it takes away the bragging rights that KD used to provide. Each improvement in SBMM has diminished that sense of accomplishment.

At the very least, players deserve some sort of feedback and recognition for their skill. This doesn't make Casual the same as Ranked, it simply acknowledges that they're playing against tougher competition, rather than pretending it's all the same.

Transparent MMR in Ranked

Competitive (not necessarily high-skill) players are under-served.

At a minimum, competitive players in Ranked should see how good the matchmaker thinks everyone is. WW2's Ranked Play (2017) was the only COD entry with transparent MMR like this.

The difference between the current hidden MMR systems and true visible MMR can seem subtle, but it's the difference between:

• making players grind to reveal how good the game thinks they are

• letting players prove how good they really are

Competitive players can tell the difference.

In a hidden MMR system, the visible skill rank is mostly cosmetic. So, a highly-skilled player (a future Diamond) at a visible Silver rank would play against other highly-skilled players, based on their hidden MMR. It seems like a nice design, but all matches feel 50/50 then, regardless of Rank. There is no tangible sense of what it means to be a Silver player or a Diamond player.

In a true visible MMR system though, a future Diamond player in Silver rank should dominate matches. Ranked matches should not be 50/50 all the time. Mechanisms like placement matches keep this deliberate imbalance in check.

Halo Infinite recognized that matchmaking on hidden MMR feels weird to players, but instead of switching to a probabilistic, transparent MMR system, they used the fake player-facing CSR (SR in COD) for matchmaking instead. This indeed looks better, but it makes the rating updates feel wrong. The way to fully solve it is at the MMR level.

Fix team-balancing / Optimize matchmaking

Players in parties with disparate skill have bad experiences when matchmaking together in Casual. There are technical improvements that could address this while increasing the similarity of team compositions within matches.

The details are out of scope here, but you can get details in this video (The Dark Art of SBMM).

Furthermore, the ideal matchmaker shouldn't need to relax constraints over time — it should incrementally improve matches with every iteration (with preference given to the older ticket). Then wait time can adjust deterministically based on the size of the pool of pending tickets.

Takeaway

Players don't always know how to fix problems, but they recognize them when they experience them. The negative sentiment around matchmaking in Call of Duty stems from real problems that are difficult to address, but not impossible.

If you’re trying to avoid the same problems in your matchmaking system, reach out to our matchmaking experts (just a convo, no hard sales).