Operation Tell Valve All The Things, 3.0

I surveyed a bunch of Steam developers and they told me how they feel about recent developments.

Operation Tell Valve All The Things, 3.0

So here's a thing:

revenue_share-2

Please read this entire article before posting a hot take. This is a long, hard, nuanced discussion.

First, some context.

For the past two years I've been running a wide-ranging survey of independent game developers. Last year the results were published in PC Gamer, the previous year they were just shared privately. I know for a fact that Valve reads these reports, and even responds to them. Of the 19 issues we raised in the past two surveys, we got a resolution for 55-60% of them, depending on how you figure. A full catalog with dates and details can be found at the bottom of this article.

I wasn't planning on doing another survey this year, but then this stuff happened:

I had recently dialed back my involvement in indie dev community organizing, primarily because I have a weird mutant brain with abnormal stress tolerances and was feeling really burnt out. But man, the community groups I frequent have never been this animated. To top it off, I reached out to my contacts at Valve and they agreed to a phone call for that Friday to discuss current events.

I had to do another survey. But this time I didn't have months to noodle around on overcomplicated details -- just two days. So, I set strict limits, threw together a quick and dirty survey, and blasted out a frantic call for respondents.

48 hours later, just two hours before my call, it was complete.

The Survey

Design

The 2016 and 2017 surveys had a lengthy "raising issues" phase where community members would submit statements, followed by an overly complicated voting phase on said issues with lots of dickering over methodology. Both surveys asked boatloads of questions and took over a month to gather responses for. I then analyzed the responses with needlessly complex spreadsheets.

I had no such luxury this time, so I skipped the "raising issues" phase entirely and just chose nine issues to vote on myself, drawing from ongoing community conversations and my own experience from the past two years. The only mandatory response was to rank these issues from most to least important (I also threw in a few optional opinion polls and demographic questions). Then, I sent the anonymous survey to gamedev communities I frequent and asked for introductions to new ones. Before long, I was surveying over two dozen groups scattered across the globe. The survey form was shared privately by creating a unique copy of the master form for each community. This way, if a survey form got leaked publicly and filled with troll responses, I could contain the damage by nuking just those responses. In the end, I heard back from 22 different community groups.

Response

In less than 48 hours I had 194 responses (compared to 230 or so in the 2017 survey). The first two surveys reached gamedev groups concentrated in the USA, Canada, and Western Europe. For this third survey I again reached out to those groups, but also new ones encountered via regional gamedev discord servers, slack chats, forums, mailing lists, etc. We got hits from Indonesia, Texas, Washington, Illionis, Israel, Russia, New Zealand, France, Malaysia, Sweden, Singapore, the Phillipines, Belgium, and more.

Caveats

This survey is neither scientific nor comprehensive, but it's the best I could do in 48 hours flat, and I hope it is a signal that's one order of magnitude clearer than "everybody screaming at once" and "random dude's unsubstantiated hot take."

For those of you who have problems with the survey's squishy methodology, I have written a companion article that sketches out a far superior approach: How to scientifically survey all of Steam's developers. That will probably take a lot of time, work, scripts, and grad students. But if you're up for it, please do it -- and share the results with me!

Also, bear in mind this is a survey of developers. I am well aware that stores, developers, publishers, and most particularly customers, have divergent views and interests, and I respect that.

Finally -- these results do not necessarily reflect my personal opinions, which I shall do my best to keep out of the survey results themselves. There will be a separate analysis section at the end where my own stupid subjective thoughts begin to enter, but those shall be clearly marked as such.

Now, let's get down to business (♫ to defeat / the Huns ♫).

Survey Results

I am still cleaning up the raw data set to remove any personally identifying info someone might have inadvertently included in free responses. I'll post that later in a follow up so people can check my math. Meanwhile, here's a summary of the results.

Edit 12/11/2018::
Just finished. I can now post the raw data set and an example of the master survey form. Please let me know if there's any errors in my calculations and I will update this article. I have removed all personally identifying information I could find from all free responses, and re-labeled community names to protect identities. Private groups were renamed something like "[Region]A, [Region]B," etc, and groups based on a specific location such as a city were renamed to be no more specific than their country. An example: New York --> UsaF (A New York group was not actually in the data set).

Who took the survey

As of this writing, 194 people took the survey across 22 different gamedev communities from all around the world -- whether or not they identify as "indie." The only requirement was that they be registered Steam developers. Of those, 148 also volunteered information about their lifetime earnings on Steam.

Lifetime gross ($USD) % of respondents
$1M gross or more 20%
$250K-999K 18%
$0-250K 62%

That's a big bottom tier, but there's also a fair number of middle and high earners, (38%). And at the top, 1 respondent had a lifetime gross in excess of $50M, and 3 had in excess of $10M (this ~3% of our sample might even be candidates for Valve's new upper-tier revenue shares). This distribution matches what I know anecdotally -- I'm friends with a lot of developers who have shared their earnings with me in private, and I know many of them took this survey.

The respondents were spread pretty widely across the world. The survey didn't ask about location, but it could be inferred based on the gamedev community group the respondent belonged to ("Indonesian gamedevs", "Chicago gamedevs", etc.) I combined these different groups into five regional clusters below, as well as the catch-all "Western" cluster. This is a really rough way to measure this sort of thing -- a future survey should just ask about location directly.

Total Western USA Oceania Europe Russia Southeast Asia
194 108 12 5 31 2 37

whereintheworld

The "Western" cluster is a mix of several large gamedev communities from around the world that have no particular regional label, but have membership that's clearly concentrated in the US/UK/Canada/Western Europe. The "USA" group consists of various gamedev communities identified with specific American cities. The rest are formed by combining individual gamedev communities identified with specific countries, bucketed by geographic regions. "Southeast Asia" here consists of Indonesia, Malaysia, Singapore, and the Phillipines.

When we combine the two revenue tier chart with the regional chart, we get something like this:

Tier Overall Western USA Oceania Europe Russia Southeast Asia
$0-249K 92 33 7 2 21 1 28
$250-999K 27 14 3 0 5 1 4
$1M+ 29 21 1 3 2 0 2

European and Southeast Asian game dev groups increased representation from the lowest revenue tier, whereas the "Western" and "Oceania" game dev groups brought the most representation from million+ earners.

What was said

2016's survey report consisted of mostly safe, boring technical issues, because people were reluctant to approach Valve with their true feelings. I know this because during the "issue raising" phases of the first two surveys, there was quite a lot of heated discussion that did not make it into the final report, specifically because many people were nervous about telling Valve something they thought the company might not want to hear.

With the first survey specifically, I got panicked emails from several developers desperately trying to get me to scrap the entire project for fear that Valve would retaliate against the entire group. It took a lot of diplomacy and back and forth to get the necessary community buy-in to proceed, ultimately softening the final report. After the first report was sent and nothing bad seemed to happen, it was easier to get community buy-in for the next one.

With each subsequent survey (and as circumstances have changed), responses have grown bolder and more honest.

What's the most important issue right now?

Users were asked to rank 9 issues from most (1) to least (9) important, without repeating a rank. I couldn't get Google Forms to enforce that, and I can tell some people gave multiple scores of "1" and "9", but most respondents complied and we got a pretty clear signal out of it, especially for the top issues. The final score is an average of each issue's numerical rank, normalized to (0.0-1.0), and flipped (1-X) so that bigger means better.

The nine issues to rank are shown below. They were presented in random order.

  1. Dealing with toxic users
  2. Better/fairer revenue share
  3. Direct relationship with the customer
  4. Liberal use of steam keys
  5. Not being punished for deals with other stores
  6. Curation
  7. Not getting screwed by "the algorithm"
  8. More transparency from Valve
  9. Something else / any and all other issues

An additional limitation: because of the rushed nature of this survey, some of those might be construed as leading questions. All I can say for myself is these were sourced from community discussions, plucked straight out of conversations. Had I had more time to do it again I would probably seek more neutral wording; if you think you can do a better survey, 1) you're right, and 2) I'd love to see it.

In any case, here's the results:

The top two issues

  1. Not getting screwed by "the algorithm" (0.73)
  2. Better/fairer revenue share (0.70)

The runners up

  1. Something else / any and all other issues (0.61)
  2. Curation (0.58)

Breakdown

Results were surprisingly consistent across revenue tiers:

Issue 0-249K 250K-499K 1M+
1. Not getting screwed by "the algorithm" 0.75 0.73 0.73
2. Better/fairer revenue share 0.73 0.68 0.69
3. Something else / any and all other issues 0.65 0.61 0.64
4. Curation 0.63 0.56 0.50

The less a dev has earned, the more they clamor for curation -- but not enough to shift their overall priorities. And each tier is in complete agreement that their #1 and #2 concerns are "Not getting screwed by 'the algorithm'" and "Better/fairer revenue share", in that order.

Given how much ink has been spilled about revenue share, it's interesting to see that topic edged out by "Not getting screwed by 'the algorithm'", and also to see this concern shared across revenue tiers -- at least in this sample, it's not just the < 250K crowd, it's everyone, even the million+ earners.

These priority holds when we slice by region too, though not quite as strongly as it does for revenue tiers:

Issue Western USA Oceania Europe Russia Southeast Asia
Respondents 108 12 5 31 2 37
Not getting screwed by "the algorithm" 0.72 0.66 0.82 0.72 0.83 0.74
Better/fairer revenue share 0.68 0.69 0.69 0.67 0.78 0.77
Something else / any and all other issues 0.54 0.64 0.58 0.67 0.67 0.73
Curation 0.53 0.55 0.53 0.70 0.67 0.66

Some of the groups (Oceania, Russia) are probably too small to be meaningful by themselves, and one shouldn't draw too many conclusions from this naive regional analysis anyways since it doesn't control for group size and makeup. But at a glance we can see that the dominant "Western" group basically agrees with the priorities of the large Southeast Asian and European groups. The Southeast Asians are slightly more concerned about revenue share, and the Europeans are the only ones to have a different top 2 issue, namely curation (but it's still not their top concern).

It merits further analysis, but I believe the chief concerns of game developers about Steam right now are fairly consistent for both developers large and small, as well as across different countries, even ones with wildly different average incomes and proximity to Valve's headquarters. I expect a more thorough survey would find similar results.

How is Valve doing?

Three of the opinion questions were repeated from last year. This indicates how Steam developers' opinions might have changed in just the past year:

postively_aligned

answers_questions_meets_needs-2

revenue_share-2

There's a few caveats to consider before we take these at face value. For one, 2018 didn't get as high a response rate as 2017, and also sampled a slightly different population.

Maybe the negative shift in sentiment was influenced by the new, more international survey population? A quick way to check this is to see if the "Western" sub-group's responses stick out compared to the overall results, since that group consists largely of the same communities I surveyed last year.

In the charts below, I mark in bold any sub-group's negative sentiment that's greater than the overall average.

"Answers my questions and meets my needs"
Sentiment Overall Western USA Europe SE Asia
Positive 24% 20% 27% 30% 28%
Neutral 35% 27% 36% 37% 50%
Negative 41% 54% 36% 33% 22%
"Has interests positively aligned with mine"
Sentiment Overall Western USA Europe SE Asia
Positive 22% 19% 9% 23% 33%
Neutral 36% 35% 55% 20% 44%
Negative 42% 47% 36% 57% 22%
"IS earning their 30%, in 2018"
Sentiment Overall Western USA Europe SE Asia
Positive 11% 13% 9% 3% 14%
Neutral 21% 14% 9% 27% 31%
Negative 68% 73% 82% 70% 56%

The "Western" sub-group actually feels more negative about Steam than the overall population. If anything, including more voices from around the world (especially South East Asia) served to dampen the impact of the old "Western" group's discontent. But even the "milder" South East Asians opinions were still significantly more negative than those reflected in last year's survey.

Other stores

Given all the talk about Epic (and various other competitors) We also asked people which stores they had already agreed to put their games on, and which they were currently considering.

These are the results:

already_signed_store

considering_store-1

I probably should have included options about consoles, and "Microsoft" is an ambiguous choice in retrospect because it was not clear to respondents if I meant the Windows store or the Xbox store (I meant Windows), which probably renders that response meaningless here.

In any case, a lot of people have already put their games on Itch and Kartridge, and people are most interested in Epic and Discord.

Here's how it breaks down by revenue tier:

already_signed_store_breakdown

consider_store_breakdown

It's unsurprising that the only developers who have already signed with the more exclusive curated stores (Epic, Discord, Twitch) are those in the higher revenue tiers. However, it's apparently not just small developers who are signing up to offer their wares on Kartridge and Itch.

In terms of interest, Epic is the most popular choice by far, followed by Itch, but within each group of developers considering a particular platform, there seems to be an even mix of low and high selling developers.

Curation

Curation is a hot topic right now, but what exactly do people mean when they say, "Curation?" Curation itself came in at position #4, but it's credible that a respondent's private definition of "curation" might influence their reason for choosing "Not Getting Screwed by the Algorithm" as their top priority. This is why we included some optional free response fields. One of these was: "What does 'curation' mean to you?"

Alas, I was not able to do a full qualitative analysis in time for this report. Suffice it to say, curation is a somewhat controversial subject among developers. There seem to be three main factions, but I haven't had time to study how popular each opinion is or who shares it, so this is a very rough outline supported by little more than visually scanning over the free responses:

  • There's too much "shovelware" on Steam, clear it all out.
  • Those calling for "curation" just want to clear out their own competition -- everyone wants "curation" to kick in right at the threshold below their own game.
  • Steam should be two-tiered; it should have a "firehose" where any game can be, but the front page should have a better filter for "quality" for games, preferably human powered.

For those who want games cleared off of Steam, proposed mechanisms were:

  • Raise the Steam Direct fee
  • Add a monthly/annual listing fee that kicks of your game if you don't pay it
  • Approve all or most games by hand

I want to emphasize again that I have not had time to measure relative support for the summarized positions above, and given the lack of quantitative questions in my survey regarding curation, I may not be able to at all from this data set. A more thorough survey could shed more light on this issue.


Analysis

Fair warning: this is where the report ends and Lars Doucet's stupid personal opinions begin. Note specifically that these are not Lars Doucet's "personal views about how things are and ought to be", they are Lars Doucet's "subjective opinions about exactly why the survey results came out this way." In short, I am doing my best to speak in the voice of those who took the survey, leaning on my own imperfect understanding of the overall forces in play, and drawing on my own personal experiences as necessary to provide concrete examples about what I'm talking about.

Also to any Valve employees reading this, please don't hate me for saying what comes next. It's my best attempt at putting forward the truth. You've always been welcoming of my criticism in the past, something I sincerely appreciate, and I hope that's the spirit it will be received in today.

Hokay:

1. Developers are at the mercy of a capricious algorithm that can change at any time

The biggest thing that sticks out in the survey is that revenue share is not developer's #1 concern. I mean it's way way up there (nearly tied for first in fact), but the top issue is "Not getting screwed by 'the algorithm.' Anyone who's followed the drama over the last few years in Youtube land will find this a familiar conversation.

Valve sells Steam as a meritocracy, but sharp discontinuities in traffic from overnight changes like The October Bug shatter any remaining faith in that promise. The "October Bug" was crystallized into a symbol that now represents so much more than the specific damage it caused (which for many developers was extensive).

I know the word "meritocracy" has become an epithet in certain communities right now, and I definitely get where they're coming from -- they believe "meritocracy" is moral cover for the Just World Hypothesis - a justification for treating personal privilege and luck as moral worthiness, and poverty or any other negative life outcome as deserved. And in this view, at its best meritocracy is just a cynical system that sets up an arbitrarily narrow (and often overly nerd-metrics-driven) definition of "merit", ruthlessly ignoring all other dimensions, wrapping itself in a false mantle of objectivity and fairness.

As a negotiator, however, I always take the other side's stated worldview at face value and then hold them to their own stated beliefs. And meritocracy is at its best when it stands in opposition to insidious nepotism -- where the quality of your offering and skills don't matter, where hard work, ingenuity, and honest labor is subverted at every turn by the "secret handshake club", information assymetry, and shady backroom deals with powerful actors. Where the proverbial "little guy" has no chance to ever make it, simply because of where they started in life. In this light, meritocracy can be a good thing, even a liberating thing.

But how can Steam be a meritocracy if a developer was doing everything "right" according to Valve's stated best practices for success yesterday (and being rewarded for it!), and then the next day their traffic and sales immediately drop off a cliff, never to recover? How can any business plan for such an event? If you want to nurture an ecosystem where the next Undertale or Terraria can spring forth, this sort of unstable environment could strangle it in the crib. Developers are clearly looking over their shoulder wondering if and when a "January Bug" or a "March Bug" is going to drop.

I get it, if you're going to use an algorithm, there's inevitably going to be changes. And we know Valve is in a constant cat and mouse game with people trying to game the system, too, so you're cagey about explaining how it works. But game studios small and large run on operational budgets with razor thin margins, and we can't handle these kinds of wild swings, particularly when they come with no advance warning. Valve can put developers out of business just by pushing a button, and it doesn't feel like merit had anything to do with it. It feels like caprice.

2. Developers aren't being listened to

Even if Valve is hearing developers' concerns, they aren't listening to them. The difference is that listening includes understanding, as well as acknowledgment. And I get it, I can't imagine what the static and noise must sound like from Valve's end of the table, it's probably deafening. It's a lot to sort through, and I'm sure a lot of the feedback consists of contradictory requests that amount to "Do this specific thing that will help my studio (even if it hurts this other studio)."

That's something this survey is meant to address -- and a big reason Valve should do their own.

Another part of the problem is that there are thousands of developers but only a few hundred Valve employees. And to Valve's credit, they have added some improvements to the developer support system (see the issue resolution catalog at the bottom of this article for details). But there's still plenty of developers who feel shut out, and as this chart clearly shows the sentiment is getting worse:

answers_questions_meets_needs-2

The typical rumor is that all the big earners have direct access to Valve employees, but the bottom tier doesn't. At least according to my data, that's not actually the case.

This survey didn't ask about respondents' access to Valve employees, but last year's did, and this interactive visualizer by my friend Jason O'Neil lets you see a breakdown which I'll screenshot here:

contact_employee

The dots are colored on a red-blue gradient, with dark red (1-$9.9K) to pink ($250k-$499K), grey ($500K-$999K), and blue shades representing increasing millions as they get darker. White dots shared no revenue information.

Clearly, high-selling devs are more likely to be able to reach a Valve employee, but there's still many high-earning devs (including several multi-million earners) with unreliable human contact or even none at all. This fact is further borne out by the kind of access to Valve employees I personally lucked into despite being a mid-tier nobody developer, contrasted with quotes like this from survey respondents:

As you can tell, quite a few of us don't have much contact with Valve (Hell, I'm in the ~$3m kind of space, and only recently even got an email address for anyone at valve... and that was by asking another dev.

I certainly don't think that big sellers deserve automatic unfettered priority access while lower selling developers should be content with a bare contact form, and I also understand that Valve employees have limited bandwidth. But regardless of how access to Valve's scarce employee attention ought to be rationed, the status quo is a haphazard system where lucky folks like me can reach out any time they want, and others are left in the cold simply by chance. However, the looming threat of a capricious algorithm capable of another "October Bug" changes what might have been mere frustration into an ongoing existential crisis with no one to turn to for support or even explanation.

I do want to thank Valve for concrete improvements in their developer support backend in recent years, but more active listening on their part could go a long way to address developers' concerns.

(I recommend a regular comprehensive survey as a starting point).

3. Valve's all-seeing data eye also blinds them

Valve has access to more and better data than any of us could ever dream. They almost certainly have credible arguments based in data for the messaging that comes across to us developers as tone-deaf. For instance, this quote from the recent announcement about revenue share changes:

The value of a large network like Steam has many benefits that are contributed to and shared by all the participants. Finding the right balance to reflect those contributions is a tricky but important factor in a well-functioning network. It’s always been apparent that successful games and their large audiences have a material impact on those network effects so making sure Steam recognizes and continues to be an attractive platform for those games is an important goal for all participants in the network.

Okay, fine. I'm sure lowering revenue share for high earners is abstractly beneficial for smaller developers. Given that the top 100 earners make half of the revenue on Steam, there would be a negative impact on downstream developers if you could no longer buy GTA V or PUBG on Steam. I'm sure this is all technically true. But, the message this actually sends to small developers sounds like:

We're willing to take a huge cut in annual revenue to keep big developers on board, and even though throwing a similar bone to small developers would cost us a mere fraction of that amount, and have a much more immediate, direct, visible impact on their lives, we ... don't care.

And I get it, given the lower earnings of a small developer, the per-unit-sold costs of supporting a small developer's game are obviously higher. But given that Valve is one of the most profitable companies on earth per employee, and that Valve has just willingly signed away what must be an enormous amount of money as a sign of good will to keep their top-selling developers, it really makes it seem like this decision is not about the money, but some principle. And what is that principle? We haven't heard.

The one thing that behavioral metrics can't show Valve is what their own developers truly think of them. Some of you may not know this, but Valve has actually been sending its "front of house" employees out on a wide-ranging speaking tour all over the world to talk to developers for well over a year now, as well as hosting endless developer luncheons and round tables to gather feedback. That's a good thing! But not only is there a selection effect at play here, the power differential means that everything said to Valve when a developer can be identified is said under a veil of fear. Anything a dev says to Valve's face should be downshifted by two octaves. Only the boldest dare complain to the King's face.

Well, consider me the court jester. My task is to tell you the truth. And the truth is:

4. The good will is spent

People used to stick with Valve because they honestly felt it was the best platform for both customers and developers. Now they stick with Valve because of power. And the second they have an alternative, they will jump ship.

answers_questions_meets_needs-1

revenue_share-1

I know a lot of people are talking about the Epic store. Epic is making a really strong developer-first push, offering 88% revenue share across the board, and reportedly giving out money by the truckload to buy exclusives and free game giveaway deals. It's a strong offering. I can say for a fact that it's VERY popular among developers right now, and the survey results bear that out.

Now, let's be honest here. Epic is not a threat to Steam... today.

I downloaded their client. It's ... fine. It's buggy, it's bare bones, and it's catching a lot of flak from customer groups for various reasons (refund policy & regional pricing among others). And even given Fortnite's huge player base, tons of those are kids playing on mobile who aren't old enough to have credit cards (it'd be great if Epic could give us clear breakdowns of their PC install base, and how many have ever spent money on anything).

But.

But but but.

A lot of developers don't care. They know that jumping ship or taking one of Epic's deals is a risk. But staying on Steam is a risk too, when the algorithm could decide without warning that all your marketing efforts (the very ones Valve told you that you should be doing) are now suddenly wrong and you deserve half as much money or less. Precisely because developers now feel that simply staying is a risk in itself, they are more willing to take other, bigger risks.

Even though Epic's launch can rightly be scrutinized, they're still the strongest offering so far, and buying your way to the top is a proven strategy. All who underestimate Epic's offering do so at their peril, and developers are jumping ship to them as we speak.

But let's not make this all about Epic. There's many other options on the table.

Gaining access to the Steam's audience in 2012 was a godsend to me. But there was always an alternative: selling direct.

I sold direct for nearly a full year before getting onto Steam. It wasn't easy, and a lot of what made it work then was leaning on circumstances specific to that time and to my game. But I did it, and I earned a decent living. It can still be done today. Many developers are doing it right now, and succeeding.

"But won't you make less money if you abandon Steam?"

Well, Lars Doucet probably would. I've been blessed by the algorithm and I sincerely hope that continues. But for other developers -- not necessarily. The 30% revenue share was established back in a day when a developer like me could put out a major update, email a rep asking for a global front page feature, and get a reply that said "Sure!" is less than a day. Nowadays, developers are being asked to "bring their own traffic" and drive metrics like wishlists in advance of launch. And I mean that's fair in a sense -- there's so many developers today, we all have to do our own marketing, why should we expect a company to do it for us when there's not enough promotional space to go around? But, if we're being asked to bring the traffic, and "the algorithm" is threatening to cut us off at the knees at any moment, and we can't reach an actual human to explain what the heck is going on while we're stressing about how to make payroll next month, maybe we should just take all that traffic and marketing we're driving, and save it for ourselves?

Valve calls us "partners." That's what developers in 2012 certainly were. But circumstances have changed, our options have changed, and our relationship with Valve has changed. Revenue share should change too, or many of us will leave. Maybe they're okay with that. Maybe we're okay with that.

But that's the truth, at least as far as I can tell.

And if you don't believe me, you can do a better survey :)


PostScript

The main article's over and you can stop reading now. But here's some extra goodies for further context that I said I'd stick at the end.

How Valve responded to the first two surveys

Here's an up to date list of all the issues that made it into the executive summaries for the previous two surveys, and whether or not Valve responded to or resolved the issue.

Operation Tell Valve All The Things, 1.0 (2016)

1. Players should be able to "follow" developers on Steam
We got this one on June 25, 2018.

2. Established developers should be able to get app IDs more easily
Resolved, for better or worse, by Steam Direct on June 13, 2017.

3. Create a developer support ticket system
I don't know when, but we got this. There's now a Steamworks developer help site that has an automated wizard for common issues, and a "contact the steam team" button that opens a traditional support ticket system.

4. Add embeddable "add to Steam wishlist" buttons for the web
We got this in the form of the embeddable Steam Widget, which shows a "add to wishlist" button if the game is not out yet. Not sure how long we've had it but the documentation showed up on November 26, 2018.

5. Can Valve provide data on whether there is any significant pattern of refund abuse for shorter games?
I don't think we ever got any hard data, so no points here. For what it's worth, they've told us repeatedly that from what Valve can see, they don't think there is any such pattern.

6. Add an “open in Steam client” button on any Steam webpage, when viewed outside of the Steam client browser
Don't think they did this. Looks like they did do this! I see it on the right column of Steam pages when I'm not logged in. If you're logged in and in a web browser, but not using the client, you still don't see it, but this feels close enough to count. A Valve rep informs me this was added June 2016.

7. Add a traditional coupon code discount system.
Don't think they did this.

8. Developers should have (limited) access to directly running their own coupon promotions.
Don't think they did this.

9. Make Steam video player work on mobile.
They did this. It went live about 12 months ago (so December 2017 I guess).

10. Allow selling "unlisted" apps on Steam with no public store page.
They did this. It doesn't seem to be documented yet, and it currently requires submitting a ticket for them to review the request and make the change.

It's taken a while (the most recent resolution dates back to just a week ago!), but it looks like we got 7 out of 10 issues resolved from the first survey, including all top four issues. I should note that although Steam Direct counts as literally resolving one of this issues, it remains a controversial decision among certain developers.

Operation Tell Valve All The Things, 2.0 (2017)

1. Remove ability for users to delete developers' comments on their reviews.
AFAIK, no change has been made. I never found an announcement for it, but apparently this is resolved as of January 2017. Here's an example -- a developer can leave a special "developer response" that can't be removed by the user, and is called out as a developer response. Good to have this #1 issue resolved. Interestingly, the claimed date for this seems like it was before the second survey was released. This may be because there used to be an exploit vector where a user could change one character in their review to clear a developer response, or just delete the review and re-post it. This no longer seems to be the case. I'm not sure when exactly this exploit was patched.

2. Clarify what a developer needs to do in order to qualify for various featuring opportunities.
An ambiguous request with an ambiguous resolution. Valve has done a lot of touring and speaking recently and they often talk about featuring. I'm not sure they've ever broken it down into a hard rubric or anything. Let's give them half a point for this one.

3. Bug reporting / technical support should be built into Steam/Steamworks. Filter issues that are Steam issues to Steam support team instead of to the developer.
AFAIK, there isn't a built in bug reporting system other than user reviews and the forums, the same as always.

4. Devs should be able to have one unified landing page for all their games.
This was a repeat issue from the previous year's survey, and we did eventually get it.

5. The Steam Community feature needs better ways to deal with toxic users.
Well, they gave us something. On September 19, 2018 there was a moderation update. Valve started taking responsibility for directly moderating Steam community forums for a developer's game -- they will respond to reported posts themselves, a behavior you can opt out of. I'm sure there is disagreement over whether this qualifies as fully resolving the issue, but it at least addresses/acknowledges it with a response and a change. Counts as 0-1 points and expressed as a range.

6. Valve should increase the overall volume of sustainable middle class developers.
A broad statement rather than a narrow technical feature to point to (it doesn't clarify how/what they should do to achieve this), but at the moment I think the consensus is that "they have not done this."

7. Make reviews on the frontpage representative of the game's review percentage.
The idea behind this one was that if a game has 70% positive reviews, then on the game's front page, it should show the 7 "most helpful" positive reviews and the 3 "most helpful" negative reviews, rather than simply the 10 "most helpful" reviews regardless of sentiment. We got this on November 20, 2017. The "Most helpful" column has a ratio that reflects the "all reviews" score and the "Recently posted" column has a ratio that reflects the "recent reviews" score.

8. The “Upcoming Games” list is becoming useless.
Unfortunately it's still pretty useless and still has a lot of problems. Also, "New releases" is largely the same -- multiple DLC items for a single game will clog out actual new releases, and unreleased games with no price or that have a clearly false release date in the past will get placement in this list.

9. Devs should have better tools and info for tracking and identifying players who abuse/hack multiplayer, bot/farm item drops, etc.
I don't know if there was ever any progress with this one. I read all the Steam developer announcements, and don't recall seeing anything about this.

10. Devs should have the possibility to send messages to forum users.
I don't know if we ever got this. We did get this actually! March 2, 2018.

Valve has had less time to respond to the second survey, but depending on how you figure they delivered on either 2.5 or 3.5 issues (which includes a point for a duplicate issue from the previous survey).

Grand Total

  • OTVATT 1.0: 7/10
  • OTVATT 2.0: 4.5-5.5/10
  • Total unique: 10.5-11.5/19

The total resolution rate is 55-60%, (or as low as 50% if you hate Steam Direct and insist it shouldn't count). Either way, that's pretty good.

The key takeaway from this is that developers should be bolder in asking specific things of Valve, because they just might get half of them.

A better survey

Here's that fancy methodology I said I would post for anyone who wants to replicate or disprove my findings. It became long enough I stuck it in its own companion article.

How to scientifically survey all of Steam's Developers