How to design (and listen to) a developer survey

A well-designed developer survey is only the first step towards measuring team health, the important part is knowing how to listen to the results.

Software teams can be a hard group to measure. Raw output metrics like lines of code written, features shipped, and bugs squashed can go some way towards telling you how a team is doing – but they leave a lot unsaid. Do your developers feel they are being used effectively? Are they able to get the focused time they need? Or are they managing to ship in spite of bureaucratic obstacles, too many meetings, and a lack of direction? To get that kind of information, you need to conduct a developer experience survey.

Developer experience (DevEx) is all about how developers “feel about, think about, and value their work.” Unlike blunter, traditional metrics of developer productivity, it can only really be measured by asking people qualitative questions about how they feel.

How to conduct a developer experience survey

Conducting a developer experience survey doesn’t need to be difficult, but it requires a fair amount of planning and forethought.

Who do you want to survey?

Start with who you want to survey. Do you want to conduct a census of your entire organization, or just a representative sample of your developers? Do you want to break down data by team? By role? Both? LinkedIn, for example, segments its data based on “developer personas”.

Deciding who exactly you want to take the survey determines what kind of questions you need to ask and how in-depth it should be. A census of all the developers in a large organization should be shorter than a survey of a representative sample if you want good data. Abi Noda, CEO at developer productivity specialists DX, recommends that a survey should take somewhere between 10 and 30 minutes for each participant to complete. 

How frequently do you want to conduct your survey?

While an annual survey is typical, you can also conduct your survey at quarterly or even monthly intervals. Companies like Google tend to survey a representative sample of one-third of the organization every year, so each individual developer is only getting surveyed every three years. While this would be extreme for a small company, it enables Google to ask more detailed questions when the time does come to ask developers how they are feeling.

What do you want to know?

Do you want to create your own survey or use an existing solution? According to Arylee McSweaney, a Director of Engineering at Etsy, Culture Amp’s developer engagement survey is a great starting point. Then you will want to tweak and add questions to any survey you take off the shelf, and Noda recommends resources like MeasuringU’s blog, which has articles on creating good survey questions and validating your metrics, which are useful if you are going to roll your own.

What’s important is asking specific questions that will get you the information you need to understand your developers’ lived experience.

Pick a platform

If you don’t use a dedicated platform like Culture Amp or DX, you will need to choose a platform to administer your survey. SurveyMonkey, Google Forms, and Qualtrics are all solid options.

Test your survey

Even if you use an existing solution, it’s a good idea to test your survey on a small sample of your participants before administering it to everyone. That way, your team has an opportunity to tell you that certain questions are unclear or hard to answer, and you can see which questions might not give you great data.

How to listen to the results of a developer survey

Conducting a developer survey is just the first step. Once you have your data back, you have to assess and understand it, and turn it into concrete next steps for your organization.

Benchmarking your results

“Benchmarking is really important,” Noda says. If you don’t have some idea of what normal looks like, then you can’t do a lot with your data. The example he gives is if 30% of your developers are dissatisfied with the technical debt, is that a real problem? Or is it just normal developer grumbling? Without some kind of benchmark, it’s impossible to tell. 

There are a few ways you can benchmark your data:

• To past surveys. If you have conducted surveys in the past, you can compare data to see what has changed.
• Within your organization. For things like general sentiment, you can compare data between different teams within your organization.
• With other organizations. With some existing paid solutions you can compare your results against industry norms.

Talk to your team

While the results of a developer survey can be illuminating, they don’t paint the whole picture. After every survey, McSweaney recommends talking to your team to understand why they’ve given the responses they gave. A few years ago, she joined a team that scored low in a developer experience survey, so she held a one-hour meeting to talk things through. “I could hear people wanting to feel more visible, more acknowledged, and more recognized for the work that they were doing,” she says.

From that, she was able to get a much deeper understanding of where the true issues lay – not just what the numerical scores were.

Prioritize a few areas

Depending on the results of your survey, there is a good chance you won’t be able to fix every issue immediately. As a management group, hone in on one or two areas to focus on initially.

Acknowledge the results

It’s vital to acknowledge the results of the survey publicly, and thank your team for participating. 

Often organizations will conduct a developer survey and not be in a position to take many actionable steps with the results. But by seeming to ignore the survey, they are communicating to their engineers that it was a waste of time. In that case, Noda recommends sending something as simple as an email saying, “Thanks for your feedback, this is really valuable. This is going to help us keep an eye on how things are trending and where we need to invest in the future.”

Keep testing and iterating

A developer experience survey is a great way to get a snapshot of how the people in your organization feel at a given moment, but a single survey can only tell you so much. 

If your organization has just had to cut costs or make layoffs, people are going to be less satisfied than if they’ve just been paid an annual bonus or received a stock grant. By conducting the same survey again a few months or a year later, you can get a much better picture of how people in your company feel and what direction that sentiment is trending.

When you combine those results with your other measures of developer productivity, you can begin to get a holistic picture of how your dev teams are doing and take actionable steps to improve things for them.