Interviewing for an engineering job in the tech world can mean you’ll be asked all sorts of questions. Sometimes, the interview questions can be pretty straightforward: “Tell me about a time that you have implemented an effective monitoring solution for a production system.” Other times, the questions are impossible to answer and designed to spark your creativity: “How many windows are in New York City?” After passing the initial interview, the candidate graduates to the next tier of interviewing: The often-dreaded technical interview.
Prospective engineers often face a challenge when it comes to preparing for the technical interview, largely because there is no playbook for how companies set them up technical. It’s unclear whether to prepare by memorizing many different topics, or focusing on specific projects. Is it better to practice with a computer or a peer engineer? There are an overwhelming number of resources available online, but with little clarity as to what the standard is for a technical interview and little guidance from the company on what to expect, most of the time engineers start technical interviews in the dark.
Inconsistencies in the technical interview process isn’t just a job candidate problem. In fact, many companies struggle to set up a technical interview process that is effective, equitable, and allows the hiring manager to compare candidates. The problem with technical interviewing compounds when a company is experiencing rapid growth.
The challenge with conducting technical interviews at a growing company
"Imagine you had a hiring target of doubling your team size and all your interviews are conducted remotely. Welcome to GitLab," says Clement Ho, frontend engineering manager on the Monitor: Health team at GitLab.
GitLab more than doubled the number of hires from around 400 in 2019 to roughly 1300 by end of 2020.
We identifed three core challenges with orchestrating technical interviews as GitLab grows.
- We didn't have enough interviewers for the pipeline of candidates.
- Our technical interviewing process was inconsistent and even a little biased.
- It was difficult to measure whether or not we were raising the bar.
"And by raising the bar, I mean making sure each candidate that joins the team makes the team better," says Clement.
These problems are by no means unique to GitLab. Any engineering company that is scaling rapidly will encounter some growing pains when it comes to hiring, and many will end up falling back on some of the typical models for conducting technical interviews.
The typical technical interview methods
During his talk, "Using GitLab to Power our Frontend Technical Interviews" at GitLab Commit San Francisco, Clement explained the four different techniques that are often employed in technical interviews. Each method comes with advantages and disadvantages from the perspective of the hiring manager.
Verbal technical questions
There are countless articles online that try to prepare job candidates for a verbal technical interview, but whether this method truly effective for evaluating the technical competency of an engineer is debatable.
In the typical scenario, the interviewer asks the candidate to describe a technical concept and tries to measure their fluency in said concept based on the quality of the conversation.
The advantage of this method is that the interviewer can understand how the candidate communicates, which is of particular importance when the engineering team is all-remote, as is the case at GitLab. The drawback? Being a good communicator does not necessarily mean the candidate knows how to code effectively.
"So I've interviewed candidates that could talk the talk, but they couldn't really write the code,” says Clement. "And that's not a great situation for an engineer to join GitLab." Clement’s team has moved away from using verbal technical questions as a method for evaluating candidates.
Live coding exercises
One of the more popular methods for evaluating engineers is through live coding. While it allows the evaluator to see how engineering candidates answer data structure questions, it also has its disadvantages.
A key advantage of live coding data structures is that it offers a fairly consistent measurement and evaluation.
"I can talk to another manager or another interviewer and be able to communicate, 'Hey, this person wasn't able to do a linked list, they got stuck here. They weren't able to understand a runtime efficiency here.' So it's pretty consistent," says Clement.
But the ability to create data structures is not always the best indicator of ability. Oftentimes engineers with a very traditional background or recent graduates will shine here, but someone who is more senior and able to do a lot of great things, but is perhaps not as brushed up on data structures, may struggle.
Live coding interviews probably aren’t going anywhere fast, but the pitfalls of this method are well documented by engineers and hiring managers. Brennan Moore, a product engineer in New York City, explains why he does not conduct live coding interviews when evaluating a prospective candidate:
"Much like the SAT when applying for college, live coding is a structured test. I didn’t go to a school that trained me to do live coding, and so will probably fail the test. As I’ve experienced it, live coding isn’t the meritocratic space that it pretends to be. Live coding interviews weed out the people who are good at live coding interviews," says Brennan in his blog post.
At GitLab, we found that live coding exercises don't accurately represent engineering capability. Oftentimes, a recent computer science graduate will outperform a more senior candidate with a lot of valuable experience. In summary, live coding exercises will often disadvantage more senior candidates, people who are nervous in high-pressure situations (read: everyone), and advantages more junior engineers or people who have practiced live coding.
A third common method for evaluating candidates is to ask the engineer to code a UI using an online editor while on screen share with the evaluator.
The advantage of this method is that it allows the evaluator to observe how a candidate builds. The drawbacks here are similar to those with live coding. First, the engineer is under pressure to build while the evaluator watches on, making it a nerve-wracking situation. The other drawbacks come from an evaluation perspective: It is challenging to measure the effectiveness of this method and it is hard to compare between candidates.
Any engineer (or writer, for that matter) can tell you, the supplemental take-home project is a very common ask when going through an interview process. The advantage here for us is that this assignment closely mimics the reality of building environments while working remotely at GitLab.
But this task comes with major drawbacks, mainly that it disadvantages candidates who may not have the time or capacity to complete the project.
"… imagine a scenario where you're a single parent and you have kids; you may not have as much opportunity to take dedicated time, a couple of hours after work to really focus on a take-home project compared to someone from a more privileged background," says Clement. "They might be able to dedicate and output something better."
Diversity and inclusion is a core value for GitLab, and anything that disadvantages candidates from underrepresented groups is not inclusive, and therefore suboptimal for evaluating candidates based on their engineering abilities.
The new way
While each method for conducting a technical interview comes with advantages, there are also numerous disadvantages when it comes to conducting an effective and measurable evaluation and creating an equitable interview process. Under the guidance of Clement, the Monitor:Health team decided to interview frontend engineers in an entirely new way using GitLab.
Now let's take a deep dive into the nuts and bolts of reinventing the technical interview for frontend engineers at GitLab. Just wondering about the key takeaways? Skip ahead. As we continue to iterate on a more effective and measurable technical interview process, we hope this inspires other engineering organizations to rethink theirs and share learnings with us.
Our first step: Standardize the interview process.
Fixing an MR on a test project
The team standardized the interview process by creating an open source test project, called
project-seeder, which seeds projects to different candidates using a GitLab Bot. Candidates are assigned a merge request to troubleshoot in the project created for the technical interview. The
project-seeder is powered by the GitLab Bot so the interviewer doesn't have to worry about API keys, and works in four steps:
- Exports the template project
- Imports template project
- Adds users with expiration
- Triggers pipeline for candidate to review MR
The candidate is sent an email with a link to the MR the candidate is assigned to fix as part of the technical interview.
Standardize the evaluation rubric
The team also created a standardized rubric for how the candidate's performance on a technical interview is evaluated.
"We don't want to be in a situation where unconscious bias or bias of one candidate over another plays a part because of our preconceived notions," says Clement.
Creating a rubric that looks at multiple categories allows the evaluator to look at the performance of the candidate from a more holistic perspective, as opposed to looking at a candidate's performance on one technology.
The team created a Periscope dashboard to create a feedback loop between the candidates and evaluators to identify opportunities for improvement in the technical interviewing process.
The frontend engineering team used Periscope to collect feedback from candidates who participate in technical interviews.
Demoing the technical interview
Inside the technical interview project
Clement created a sample project to demonstrate how we use GitLab to power our technical interviews.
In the gl-commit-example group, there is a subgroup with all the interview projects we are seeding to the imaginary candidates, a template, and a project seeder.
The interview project's subgroup, template, and project seeder application lives inside the sample project for the technical interview.
The assignment here is pretty simple. The candidate needs to update the website to say "Hello GitLab Commit SF," but in order to accomplish this, the candidate will need to fix the failing pipeline.
We use variables from GitLab CI to configure the project-seeder application.
Inside the project-seeder application which seeds the interview projects to job candidates.
new-project-example-two, and I'm adding this bot user that I created and the expiration, so I can just easily run this pipeline and it'll seed this project," says Clement.
The next step is to run the setup pipeline, which will create the project, import the project, export the project, and share it with the job candidate.
A look inside the pipeline that will create the test project.
Looking inside example-one, we can see there is a project and broken MR.
"And an example for a candidate – they would probably look at the CI and see, 'Oh there's a failing test. Let's see what that's about. Oh, it looks like it's checking for "hello world". So since we changed the message earlier, we can just change this and get this test passing and then pass this interview,'" says Clement.
Why this new model for technical interviews is better
The new model surpasses the old model because we created realistic scenarios that reflect what it's like to actually work for GitLab, and we established a more consistent method of measurement.
"So we're able to get better candidates overall. Candidates that pass through this technical interview, we're sure that they're going to be successful at GitLab," says Clement.
By designing our technical interviews this way, we can ensure that the interview project matches our actual product architecture at GitLab, which in this case is Ruby on Rails for Vue JS.
We also struggled in the past with finding a good way to check that the candidate knows how to use Git, and can navigate pipelines and testing. By using GitLab for interviews, we're able to confirm a candidate's competency with Git implicitly by evaluating their performance on the technical interviews.
We wanted to mirror the actual experience of troubleshooting a broken MR while working at GitLab, so we allow our candidates to use the internet during their technical interview. This allows the evaluator to see how the candidate solves problems and see their resourcefulness.
"If you're already using GitLab for your tooling, you're just exposing them to what it's like to work at GitLab; it's a more accurate representation," says Clement. "And you can also make sure you're measuring testing proficiency and you make sure they understand how that works before they join your company."
Four key takeaways from our technical interview update
Whether or not a company uses GitLab, there are a few key lessons that we learned by iterating on how we conduct technical interviews for engineers.
Make technical interviews as much like real work as possible: Nine times out of ten, an engineering manager isn't going to sit back and watch an engineer break a sweat in a live coding exercise, any more than they will watch on as an engineer builds in UI. Create realistic scenarios based on the actual work and evaluate based on the candidate's performance.
Make any technical interview process "open-book": Engineering doesn't involve much rote memorization. Instead, allow the engineering candidate to use the internet (and in our case, the GitLab Handbook to look up their questions. It's better to see how a candidate applies their knowledge and troubleshoots the inevitable problems that may arise.
Standardize your rubric: However the technical interview is done, make sure that the rubric is as objective as possible and that the candidate is evaluated based on various criteria, not on their familiarity with a particular technology. A strong rubric means a stronger, more valid method for evaluating candidate performance.
Create an inclusive process: Think critically about how the technical interviewing process and evaluation is structured so a diverse group of candidates can be recruited and evaluated based on their merits. When in doubt, ask a diversity, inclusion and belonging expert or turn to your human resources team for advice. Still coming up empty? Hire a diversity consultant; it will be worth it.
Interviewing at GitLab? We encourage you to use the resources GitLab creates during your technical interview. We don't publish our evaluation criteria publicly, but we do have the Periscope dashboard which can provide some insight.
Watch Clement's talk from GitLab Commit San Francisco to learn more about how we used GitLab to power our technical interviewing process.