Why you’ll NEVER nail that DevOps interview

Source – venturebeat.com

Many companies these days are constantly complaining that they are struggling to recruit good talent. There are rumored talent acquisition wars and agreements between large companies like Google and Apple, and I’ve often heard these types of issues referenced in talks. Just recently I heard Battery Ventures’ Adrian Cockcroft say that, while he was at Netflix, large companies often asked him, “Where does Netflix hire such great talent?”, and his reply to the Comcasts and others was that “We hired the same people you hired, we just got out of their way.”

Another person who discusses this constant gripe is Andrew Clay Shafer, Senior Director of Technology at Pivotal and a known figure in the DevOps world. In a 2013 talk titled “There is no talent shortage,” he argued that the talent we need is already in front of us, and we’re just looking for the wrong things.

I’d like to assert that we can’t even know if there’s a talent shortage or not, because the process is completely broken — on two fronts: First, recruiters don’t know what they need to be looking for. Second, job seekers, as a result, simply are not presenting themselves correctly.

Note: I choose to use the example of DevOps for the sake of demonstration (and in particular “DevOps Engineers”) throughout the article, but what I’m writing here is also true for “Fullstack Developers” or whichever title we’ve mistakenly created to find our “technically experienced out-of-the-box any-problem-whatsoever solvers.” Another note: The term “DevOps Engineers” is an irony and a paradox, and I’ve spoken against using it in a job descriptions since it contributes to the delusion that such people exist. You’re invited to read about it here.

The Ops Transition

Throughout the last decade or so, due to the Web Operations blowout, there has been a shift in the Ops paradigm. While system administrators used to have expertise in a specific subsystem in an organization, nowadays, Operations Engineers are responsible for a much wider subset of the production system, and sometimes, its entirety.

Just to make sure we’re on the same page, I’m referring to those superhero 4-6 person teams managing a very large system — from the infrastructure and monitoring pipeline architecture to version deployment, configuration management, and whatnot (pager duty at 3 a.m., anyone?). So we now have teams of five doing what was once the domain of tens of people.

HR Ripples

This shift has caused ripples. Well … ripples might be too soft a word to describe the state of Ops in LinkedIn.

If you’re an Operations Engineer, have Puppet, Chef, AWS, Linux, Docker and, of course, the notorious “DevOps” in your resume, Amazon, Yahoo, and many other Fortune 100s will pounce on you readily. Actually, lots of good companies, whether they’re big and progressive or small, SaaS-y, and cool, are completely emptying out the Operations Engineer market.

Meanwhile, let’s say you just left a decent job at a “Good SaaS Company” to look for something new and opened up your LinkedIn account to update your profile. You threw in whatever you could: Nagios, PHP, Shell, AWK, Apache, Hadoop, thousands of servers, Ruby, virtualization, CFEngine, all known distributions of Linux, MySQL, NoSQL, YesSQL, WhySQL …

But it’s not enough. You don’t have a degree; you’ve never worked with “The Cloud,” and you’re not “DevOps” (whatever the heck that is — at least the last time you checked).

You might be rejected — that’s assuming you get to the interview at all. Why? Because they’re looking for the best, the creme de la creme. They’re looking for… *drumroll please*… The revered “DevOps Engineer.”

I recently attended the DevOps Days 5th Anniversary event in Ghent, Belgium, and as with other DevOps Days conferences I’ve attended in the past, much of the talk was about what DevOps actually is. Tools? Culture? Automation? Continuous Delivery? Collaboration? A Person? A Team? After five years of DevOps Days, five years after the term was first coined by Patrick Debois, we’re still trying to define it.

Here’s a sample of the “DevOps Engineer” in all its glory, found through the job search option on Linkedin:

Requirements
The desired candidate will have B.Sc from 4 years of college or university, at least 10 years of infrastructure development experience and/or training, along with:
– 3+ years of AWS experience.
– 5-7 years of general DevOps/WS/DB/infrastructure experience.
– Practical experience with most, if not all components, of AWS: EC2, Beanstalk, ELB, Route53, S3, Cloudfront, SNS, SWF, SQS, RDS, DynamoDB, ElastiCache, IAM, CloudFormation.
– Familiarity with SOA principles and practice.
– Experience reading, analyzing, and absorbing AWS white papers, architectural documents, and technical briefs.
– Service rollout plus complete multi-cycle SDLC experience, using Chef, Puppet, CloudFormation, etc.
– Exposure to AWS auxiliary tools and practices (e.g., Chaos Monkey, Asgard, etc).
– Hands-on experience with monitoring tools, intrusion detection mechanisms, and vulnerability assessment tools and practices.
– General scripting wizardry.
– Self-motivated team player who demonstrates initiative and flexibility.
– Strong organizational skills, with the ability to handle and prioritize multiple tasks.

Additional preferred skills
Beyond the requirements above, the preferred candidate will have:
– Familiarity with iTunes/App Store fundamentals.
– SQL and NoSQL experience.
– Familiarity with JDBC and various flavors of SQL/NoSQL.
– Experience designing solutions for access control, user authentication, and service security in general.
– Strong verbal and written communication skills.

Being a recruiter, you’d have to rifle through endless amounts of resumes in order to find someone who fits the job description posted.

From browsing through countless resumes on paper and online; from hearing about DevOps in many different contexts; and from looking at hundreds of job descriptions I conclude that we’re not searching properly. I believe that this overloaded term (alongside Fullstack Developer and some other industry favorites) is creating a vicious cycle in which both recruiters and job seekers are missing many opportunities.

I would like to point out several common mistakes and propose some possible solutions to the problem.

Mistake 1: Recruiters are posting job descriptions of professionals who do not exist

“We want the best,” you say. Naturally, everyone does. Since “DevOps” has become such a widespread term which we tend to think sums up CI, CD, CM, Architecture, Automation, Cloud, and any other technical experience, we’re looking for the person who can fill this role. Who wouldn’t? Any organization would rather find and pay someone who knows everything, than five people who know something.

An Operations Engineer, in its modern form is a new profession. It’s been gaining momentum since the cloud and on-demand systems that require 24/7 uptime. It requires the ability to design, build, monitor, analyze and maintain a complete, complex, potentially large-scale and highly-available production environment. Can you become an expert in all of these in just a mere 3-4 years of practice?

Those fortunate people who know so much that they somehow manage to closely match that definition are probably already taken. What’s more, I’d venture to say that you likely wouldn’t be able to match the Google/Amazon terms of employment or provide them with the satisfaction of working for a company that’s changing the world.

“But we’re willing to wait until we find the perfect candidate.”

Pssst. So is everyone else.

“We’re looking for candidates with a degree from a major university.”

Do you expect people to learn to be good engineers who solve real world problems at the university? Is there an institution in which you can learn how to monitor a system? There has been much said about academic institutions lag behind actual technology. It’s nearly impossible for these curricula to move at the speed of the market. I recommend reading Ken Robinson’s Out of Our Minds, where he discusses this gap in detail. He focuses on how universities may no longer provide the real value employers are seeking.

I, personally, studied to be an electrical engineer. Engineers are for the most part taught math, physics, electricity, chemistry, electronics and other disciplines. As far as I know, they are not taught HOW to do their job. They’re given theoretical knowledge and practice, where some of that knowledge may be applied or serve them in the future for doing so. But in this new profession, we don’t even have that. There is no school. Literally.

The common demand that the candidate needs to have at least X years of experience is flawed for so many reasons. Can we truly assert that someone who has four years of experience is better than someone who has two? Can’t someone be really bad at what they do for five years, while another exceptional for two? Should we judge based on quantity or by quality?

Mistake 2: Recruiters are asking the wrong questions in interviews

Considering how broad these new roles are, should you expect people to remember how to configure a GRE Tunnel in Junos, or remember which port Kerberos requires, or remember how to move a zone file between DNS servers?

Mistake 3: Job seekers are presenting the wrong skills in their resumes

Let’s look at John Doe’s resume: Self-motivated DevOps Engineer with over 3.7 years of IT experience in Systems Engineering, Development and Operations. Looking to work on Free/Open Source Software.

Career Summary:

3 yrs of experience in DevOps and Systems Engineering on several flavors of Linux
2 yrs of extensive experience on virtualization and cloud management.
2 yrs of work experience with Agile development methodologies.
Proven ability in leading the teams on technical operations.
Experience in writing the automation scripts with Python, Ruby, C, and Shell Scripting.
Experience on LDAP management and integration with other applications.
Extensive experience on web applications (LAMP) configuration and administration.
Knowledge on mobile platforms like BREW and Android.
Proven ability to deliver sessions and conduct workshops on various technologies.
Technical Skillset Summary

System Administration: GNU/Linux (Debian, Ubuntu, Arch, CentOS, Redhat and Zentyal)
Virtualization and Cloud Services: LXC, KVM, OpenVZ, AWS, Droplets and CPanel.
Configuration/Build and Release: Ansible, Make, Projspace, Go (learning), Chef (learning)
Programming Languages: Python, Ruby (learning) , Shell Scripting, C, C++ and Java
Database and Directory Services: OpenLDAP, Mysql, MariaDB
Version Control Systems: GIT, Bazaar, Subversion, GitLab
Networking Tools/Services: Nagios, ntop, SSH, DHCP, DNS, FTP, Rsync, Squid, Iptables
Web Tools/Technologies: Apache, Nginx, Drupal, Mediawiki, Redmine, Piwik, PhpBB
Even if we assume that John Doe somehow managed to get some decent experience with all of the above in such a short time span, we still don’t know if he can apply the correct tooling to an arbitrary problem. We don’t know if he can solve problems at all. We don’t know if he can even identify there IS a problem.

Wouldn’t it be more useful if, instead of “Experience in: Nagios, Statsd, Graphite, Logstash, Elasticsearch, Kibana, collectd,” John’s resume said “Loves researching and implementing monitoring solutions for challenging architectural problems”?

And wouldn’t it be more useful if, instead of “Four years of experience programming with Ruby, Python, AWK, Shell, Batch, VBS, C, C++, C#, Java, Javascript, JRuby, Jython and everything else,” it said, “Knows Ruby well but am passionate about learning new languages on demand. Here’s a project I’ve been working on: <LINK>”?

Mistake 4: Job seekers talk about the wrong things during an interview

Let’s say you’re being interviewed and the interviewer says: “Here is a system .Please describe the most secure pipeline for sending requests from the application server to the database and the most fitting type of tooling you’d need to implement the solution. Please also provide a step by step documentation of your research process and sources you’ve used along the way.”

Do you think the number of servers you’ve managed or the diversity of IaaS worked with will provide good insight into an ability to solve this problem?

The problem isn’t that tools and experiences are part of a job description or a resume. It’s that they seem to take precedence over potential and achievements. Is experience important? Yes, but how many experienced people can you expect to find in environments that are only just materializing?
Look for passion, don’t ask for it

I’d suggest recruiters venture out into the wild and really find those passionate individuals who care enough to enrich themselves. You’ll know them by their digital footprint on sites like Stack Exchange, or by the fact that they’re taking an active part in the community — anything from attending meetups, DevOps Days, or VelocityConfs to taking part in active discussions online in mailing lists or open spaces. Listen. You might just find what you’re looking for.

I agree with Andrew, and Adrian. I do not believe we’re missing talent, we’re just looking for it in the wrong way. We need to remember that we’re not ACTUALLY looking for someone who has several years of experience with a bunch of systems; we’re looking for someone who will solve your problems.

The word “DevOps” in itself isn’t the issue. I’m not preaching against using such terms in a job description or in a resume. I think that the excessive use of the term is a symptom of recruiters trying to find an easy solution to a complex problem; and job seekers enable this problem by trying to position themselves as ideal candidates based on the wrong strengths and capabilities.