I've been doing interviewing and hiring at our technology startup for almost a year now. When we began hiring, I found that I was basically the only person who would do it -- partly because I was the only person who displayed any sort of natural aptitude at judging another Engineer's technical and personal abilities.
It didn't take me long to create my own system which I thought was both fair to the interviewee (whom I empathized with, having been one for so long), and gave me a decent understanding of the candidate's qualities. With a little bit of intuition, and some trial-and-error, I evolved my system.
As the people I had interviewed and hired began to join me in the interviewing process, I got to notice how other people handled going from interviewee to interviewers, and how their outlook on a candidate's performance differed from mine.
This evolutionary process has lead me to solidify the following as my Golden Laws (tm) of hiring. Most people who are experienced with interviewing will not be surprised I think, but when a new person joins our team, I find that I need to explain these rules each time anew, so I thought I'd write them down.
1. Hiring is the single most important thing a company can do because it directly affects all other things that are done thereafter.
2. Hire only people you completely trust to fulfil a given role. Then completely trust those people.
3. Your job as an interviewer is NOT to hire "good" people -- that's an undecidable problem. Your job as interviewer is to REJECT substandard people.
4. The most critical personality trait to look for in a candidate is Honesty. All other important properties flow from that font. Dishonesty in an interview is the Kiss-of-Death.
5. Every thing you need to know about an Programmer, you can learn from having them explain their own code to you.
(And lastly, the Green Interviewer's Trap: Hiring is not a popularity contest. Most interviewees are a nice people.You are not required to hire nice people just because they're nice.)
The last law is the title of this entry because its perhaps the most controversial item. Yet I hold it to be the most essential truth I have discovered about hiring Programmers. So much so, I am going to let you in to my entire technical review process, and how it is centred around item 5:
The Sample
After reviewing the candidate's resume, and mentally making note of potential strengths and weaknesses, I always request a code sample from the candidate, in the following order of preference:
1. Open Source or otherwise Publicly Available Source
Open source is the best way to possibly find out how good a programmer is. A public SCM server has all history and contribution information regarding that project. You can graph a programmer's entire history with that code base to the most minute detail.
Mailing lists have discussions of technical matters, and help show how well the candidate gets along with other people.
Open Source is usually a work of passion, and outside the corrupting influence of bad company management, so you have already factored out the naked element of talent.
2. Redacted Source code taken from a company project (without violating any applicable laws, be careful)
Proprietary (but redacted in order to comply with confidentiality laws) source code is good because it shows what the candidate produces when given a real-world problem in real-world circumstances. Particularly it shows if they tend cut corners to meet deadlines or other practical constraints.
3. School Coursework (if a semi-recent graduate)
School coursework, if not too dated, is good because it shows what the candidate can do when asked to really apply their intellect to a serious problem. It's also an opportunity for the candidate to explain how his coding style has evolved over time, and why he would make different decisions today.
4. Toy Project selected by Candidate
Candidate-selected toy project is good because it shows what interest the candidate has if someone is not over their shoulder telling them what to do.
Do they choose something just difficult enough to meet your requirements? Do they choose something way too hard to complete in a reasonable amount of time? Do they report enjoying the exercise?
5. Toy Project selected by Me
One reason why I neglect this option to the last is that one must be respectful that candidates often are busy people, and they can and will turn down the interview because they feel they have better things to do than jump through your recruiting hoops.
6. Source written by Me
This is fun for showing off your work, and is realistic in the sense that programmers are routinely asked to do exactly that, however its rarely under such pressure situations. It's too easy to have a flustered programmer get thrown off by some minor detail, and thus you end up with incomplete reading of their true ability.
Either way, it is absolutely critical that the code is either hand-written by the candidate (or in cases where external code sources were used, clearly attributed as belonging to someone else), because we are going to ask them to explain its working to us. Clear attribution is critical because it is an unalterable part of our standard of honesty.
The Interview Prep
Once you have a sample of their code, you can of course review the code before the Interview to look at issues of Style, Correctness, Performance, Good Design. This is important. However the real purpose of reviewing the code is to generate questions for the candidate once he arrives to the interview. Ask yourself, "why did this person choose to make these decisions when writing the code?"
The Interview
When the Candidate arrives, I open the interview with a stupid joke or sarcasm. Partly because I often am as nervous as the candidate, and I want to break the ice. I also tell him that there are no "Right" answers in this interview, and no trick questions. It is important for the dignity of both parties to keep the interview cooperative, and to make it understood that you are both on the same side -- after you hire the candidate, you certainly will be counting on their cooperation and good will then!
Next I have a short list of simple, no-nonsense, general questions that I ask every candidate. For example "We are in the technology field, and as we both know, technology changes at a staggering rate. What methods do you use to keep on top of that change? Please be specific."
Next I have the list of questions generated from their sample code with me, and I move across the table to sit beside to the candidate with my laptop. I open their source code opened in a text editor (their preferred on if possible).
I ask the candidate to pretend that I am a Junior Engineer, and he must explain his code to me. As a curious person, I will stop them at various points to clarify the concerns I compiled during the Interview Prep, and put to rest any questions that happen to come into my mind as they are speaking.
What an observer can gather from this simple exercise is essentially everything you'll ever need to know about that person as a Programmer.
- A person who remembers even very old code is a person that was paying close attention when they wrote it, and understood the nuance of every line.
People who have trouble recalling what they wrote either didn't write it, or never understood it clearly when the wrote it.
- A person who speaks confidently and at a decent pace is a person at ease with their code and the decisions that code embodies. Even if the code is bad, a good programmer can justify their compromises in a rational manner.
People who speak in broad generalities, ramble incoherently, or carry off on tangents are more than just disorganized. They are trying to distract you from the fact that they don't know what they are talking about by burrying you in an avalache of verbage.
- A person who identifies their own mistakes and errors is more than just aware of their own mistakes, they are showing an honesty and desire to learn and improve. Similarly, people admit their don't know or understand some facet of the system.
A person who steadfastly insists on issues where you know him to be mistaken are not capable working with others, and should be rejected immediately.
- A person that can turn a complicated algorithm or technical trick into something easily understood by any simple engineer has not just a gift for teaching, but likely a deep and intuitive understanding of domain.
People who cannot relate technical matter without speaking entirely above your level may be brilliant, but could prove problematic if they have to work with others.
That said, even simple body language can tell you a lot about a candidate. One of the most dependable Engineers I've ever worked with failed horribly in trying to understand some source code I had written (possibly because I hadn't written it nearly as well as I thought I had!). Every time I asked him what a certain peice of code was doing, he politely answered "I don't know". What impressed me was the way in which he studied my code so closely, determined to fathom out my design through force of will. To this day his tenacity to reason through a problem completely is the hallmark of his good work with our company.
The wind-down
If I have come to a negative conclusion about the candidate, and decided not to continue, I ask the if he has any questions, then thank the person for their time, and ask them to leave.
If I have finished my walk-through on a positive note, I wrap the interview up with a brief explanation of my company, then invite the candidate to ask his own questions of us in a reversal of the interview process.
A complete and incisive set of questions for me to answer is a sign of a good candidate.
If I conclude that a person is technically competent, the next round of interviews include various members of our team, and are mostly about making sure that personalities match well, and that company culture is maintained.
If any one interviewer registers a "refuse" vote, then the candidate is refused.
Thursday, June 19, 2008
Every thing you need to know about a Programmer, you can learn from having them explain their own code to you.
Subscribe to:
Post Comments (Atom)
12 comments:
Oh my, I'm so glad I've never had to be interviewed by you. Interesting insights though, I've always thought about specific answers I've given during (failed) interviews and wonder what it was that earned me the big red "X".
Why are you glad? I personally think my method is the most humane possible. :)
It's difficult to say why one didn't pass, since it's a game for 2+. There are likely a lot of bad interviewers out there, and judging by stories, there is just no telling what some of them are thinking.
However if someone like me rejected you it's likely an issue of confidence.
No one knows your true abilities better than you, and if you come in with a uncertain appraisal of your own ability to work with us, I am not going to override that judgement lightly.
Best advice I can give to a programmer lusting for a job they're not sure if they can get: write some toy project in the domain in question to build confidence, and show up ready to explain your reasoning.
You're totally right about the issue of confidence, Ryan. It's come up several times in my performance reviews. (Yes, outgoing me afraid of her own code!)
P.S. I'm forwarding this article to other people, I think it's great.
I come back to this post every now and again as it's such a nice example of how software engineers are interviewed. Useful for interviewers and job candidates alike.
Don't forget to praise your company too and how the candidate would do well to choose to come to work with you :
Once I had a great candidate. I had a good feeling immediatly. I asked all those questions and he had insightful response to most of them.
Great candidates have more than one option usually. It's the interviewer responsability to give candidate the wish to work with you. My candidate turn out the offer. Fail.
I completely disagree that a person who can't remember code he's written didn't write it or didn't know what he was doing.
I can easily forget what I wrote in the space of about a week. Obviously it's easy for me to relearn because my thought process is the same, but I wouldn't necessarily be able to tell you what I was doing without studying it first.
As an example, I was modifying some code that I presumed a colleague of mine had written. At some point I went back to the commit logs where I discovered (and subsequently remembered) that I'd written the code myself. I'm glad to say that, up to this point, I was thinking my colleague had done a good job. :)
"I completely disagree that a person who can't remember code he's written didn't write it or didn't know what he was doing."
I understand your point -- many a time I've been unable to recognize my own code!
However, I am assuming that the person has specifically chosen to give you a piece of code that represents his skills, and should at least be familiar with what he's chosen.
If he gives you some code, or the code is something you've selected from an open source project, and he can't remember what some section does -- it's no problem.
However, if he can't remember writing any of it, or at least replies that he can't remember how any of it works, then there are red flags.
There reason why I mention that in this post was that I guy was showing me the code for his doctoral thesis, but couldn't recall how *any* of it worked.
I like how you balance personal and technical skills in your interviews.
I'm someone kinda shy and i don't very like talking about myself but i like talking about my code, how it works, etc. It would give me some points if i would get interviewed by you :p
I'm still young in the field; i don't have much experience with interviews, but i would say that i would prefer to explain my own code than passing a technical test.
Nice article.
@Benjamin
I try to factor nervousness and shyness out as much as humanly possible -- I'm no charismatic movie star either.
Basically I just need to know you can communicate professionally, are not mentally unstable, and won't poison the working environment -- and that's a fairly low bar.
Of course conversely, a charismatic guy can really enliven and bring together a work environment, so if all things are equal such a person might get the job.
Above all else, remember that even if he won't admit it, each interviewer is looking for someone who mirrors their own personality. If your interviewer is an outgoing people-person, he may not be as forgiving of shyness. That said, you're also probably not going to enjoy working for him either, so don't view that as a loss if you don't pass.
Very insightful post Ryan. I learnt something from it.
I have always formed my opinion of people i work with by looking at their code, everything else to me is secondary. This technique seems to take it to the next level , do the check right at the beginning.
My question is - do you insist on a specific "type" of code for the candidates to bring in ? For example one candidate brings in implementation of a wait-free data structure,partially done versus someone who brings in a very well-done implementation of a CRUD application. Obviously one is trying to solve a harder problem. How would you judge in such a scenario ?
Excellent! You are going to get people who can explain what they are doing and why. At least some of those are going to be people who like working with people who can do that.
Hm, I tend to be a bit disorganised and tangent a bit, but it's because I'm really interested in the technology and the possibilities and have learned *about* a lot of them, and am interested in trying them out.
I try not to do too much verbiage, but it does come out - always try and make sure it's stuff people understand and have used, and prefer plain English...
If anything my last boss said I was difficult to work sometimes because of this and being "very technical".
Post a Comment