Saturday, June 16, 2007

On the severely misguided nature of programming tests

Part 2: Intermediate Questions

Question 6:

Consider the following function:

void foo(const char* name)

{

char* Name1 = name; // Statement 1

const char* Name2 = name; // Statement 2

char* const Name3 = name; // Statement 3

char const* Name4 = name; // Statement 4

}

Which of the following is true?

  1. Statement 1 fails to compile.
  2. Statement 1 and 3 fails to compile.
  3. Statement 3 and 4 fails to compile.
  4. Statement 2 fails to compile.
  5. Statement 3 and 4 fails to compile.
  6. If you answered any of the above you have no place being around computer code. A person who programs as a profession needs to first ask them self if they need: a variable pointer to something that is itself constant, a constant pointer to something that is itself variable, or a constant pointer to something that is itself constant, then look up the correct syntax in the book. And if you're still worried, finally test it on the compiler -- THEN NOTE IF IT DOESN'T COMPILE! Anything else is pure nerd dick-waving.
And if I ever catch anyone using such unnecessarily confusing, dusty corners of the C++ standard in my code, or my job interviews, I will consider it an intentional attempt to trick the reader, and toss you out the door for being deceptive and a danger to the code-base. Seriously, what sort of programming job requires you to outwit an adversarial co-worker?!

Because, if programming tests are an accurate reflection of the work you will do as a programmer, then it must follow that the programming test itself is an accurate reflection of the kind of people you will be working with, by virtue of being the kind of people who would pass such a test.

Bonus points for anyone that can translate the following from its current wording to some form that would make me care.

Question 15:

What best describes virtual inheritance?

  1. A derived class which does not implement abstract methods.
  2. A derived class which adds abstract methods to a concrete base class.
  3. A multiply inherited class with multiple copies of its base class.
  4. A multiply inherited class with one copy of its base class.
  5. All of the above
ps. (d)

9 comments:

Anonymous said...

If you answered any of the above you have no place being around computer code.

I totally disagree. If you've been asked a question like "Which of the following is true?", then you should give an answer. Responding with a "what kind of question is that?" is most likely going to help prevent you from getting the job. The question above, gives a scenario in which to test the knowledge of const placement, it's not designed to determine if the programmer is stupid enough to write code like that. I would prefer to hire a person who does answer the question rather than someone who complains about being questioned on something they consider to be unnecessary.

While I agree with the idea that in general most programming tests are crap, I really don't see anything wrong with testing the programmer's knowledge of const placement. I also don't see how const placement can be considered a dusty area of the standard, nor is it unnecessarily confusing.

Personally, I'd want to know if someone knew about how to deal with the placement of the const in pointer scenarios, and I wouldn't be offended if someone asked me the above question.

Anything else is pure nerd dick-waving.

That's a fairly typical comment from someone who doesn't know enough about the language and who is embarrassed when questioned on an area they don't understand. The question is about understanding, not about the method you'd use to make your code compile. There's nothing wrong with querying a candidate's understanding of a certain topic.

As a parting note, I find it appalling that you'd complain so heavily about being questioned on const placement, and yet you have no issues with the question on multiple inheritance. To be honest I find this post quite misguided, not the questions therein.

Ryan McDougall said...

oj,

Thanks for commenting.

I did give an answer, but given I had 20 minutes to finish the whole test, I didn't have a lot of time to think about it, and got it wrong.

You're right in the sense that if I had gotten it right I never would have bothered to care, but the fact of the matter is memorizing this stuff is useless, so I don't do it. I've never been good at memorizing things, thats why I own an extensive collection of books.

As for preventing me from getting the job, you're quite right, it likely will. However you may be mistaking just how much I'm interested in such a job. I currently work for a prestigious japanese company whose rigid thinking I'm trying to avoid. I really don't think this is good advice for test takers, as much as its "the truth" that might make sense to the people who devise and use such tests.

As for me, I would rather give a Pre-Screening test that focuses on pure knowledge questions that are important., even if that makes the test "easy". Especially given how quickly "hard" multiple choice tests devolve into guessing: http://unexpectedtruths.blogspot.com/2007/06/fallacy-of-hard-tests.html

In my opinion C++ is a dangerous land filled with areas that compile, but offer little reward for their usage, otherwise why would we need a series of "Effective *, Nth Edition"? My reference was to const char*, vs char* const being the exact same thing. You don't think that is possibly confusing code that shouldn't be in production code?

Personally, I'd prefer a candidate that said "not sure, I'm going to look in the book" over *any* candidate that said "for SURE, I KNOW that const syntax means Variable pointer to Const data!". But there is no option for that on the test is there.

Whats wrong is not trying to find out the candidates competence, whats wrong is thinking memorized knowledge of dusty C++ syntax or the formal definition of virtual inheritance is equivalent to competence, when its just NOT. Period.

I have TONNES of issues about the virtual inheritance question, which is why I added it. I think you misread.

Thank you for offering your honest opinion. Feel free to come back and tear into me again. ;) Better yet, post a reply on your own blog and link to my foolishness. :)

Ryan McDougall said...

Oh forgot one thing: "That's a fairly typical comment from someone who doesn't know enough about the language and who is embarrassed when questioned on an area they don't understand."

You really hit the nail on the head with the problem here. You're deeply confused thinking knowledge of syntax equals with understanding of programming.

I don't "know" the exact syntax rules because what I need when programming is 99.9% "pointer to const data, or const pointer to const data". You really think that someone who frequently uses corner-case code whether they need to or not (this was not a job for embedded programming, it was for a top international bank), is a good candidate for systems that make or lose millions of dollars per day?

Anonymous said...

Hello again,

Let me start by stating that despite me giving a blunt and different opinion on the post, I didn't intend to "tear into you". I was merely pointing out that the post reeked of frustration, and that I didn't agree with what you implied.

but the fact of the matter is memorizing this stuff is useless, so I don't do it. I've never been good at memorizing things, thats why I own an extensive collection of books.

I do not think that it's useless. I didn't 'memorise' the information about const when I was learning C++, through constant use (no pun intended) I got used to the rules. Plus, when I maintain other people's code, or write my own, I do not need to waste time referencing books over and over again because the information is already in my head. To me, it's a timesaver, and it makes me more productive. That, in my view, makes this stuff very helpful, and not in any way useless.

However you may be mistaking just how much I'm interested in such a job.

I hadn't really considered how interested you were in the position. All I had considered was that you went for the position intially and hence were interested enough to make it to the interview process. In my previous comment I assumed, for the sake of argument, that you were interested in pursuing the opportunity further. That may be a false assumption, but since we're having a general discussion on the merits of tests in interviews it wouldn't matter whether I assumed it or not.

I really don't think this is good advice for test takers, as much as its "the truth" that might make sense to the people who devise and use such tests.

Again as I've already stated, I feel that most programming tests are way off the mark. I'm not saying that I'd be happy with a test that contained nothing but questions that fit into the same category as the ones above, but I wouldn't mind having one or two of them for the sake of coverage. Programming tests in interviews generally do not cut it as far as determining candidate quality.

As for me, I would rather give a Pre-Screening test that focuses on pure knowledge questions that are important.

I would agree. In my time I've only seen one programming test that really covers things in enough detail to determine the experience level of the candidate without forcing them to recite the C++ standard off the top of their head. Multiple choice was part of the test, but I found it wasn't the same as so many other tests that I've taken. I think attitude, aptitude and personality are more important than standards regurgitation, and programming tests don't really cover those areas very well.

In my opinion C++ is a dangerous land filled with areas that compile, but offer little reward for their usage, otherwise why would we need a series of "Effective *, Nth Edition"?

That's fair enough. C++ can be a nasty world to work in. Most of the time you're in sh*t, it's just a matter of how deep. But that doesn't mean that it shouldn't be used, as there are cases where it's the ideal language to use to solve a given problem. Finding the right candidate to work in such a world isn't an easy task.

My reference was to const char*, vs char* const being the exact same thing. You don't think that is possibly confusing code that shouldn't be in production code?

Well, that's a tough question to answer. In my view it's important to have const for both the pointer AND the data that's pointed to. Keeping that kind of functionality while retaining consistency across the syntax wouldn't be easy. While I don't think that it's necessarily intuitive that const can appear either side of the pointer symbol, I still think that this feature should exist. Should it be production code? If it fits a given scenario then yes. Though I can't picture that scenario right now :) Depending on the view of the programmer, he/she might want to make sure that a pointer argument to a function always points to the same chunk of data, but doesn't want that data to be const as its contents might be modified. Again, I might not use such a mechansim, I'm just thinking out loud.

Confusing? Partially. Again to those who don't really know enough about this particular area of the language they will have to look up the book. But I don't think it should be avoided for the sake of people who aren't as familiar with the language. If it's used, then perhaps a meaningful comment alongside the code would help when it comes to maintenance? There are a few ways to deal with the problem, but avoiding it for the sake of people of lower level of syntax understanding is a bad idea in my view, as it's not obvious as to where to draw the line between what you refer to as 'dusty' and the features that are, shall we say, 'non dusty' :).

Personally, I'd prefer a candidate that said "not sure, I'm going to look in the book" over *any* candidate that said "for SURE, I KNOW that const syntax means Variable pointer to Const data!".

I probably wouldn't have a preference. I'd be as happy with both options that you've listed. However, if I heard someone say "I have no idea" without stating how they'd find the information out, then I'd be concerned.

whats wrong is thinking memorized knowledge of dusty C++ syntax or the formal definition of virtual inheritance is equivalent to competence, when its just NOT. Period.

Agreed. Competance, knowledge and experience are all different. Again, I don't agree with your use of 'dusty'. It implies that the feature of the language isn't used, but in my experience I've seen it used very often. Sure, most of the time it's const pointer to const data, but the principle remains.

I have TONNES of issues about the virtual inheritance question, which is why I added it. I think you misread.

Then I apologise :) Perhaps the reason I was stunned was that you were so vocal about the const issue but didn't say a word about the inheritance issue. My bad.

Thank you for offering your honest opinion.

Isn't that what blogs are all about? :)

Feel free to come back and tear into me again. ;)

I don't see how I tore into you the first time, and I'm sorry you feel that way.

Better yet, post a reply on your own blog and link to my foolishness. :)

Stating my different opinion doesn't immediately imply that I think you're foolish. In fact until you said that I didn't think you were. I'm not being petty, and you don't need to be either. Let's just discuss the points you've raised, as it makes for an interesting discussion that I think we can both learn from. I am not the kind of person to run off an whinge about you behind your back on my own blog when I can discuss it here with you and hopefully come to a meaningful conclusion from which I can learn something.

You're deeply confused thinking knowledge of syntax equals with understanding of programming.

Actually, I don't think I am. I haven't in any way stated that I believe knowledge of syntax is the same as understanding of programming. Perhaps you can point out where I said that before? I do believe that it's an important tool in the box of the programmer, but I do not believe that the person with the best knowledge of the language/standard is the best programmer. Please don't put words in my mouth.

.... is a good candidate for systems that make or lose millions of dollars per day?

Whether or not the corner case is used isn't really a concern for me. I certainly don't write code that uses the corner case, but I do see it in other people's code. Having the understanding of those cases is a good thing in my view. Would I base my decision on whether they were a 'good' candidate purely on questions like this? Most definitely not. So I think your question is unfair :) Good and bad candidates are just as likely to use those cases, it's other areas of their skillset that I'd base my decision on.

Thanks for responding.
OJ

Ryan McDougall said...

Thanks for coming back OJ. I was mostly being playful when I said "tear into". I did feel offended at all by your remarks. :)

I think you are right there was frustration, but it was more of that "why do people take these things seriously?" frustration.

The recruiter for the bank was keen on me until the after test, but now she has been a little quiet, so at least she seems to be taking it seriously, outside the employer itself...

I appreciate your detailed reply, but unfortunately, you've taken the uncontroversial tack of saying things I all agree with, so I don't have much left to argue with. :)

I tried heading to your blog, but your profile is not public. Is there a way find your blog?

Anonymous said...

Hi again,

Well I'm glad that's clear then :) I do definitely agree with the fundamental point of the post, it's been the bane of a good programmer's existance for many years and probably will continue to be.

Sorry to hear it's gone a bit cold on the job front, perhaps it's not such a bad thing considering the methods they use.

My blog can be found here, for some reason I was posting with my Google account rather than just posting with a generic name and URL. I won't be making that mistake again.

Cheers!

Anonymous said...

Interesting discussion :)

I would have to say that it is pretty vital that you understand the ins and outs of the language you're working with. That includes syntactical weirdness, common idioms, and dangerous pitfalls.

When I'm interviewing I'm not necessarily looking for an encyclopaedic knowledge of every language feature, but I am vastly reassured by it - particularly since it tends to suggest that you have either:

- real world experience
- an enquiring mind
- both of the above :)

C++ is certainly a mess of contradictions and inconsistencies, but if I've got to hire people to use it, I want to be damn sure that they know what they're doing.

I also want to know that they aren't complacent enough to think that because they've absorbed the 25% of most commonly used features, they're done now and can stop learning. The existence of books like Effective C++ is just a reflection that you can't learn everything in one go - there are levels and levels of sophistication. For me you get a point just for wanting to read that book (and another for reading it, another for understanding it, another for remembering to apply some of the knowledge, and further points for having serious issues with Myers' hair cut in the author picture).

Ryan McDougall said...

Thanks Sam,

One definitely prizes someone who knows the syntax by heart, but my point might be that even someone who *doesnt* know the syntax could still be a very good programmer. I think employers are unnecessarily throwing out good candidates if they take such tests seriously.

However, I think one definitely has to bow to the requirements of practicality sometimes, that there are only so many good questions you can use in pre-screening, but perhaps I can suggest that pre-screening tests should only test critical knowledge, and more subtle things should be saved for the interview?

For example, if you value people who aren't content with knowing only the basics, why don't you ask how many computer books they own, and what their favorites are and why? I think that would tell you 100x more than a question on "dusty syntax".

Anonymous said...

* ignores yet another 'dusty syntax' comment *

:)

I stumbled across this on dzone today, thought it was quite interesting. While it's not exactly on topic, it has some interesting things to say about const.

OJ