Episode 36

Artjom Hatsaturjants on Talent Acquisition in the Age of AI and Challenges & Strategies in Hiring for Technical Roles

TIM: Great to have you here on the objective hiring podcast. Thank you so much for joining us.

ARTJOM: Thank you very much, team. Glad to be here.

TIM: I'd love to start with a topic that I find really interesting, which is around talent acquisition and HR hiring for technical roles. How well equipped do you think they generally are to hire for some of these more specific data roles? Have you seen any challenges in the past in talent teams effectiveness in doing the screening, especially for roles that are so far outside of their own skillset? I'd love to hear your thoughts there.

ARTJOM: I think there's been quite an explosion of data-related roles and insights-related roles. I wouldn't want to say as well, which sort of sits on the border of hard data and hard kind of data engineering and a more consulting type, where in both cases you need to understand the hard skill sets that a person needs to possess in order to hire for that position. and I think there's been really a disconnect between what the both internal hiring teams or even recruiters have been able to deliver in terms of good candidates or a pool of candidates where you are able to deal with the volume of CVs because quite often you'd advertise for a role Maybe a recruiter or your internal talent team will send you a million CVs that are in the ballpark. Kind of. They tick a few boxes, but you can't really make sense of Is that person good enough or not just from this CV? So there's definitely been a challenge in that regard. Because people are not quite as equipped to parse those CVs or even express their own skills in a way that's easy for people to understand and say, Yes, that's my guy or my gal.

TIM: Yeah, you're right; there are so many different issues with that CV screening step. I always think of the CV as like a pretty weak data set to decide who to take to that next stage. I guess you don't really have much more than that normally, and I do have some empathy for talent teams and recruiters because I try to put myself in their shoes. and if I was trying to hire a role that was deviations away from my skillset, for example, lawyers, I would have no clue what I'm looking for. I would at best ask a friend who's a lawyer what I should look for, but I wouldn't understand the nuances between X kind of law and Y kind of lawyer. The distinction between, I don't know, a data engineer and a data analyst to us might be pretty obvious, but to someone in talent, it's just there are just words, really.

ARTJOM: Yeah

TIM: Are expectations over what they could do almost too high? Like, how can people completely outside of a field really recruit in that field? That must be very difficult, I would have thought.

ARTJOM: Yes, and sometimes it can be easy for the talent teams because maybe a job requires a particular data stack, and that's all it requires. Yeah, and they tick the boxes and say, Yes, that person has that on their CV, so let them put them through, but even if they have the tech stack, you never know. Okay, what skill level do they have? Have they been able to do left joins or right joins in SQL, or are they a SQL master? What sorts of technologies have they been exposed to, and at what level of seniority or level of experience? So all of that, the CV can't really tell you. and one thing I got to feel for the candidates themselves is that not everyone is a good storyteller, so not everyone can actually put together a CV that actually puts their skills forward in the best way, so I just got to feel for people who get overlooked because a recruiter or a hiring process just glosses them over because they've had a They haven't put it exactly right on the CV, or maybe if they had a conversation with a recruiter or even with a hiring manager, and the person just couldn't get that across, that they have that particular skill or they're the right person for the role, so it can be really tricky for both sides.

TIM: Yeah, absolutely, and I wonder whether this problem is being exacerbated now in a way with ChatGPT because I hear of a lot of candidates obviously using it to improve, optimize, or whatever you want to call it, the CV, especially against the job description, and so the feedback I've heard anecdotally a lot in the last few weeks is that now a lot of the CVs seem to look really good, as in there seems to be a high match score, if you will, against the JD, but then probably that might not represent the reality of the candidates. So I wonder whether the CV might end up being even less valuable than it was, which was not very valuable at all in my view.

ARTJOM: I think that's almost the level one of kind of the new world level two, in my opinion, is when, for example, they are set a technical task and the second stage, or the first stage, or a presentation they need to do, and they would use ChatGPT or similar technologies as well for that. I think that's quite prevalent, and people don't talk about it because that kind of depends on the entire hiring process and the way things have been done. but I think that's what recruiters need to deal with: they might be hiring a person who has had assistance, which is not a bad thing in itself, because I would assume that engineer would use tools in their career once they get hired, so nothing is wrong with using tools to help you, but it's the distortion of the skills in the hiring process and the assessments of those skills. That is the problem.

TIM: And in terms of ChatGPT in the hiring process or any other such tool, do you have any views on any limits where a candidate shouldn't use it? So, for example, is improving their CV fair game? Is using it on a take-home test fair game? But maybe in a live interview itself, if they were sitting there interacting between an LLM and talking to you, would you find that almost like a bridge too far?

ARTJOM: Certainly that example is a good way to describe it. If they were set a technical task and they were using an automated system like ChatGPT to give them answers, that's certainly a bridge too far for me. I don't mind if someone uses LLMs to maybe polish the language or just check kind of summarize the requirements of the AG to make sure that they haven't missed anything. I think that's fair game, but it's certainly a bridge too far if they use external tools to that extent in a test because is that really an inaccurate and fair evaluation of their own skills, or is that just a tool helping them out? I don't know. I don't know exactly, but it feels a bit on the edge to me.

TIM: Yeah, I agree, and I don't have a clear picture myself in my head. I guess that's what happens when there's this kind of fast-moving, groundbreaking technology; it's hard to know what to think about it. We were hiring some people recently into our team, and as part of the evaluation process, we had a question in there that was open-ended, and it wasn't really a test question. It was just a question that we wanted to get an answer from them on, and it was something like, Imagine it was your first day in Alooba, like you got successfully hired. What are the three things you would need from us to be as successful as possible? So the question I was writing was very specific; like, I wanted to know that individual's exact view on this because I wanted to tailor an onboarding program to that individual. What I was struck by was how many candidates use ChatGPT, even in that very crude copy-paste way. and I that annoyed me I have to say, because it was like I don't give a shit what the large language model thinks, I can send the question into I'm not evaluating the question, the answer to the question. I want to know what you personally think. And so I feel like that was a bridge too far personally for me.

ARTJOM: Certainly that feels like it's not a gotcha question. They're not trying to get you; they want your honest opinion; they want your personality, not the LLM. Yeah, I agree that that should not happen, and it's such an obvious thing not to use ChatGPT on; just write your own thoughts. You're like, you're not going to fail on that. In my opinion, if I were a hiring manager in your position, I wouldn't have failed the person if I disagreed with something they wrote there. It's just their opinion in the end.

TIM: Yeah, exactly. What about the bigger picture now? So if we talk about hiring challenges for you right now, what are the biggest challenges in the market in hiring great data professionals? Do any spring to mind? I think it's aligning skills and the volume you have in the market; you have great talent out there, and they are available. They're ready to go; they want to work for you, but finding them in the great sea out there is quite challenging. I literally get hundreds of CVs for roles that I've advertised with very low quality on average.

ARTJOM: Sifting for that is a big challenge; obviously, that's what the talent teams are for. That's what recruiters are for in some cases, but not all companies can afford that. I come from a background where I work mainly for smaller to medium-sized businesses; a lot of them cannot afford to have a talent team. A lot of them are using LinkedIn or similar platforms to recruit people just because they want to save a bit of money on the recruitment process, so sifting through those hundreds of CVs and then finding the right candidate is incredibly challenging because the market is so open. there's lots of talent there, but there's lots of people There's lots of supply, if you will, out there, and finding the right person is very difficult.

TIM: You mentioned mainly hiring into SME-sized companies. I'd love to unpack that a little bit more and understand more of the specific, let's say, challenges in smaller businesses. One thing I'm interested in is for the candidates coming into the roles, often for smaller businesses, the kind of scope of a role might be a little bit vaguer; they might have to wear more than one hat. Can you talk through that relative to, let's say, a larger organization where normally the roles are a bit more defined?

ARTJOM: That's been my experience personally as well. When I came into my last company, I had to take on the entire insights and data function essentially on my own. I had support from contractors and agencies, but in the end, I was the master of that domain. speak exactly like the one-man band So I had to wear many hats. I had to support the white team with things that were maybe outside of my initial remit, and that's been the case with lots of other people who would join the team or who would come in as contractors. Yes, you are hired maybe for a specific task, but you are sometimes almost expected to support other projects. A lot of times when I worked for SMEs, the budget was there, but the resource was not because the company could not sometimes afford to hire full-time employees, so they would rely on contractors, and sometimes contractors are hired for a certain task, or sometimes they're hired to plug a hole in the team that they couldn't have a full-time staff member for. Yes, it is a challenge where JTs can be muddled, and someone would be hired for just general insights and data wrangling, not for a specific project that requires specific skills. You do need to be a bit of a generalist and willing to take on more than maybe was in the job description. if you're willing to succeed in that company

TIM: I would love to chill down a little bit further than on any other advice you would have for such candidates. You just mentioned a couple there, like willingness to take on more. I assume a growth mindset is probably going to be helpful because you're just going to have a wider kind of role. Are there other places where maybe a typical corporate kind of candidate who's worked in only massive companies might fall down in the hiring process where you might commonly think, Oh, they're not going to be a great match for this smaller environment?

ARTJOM: When you are a contractor and you are joining an SME for a specific project, which has been an experience that I've had over the past couple of years, you are hired maybe for a specific task, but then your role will grow in the process, and you might actually be hired for a permanent position if you succeed in that. I'm just going to do it. You just have to almost think for yourself that I'm here to help this company in whatever it is. Yes, my skill set is inside data analytics, kind of data science engineering, etc., and that's will be my primary focus, but how can I use those skills to help the entire business? and that's where people need to be strategic and commercial. Yes, they're coming from a corporate background perhaps, and maybe they're used to doing kind of their small thing, but in a small business you have to think about, okay, what's the bottom line for the entire business because any small business will tell you they are this close to getting completely wiped out. So everyone needs to put in an extra effort, and I think a lot of corporate people who are able to master that one-for-all and all-for-one mindset Can actually succeed in those businesses by then transitioning to a full-time role I've often had in my business contractors join and then take on a full-time position that was created just for them because they were perfect for that specific position that wasn't advertised initially, but we found out in the process that this is what they're great at, and that's how they can best support the company.

TIM: That's a really interesting transition then from consultants to a full-time role, and I guess that de-risks it a lot from the company's perspective because there's now no longer a question that the person could adapt to a smaller business because you've just seen them work on a project for six months or whatever it is. You already have that evidence; you've already battle-tested them in the field, so

ARTJOM: Yeah, and not everyone wants that; some people want to do a 6-month or a 12-month project and then go do their own thing, but some people do grow into a company with an SME. I think there is a bit of a family mentality sometimes that you get used to people, and you start liking people that you work with. so you get attached to the company as much as to the skill set or to the role, and you end up growing into a role. That is full-time. There is a pitfall and a danger there of kind of you—you know how recruiters tell you that your coworkers are not your family, so you never should assume that but you do end up having feelings of closeness to people because you've sweated and toiled alongside them, and you've taken on tasks that are not in your remit, but you've done it because you love the work, you love the company, you love what it's doing, so you end up achieving more for the company than you would have if you were just a regular corporate staff member.

TIM: Are there any people that you can think of in the past that have been hired either by you or someone else from a kind of bigger corporate type of environment, and it didn't work out for them? And if so, can you think of why, like what was the lack of adjustment? Was it that they were still too fixated on just their narrow set of tasks and they didn't have that wider view of the business? They weren't as quick; they didn't like it; they didn't trade off quality for speed enough, or was there any particular pattern?

ARTJOM: I think in a lot of cases some people are looking for a transition and maybe a slightly different environment, and they were tired of the kind of big corporate juggernaut that they were working at previously, and they think that an SME or a more and maybe a founder-led business is the right place for them, but it turns out they wanted still that growth, the career growth. They wanted that kind of progression that was not just available in a smaller business because it really depends on the type of environment that they're put in. I've had experiences where people have come in and they were just frustrated by the fact that they might never progress on the career ladder beyond where they currently are. They will keep doing what they're doing, and it's up to them almost to find a new meaning for the role because the business can't really deliver that to them. They can't be promoted to a level above because there isn't really a level above. It's a small business, but they need to essentially reinvent themselves within that business to say their skills are also good for that. and that's what else I can do, and let's think about where my career can take me if I take on those responsibilities and pivot myself to that so they couldn't adapt quite from that perspective.

TIM: So it's that there might've been opportunities, but not the ones they were thinking about in this linear way. I'm just going to go up the ranks in my exact function. If they were thinking more widely, they might have seen opportunities, or if they had a more proactive mindset, they might have been able to have more success.

ARTJOM: I think so as well; like in a corporate environment, it can be quite predictable. You join as a data analyst; you progress to maybe senior; you progress maybe to principal, etc. The same with consulting; it can be quite linear and quite predictable where you stand and how your career develops. In a founder-led business or a smaller business in general, you need to make your own niche. You almost need to say, This is my skill set; this is what I want to do with my skills, and be proactive about it. Talk to your line manager; talk to the business; see how you can adapt your skills and your aspirations. It won't work out for everyone; sometimes some companies are just rigid, and there's nothing to be done about it, and then you just need to make a decision that this is not the right place for you, and there's nothing wrong with that. Honestly, we've had those situations, and I wish those people well, and this was not the right place for them. and they were amazing insights people but just not suited for the company I worked for, but sometimes you can reinvent yourself, and that's how you make it work.

TIM: Yeah, life is too short definitely to stay in a role that isn't the right one for you, and yeah, it's a good mentality to have no hard feelings; sometimes it's just the wrong match. It's not that nobody's done a bad job; it's just not the right fit for either party. What about when it comes to hiring, especially into SMEs? What's your general approach? I can imagine a spectrum from very data-driven to pure intuition. Where would you place yourself, and how do you think about hiring in terms of either gut feel intuition versus data?

ARTJOM: I'd say it has to be a bit of both, and I tried to split the two. I tried not to get opinionated or intuition-led on skills on hard skills if it's a question of does the person possess those skills. I want to be a starter-driven person and as analytical as possible. I think some things you just can't do, or you can't, and you can gauge a level of experience and the level of kind of this is how much I can do with that topic, but you either can or can't, and I'd rather be super straightforward and super analytical about it; however, that doesn't mean there isn't a level of cultural thirds. and does that person actually work well within that environment, which is a separate question, and I prefer to have that as a separate stage in an interview where we've established that person is a good technical fit, they would work on that project, and they can do it, but it doesn't mean that they will be happy in it or that the company will get a good result in it because they were unhappy. If a person doesn't have a cultural fit, we need to establish that, and that's harder to evaluate objectively, and you just have to use your own experience and sometimes gut feeling to do it. I don't want to disadvantage any candidates by just saying I don't feel like he's the right fit. and because that's unfair to a person, there needs to be still a benchmark. Okay, what are you measuring? What are the soft skills that you're trying to evaluate? They're separate from hard skills, but still, they're skills, and it is harder to measure objectively, and I guess we need to, but I personally, for myself, haven't figured the way to do it. I just know I need to keep something separate.

TIM: And so for almost like a combination, then, of the soft skills and the cultural fit, are you evaluating that in an interview, and if so, what does that look like? Do you have certain questions that are trying to evaluate different things, and then you're expecting certain answers? Like, how do you evaluate whether it's with or without data, or how do you think about that?

ARTJOM: I think in most cases, or at least the way I've done it, is with almost like case studies, so I would ask a question about what the person would do in a situation X, Y, and Z, or have they faced a situation like a challenging situation where they had to deal with a certain problem. I know from personal experience that questions like that can be a bit vague and annoying because of that, because they can be really broad. So I try to drill it down to maybe a very particular kind of situation but still something that anyone can encounter, so something like, Have you ever had an experience with a difficult client where the onboarding process was really smooth, but then you run into kind of operational problems where people down below something like that? It's broad enough that almost anyone coming into this job would probably have experience with that sort of situation, but it's still specific enough that they can give examples from their career, or even if they worked in a business where they wouldn't have encountered that sort of situation because they maybe worked in the back office, not in the client-facing I want to hear their thoughts. How would they deal with that situation? I know it's not ideal; it's not 100 percent subjective. I would still gauge their response based on my own experiences and what I would expect to do in that sort of situation, but giving those kinds of case studies or examples is probably the best way I've figured out how to do it. Perhaps there's a better way. I certainly would welcome anyone who can figure out an objective way to gauge cultural fits or attitudes. One thing I've learned is that you can teach skills; attitude is something that's essential, and it's a lot harder to teach, and evaluating attitude is essential when you're hiring, whether it's for any role, data, marketing, place management, or what have you.

TIM: Yeah, a hundred percent, and I feel like part of the challenge with evaluating something like attitude—let's call it that—is we're a bit reliant on the candidate giving us examples in this kind of behavioral interview style where it's where a well-practiced candidate might come up with reasonably convincing answers that might not necessarily be representative of how they would really act. and so one conundrum we're always in is like actions speak louder than words. I'd love to see how the candidate actually acts in real life, and so then you're always thinking about, like, the ultimate interview would be to get them in to work for a week, but unfortunately it's very impractical. and always thinking about are there ways we could hack the interview to somehow force them to behave in a real way as opposed to give us stories, which I've got one interesting example to share of something we did that was along the same path we figured that for us the single most important thing we were interested in a candidate was their ability and willingness to learn new things Because we're a startup, we thought we had no idea what they were going to be doing in three months; it could change drastically, so as long as they're open to change and they can change quickly, then probably they can figure out if they're clever, so for our software engineers One of the projects we gave them was to build a little algorithm. Nothing that complicated, but in the programming language, they like the statistical language because none of them would know that because they're software engineers. and so what we were looking for was like, Oh, what is their reaction to this? Are they going to say, I didn't give a shit about this; I don't want to learn R ? That would probably not be a good response, or they just can't do it, like they just can't wrap their head around a new language, which also would be a problem because it's not that complicated. They could probably have Googled it; Stack Overflow put in a bit of effort, and yeah, we found that as a fascinating way to unlock their true behavior. The problem is that's quite unscalable, and I don't know of a way to repeat it across some of those other values as other cultural fit things. like we have eight values that we want to select for I only figured out one way for one

ARTJOM: You're ahead of most people, let's say, with trying things objectively. I think that's the key thing. You do want to test the real-world scenario, or almost like a near real-world scenario, but you need to scale it. I mentioned receiving hundreds of CVS, and maybe you whistle those hundreds of CVS to a dozen Kind of a list of candidates and maybe several B lists as a backup How do you then evaluate those couple of dozen in an objective way that doesn't make you spend a lot of time? I think that's the challenge for a lot of recruiters and a lot of hiring managers because, in the end, that's an advanced level of process that's probably stage two or stage three, so you are at the hiring manager level where people have even less time to spend than this. So how do you do it? It is a great conundrum, and I hope someone will solve it.

TIM: Yeah, the ultimate way to measure things that are subjective. Yeah, it sounds challenging. I feel like for a lot of companies, a fairly quick win would be just having a matrix, like, just what are the skills you want, how much are each of them worth, and then some interview questions for whatever those things are that you're evaluating and then scoring the candidate on a scale of, I don't know, one to five or whatever. Even just that I feel would be a fairly drastic improvement on the way most companies do it because then at least you'd then have some numbers to look at, very subjective still down to your opinion as an interviewer, but like you're the expert, you're in the best position to evaluate them, so I feel like that's Yeah, at least then you've got that comparability as well among the candidates. You can say this one's really strong here and weak here. This one's strong here, weak here, et cetera.

ARTJOM: I think that's a great point because in my experience with some hiring managers, they sometimes tend to evaluate different candidates differently. You feel slightly different on that day, and you just heard something that candidate said, and you just said, That's a kind of deal breaker for me. I'm not going to go with that, or that's not strong enough; it's something that maybe some skill set or some mentality that they've put forward that you said, Okay, that's just not going to work for me with another candidate. They've put emphasis on something else, and when you have those uneven scales, it's hard to evaluate candidates fairly. So I think the matrix you proposed is a great tool to make it a bit fairer to the candidates but also a bit more standardized and maybe predictable for the recruiter or a hiring manager.

TIM: Yeah, I think so, and to play devil's advocate, when I've spoken to people about having this kind of structured interview approach, the feedback I've sometimes gotten is they've seen examples of where that's maybe gone too far, where it's almost like a robotic box-ticking exercise. Someone I spoke to last week had gone through a process at one of the FAME

ARTJOM: Yeah, because they dictated exactly every single question, and those questions have been leaked on the internet.

TIM: Once they're actually in the interview, the recruiters are like, I meant to ask you these questions, but everyone already knows what these are, and this is the box-ticking exercise. I'm going to ask you something else, so some kind of loop where the processes end up breaking, but apart from that rare scenario, I feel like, yeah, if you have just a structured set of questions, then just everything becomes so much easier. Because, yeah, you get that more of an apples-for-apples comparison; maybe it's not perfect, like each interview is still going to go down different paths, but if the general structure is the same, I think it makes it

ARTJOM: And it doesn't have to be equally weighted if you're testing, like if your measurement KPIs are hard skills, soft skills, maybe cultural fits, et cetera., and you want to put more emphasis on the hard skills that you need to, like a hardcore data engineer who would have kind of no interaction with the team; it would just do that six-month coding project. That's it. You can just weigh it towards that; it doesn't have to be equally distributed across all roles. You just need to apply some kind of judgment to what's important to you in your company when you're hiring a person. Is it having someone really personable, someone with a can-do attitude, someone who may not have all the technical skills but is willing to learn and has that kind of like mentality? That kind of will help them succeed. Maybe that's more important for your business than having some kind of knowledge of SQL coding. Which is, by the way, a thing. Apparently, you can code in SQL, which is amazing, but that's another subject altogether.

TIM: You don't count SQL as coding as it's too much scripting. I've seen some interesting programs built in SQL over the years by wizards.

ARTJOM: It has to be literally a wizard, or at least from my perspective, I've always only used it as a table querying tool, but hey, I've never considered myself a data engineer by any means.

TIM: One other thing to touch on here is I feel like there's value even in just the initial approach of getting down on paper what exactly you're looking for; even just that process itself, I think, is almost a forcing function to make you really think about what you're looking for because without that I've seen so many hiring processes that will get derailed because a candidate will get to some stage where some other interviewer is looking for something that's a bit different, and they'll come in and have their own lens of what they think the analyst should have or whatever the role is. and if you haven't gotten that all on the same page initially, then it can just collapse so, so quickly into chaos.

ARTJOM: Yeah, exactly, and the other problem that I've unfortunately faced with both recruiters and myself is that obviously recruiters wouldn't be able to exist without automatic talent systems and AI-based filtering of all those hundreds and hundreds of CDs, so how objective is that in your opinion? Is there a better way to sift through that initial box-ticking exercise?

TIM: Yeah, so at the moment, or at least maybe until the last few months, I have heard of quite a number of companies spinning up their own hacked-out Claude or GPT screenings, so let's park the last few months until then, as my understanding is that 99.9 percent of roles would have been a human looking at a CV as a manual first step for maybe 5 to 20 seconds and making a yes-no decision based on probably some information they got from the hiring manager or job description and some other criteria. I feel like that process is so flawed on so many levels that I could easily imagine AI-based solutions that are better at that on numerous dimensions, and I could suggest a few. One is speed; obviously, you could automate it as soon as the application comes in, and you could score it. So you're not going to have the scenario where the talent person's off on Monday, a candidate applies on a Friday afternoon, and you have to wait until Tuesday to hear back, like just basic time-saving Transparency as well, I think it is very easy for an LLM to grade a CV comparing it to a job description. You can just ask it. Score it against these criteria, give it a score out of 10 across these 10 things we're looking for, and then calculate an overall score. That is already possible right now, and so that could easily be used internally to then have history over why you'd screen candidates in certain ways, and it can be shared with the candidates themselves, who are always complaining about a lack of feedback. I think based on current technology there should be, in theory, a much better AI screener of the CV and application data than humans. And it's just inevitable that it will be introduced, I think, if for no other reason than because the volumes of applications are now so high because candidates are using large language models to apply. It's a game of cat and mouse, and I think companies are going to have to do it, actually.

ARTJOM: It's almost going to be LLMs checking LLM work against their own kind of criteria, but yes, I agree that more objectivity in the scoring is definitely better because that hiring, sorry, that recruiter or that talent manager might have hundreds of CVs to look for, and they are human. They will get tired; they will overlook great candidates just because either that candidate forgot to mention something on their CV or they haven't put it in the right way that kind of directly spoke to the talent manager, so definitely using automations, I think, is a great way to do it to maybe make it a bit more objective. And better for the companies because in the end they will get that talent that they might have overlooked, or they might have filtered out the people with CVs of lower quality where it's just not worth it to have that conversation with them.

TIM: Yeah, a hundred percent. I'm very confident that it should work better than the current scenario. One thing that I'm wary of is in the current kind of discussion around AI in general, there's a lot of fear, which is fair enough because there's this amazing technology that could be used for good or evil. and it's right that there are concerns; however, the point I feel like I keep making is that the current way things are done is so bad across all the dimensions that people are complaining about with AI that that's the current scenario: it's biased massively, and there's no transparency over who gets rejected and why. That's the current scenario, so it's hard to imagine how we could make it worse in my view.

ARTJOM: Yeah, and the point you raised about the ability to automate feedback I think is amazing. It's always so gratifying as a candidate to receive that no, but not just no, but why? What is it about your CV that didn't work for this role, and not just the standard boilerplate, We found other people more suitable? So that is a great way to actually improve the experience for both candidates and companies.

TIM: Yeah, exactly, and that is currently technically possible. What is interesting is to see where the legislation might go, so I know in some markets, for example, in New York City, Even a city has its own specific legislation around automated decision tools in employment, so now you can imagine how many different laws there are around the world that companies now have to adhere to. and so there are a lot of complications with using any kind of automated tools, which hopefully will get worked through over the next couple of years as we start to understand how these things work, but certainly from a technical perspective, I don't think there's any barrier at all to doing this currently.

ARTJOM: Yeah, what about the next stage in the interview process when you test for actual hard skills, like someone's knowledge of coding in a particular language or someone's knowledge of a particular tool? How can you possibly automate it? Because there are so many tools out there, like the company might require a very particular tech stack. And yes, a lot of these things are standard, but you need Power BI, you need SQL, you need R, you need Python. Maybe that's standard, but some might require something completely different because they use a very specialized tech stack or their client wants something very particular. How do you test those skills in an objective way where then either an automated system or a semi-automated system with some human involvement?

TIM: Yeah, that's a good question. So we've played in this space for a few years, and the way we approached it initially was, well, to your point, There are so many different tools. Like how many different visualization tools are there? And then imagine any other layer of the stack; suddenly there's thousands of tools that a company might, in theory, want to test a candidate in. So it'd be unscalable for us to develop tests to cover all of those things. So instead we went one level up, abstractly, to more like the concept level. So testing those data management skills as opposed to a data management tool, for example, has worked okay. I wonder whether now with large language models there's the opportunity to generate questions to cover many skills very cheaply, like the economics of content generation have changed that maybe now it's possible to bang spin up 10,000 questions for 10,000 tools. That's something we're going to investigate, and if that is the case, that would be maybe a way to combat that at scale.

ARTJOM: I think you make a great point about abstracting some of the testing. One great way that I've personally employed and had employed against me, or where I was a candidate, for example, was a presentation-style case study where you're given a task that mimics a real-world scenario. like you are before a client and you need to make a presentation on a certain topic and you're asked to propose specific methodology, specific analytical tool sets, and specific deliverables and present it all to a panel as if they were a client, so that's almost testing in the real-world scenario. How would you behave by combining all the skills you have, both technical soft skills and client management skills? in a single package it probably does warrant a kind of a further stage almost like a final or a semi-final stage in the recruitment process, but I think that's a great way to test kind of candidates when you maybe are selecting between two or three.

TIM: Yeah. for sure, and I'm assuming for this role it was like what you were doing in that case study was representative of what you would do in the role, like you would be dealing with clients, so in that case that seems like a great test to me because the closer you can get the interview process to the actual job, surely the more predictive it's going to be.

ARTJOM: You would be presenting to essentially your line managers and the head of department, so it's senior people. It would simulate the level of seniority where you will be presenting to a client, but it does create a situation similar to what you would be doing in the real-world scenario. Obviously, not every data analytics person would be client-facing, so you need to maybe devise a similar scenario that's internal. You would still be expected in most roles to present the results of your work and defend it, showing people how it would work, so you can devise those kinds of scenarios that would simulate the way you would be doing business within the company from a holistic point of view, both the tech stack, the presentation skills, and the kind of interpersonal skills, if you will.

TIM: Yeah, and one interesting aspect of this that might be helpful for people who are concerned about candidates using maybe too much chachibity in the take-homes is that you give someone a take-home test, and you're worried that is this really them who has done this? What am I really getting? Having that follow-up stage where the candidate's presenting their work and you're digging into it makes it abundantly clear, very easily, how much of the work they've done themselves. because even before Chachapiti there was always the question mark of, Oh, have they done this themselves? Did they get their friend to help them? It's the same now, but just everyone's got a friend called Chachapiti, and so if you, as a hiring manager, can then really drill into their thought process and why they did certain things, either they will be able to explain it or they will crumble very quickly under the pressure of having outsourced the entire thought process to a large language model.

ARTJOM: Yeah, and I guess you need to be fair about it just because different roles require different levels of external exposure, and not every analyst or data person is, in terms of their personalities, used to dealing with outside people, so you need to deal with different kinds of personalities. Some people are neurodivergent, so you need to be able to accept them as well and fit them within your organization. Not everyone will work for that kind of scenario, but I think you can devise those kinds of case studies where you test holistically not just the tech skills but also everything that goes around it.

TIM: Artyom, this has been a wide-ranging and interesting conversation. I hope you've enjoyed yourself. I've certainly enjoyed chatting with you. Thank you so much for joining us today.

ARTJOM: Yeah, thank you very much, Tim. That's been a great conversation, and I come out of it with lots of new ideas and new thoughts. You've given me quite a few things in terms of inspiration for how to potentially approach roles, clients, and candidates, so definitely a great conversation, and thank you for having me.