Interviewing Engineers in the AI Era
What is a reasonable framework for interviewing engineers? How has this changed in the AI era?
Lets start by acknowledging that there is no one good answer for everyone. Startups can express their identity more during interviews. Enterprises are required to handle scale and consistency. Each startup and enterprise has its own quirks.
We also acknowledge that interviewing is the “best available option” and not necessarily a perfect selector. This is why we obsess over false positives and are OK with false negatives as the cost to pay. Referrals can hence be gold, particularly when intrinsic incentive kicks in (“this place is so awesome I want my friends to work here”).
And the last disclaimer is that AI-assisted engineering is still a fast-evolving field, which just means we take the stance of observe and adapt, as opposed to baked and done.
We now methodically understand the pre-requisites and then present an iteration.
- what does an ideal engineer look like
- what challenges in interviewing have we had before AI
- how does AI affect all this
- what could the iteration for interview in the AI era be
Understanding an Engineer
Our goal in assessing an engineer is to “predict their net impact on our org.” The industry approaches this through two lenses: Functional (individual impact) and Leadership (leveraged impact).
It helps to thoughtfully describe what your ideal engineer looks like. Just like we push candidates to be specific, be specific in detailing your own model.
By default, I lean towards the “entrepreneurial engineer.” This is someone who, beyond being a brilliant developer, has figured out markets/needs, product ideation/design, research/rapid pivoting, building for different scales, selling to users/investors, and is humble, happy, and healthy doing all that.
In a successful startup, this engineer represents the essence of net impact. This engineer and Jeff Dean have common core engines, and Jeff was this engineer at some point. Your best engineers belong to this class of skills. Someone who is creating continually growing impact for your org at all levels (self and intra/inter-team). They are not just super strong at Functional, but they are Leading in more ways than one. IOW, your best engineers are “entrepreneurial engineers” who are doing micro-startups for your org. This is also an aspirational goal, a model for understanding, and should not devalue other meaningful engineering participation.
Challenges with Interviewing
Tech screening, coding, SDI, and behavioral are the empirical, pragmatic structures with which we build a picture of the engineer. We call these interview Formats. Functional is addressed in multiple Formats such as coding/SDI/tech-bar-riser, and Leadership typically gets one focused slot (Behavioral). Each Format drills into a specific cluster of Skills. It is these skills that are shifting as a result of AI.
Classic interview formats have difficult-to-solve weaknesses.
TIME: We have limited time with each candidate. Investing 8h in interviews and twice that in logistics/prep is not a bad start, but it is not the same as revisiting your decision honestly after a quarter of working with your new colleague.
PRECISION: The top false positive descriptions are “lack of technical maturity” and “lack of effective leadership.” IOW, “we failed at interviewing for our production and culture reality.”
DRIFT: Candidates have memorized LeetCode. They are using AI to cheat. The SDI no longer represents the skills we need. The candidate is “ahead in gaming” the system.
SUPPLY: Good engineers are few and hard to find.
Challenges from AI
Coding has several properties that fit LLMs: verifiable outputs, strong training data coverage on the internet, and smaller complexity compared to the natural language world of usage. AI coding skills are hence leading commercially. It is important to note that AI skills, including coding, are not perfect, but evolving. Understanding ambiguity in AI is the professional’s starting point. The takeaway is that coding is a natural fit and a massive market.
Since coding is the core truth for engineering, this has broadly affected engineering to the extent of justifying the term “AI Assisted Engineering.” The questions it has raised are: “what does AIAE look like?” and “how do we interview for it?”
AI impact points in Engineering
A quick tour of common impact scenarios we see with AIAE. We outline the effects we have seen, and indicate a direction of thinking that hopefully helps you to build a solution for your org.
AUTOMATION: While Python and Playwright have unlocked automation that much more, coding agents’ ability to write code and validate quickly means automation is a natural sweet spot. Structure and automation are fundamental to an engineer’s thinking. So expect your engineers to automate so much that the actual challenge becomes learning when to stop creating automation slop.
LEARNING: You can learn an engineering topic an order of magnitude faster IF you know how to learn. This is the art and experience of asking the right questions, not skinning the yak, and validating your knowledge. The key here is to practice “goal-oriented learning.” Be clear about what you will do for your org with that learning, and when.
CYCLES RELEASED: This is the core effect of AI that is deceptively complex. Should an engineer write more code with the time released? Should they instead invest more cross-functionally? What is the right balance of org guidance vs. engineer agency? The entrepreneurial engineer would invest the new time and speed in building things that scale engineering, help PMs build better prototypes, and so on. Essentially, going back to unlocking value and increasing net impact.
VELOCITY VS QUALITY: The trend is still “move fast and break things” when it comes to AI adoption. Engineers need to master the art of choosing between determinism and agency. Should your AWS infra changes be chat interactions, slash commands, or tools written and maintained by agents? While the news is high on failures, note that contained failure is likely essential for optimizing your AIAE velocity.
DECISION FRAMEWORKS: Mapping a problem instance to a class of problems, finding the shape of that class, building and validating hypothesis, and pre-defined success KPIs are a decision framework that engineers need to be strong at. They are effectively conducting ‘experiments’, deciding under ambiguous conditions at a higher rate. Strong decision framework practice is needed to translate to productivity.
PAIR ENGINEERING: AI is not just for coding. Claude Code is effectively a pair engineer — someone you can develop/validate ideas with, discuss architectural decisions, and most importantly ask to brutally review your decisions. This also happens to be a good way of building anti-hallucination skills. AI does not have the domain knowledge needed to define your contracts or design your system, but it has the generic capability. So a long arc in your interaction is providing that domain knowledge as you move along, while figuring out how to capture it in tools, skills, and contextual AGENTS.md files. This domain knowledge lifecycle is the broader area for “long-term agentic memory.” While frontier labs continue to help with this, engineering should understand and invest in this, and engineers should build this today. Treat them as assets that are shared, reviewed, versioned, and have a defined lifecycle so that you can scale.
COGNITIVE DENSITY: Working with coding agents requires high cognitive intensity and focus. One of the reasons is the ability to run multiple agents in parallel and play the role of an orchestrator across these sessions. We typically avoid context switching when in the “zone” and at cognitive depth. This is no longer a luxury we can afford. The ironic twist is that manual coding is now a relaxing job that you can only switch to when needed (and it is needed more often than typically imagined).
CODE REVIEW FATIGUE: Using agents for code review, PR merge gates, etc., are already best practice. However, code review is a critical human backstop, quality checkpoint, and monitoring/learning mechanism in AIAE. Agentic reviews should be one layer in a multi-layered strategy for code review. Push your conventions and domain-specific code-review checklists to this layer, and think about how to solve this bottleneck for your org without dropping quality.
COMPRESSION: The middle layer is compressing, and the ladder is broken. Juniors are AI-native but lack production-grade engineering maturity. Seniors are scaling productivity using agentic solutions. A non-zero risk is that the “learn while you do” ladder for juniors might become too tall for most.
These AI impact points are listed in no particular order since these are early days and it is pragmatic to observe these changes, and drip feed them into your interview questions as opposed to make any structural changes right away. Some of these drips are indicated below.
Interviewing for Engineers in the AI Era
We build on the empirical and pragmatic value of the classic interview format, and add AIAE specifics. Examples and topics are intermingled, to avoid being prescriptive.
Coding
CLASSIC: 1. early-stage, high-scale filtering. 2. Take-homes are more effective, but more expensive.
AIAE (move closer to real world coding): 1. writing code where agents fail is the new bar 2. take-homes with follow-through (THFT) interviews that drill deep on why, how, change-this, etc. 3. THFT is a productive middle ground between coding and SDI that tests the real world ambiguity of implementing features/SDI. 4. agents can generate and validate these, with a human stepping in later to assist scale
AI for coding
CLASSIC: NA
AIAE (usage experience & best practices to derisk AI coding): 1. when did you have to switch to manual, and why? 2. give me examples of when agent wrote bad code and you detected it. why did it write bad code? 3. describe your highest-impact AI-assisted coding experience 4. how have you used AI to improve QA? 5. Spotify fleet upgrades, Cloudflare Next.js rebuild are good indicators of what is possible today.
Systems Design
CLASSIC: 1. working through high-level details of building a production system. 2. deep dives allow increased validation like with THFT.
AIAE (agents for monitoring, support): 1. using agents for DevOps, DevSecOps (if applicable) 2. using agents to debug production failures
AI systems design (non-agentic and agentic)
CLASSIC: NA
AIAE: 1. explain when to choose agentic solutions and why 2. walk through a unstructured to structured data that uses LLMs to improve a regex solution 3. every aspect of SDI (e.g., reliability) has an AI-specific wrinkle 3. solve an engineering leverage problem using an AI driven solution.
AI Engineering
CLASSIC: NA
AIAE (umbrella topic): 1. most impact points mentioned above are part of this new category 2. can you validate design using AI? how? 3. how do you decide between prompts, skills and tools? 4. give me a couple of examples of MCP tools you have written 5. how to you maintain token efficiency? 5. how are you handling code reviews for agent generated code? 6. Do you have a frontier lab model preference? why? 7. Explain with examples, guardrails and ensembles and how you could use them in the developer workflow 8. detail your anti-hallucination process 9. what new features would you like to see in Claude Code? 10. how do you manage session continuity?
Leadership
CLASSIC: 1. self: failure handling, decision under ambiguity (ex: one-two-doors), time management. 2. intra-team: conflict (ex: disagree-commit) influence without authority, mentor, center-of-excellence, conventions/best-practices, leading by example 3. inter-team: intra-team+, XF investment, empathy/big-picture, support/communication, soft contracts, protocols
AIAE (using AI to improve XF impact): 1. what internal tools have you built for your org? 2. how can engineering support XF given the ability to rapidly develop code? 3. stress-test for intensity and context-switching — prevent burnouts 4. using AI to improve communication (analyzing transcripts) 5. modeling you leadership explicitly and asking AI to assess instances of execution
Leadership is a broad topic, and is relatively more abstracted here than others.
Conclusion
We are still in the early stages of AI-assisted engineering. But it is inevitable to adopt AIAE and interview for these engineering positions eventually if you are in any way involved with software creation. Consider evolving your interview framework iteratively. Make sure you measure, since AIAE adoption must be empirical and scientific like any other.
Good luck with Predicting your next engineer’s net impact!