#26 - Key Insights from dbt CEO on AI and Data Engineering
Simple breakdown of the trends shaping data teams in 2025
Read time: 4 minutes.
Hi Data Modernisers,
I just listened to a great conversation between Tristan Handy (CEO of dbt Labs) and the team at a16z about where data engineering is headed.
Instead of the usual AI hype, Tristan shared some really practical insights about what's actually working in data teams right now.
Here's what I learned that might be useful for your team:
How AI is changing data work (but not replacing it)
Why do some AI projects succeed while others create problems
What dbt is building for the future of data engineering
Let me break down the key points in simple terms.
3 Key Ideas from the Conversation
Based on the podcast discussion, here are the main points that stood out:
1. AI Helps Data People Work Better (It Doesn't Replace Them)
What Tristan said:
80% of data professionals now use AI in their daily work
AI is best at automating routine tasks, not making business decisions
Data teams are growing because AI creates more demand for quality data
What this means for your team:
Focus on AI tools that help your current team be more productive
Don't expect AI to replace the need for people who understand your business
The most valuable skill remains knowing what questions to ask and how to interpret the results.
Examples of what's working:
AI is helping write SQL code that humans then review
Automated documentation generation
AI-assisted debugging of data pipeline failures
Tools that suggest optimisations for existing queries
2. The Success Factor: Human-in-the-Loop vs Human-out-of-the-Loop
Tristan's framework:
Human-in-the-loop: AI generates something, and an experienced person reviews it
Human-out-of-the-loop: AI gives answers directly to people who can't verify if they're correct
Why this matters:
Most successful AI projects keep humans involved in validation
The dangerous projects are ones where non-technical users get AI results they can't check
As Tristan put it: "Without a human to verify the result, that's a very scary thing"
Questions to ask about your AI projects:
Who's checking if the AI output is correct?
Do they have the skills to catch mistakes?
What happens if the AI gives the wrong answer?
Real example from the conversation:
dbt built an AI system that can write SQL to answer business questions
But it only works because it connects to their "semantic layer”- a system that defines exactly how your company measures things like revenue.
Without that context, AI just guesses at what you mean.
3. Data Engineering is Becoming More Like Software Engineering
What DBT is building:
They acquired a company called SDF that built a SQL compiler in Rust
This lets data engineers test their code locally instead of only in the cloud
It can translate between different SQL dialects automatically
They're adding features like automated refactoring and better error checking
Why this matters:
Right now, data engineering is more complicated than it needs to be
You can't easily reuse code between different data platforms
Testing changes is slow and risky
The new tools will make data work more reliably and faster
What Tristan said:
"Software engineering tool stack was maybe two decades ahead of data"
The goal is to give data engineers the same quality tools that software engineers have
This includes things like package management, version control, and local development
Changes coming:
Better debugging tools that can automatically find problems in data pipelines
Reusable components that work across different cloud platforms
Faster feedback loops when developing new data transformations
More standardised ways of building and sharing data logic
Key takeaways:
AI works best when it helps skilled people do more - Don't try to replace expertise, amplify it
Keep humans involved in validating AI outputs, primarily when business decisions depend on the results
Data engineering tools are continually improving. The next few years are expected to bring significant productivity improvements.
What to do next:
Listen to the full podcast if you want more details on any of these topics
Look at your current AI projects and ask if they're human-in-the-loop or human-out-of-the-loop
Consider how better development tools might help your data team work more efficiently
The main message: AI isn't going to replace data teams, but it will change how they work. Companies that use it thoughtfully will have significant advantages.
What did you think of this breakdown?
Helpful summary?
Too basic?
Want more technical details?
Let me know what would be most useful.
PS... If you found this summary helpful, feel free to share it with your team. These kinds of industry insights are worth discussing.
And whenever you are ready, there are 3 ways I can help you:
Free Data Flow Audit - 60-minute deep-dive where we map your current data flows and identify exactly where chaos is killing your AI initiatives
Modular Pipeline Migration - Complete rebuild from spaghetti scripts to dbt + Airflow architecture that your AI systems can actually depend on
AI-Ready Data Platform - Full implementation of version-controlled, tested, modular data pipeline with real-time capabilities designed for production AI workloads
That’s it for this week. If you found this helpful, leave a comment to let me know ✊
About the Author
Khurram, founder of BigDataDig and a former Teradata Global Data Consultant, brings over 15 years of deep expertise in data integration and robust data processing. Leveraging this extensive background, he now specialises in helping organisations in the financial services, telecommunications, retail, and government sectors implement cutting-edge, AI-ready data solutions. His methodology prioritises pragmatic, value-driven implementations that effectively manage risk while ensuring that data is prepared and optimised for AI and advanced analytics.