After 10+ data migrations across three continents, I have noticed something uncomfortable.

The organisations with the strongest balance sheets consistently build the most sophisticated data capabilities.

At a major bank, we had dedicated data quality teams and budgets for cutting-edge platforms. At a mid-sized company, we struggled to get approval for basic monitoring tools.

The difference wasn’t technical expertise. It was Maslow’s hierarchy playing out in real-time.

This Week’s Deep Dive:

• The 5-stage data maturity model mirrors human psychology

• Why “AI-first” strategies fail for cash-strapped organisations

• A practical assessment you can use on Monday morning

• Real patterns from my enterprise implementations.

Let’s break down the uncomfortable truth about data capability.

The Story: Two Migrations, Two Worlds Apart

I have worked on two strikingly similar projects.

Both were migrations. Both had identical technical complexity and timelines.

One at a major bank. One at a mid-sized retailer.

Completely different outcomes.

The Bank Project:

• Dedicated teams for data quality

• Automated testing pipelines

• Redundant systems during migration

• Performance issues? Additional resources approved in 24 hours

The Retailer Project:

• Every decision went through three approval layers

• One analyst wearing multiple hats as the “data quality team”

• Extra processing power during critical migration? Two weeks for sign-off

Result:

• Bank: Delivered ahead of schedule, zero data loss

• Retailer: Six months over, lost two team members to burnout

What I realised: Organisations can’t skip levels in the data maturity pyramid.

Just like humans can’t skip basic needs.

The Data Maturity Hierarchy: 5 Stages Every Organisation Climbs

Here’s the framework that explains everything:

Level 1: Survival (Basic Data Collection)

“We need reports to operate”

What it looks like:

• Spreadsheet-driven reporting

• Manual data extracts

• Basic storage systems

• Reactive “reporting as requested”

Organisations here: Startups, struggling companies, resource-constrained teams

Investment required: $50K-$200K annually

Time to progress: 6-18 months with dedicated effort

I’ve seen companies spend years stuck in this position. They think they need AI when they actually need consistent month-end reporting.

Level 2: Security & Trust (Reliable Foundation)

“Our data needs to be accurate and safe”

What it looks like:

• Data validation processes

• Backup and recovery systems

• Basic governance frameworks

• Consistent data definitions

Organisations here: Established businesses with regulatory requirements

Investment required: $200K-$500K annually

Time to progress: 12-24 months

Real example: One financial services project spent eight months establishing data lineage before touching analytics. Worth every hour.

Level 3: Collaboration (Breaking Silos)

“Everyone needs access to the same truth”

What it looks like:

• Cross-departmental data sharing

• Self-service analytics tools

• Standardized dashboards

• Data literacy programs

Organisations here: Growing companies with multiple business units

Investment required: $500K-$1M annually

Time to progress: 18-36 months

Level 4: Insights & Recognition (Advanced Analytics)

“We’re making data-driven decisions”

What it looks like:

• Predictive modeling

• Real-time dashboards

• ML-powered recommendations

• Industry recognition for data practices

Organisations here: Market leaders, well-funded scale-ups

Investment required: $1M-$3M annually

Time to progress: 2-4 years

Key insight: This is where most “AI transformation” projects actually begin. Not where they’re sold to start.

Level 5: Innovation (Data-Driven Transformation)

“Data is our competitive advantage”

What it looks like:

• AI-powered product features

• Real-time personalization

• Automated decision systems

• New revenue streams from data

Organisations here: for example Netflix, Amazon, Google, and mature fintech companies

Investment required: $3M+ annually

Time to maintain: Continuous evolution required

The Uncomfortable Truth: Money Talks, Data Walks

Here’s what I have observed across 15 years:

Level 1-2 organisations ask: “What’s the cheapest way to get insights?”

Level 3-4 organisations ask: “How do we scale our data capabilities?”

Level 5 organisations ask: “How do we stay ahead of disruption?”

The Pattern:

Financially stable companies don’t just have better data.

They can afford to fail fast, learn quickly, and iterate.

At one of our enterprise clients, we tested three different approaches to forecasting simultaneously.

At a struggling company, we had to pick one approach and hope it worked.

Resource availability directly correlates with the progression of data maturity.

Your Monday Morning Assessment

Rate your organisation (1-5) on each dimension:

Financial Stability:

• Budget approval timelines: Immediate (5) → 6+ months (1)

• Risk tolerance: High (5) → None (1)

• Team investment: Growing (5) → Shrinking (1)

Data Maturity:

• Data quality: Automated validation (5) → Manual checking (1)

• Analytics capability: Predictive models (5) → Excel reports (1)

• Decision speed: Real-time (5) → Monthly reviews (1)

The Pattern:

Your financial score predicts your data maturity ceiling.

Organisations consistently score within 1 point between financial stability and data maturity.

I have never seen a Level 1 financial organisation sustain Level 4 data practices.

What This Means for Your Career

If you’re at a Level 1-2 organisation:

• Focus on foundational skills: SQL, data quality, basic automation

• Build systems that show immediate ROI

• Document everything; you’re building credibility for future investment

If you’re at a Level 3-4 organisation:

• This is where careers accelerate

• Learn advanced analytics and cloud architectures

• Lead cross-functional initiatives

• Position yourself for the transition to Level 5

If you’re at a Level 5 organisation:

• Stay ahead of the curve—real-time systems, AI integration

• Share knowledge externally—speaking, writing, thought leadership

• Consider consulting back to lower levels

The key insight:

Align your skill development with your organisation’s realistic trajectory.

Not their aspirational goals.

The Bottom Line

Data maturity follows a similar progression to human development.

Organisations trying to jump from basic reporting to AI are like startups trying to offer luxury products.

The foundation isn’t there.

Your next promotion depends on understanding where your organisation really sits on this pyramid.

Not where leadership thinks they are.

I have seen talented data professionals burn out trying to build Level 5 capabilities with Level 2 resources.

The Solution:

Match your ambitions to your organisation’s actual maturity level.

Then systematically help them climb the pyramid.

What level is your organisation really operating at?

PS...If you’re enjoying this newsletter, please consider referring this edition to a friend. They’ll receive weekly insights, backed by industry research, on making data modernisation more predictable and profitable.

Share Data Modernisation Journey

And whenever you’re ready, there are 3 ways I can help you:

1. Migration Readiness Assessment & Roadmap - The essential first step to clarity. Fixed-fee, 3-4 week engagement for leaders who need a comprehensive architectural blueprint and de-risked plan before committing to full-scale legacy data migration.

2. Fractional Data Architect Retainer - Ongoing senior architectural leadership to guide your team through major projects. Consistent expert oversight that ensures design integrity, manages technical risk, and keeps complex initiatives aligned with core business goals.

3. Advisory & Review Sessions - Expert guidance on demand. Prepaid hours are ideal for reviewing internal plans, evaluating vendor proposals, or workshopping specific architectural challenges with a seasoned expert who has over 15 years of experience in the field.

I've been asking data teams the same question for months: "How long does it take you to figure out why something broke?"

The answer is always some version of "too long."

Here's what I learned from analysing this pattern across teams, plus 5 tools and 1 framework you can start using today.

🔥 This Week's Big Insight

The Problem: 73% of data teams monitor when things break, but have no visibility into why they break.

The Solution: Orchestration-native observability (monitoring pipelines, not just warehouses).

The Impact: Teams implementing this see 60% faster incident resolution on average.

🛠️ 5 Tools You Should Bookmark This Week

For Task-Level Monitoring:

1. Dagster - Asset-oriented orchestration with built-in observability

   • Use case: See exactly which pipeline task failed and why

   • Quick start: Their 10-minute tutorial shows task-level visibility

2. Prefect - Modern workflow orchestration with detailed logging

   • Use case: Clear failure information and automated retries

   • Quick start: Free tier for teams under 3 users

For Data Quality Monitoring:

3. Monte Carlo - Data observability platform focusing on the "5 pillars"

   • Use case: Automated anomaly detection across freshness, volume, and schema

   • Quick start: They offer free data health assessments

4. Metaplane - Lightweight data monitoring with fast setup

   • Use case: Real-time alerts without heavy configuration

   • Quick start: Connect in under 30 minutes

For Schema Change Detection:

5. Great Expectations - Open source data validation framework

   • Use case: Catch schema drift before it breaks downstream

   • Quick start: Their schema validation tutorial takes 15 minutes

📋 Copy This: 5-Point Readiness Checklist

Rate yourself 1-5 on each:

☑︎ Task Visibility: Can you see individual pipeline task status in real-time?
☑︎ Schema Monitoring: Do you catch schema changes before they break downstream?
☑︎ Impact Analysis: Do you know what breaks when a pipeline fails?
☑︎ Cost Attribution: Can you track compute costs per pipeline?
☑︎ SLA Tracking: Do you monitor data freshness and completeness automatically?

Score:

• 20-25: You're in the top 10% of data teams

• 15-19: Solid foundation, some gaps to fill

• 10-14: Standard setup, big improvement opportunity

• Below 10: High risk zone - prioritise this

🎯 Quick Win: 15-Minute Task Monitoring Setup

If you're using Airflow, add this to any DAG for instant task-level visibility:

import logging
from datetime import datetime, timedelta
from airflow.models.dag import DAG

def log_task_metrics(**context):
    """Production-ready callback for task monitoring"""
    try:
        task_instance = context["task_instance"]
        start_time = task_instance.start_date
        end_time = task_instance.end_date
        
        if start_time and end_time:
            duration = (end_time - start_time).total_seconds()
            logging.info(f"Task '{task_instance.task_id}' finished. "
                        f"State: {task_instance.state}. Duration: {duration:.2f}s")
        else:
            logging.warning(f"Task '{task_instance.task_id}' finished. "
                           f"State: {task_instance.state}. No timing data.")
    except Exception as e:
        logging.error(f"Monitoring callback error: {e}")

# Apply to ALL tasks in the DAG via default_args
default_args = {
    "owner": "data_team",
    "retries": 1,
    "retry_delay": timedelta(minutes=5),
    "on_success_callback": log_task_metrics,
    "on_failure_callback": log_task_metrics,
    "on_retry_callback": log_task_metrics,  # Track retries too!
}

# Now every task gets monitoring automatically
with DAG(
    dag_id="monitored_pipeline",
    start_date=datetime(2025, 8, 9),
    schedule="@daily",
    default_args=default_args,  # This is the magic line
) as dag:
    # All tasks inherit the monitoring callbacks
    extract_task = BashOperator(
        task_id="extract_data",
        bash_command="your_extract_script.sh"
    )

Implementation time: 15 minutes
Value: Immediate visibility into task performance patterns

📚 This Week's Reads

Must-read articles I bookmarked:

• Netflix's observability lessons - How they built tools to monitor petabytes of data

• Airbnb's data quality evolution - Their innovative DQ Score approach

• Spotify's data platform architecture - How they process 1.4 trillion data points daily

Tools/Resources:

• Data Engineering Weekly - Best curated data engineering content

• dbt's Analytics Engineering Roundup - Weekly analytics engineering insights

💡 Industry Intel

What's trending this week:

• Snowflake just launched enhanced pipeline monitoring with built-in telemetry and better Monte Carlo integration

• Monte Carlo raised $135M Series D, pushing valuation beyond $1.6B (validates the data observability market explosion)

• Major companies migrating from Airflow to Dagster specifically for observability features (names confidential, but momentum is real)

Who's hiring data engineers with observability skills:

• Netflix (Senior Data Platform Engineers + Observability specialists)

• Stripe (Data Infrastructure Engineers with platform reliability focus)

• Databricks (Solutions Architects - observability expertise increasingly required even when not in job title)

🚀 Next Week Preview

"The $500K Snowflake Bill: 3 Cost Optimisations That Cut Warehouse Spend by 40%"

Including:

• The warehouse query that's probably costing you $10K/month

• 5-minute optimisation checklist

• Cost monitoring dashboard template you can copy

💬 Community Question

This week: What's your biggest pipeline monitoring blind spot right now?

Reply and tell me - I'll feature the best insights in next week's issue (with your permission).

🎁 Subscriber Exclusive

Free resource: My "Orchestration Observability Setup Guide" (10-page Guide)

Includes:

• Tool comparison matrix with scoring framework

• Implementation timeline template

• Cost-benefit calculation spreadsheet

• 30+ monitoring queries you can copy-paste

Newsletter subscribers only - Subscribe OR if you are already subscriber comment with “Observability” and I will send you the link.

Found this useful? Forward to a colleague who's tired of debugging data incidents at 2 AM.

Share Data Modernisation Journey

Want more like this? Hit reply and let me know what data engineering topics you want me to dive into next.

That’s it for this week. If you found this helpful, leave a comment to let me know ✊

Leave a comment

About the Author

Khurram, founder of BigDataDig and a former Teradata Global Data Consultant, brings over 15 years of deep expertise in data integration and data processing. Leveraging this extensive background, he now specialises in organisational financial services, telecommunications, retail, and government sectors, implementing cutting-edge, AI-ready data solutions. His methodology prioritises value-driven implementations that effectively manage risk while ensuring that data is prepared, optimised, and advanced analytics.

Hi Data Modernisers,

I just listened to a great conversation between Tristan Handy (CEO of dbt Labs) and the team at a16z about where data engineering is headed.

Instead of the usual AI hype, Tristan shared some really practical insights about what's actually working in data teams right now.

Here's what I learned that might be useful for your team:

• How AI is changing data work (but not replacing it)

• Why do some AI projects succeed while others create problems

• What dbt is building for the future of data engineering

Let me break down the key points in simple terms.

3 Key Ideas from the Conversation

Based on the podcast discussion, here are the main points that stood out:

1. AI Helps Data People Work Better (It Doesn't Replace Them)

What Tristan said:

• 80% of data professionals now use AI in their daily work

• AI is best at automating routine tasks, not making business decisions

• Data teams are growing because AI creates more demand for quality data

What this means for your team:

• Focus on AI tools that help your current team be more productive

• Don't expect AI to replace the need for people who understand your business

• The most valuable skill remains knowing what questions to ask and how to interpret the results.

Examples of what's working:

• AI is helping write SQL code that humans then review

• Automated documentation generation

• AI-assisted debugging of data pipeline failures

• Tools that suggest optimisations for existing queries

2. The Success Factor: Human-in-the-Loop vs Human-out-of-the-Loop

Tristan's framework:

• Human-in-the-loop: AI generates something, and an experienced person reviews it

• Human-out-of-the-loop: AI gives answers directly to people who can't verify if they're correct

Why this matters:

• Most successful AI projects keep humans involved in validation

• The dangerous projects are ones where non-technical users get AI results they can't check

• As Tristan put it: "Without a human to verify the result, that's a very scary thing"

Questions to ask about your AI projects:

• Who's checking if the AI output is correct?

• Do they have the skills to catch mistakes?

• What happens if the AI gives the wrong answer?

Real example from the conversation:

• dbt built an AI system that can write SQL to answer business questions

• But it only works because it connects to their "semantic layer”- a system that defines exactly how your company measures things like revenue.

• Without that context, AI just guesses at what you mean.

3. Data Engineering is Becoming More Like Software Engineering

What DBT is building:

• They acquired a company called SDF that built a SQL compiler in Rust

• This lets data engineers test their code locally instead of only in the cloud

• It can translate between different SQL dialects automatically

• They're adding features like automated refactoring and better error checking

Why this matters:

• Right now, data engineering is more complicated than it needs to be

• You can't easily reuse code between different data platforms

• Testing changes is slow and risky

• The new tools will make data work more reliably and faster

What Tristan said:

• "Software engineering tool stack was maybe two decades ahead of data"

• The goal is to give data engineers the same quality tools that software engineers have

• This includes things like package management, version control, and local development

Changes coming:

• Better debugging tools that can automatically find problems in data pipelines

• Reusable components that work across different cloud platforms

• Faster feedback loops when developing new data transformations

• More standardised ways of building and sharing data logic

Key takeaways:

• AI works best when it helps skilled people do more - Don't try to replace expertise, amplify it

• Keep humans involved in validating AI outputs, primarily when business decisions depend on the results

• Data engineering tools are continually improving. The next few years are expected to bring significant productivity improvements.

What to do next:

• Listen to the full podcast if you want more details on any of these topics

• Look at your current AI projects and ask if they're human-in-the-loop or human-out-of-the-loop

• Consider how better development tools might help your data team work more efficiently

The main message: AI isn't going to replace data teams, but it will change how they work. Companies that use it thoughtfully will have significant advantages.

What did you think of this breakdown?

• Helpful summary?

• Too basic?

• Want more technical details?

Let me know what would be most useful.

PS... If you found this summary helpful, feel free to share it with your team. These kinds of industry insights are worth discussing.

And whenever you are ready, there are 3 ways I can help you:

1. Free Data Flow Audit - 60-minute deep-dive where we map your current data flows and identify exactly where chaos is killing your AI initiatives

2. Modular Pipeline Migration - Complete rebuild from spaghetti scripts to dbt + Airflow architecture that your AI systems can actually depend on

3. AI-Ready Data Platform - Full implementation of version-controlled, tested, modular data pipeline with real-time capabilities designed for production AI workloads

That’s it for this week. If you found this helpful, leave a comment to let me know ✊

Leave a comment

About the Author

Khurram, founder of BigDataDig and a former Teradata Global Data Consultant, brings over 15 years of deep expertise in data integration and robust data processing. Leveraging this extensive background, he now specialises in helping organisations in the financial services, telecommunications, retail, and government sectors implement cutting-edge, AI-ready data solutions. His methodology prioritises pragmatic, value-driven implementations that effectively manage risk while ensuring that data is prepared and optimised for AI and advanced analytics.

I've spent over 15 years in the trenches of data modernization, and I'm observing something fascinating: serverless is absolutely consuming the ETL world.

I appreciate elegant SQL and thoughtful migrations, yet the developments in serverless ETL for 2025 are extraordinary. After exploring numerous intricate migrations and observing the emerging patterns across various sectors, I'm identifying clear trends:

• Conventional ETL tools are becoming less popular as new initiatives predominantly prefer serverless solutions.

• The adoption of AWS Glue is rapidly increasing, particularly among mid-market companies seeking greater agility.

• The industry is clearly moving towards serverless-first strategies for data integration.

• Organizations are moving away from the legacy middleware they have depended on for years.

• Even historically conservative industries such as finance and government are embracing change.

Today, I'm exploring how AWS Glue is spearheading the serverless integration revolution and its implications for your 2025 data strategy:

• Why the transition to serverless ETL has become essential for competitive businesses.

• How AWS Glue's maturity has created a compelling leadership position

• The 2025 data integration landscape for innovative organizations.

Let's dive in.

3 Reasons Serverless is Eating Traditional ETL in 2025

The transformation of data integration through serverless technology is not merely a trend; it's a profound shift in how organizations handle ETL. As we approach 2025, here are the reasons AWS Glue is at the forefront of this movement:

1. Serverless Economics are Reducing Traditional ETL Expenses

The cost dynamics of serverless ETL have reached an essential threshold that cannot be overlooked. Traditional ETL solutions now serve as a financial burden on IT budgets.

AWS Glue's serverless approach excels on multiple fronts:

• Pricing by the second is taking over costly annual licensing models.

• Infrastructure expenses have drastically decreased, eliminating the need to maintain idle capacity.

• The overall ownership cost dropped by 60% in my recent projects.

• Capital expenses are transformed into operational expenses to enhance budgeting.

• Elastic scaling removes the overprovisioning issues found in traditional ETL.

2. The Democratization of Enterprise-Level Data Integration

By 2025, the technical barriers that previously distinguished enterprise ETL capabilities from mid-market organizations will have entirely diminished.

AWS Glue has dramatically advanced its visual development methodology:

• Non-technical users are creating intricate ETL workflows without the need for coding.

• Domain experts execute business logic smoothly, without bottlenecks from developers.

• Visual data lineage and impact analysis compete with conventional enterprise tools.

• Templates and patterns speed up implementation by 5 to 7 times.

• The learning curve has significantly flattened due to an enhanced user experience.

3. AI-Powered Automation is Transforming Possibilities

Incorporating AI features into AWS Glue marks the most transformative development in ETL since its inception.

By 2025, these capabilities have matured significantly, transforming possibilities.

• Generative AI helps enhance ETL development by recommending the best transformations.

• Automated data quality rules automatically identify and correct anomalies without human input.

• Self-optimizing job performance enhances automatically, eliminating the need for manual adjustments.

• Natural language data transformation enables business users to articulate modifications.

• Automated record-keeping and lineage monitoring lower governance burdens.

That's it.

Here's what you learned today:

• In 2025, serverless technology will have significantly transformed the ETL market.

• AWS Glue drives this change with robust, enterprise-ready features.

• The economic benefits far surpass those of conventional methods.

• Democratization has removed technical obstacles for non-experts.

• AI automation signifies a new chapter in the evolution of data integration.

In 2025, organizations thriving with data won't be those holding onto outdated ETL tools; they'll be adopting serverless integration platforms such as AWS Glue, which fit seamlessly with contemporary data needs.

PS… If you're benefiting from these insights on the serverless data revolution, consider sharing this edition with a colleague struggling with traditional ETL issues. For every five referrals, I'll provide exclusive access to my 2025 AWS Glue Performance Optimization Playbook.

That’s it for this week. If you found this helpful, leave a comment to let me know ✊

Leave a comment

About the Author

With 15+ years of experience implementing data integration solutions across financial services, telecommunications, retail, and government sectors, I've helped dozens of organizations implement robust ETL processing. My approach emphasizes pragmatic implementations that deliver business value while effectively managing risk.

Welcome to this week's edition, where we tackle one of the most challenging decisions in modern data architecture: ETL vs. ELT? This is an ongoing debate among data leaders, and most organizations waste millions on the wrong data transformation approach. The debate has become more confusing than helpful.

Introduction

According to a 2023 IDC survey on Data Integration Strategies, 68% of organizations reported making ETL/ELT architectural decisions without thoroughly evaluating their specific workload needs. The same study found that these organizations were 2.5x more likely to experience budget overruns and performance issues.

The stakes are high—choosing without a proper evaluation framework often results in costly rework, extended timelines, and team frustration.

Let's cut through the noise with research-backed guidance:

• A practical framework based on industry benchmarks

• A simple decision tree validated by successful projects

• The hidden costs that vendors don't talk about

Ready to transform your approach to data transformation?

Let’s dive in.

5 Steps To Choose Between ETL/ELT With Confidence Even if You're Not a Data Architect

To achieve a successful migration and data transformation strategy, you'll need a systematic approach based on verifiable evidence rather than following technology trends.

Based on aggregated insights from industry analysts and successful enterprise migrations, here's the structured framework that consistently delivers better outcomes:

1. Define Your Migration Goals & Constraints

Start by clearly articulating what success looks like for your data modernization project. List your objectives in order of priority:

• Operational cost reduction → Favors ELT if your target platform is cloud-based with optimized pricing

• Real-time analytics capability → Often better with ELT due to faster raw data availability

• Enhanced data quality → May lean toward ETL for upfront cleansing and validation

• Regulatory compliance → Usually better with ETL for audit trails and controlled transformations

• Increased business agility → Typically favors ELT for faster iterations and explorations

Industry Insight: Financial services organizations prioritize compliance and data quality over cost reduction, often favoring ETL in regulatory-sensitive workflows. Meanwhile, retail and e-commerce businesses prioritize speed-to-insight, favoring ELT approaches for customer analytics.

Actionable Example: A financial services client created this simple prioritization matrix:

• Priority 1: Compliance with GDPR and financial regulations

• Priority 2: Improved data quality for customer reporting

• Priority 3: Reduced operational costs

• Priority 4: Faster analytics capabilities

Quick Action Item: Create a simple table with your priorities (1-5) and constraints (budget, timeline, team size) to reference during architecture discussions.

2. Map Your Data Landscape Honestly

Create an inventory of your data sources and transformation needs with these critical metrics:

• Data volume per source → Flag anything over 5TB as potentially problematic for pure ELT

• Data change frequency → Real-time/streaming sources may need specialized ETL pipelines

• Current data quality score → Sources with <80% quality scores usually need ETL preprocessing

• Transformation complexity → Count steps required for each significant transformation

• Target platform limits → Document compute/storage costs and scalability boundaries

Industry Insight: Organizations processing over 10TB daily often experience significantly higher cloud computing costs when using pure ELT approaches versus hybrid ETL/ELT architectures. Telecommunications companies typically show the most dramatic difference, with pure ELT approaches costing substantially more for call detail record processing due to the high transformation complexity.

Quick Action Item: Identify your 3 largest data sources and run a 1-day sample through your ETL tool and target system to compare processing times and costs.

3. Evaluate Your Transformation Complexity

Catalog your transformation requirements and categorize each into these complexity tiers:

• Simple transforms (ELT-friendly)

  • Basic filtering (WHERE clauses)

  • Column renaming and basic type conversion

  • Simple calculations (currency conversion, etc.)

  • Direct 1:1 mappings

• Moderate transforms (ELT-possible but evaluate)

  • Aggregation and grouping operations

  • Basic joins between 2-3 tables

  • Window functions and time-based calculations

  • Data quality checks and simple validations

• Complex transforms (Usually ETL-better)

  • Multi-step data cleansing routines

  • Machine learning feature engineering

  • Fuzzy matching or record linkage

  • Complex business rules with multiple dependencies

  • Slowly changing dimensions (Type 2+)

Industry Insight: Complex transformations in ELT pipelines frequently consume substantially more compute resources than equivalent ETL processes for datasets over 1TB. Particularly notable is that slowly changing dimension (SCD) Type 2 operations on large customer tables are significantly more expensive when performed in-database than in traditional ETL tools. However, simple and moderate transformations typically show minimal cost differences and faster development cycles in ELT implementations.

Quick Action Item: Use this complexity scale to rate your top 10 most frequently used data transformations. If more than 50% are complex, consider retaining ETL for those specific workflows.

4. Assess Your Team's Readiness

Conduct an honest skills assessment of your team across these critical dimensions:

• Technical Skills Inventory

  • ETL tool proficiency (Informatica, SSIS, etc.)

  • SQL expertise level (basic, intermediate, advanced)

  • Cloud platform experience (AWS, Azure, GCP)

  • Modern data stack familiarity (dbt, Fivetran, Airflow, etc.)

  • Programming skills (Python, Scala, Java)

• Working Style Assessment

  • Comfort with agile, iterative approaches (ELT-friendly)

  • Experience with test-driven development

  • Preference for visual tools vs. code-based solutions

  • Adaptability to new technologies and frameworks

• Readiness Gap Analysis

  • Identify the 3 most significant skills gaps for your desired architecture

  • Determine training requirements vs. hiring needs

  • Estimate the ramp-up time for new technologies

Industry Insight: Organizations that assess team readiness before migration are much more likely to meet project timelines. Banking teams with strong traditional ETL backgrounds typically need several months to become proficient with modern ELT tools like dbt. Interestingly, teams with strong SQL skills but limited ETL experience tend to adapt to ELT approaches in just 6-8 weeks. Companies that implement structured upskilling programs generally report higher satisfaction with migration outcomes versus those that rely solely on hiring new talent.

Quick Action Item: Ask team members to self-assess their comfort level with ETL and modern ELT tools on a scale of 1-5. Use the results to identify potential champions and areas needing focused training.

5. Calculate Total Cost of Ownership

Build a comprehensive TCO model comparing ETL vs. ELT approaches with these specific line items:

• Direct Infrastructure Costs

  • ETL server costs (on-prem) or compute resources (Cloud)

  • Data storage costs (staging areas, final destinations)

  • Network egress chaCloudfor data movement

  • Scaling costs during peak processing periods

• Software and Tooling Costs

  • ETL tool licensing (often per-core or per-server)

  • ELT tool subscriptions (often per-user or consumption-based)

  • Monitoring and observability solutions

  • Version control and deployment tools

• Human Resource Costs

  • Development time (hours per pipeline × hourly rate)

  • Ongoing maintenance (hours per month × hourly rate)

  • Training and upskilling costs

  • Support and operations staffing

• Hidden Costs

  • Data quality remediation costs

  • Business impact of delayed insights

  • Compliance and governance overhead

  • Technical debt accumulation

Industry Insight: Three three-year TCO comparisons show varied industry results for enterprises processing daily 5-15TB of data. Financial services organizations with strict compliance requirements often see better economics with ETL approaches, while retail and media companies typically benefit from ELT approaches. Notably, hybrid approaches frequently yield the lowest overall TCO across all sectors when workloads are optimally distributed based on transformation complexity.

Quick Action Item: Run a 2-week sample of your most significant data flow through both approaches and multiply the resource consumption by 26 to get an annual estimate. Add this to your licensing costs for a quick TCO comparison.

That's it.

Here's what you learned today:

• The ETL vs. ELT decision isn't one-size-fits-all—it requires a systematic evaluation of your specific context

• Your migration goals, data landscape, transformation complexity, team skills, and total costs all factor into the optimal approach

• A hybrid strategy combining elements of both approaches is often the most practical solution.

Decision Tree Cheat Sheet

I promised you a decision tree, so here's a simplified version you can apply immediately:

1. Need Super Clean Data Before Loading (e.g., compliance)?

• YES → Start with ETL for that data.

• NO → Keep going.

2. Got a Super Fast, Powerful Target Platform (like the Cloud)?

• YES → ELT could be a great fit!

• NO → Think more about ETL or a mix.

3. Do you Let Your Data Team Explore and play with the Data Easily?

• YES → ELT helps with that!

• NO → Either way works.

4. Is Your Team Really Good with SQL and Cloud Tools?

• YES → ELT might be smoother.

• NO → Stick with what you know (ETL) while learning new skills.

5. Dealing with HUGE amounts of data and really tricky transformations?

• YES → Maybe use ETL for those tricky parts.

• NO → Keep going!

Bottom Line:

• Mostly "YES" to ELT questions? You can probably lean toward ELT!

• Mostly "YES" to ETL questions? ETL might be your sweet spot.

• Mix of answers? A hybrid approach (using both) is likely the best way to go!

Industry observations consistently show that organizations following a structured decision framework are much more likely to meet or exceed their data modernization ROI targets than those making technology-driven decisions. Most successful migrations implement hybrid approaches that optimize for specific workload characteristics rather than forcing all data through a single pattern.

That’s it for this week. If you found this helpful, leave a comment to let me know. ✊

Leave a comment

PS...If you're enjoying Data Transformation Insights, please refer this edition to a colleague struggling with their modernization strategy. They'll thank you when they save thousands on their migration project.

About the Author

With 15+ years of experience implementing data integration solutions across financial services, telecommunications, retail, and government sectors, I've helped dozens of organizations implement robust ETL processing. My approach emphasizes pragmatic implementations that deliver business value while effectively managing risk.