Sunday Scaries: who's migrating who?
It's all about to come crashing down
I often get asked why anyone would use Orchestra when many large companies have already invested in large teams using stuff like Airflow.
In reality, it’s not really Airflow they’re invested in.
What they’re invested in is a massive army of developers using a framework that means only a very small number of very smart very expensive and very difficult to hire people can make workflows. It also means buying in to a central team that are a bottleneck to the organisation who also need additional budget for more observability tools, datadog, lineage tools, catalog etc. yada yada you’ve heard this before.
But it did kinda stick with me- why don’t people just migrate to new tools straight away?
Hello Economics
The answer of course lies with how companies discount the future.
Let’s say your data org does $1m of value a year and it will take everyone 50% of their year to migrate from A to B. But if they migrate, they’ll be 60% more productive so they can do $1.6m of value a year. Should they do it? That depends.
As you can see from this very basic discount model, choosing to migrate today vs. not basically ends in the same payout if you discount the future by 10% and you only care about today and tomorrow or this year and next year.
I think this is a realistic assumption since, let’s face it - everyone hops jobs in data all the time, CEOs have revenue targets and no-one likes doing migrations.
But at some point - the juice needs to be worth the squeeze.
What happens when the migration doubles productivity? When it doubles output and anyone can build a workflow?
In that scenario, the migration cost needs to roughly take the team 90% of their year for the team to be indifferent between migrating something from A to B for a whole year and doubling productivity.
Big Migrations to watch out for
Legacy solutions: Talend, informatica etc.
Super expensive, super hard to hire for, super slow. Things have come on so much.
Watch out for: people skipping tools like Databricks and Snowflake entirely and starting with the Orchestrator and perhaps object storage and iceberg
Matillion
Just quite expensive, quite hard to hire for, but moreover - easy to migrate.
Watch out for: BI Developers clamouring for dbt
Airflow
The Airflow legions are still there en masse and already we see 2025 being a year where more and more people are churning from Airflow. Happy 10th birthday!
We are in an age where a single platform engineer and a bunch of mildly technical people can build incredible Data and AI workflows with a simple stack. Let the migrations commence.
It is almost inevitable. The speed and drastically lower costs of not using a legacy hard-to-use orchestrator is enormous. If the question is not if but when, we know that we draw ever closer to the day when everyone mutually decides the music’s got a bit experimental and it’s time to leave the club.
Observability tools and catalogs
With more and more consolidated providers of these vendors and less and less juice from these proverbial lemons, expect to see massive churn in these tools in 2025.
Stuff that isn’t going anywhere
dbt
I was amazed people even moved away from stored procs to dbt until I realised there is a tendency to overcomplicate and many data people don’t even know what orchestration is.
Why would I move from dbt to something else now? Migration cost is high. I just don’t see it as a bottleneck. Make it make sense.
Cloud providers
Azure, GCP and AWS are here to stay in the data stack.
Even Azure are investing in making people never want to leave Fabric ever again.
GCP is the stickiest ecosystem of them all.
Snowflake and Databricks
I’ve written a few pieces recently about what could be the undoing these two companies but it’s not happening in 2025.