Do Databricks take all the pie? How organisations should be Partnering in 2026
Modern Data Stack Alliances are faltering
What has always fascinated me is the difference in sales culture vs. product in the data industry.
Companies like Snowflake are very difficult to partner with, but in practice a bit easier as their ecosystem is missing core functionality that naturally lends itself to partnership.
Companies like Databricks commercially are easier to partner with, but in practice finding the niche is harder.
This week, we announced that Orchestra is officially a databricks partner (woo!). But it hasn’t been easy. Partnering with the megaton gorilla in the room is hard - especially when they are building everything under the sun.
There are a few interesting things to note here.
Outcomes matter
Companies like Sigma have done an exceptional job at partnering with organisations like Databricks, not least because they send a ton of compute down there. This is key, but something that is also key is outcomes. The BI Tools are able to drive outcomes and adoption metrics for Databricks where “Apps alone” doesn’t quite get the buy-in a single vendor might push for.
Similarly at Orchestra, we send a ton of compute via the pipelining and transformation layer. This is attractive at scale, because databricks workflows doesn’t cut the mustard or scale well across multiple domains
Partnerships must find a niche
Ultimately there must be a niche and a scale problem for people to partner with companies like Databricks. They almost begin to resemble Azure, offering everything without one thing being absolutely killer or standalone.
AI is critical
There are rumours spreading that Databricks recently signed its first $1bn dollar deal which is pretty incredible and almost certainly has AI as a critical part. Unity Catalog is not the perfect context store for data teams, and running all your AI on Databricks does not meet the definition of “Meeting people where they are”. Embedding Databricks Services as first party citizens in AI products will, I think, be an increasingly important part of how organisations partner with large companies in the data space going forward.
Anyway we are very excited about this. If you know folks managing multiple teams on Databricks send them this! Here is the full announcement.
Orchestra is Officially a Databricks Partner
Today, we are excited to announce Orchestra’s partnership with Databricks.
Databricks has become the platform on which many of the world’s largest organisations build their most important data and AI workloads. Orchestra gives those teams the control plane they need to build, run and monitor those workloads alongside the rest of their data stack.
Together, Databricks and Orchestra provide a powerful foundation for enterprise data and AI: Databricks supplies the scalable compute, storage, governance and intelligence layer, while Orchestra coordinates the end-to-end workflows that make everything work reliably in production.
“Databricks gives data teams an incredibly powerful platform for building data and AI products. The challenge is coordinating everything around it: dbtTM, Lakeflow, ingestion, BI, cloud infrastructure and now AI agents,” says Orchestra CEO Hugo Lu. “Our partnership with Databricks gives enterprises a simple, scalable control plane for doing exactly that—without forcing them to build and maintain another orchestration platform themselves.”
The partnership formalises work Orchestra is already doing with some of the largest and most advanced data teams in the world.
Octopus Energy, the UK’s largest energy provider, uses Orchestra with Databricks to power a multi-tenanted analytics platform containing more than 5,000 dbt models across over 10 business units. Instead of maintaining a custom fork of Airflow, its teams can use Orchestra’s state-aware orchestration to run dbt intelligently on Databricks and give engineers a single place to understand the health of the wider platform.
Engineers can work in YAML and Git, while analysts and other data practitioners can use Orchestra’s visual interface. Platform teams retain control through environments, workspaces, RBAC, SSO and SCIM without forcing every contributor to become an Airflow expert.
That is exactly what this partnership is about: helping enterprises get more from Databricks while reducing the orchestration code, infrastructure and operational overhead required to run it.
What is Orchestra?
Orchestra is the agentic control plane for data and AI workflows.
It gives data teams a declarative, code-first and GUI-driven environment for building pipelines, running dbt, coordinating infrastructure and executing AI agents. Orchestration, lineage, observability, alerting, data quality and incident management are included in one platform.
Orchestra is not the data-processing engine. Workloads continue to run on the customer’s own compute—including Databricks—while Orchestra coordinates execution and collects the operational context teams need to understand what happened.
This serverless architecture is particularly valuable for enterprises using Databricks as part of a wider ecosystem. A pipeline may start with ingestion through Fivetran, Airbyte or a custom Python service, continue through Databricks and dbt, and finish in Power BI, Tableau, Sigma or an operational application. Orchestra coordinates the entire workflow rather than stopping at the boundary of a single platform.
Data teams therefore get the power of Databricks without having to force every workload into Databricks or maintain a separate orchestration framework around it.
How does Orchestra integrate with Databricks?
Orchestra provides a deep integration with Databricks that combines workflow execution with metadata, lineage, observability and data quality.
Teams can use Orchestra to coordinate Databricks SQL, notebooks, one-time runs and Lakeflow Jobs inside larger end-to-end pipelines. Runs, dependencies, logs, retries and alerts are available in one place, including the steps that execute outside Databricks.
This is not simply a thin API call wrapped in a generic task. Orchestra understands what the underlying operation is doing and brings the resulting operational context into the control plane.
The result is a much better developer and operator experience: less boilerplate, faster root-cause analysis and complete visibility across the data stack.
The best way to run dbtTM Core on Databricks
Databricks is an exceptional compute engine for dbt, but operating a large dbt project still creates difficult scheduling and observability problems.
As projects grow, teams typically create more tags, selectors and scheduled jobs. Models are frequently rebuilt even when neither their code nor their upstream data has changed. Different teams create overlapping schedules, while engineers are left piecing together dbt artifacts, Databricks logs and orchestration metadata when something fails.
Orchestra’s state-aware orchestration solves this problem without requiring teams to replace dbt Core with a proprietary transformation engine.
Orchestra stores and understands the previous state of the dbt project. It can then rebuild a model when its code has changed or fresh upstream data is available, while safely reusing models that do not need to run again.
This gives teams running dbt on Databricks several immediate benefits:
Faster dbt runs because unchanged models can be reused
Lower compute costs because Databricks does not process data unnecessarily
Simpler scheduling without maintaining a growing collection of tags and selectors
Clear lineage showing which models ran, which were reused and why
A shared state across pipelines within an Orchestra workspace
Octopus Energy demonstrates what this looks like at genuine enterprise scale. Orchestra state-aware orchestration helps the company coordinate more than 5,000 dbt models on Databricks across a multi-team, multi-domain environment. This is not a small proof of concept; it is an operating model for one of Europe’s most ambitious technology businesses.
One control plane for Lakeflow, notebooks, SQL and dbt
Enterprises rarely standardise on a single way of building inside Databricks.
One team may use notebooks. Another may use dbt Core. A third may build batch or streaming workloads using Lakeflow Spark Declarative Pipelines. Platform teams may package jobs and pipelines through Declarative Automation Bundles, formerly known as Databricks Asset Bundles.
Orchestra provides a common control plane above them. Teams can coordinate native Databricks workloads alongside external ingestion, transformation, BI and activation services. They can define dependencies declaratively, pass outputs between tasks, trigger workflows from events or schedules, and rerun from the point of failure.
This allows teams to use the right Databricks capability for each workload while retaining one consistent approach to orchestration, monitoring and incident response.
Extending Unity Catalog as the Context Layer for AI
Unity Catalog provides a governed foundation for discovering and controlling data and AI assets in Databricks, but it lacks specific operational and business context often living outside of Databricks. Through the Context Layer, Orchestra provides this metadata which acts as a foundation for the Agent Runtime - a single place to run AI Agents leveraging Databricks.
Orchestra can track tables and views registered in Unity Catalog, including column metadata and data quality test activity. It then combines this asset context with pipeline runs, dbt lineage, logs, failures and dependencies from across the wider stack.
This matters because enterprise lineage does not begin and end inside a single compute platform. Source data may arrive through an external ingestion service, while the final asset may be consumed by a BI dashboard or operational workflow elsewhere.
This means that the Agent Runtime uses less tokens than claude, achieves its goals in a fraction of the time, and is able to debug and diagnose pipeline failures end-to-end with greater reliability.
Who uses Orchestra and Databricks?
Orchestra is designed for enterprises and fast-growing teams using Databricks for analytics engineering, data engineering, machine learning and AI.
It is especially valuable where:
Multiple teams or business domains share a Databricks platform
dbt projects have grown too large for simple cron-based scheduling
Databricks workloads must be coordinated with tools outside the Lakehouse
Teams need end-to-end lineage and observability without building it themselves
An existing Airflow, Azure Data Factory or custom orchestration estate has become expensive to maintain
Organisations want to introduce AI agents without sacrificing security, governance or control
For migration and transformation programmes, introducing a unified control plane early prevents orchestration logic, alerts and metadata from becoming fragmented across notebooks, scripts and individual tools.
See Orchestra customer stories to learn how enterprise data teams are simplifying their platforms and delivering data products faster.
What is next for Orchestra and Databricks?
This partnership is the beginning of a much deeper collaboration. We will continue investing into providing greater visibility for data teams into other parts of the platform such as Lakeflow declarative pipelines, Databricks Apps, and Agents.
We will look to support models hosted in Unity Gateway via the Orchestra Agent Runtime. We will continue to invest into the Orchestra Context layer as a means of hydrating Unity Catalog to create a single place for engineering Context as well as pipelines.
Databricks gives enterprises the platform to build the future of data and AI. Orchestra gives them the control plane to run it. We’re very excited to continue this awesome partnership.
Find out more
Explore Orchestra for Databricks
Read the Databricks integration documentation
Learn about state-aware orchestration for dbt Core
For partnership, customer or press enquiries, contact legal@getorchestra.io.