The importance of CI/CD flows for Data and AI Orchestration processes

Splitting up Orchestration logic has never been so powerful

There is a stat below that blows my mind, which is that in a survey of Airflow users about a quarter of people only use one instance and over half use 3 or more.

For all the chatter around software engineering best practices, it’s absolutely mind-blowing that CI/CD has got left behind. While we’ve spent a lot of time working out how to CI/CD frameworks like dbt, the most fundamental layer - orchestration and end-to-end, has been left out.

If you have one airflow instance, it means that if you’re doing CI/CD you’re using that single instance for all your stuff or just not doing CI/CD at all. It is a bad idea to be using the same piece of infra for testing and production - testing can bring the whole thing crashing down.

On the other side - multiple airflow instances represent a linear increase in cost both in terms of compute and resources required to maintain. Some people we know even have 18 or more instances which is staggering.

A scalable approach to this problem is to leverage environments-as-configuration and to use a scalable, serverless architecture. This kinda has parallels in some parts of software engineering (event-based systems) but not where durable infrastructure is necessary (like an app for a website or something).

Simply replicating everything in data engineering is obviously a bad idea and leads to unnecessary maintenance and spiralling costs. It’s a huge blocker for anyone trying to adopt anything other than a “Centralised Platform Team” Approach, which leads to the debilitating silo trap.

How are you doing CI/CD and making your workflows best-in-class?

Introduction

Continuous Integration and Continuous Development is one of the last great unsolved problems in data.

For all the fist-shaking and rhetoric blasted at engineers, one thing that has been ostensibly missing is how to check code changes aren’t going to break things.

At Orchestra, it is something we have thought about deeply from day 1.

A new Paradigm for Data: Continuous Data Integration and Delivery (Miniseries Part 5)
Throughout the series, we’ve examined what best practices look like in Software and Machine Learning. While there are…medium.com

Principles of effective data delivery: How CI/CD should look for Data teams (Miniseries Part 3)
Data Teams have grown exponentially over the last 5 years, but there is little consensus in terms of what principles…medium.com

Avoid bad data completely: Continuous Delivery Architectures in the Modern Data Stack (Part 1)
There is no excuse for end users accessing Bad Data. Thanks to dbt’s technology and recent advances in Snowflake’s…medium.com

In this article, we will demonstrate how a simple architectural change can create drastic improvements in code quality, time-to-resolve, and productivity by leveraging a modular architecture and Orchestra CI.

State of Play: Monolithic, slow and complex

The current state of play requires involve a large monolithic Airflow cluster responsible for data ingestion, data transformation (possibly using dbt) and downstream operations such as sending alerts, moving data back into source systems or activating dashboards.

A self-hosted Airflow/Prefect/Dagster architecture leads to unnecessary complexity

This leads to a few problems on the infrastructure side:

• Engineers need to maintain multiple sets of infrastructure (Kubernetes) to enable environments. Dev, Staging and Prod will be 3x the work of just having prod

• Teams need to pay thrice for the privilege of being best in class

• Engineers must have deep python and DevOps expertise to debug failures

• Bottlenecks arise because of the disconnect between business users (who may write SQL, change schedules, and do other rudimentary data activities) and “Infrastructure Team” who maintain the frameworks and infrastructure used to execute these things

For CI/CD specifically, there are a few data-specific problems that arise:

• CI/CD flows are complicated: as Orchestration logic (A → B) is combined with transformation logic (select * from table) what to test when code changes is not obvious. This leads to longer development cycles as opaque CI/CD actions fail; or no CI/CD at all.

• CI/CD flows take a long time: the most common approach is simply to execute a DAG in a staging environment when there is a code change to anything in that DAG. If this DAG succeeds, then you merge the code to main, and run it in prod, perhaps with a parameter full-refresh=true . You can already see how in this rudimentary set-up, flags relating to refresh logic and code-diffing is imperative to avoid this DAG taking ages (which it inevitably will).

Having monolithic Airflow or ephemeral clusters for running CI/CD flows is very sub-optimal, very slow, and forces teams to choose between being best-in-class (having CI/CD) or being productive (not being blocked by slow CI/CD).

On the first point — understanding what to test is non-trivial. Suppose you change the logic for a dbt model and recalculate a field. Should you:

• Run your entire dbt-core project in staging

• Just run that model in staging

• Run that model, tests, and downstream dbt dependencies in staging

• Run that model, tests, and downstream dependencies in staging

• Run the entire DAG end-to-end (for example, to ensure that if the new calculation makes use of a new column, that column exists in staging i.e. by ensuring your ingestion code works)

This is really non-trivial and I haven’t seen a good solution for this yet. It also requires you to have an instance of every piece of infrastructure for every environment — this idea is obviously farcical.

Snowflake, for example, allows the simple creation of zero-copy clones or even full copies of the data in an inexpensive way. There is no need to provision an entirely new Snowflake instance for every environment. This is one nuance software engineers seem to fail to understand — in pursuit of software engineering best practices multiple environments lead to multiple instances of infrastructure (“Staging is a direct replica of Prod”), despite the fact that data flows can be serviced by environments-as-configuration.

Environments-as-configuration or cloud deployment environments mirror services like AWS; where different accounts have the same environment variables that point to different values or objects. This idea is seen in Virtual Data Builds in tools such as Y42 or projects like Nessie.

By contrast, an Airflow architecture necessitates multiple instances of Airflow to enable multiple environments. If there is only a single Fivetran instance or connection / airbyte server / set of Power BI reports it is an anti pattern as illustrated below.

Airflow Anti Pattern

Here we can clearly see there is a duplication of resources while there is no duplication of Fivetran, Airbyte, Power BI etc. instances as that would be ridiculous (these tools are already expensive).

Best-in-class architecture

The best-in-class architecture we are seeing Teams implement is more akin to:

Leveraging Cloud Deployment Environments via Configuration

This is much more powerful. By having different configurations for different environments, the Control Plane passes these configurations through to different jobs, which are ideally designed in a serverless or scalable way.

For example, you could have a single ECS Container that runs python jobs that can be parameterised via config file in an Orchestrator to “point” to different resources. This means only one piece of infrastructure is required to facilitate infinite environments.

Unfortunately, many SAAS tools like Fivetran or Airbyte only really support this pattern by having multiple instances (the same is true of BI tools); but really these tools should support multiple environments. Snowflake and modern Transformation tools like Coalesce or dbt support multiple environments and are “ahead of the game” in this sense.

How to enable a best-in-class architecture with Orchestra

Orchestra is the easiest way to enable this architecture. By enabling different environments on a scalable and modular architecture, we unlock the following:

1. There is only one set of infrastructure to manage, which removes the need for teams to handle dependencies and requirements in different environments. This is especially true with managed Airflow services such as MWAA, where upgrading versions can be difficult and requires careful planning.

2. CI/CD becomes simple. By separating Orchestration logic, when there are changes to orchestration logic CI/CD ensures only that is tested. When there are changes to dbt, you can just test the dbt aspect. When there are changes to orchestration logic, you can just test that etc.

3. CI/CD becomes fast. Instead of relying on slow, monolithic Airflow clusters to test rudimentary things, Data Teams get faster feedback loops and can test more quickly / not get blocked by PRs.

4. Cost is reduced. Engineers do not need to maintain multiple instances of the same thing, which reduces cost. The current approach in a federated “hub and spoke” model (where different teams get access to the same infrastructure to self-serve) is particularly costly here.

5. Risk is reduced in skillset. Configuration files are straightforward to manage, so the requirement for specialist, niche devops skills is reduced.

6. Bottlenecks are eliminated: somewhat unique to Orchestra, but by having a Control Plane instead of just an orchestrator, anyone can view the results of CI/CD runs in a governed manner (you don’t want to give the finance team access to an Airflow UI, but you can give them read-only access to Orchestra) which means anyone can debug and maintain. You won’t need a central platform team to “step-in” every time something fails.

The average number of Arflow instances; clearly almost half of teams do not operate with SWE best practices since they only have at most 2 instances. The rest are overpaying by an order of at least 3 to facilitate robust pipelines

There are a few nice other benefits:

1. Flows across teams can be managed: in a “Data Product” pattern, where different teams all have access to similar pieces of infrastructure and create Data Products consumed by others, webhook triggers can be deployed at an environment level which ensures federated CI/CD without the need to wrestle with complex intra-team networking issues or finicky webhooks in between Airflow instances

2. Trust: trust gets built. Everyone can see the end-to-end lineage of a Data Product which means stakeholders can be kept reliably informed and have a single pane of glass to view results

Visualising e2e lineage for Data Products in Orchestra

How does this affect AI?

This pattern is clearly a prerequisite for AI. Where feedback is imperative, AI models will need to run and have manual intervention in staging environments to be reliably deployed into Production.

Without this, it will be impossible to efficiently scale AI workflows because every AI Product will require deep knowledge and swift iterations in monolithic Airflow-like code repositories.

This will eat-up developer time. Developers will spend time trying to push changes into staging environments which will quickly become disjointed from Production state. This means AI deployments will not be reliable and end-users will inevitably be effected when enterprises try to deploy more models and adapt existing ones to changing data.

Conclusion: modular, scalable and robust workflows

In this article we outlined the current de facto architecture for Data and AI workflows. We saw how almost 50% of organisations self-confess to not deploying Airflow in a best-in-class way.

Those that do pay many multiples of what they should do, both through infrastructure costs ($) and maintenance costs (🕒).

The solution is to enable a modular and scalable architecture that is compatible with environmnents-as-configuration and to use a Unified Control Plane that is compatible with config-driven cloud deployment environments.

The benefits of this are huge. Developers get to move faster, and spend less time maintaining duplicative infrastructure. Workflows are democratised, which means end users can finally make changes without being bottlenecked by platform teams, and engineering leads do not need to worry about unwanted changes to production due to granular role-based access control.

Finally, enterprises benefit from finally having a scalable architecture capable of supporting Data Workflows at scale, as well as serverless, easily-improved AI Applications. Where these feed into Product features, this has a clear and tangible impact to top-line growth.

To learn more about Orchestra, please check-out the resource below or reach out to me on Linkedin.

🌐Website

📚 Docs

🗲 Try it now

📊 Demo

Note: after writing this article some folks mentioned they use their single Airflow instance for multiple environments. The issues they have with this approach is that different environments are not segregated. They mentioned multiple occasions where a job in staging took down the entire cluster, impacting production.

Given the whole point of environments is that they’re separate/independent and don’t impact each other, we didn’t address this point in the original article because it’s such a bad idea.