dbt cloud web IDE vs develop dbt locally | Top tips

With dbt focussing more on dbt Cloud, I examine the Cloud IDE and compare it to its unfashionable older brother, your laptop

Substack note

I’m Hugo Lu —I’m CEO at Orchestra, which is a unified control plane for data teams.

I write about data architecture and how to effectively build and scale data teams. This includes diving into specific features of some products, such as the dbt Cloud IDE. I also like talking about best practices approaches, which are relevant here c.f. developer environment

---

Want to see how Orchestra is changing the game by delivering unparalleled cost savings and visibility? Try our Free Tier now.

Introduction

I’ve been using dbt-core about 6 years and my enjoyment for using it for simple data transformation tasks is just as it was back in the day.

All the good stuff is still the same.

• The Schema for the Opportunities Table from Salesforce

• The functionality I use in dbt-core (dbt build, tests, macros, and incremental loading)

• BigQuery’s UI (smh — guys you could have made it slightly better???? Slightly less cluttered? some stuff slightly bigger? no???)

With it being common knowledge that the dbt labs is spending resources improving dbt Cloud, the question becomes — why would you buy dbt Cloud? 

One oft-touted answer is the dbt IDE / dbt cloud IDE / dbt web IDE / web IDE — sorry — Integrated Development Environment.

In this article I will cover it in very light detail, compare it to how 99% of people develop (including me), locally, and offer some guidelines for choosing what is right for you

Explaining the dbt Cloud Web IDE

Head over to Develop. This takes you to the dbt web IDE:

Hello IDE

This is essentially, a browser-based version of VS Code.

You can edit files, push changes to git, and also leverage the CLI // command line interface. What does this mean? Let’s take a look below

Some basic data 

In the Cloud IDE, you can:

• Write code

• Test that code by executing the query

• See Lineage (dbt explorer)

• Lint your code by pressing a format button (cool)

• View the Data in the Cloud IDE (the big grey box above I redacted)

• Push changes to Git from the web without downloading anything

These are nice features. Now let’s compare with doing this locally

---

What about running dbt?

That’s an article for another time and not related to the dbt web IDE! Of course, there are many ways. Running dbt means you need a computer running somewhere. You can pay dbt for this in dbt Cloud. While convenient, it’s not a substitute for a full Orchestrator like Orchestra. Orchestra offers a bring-your-own infrastructure model (see EC2, ECS) for dbt, and works really well with dbt Cloud too.

---

Developing dbt Locally

Developing dbt locally requires you to open up your terminal, pip install dbt

, download VS Code, and also a few other plugins to get the same experience (all free):

• SQLFluff (Linting)

• Turntable for column level lineage (see here)

• vscode-power-user for the dbt explorer lineage bit (and here)

Thanks to whoever curated this, like genuinely. Lots of good stuff in here, well done. https://github.com/Hiflylabs/awesome-dbt

The first thing you need to do is SET UP dbt. This takes a bit of time, but no more than setting up dbt Cloud. 

Here is what you get when you’re running dbt locally

Running dbt test when developing dbt locally

As you can see, it looks really like the dbt web IDE! Apart from the command line is in a separate window and it has that retro nostalgic look — sorry one thing I should have added that also hasn’t changed for six years

• How the dbt CLI looks (still love it)

To view data, you’ll probably need to go to your warehouse (BigQuery, Snowflake, Databricks’) UI or you could use Duckdb locally 

Parsing some json using duckdb locally and viewing it in a powershell screen

dbt locally vs dbt Cloud web IDE

Feature differences

Unless there is something new or something I’ve missed, pretty much everything in the IDE you can replicate locally (for free)

Set-up

Installing dbt locally requires you to be able to do a pip install

Ideally you should also know how to do a virtual environment but if you’re new to analytics engineering, even that may not be necessary.

The IDE also requires the same auth config.

Familiarity

If you are one of the few people that write code but don’t use a code editor like VS Code, then you could make an argument for the IDE being more familiar to you and therefore faster…but damn it does look similar to VS Code.

Anything else, Hugo?

One of the great benefits of codifying your SQL is you go from this:

Running a SQL Query to create a task that runs a SQL Query with no dependencies, versioning or governance

To this:

a nice dbt-related github repo

Version-controlled, governed, well-written, well-structured, codified SQL.

The Developer Environment and its importance is well documented in software. If we are looking to learn from that profession, and extract its best bits for our own, it feels like starting with VSCode is going to be a pretty good move

Price

Developing locally is free, as long as you already have a computer. WIth the IDE, you pay for. $100 per user per month. Here’s what you can buy your team for that

• Lunch every day for two weeks every month

• A new iphone for your analysts every year

• A holiday

Ask your team what they’d prefer — the IDE or team lunches. They’ll appreciate being consulted, you’ll grow closer as a group, and they might even enjoy the food. There are, of course, a lot of other features in dbt Cloud that are worth exploring.

Conclusion

That’s it! Hope you enjoyed. Hit me up if you’d like to chat data, dbt, or anything else. 

Writer’s note: I actually used IntelliJ until 2022 and though vscode has a bit more going on and is a bit harder to get into, it’s so much better

Find out more about Orchestra

Orchestra is a platform for getting the most value out of your data as humanely possible. It’s also a feature-rich orchestration tool, and can solve for multiple use-cases and solutions. Our docs are here, but why not also check out our integrations — we manage these so you can get started with your pipelines instantly. We also have a blog, written by the Orchestra team + guest writers, and some whitepapers for more in-depth reads.