Considering Microsoft Fabric? Do these 5 things first | Lessons learned
Lessons learned from Microsoft Fabric implementations in the last 12 months
I wrote this with my good friend Christian at Steinert Analytics - we’ve both been doing Fabric implementations in the last year. Us to provide orchestration atop Fabric and other tools (Fabric is not a panacea, surprisingly) and Christian to implement the entire stack with business logic.
There are definitely some issues with the platform but a lot of the pain can be avoided by considering a few things below before diving into it Head-first. Every platform wants to be the “AI and Data platform”. Fundamentally I do not believe Fabric is in the Top 3 yet and the personal frustration it might cause over the next year or so combined with the difficulty around pricing make it a risky choice for many enterprises in traditional industries that already spend a lot with Azure and think “Oh well Fabric looks good enough to me”.
Anyway please enjoy everything below!
Edit: we shared this on reddit and the response was more mixed than the discussion below from about 6 months ago. So we are clearly getting more “Production-ready”, whatever that means ;) New Discussion here.
Introduction
As organizations increasingly turn to unified data platforms, Microsoft Fabric has emerged as a compelling solution that promises to consolidate data engineering, analytics, and business intelligence into a single environment.
We know that Microsoft have an army of people dedicated to this. Numerous high-profile figures like Tino from Motherduck have joined the Fabric team of late.
However, successful implementation requires careful consideration of several critical factors that go beyond the marketing promises. We’ve been working on Fabric implementations at Banks, Financial Institutions and massive Healthcare orgs in the past few months - here’s what we’ve learned.
Based on our extensive hands-on experience with enterprise deployments, here are three essential considerations that will determine the success of your Fabric implementation.
1. Assess Your Technical Expertise and Infrastructure Requirements
Be prepared for specialization. Microsoft Fabric requires strong expertise in the Azure ecosystem. Contrary to what marketing materials suggest, it's not a straightforward data tool to pick up. Having experience in Azure Data Factory is extremely useful when assessing data pipeline architecture and advanced functionality (variable libraries, dynamic content, and stored procedures).
When evaluating Fabric for your organization, ensure you have access to subject matter experts who can guide a successful implementation. Whether that's hiring an internal Microsoft veteran or contracting a highly skilled data consulting firm, consider these resources as part of your budget from day one.
Understanding Data Movement Complexity
When I first started using Fabric, I struggled with data movement decisions. There are numerous considerations when choosing between Dataflow Gen 2, PySpark Notebooks, Copy Activities, or Copy Jobs for pipelining. Each has specific use cases and performance characteristics:
Copy Jobs vs. Copy Activities for Incremental Processing:
If you have incremental merge as a requirement, leverage Copy Jobs or PySpark Notebooks
Copy Activities can perform upserts, but they don't execute true incremental merges—instead, they reload entire tables using unique ID keys
Copy Activities work well for tables with fewer than ~5 million records
Copy Jobs excel at processing tables with millions of rows and are extremely compute-efficient after the initial full load
While Copy Jobs consume 3 compute units versus 1.5 for incremental Copy Activities, they process significantly fewer rows incrementally, offsetting the additional cost
PySpark Notebooks Considerations:
Offers the most control but can quickly burn through Spark compute credits at lower capacities
Best suited for complex transformations requiring high-code solutions
Consider low-code alternatives first for speed and maintainability unless advanced processing is required
Industry-Specific Advantages
If your organization is already a Microsoft shop, Fabric integrates seamlessly into existing ecosystems. This is particularly valuable in industries like healthcare, where Microsoft's dominance means users are already familiar with Excel and Office 365. Fabric's Excel connectivity to Power BI semantic models ensures stakeholders use validated, governed data even in their Excel analyses.
The unified OneLake architecture eliminates the typical segmentation between data tools, creating what feels like a truly monolithic architecture rather than the multiple storage and query layers found in traditional modern data stacks.
2. Plan for Development Lifecycle Management and CI/CD Early
Start with the end in mind. One of the most critical—and often overlooked—aspects of Fabric implementation is establishing proper development lifecycle management and CI/CD deployment pipelines from the beginning.
Team Structure Considerations
Team Size Impact:
Single-person teams may operate effectively with one Fabric workspace initially
As you scale, having Dev/Test/Production environments becomes crucial for maintaining code quality and preventing production errors
Without formal software development frameworks, expect more pipeline failures and cluttered workspaces
Organizational Maturity:
Startups typically move faster and can operate more efficiently with simplified environments initially
Enterprise organizations require more regulation and formal processes to enforce quality control and best practices
Company size often determines how quickly you can move and adapt
Best Practices for Environment Management
Avoid Production Staging Tables: I've operated in data teams that used staging tables in Production environments, which consistently led to:
Inconsistent numbers across reports
Derived queries that were impossible to reverse engineer or validate
Loss of data lineage and governance
Implement CI/CD Early:
Establish formal software development practices as soon as it makes organizational sense
Plan for future team expansion—starting with proper frameworks ensures Fabric scales with you rather than against you
Use workspace separation for Dev/Test/Production to maintain code quality and deployment consistency
Consider Microsoft's guidance on CI/CD overview for Fabric as your starting point for implementation planning. Also note these features are still super early and may not be fully supported. One of our Orchestra / Bank implementations ended up getting massively delayed because the Team realised they wanted to have CI/CD for their production environment, but this wasn’t even possible for Fabric (e.g. with Notebooks)!!!
3. Understand Compliance and Security Requirements (Especially for Healthcare)
Apply the minimum necessary standard. In healthcare and other regulated industries, data compliance isn't just about technology—it's about following established frameworks that the technology must support. Microsoft Fabric provides enterprise-grade security with encryption by default, Microsoft Entra ID authentication, and comprehensive governance tools including data lineage, information protection labels, and data loss prevention.
Core Security Framework
Fabric offers always-on security where every interaction is encrypted by default and authenticated using Microsoft Entra ID, with all communication traveling through the Microsoft backbone internet and data at rest automatically stored encrypted. The platform is HIPAA compliant and meets industry standards such as ISO 27001, 27017, 27018, and 27701.
Three Critical Security Implementation Areas
1. Row-Level Security (RLS) Implementation
Row-level security in Fabric enables you to use group membership or execution context to control access to rows in database tables, with the access restriction logic residing in the database tier rather than individual application layers. However, implementing RLS in Fabric requires understanding its complexities:
Warehouse and SQL Analytics Endpoint RLS: Implemented using CREATE SECURITY POLICY Transact-SQL statements with predicates created as inline table-valued functions
OneLake Security RLS (Preview): Uses SQL syntax in WHERE clauses but has significant limitations—no dynamic user identity functions like USERPRINCIPALNAME(), requiring separate roles for each access requirement. Works with tabular data.
Power BI RLS: Offers more mature dynamic filtering capabilities with DAX functions for user identity
Key RLS Considerations:
RLS simplifies security design by applying restrictions at the database level, making the system more reliable and robust
Plan for the complexity of managing multiple RLS implementations across different Fabric workloads
Current OneLake RLS preview lacks dynamic capabilities, requiring manual role creation and assignment
2. Information Protection and Data Classification
Fabric leverages Microsoft Purview Information Protection with sensitivity labels (General, Confidential, Highly Confidential) that automatically follow data from source to business user across the platform. Key features include:
Automatic Data Classification: Purview DLP policies automatically detect sensitive information as it's uploaded and provide risk remediation actions
Label Inheritance: Sensitivity labels follow data automatically from item to item as it flows through Fabric, even when exported to supported formats like PBIX, Excel, PowerPoint, and PDF
Compliance Monitoring: Administrators receive audit logs for every DLP detection with customizable alerts and user guidance messages
3. Access Controls and Audit Requirements
Fabric administrators and compliance teams can track and investigate user activity using Purview Audit, which is available in the Purview compliance portal and often required for regulatory compliance. Essential capabilities include:
Multi-Level Access Controls: Object-level security, column-level security, row-level security, and dynamic data masking work together to provide comprehensive data protection
Identity-Based Authentication: All authentication handled by Microsoft Entra ID with support for conditional access, multifactor authentication, and device compliance policies
Comprehensive Auditing: Activity and audit logs are enabled by default with 30-day retention, supporting compliance and forensic requirements
Healthcare-Specific Implementation Guidelines
For healthcare organizations handling PHI, implement these additional protections:
Data Residency: Use Multi-Geo capacities to ensure data never leaves required geographical boundaries, with options to disable shortcut caching for S3, GCS, and S3-compatible shortcuts
Enhanced Access Controls: Enforce multifactor authentication and network location restrictions through conditional access policies, with support for bring-your-own-device scenarios through Microsoft Intune
Encryption Management: While Fabric uses Microsoft-managed keys by default, organizations can implement customer-managed keys (BYOK) for imported semantic models in Premium capacities when required for regulatory compliance
4. Navigate Pricing and Capacity Planning Strategically
The largest challenge in our experience is estimating which capacity will be suitable for your organization's Fabric implementation. Unlike traditional per-user licensing models, Fabric operates on a capacity-based pricing structure using Capacity Units (CUs) that represent a pool of compute power shared across all workloads—from data engineering and warehousing to business intelligence and AI experiences.
Understanding the Capacity Model
Microsoft Fabric uses Stock Keeping Units (SKUs) ranging from F2 to F2048, with each SKU providing a specific number of Capacity Units (CUs). For example, F2 provides 2 CUs, F4 provides 4 CUs, and so on, with each level doubling the previous capacity. All CUs are pooled and shared across workloads without pre-allocation, reducing costs by avoiding idle capacity locks.
Key Pricing Characteristics:
Pay-as-you-go billing calculated per second with a one-minute minimum, allowing you to pause capacities when not in use
Reserved capacity offers approximately 40% discount over pay-as-you-go rates with yearly commitments
OneLake storage billed separately at pay-as-you-go rate (~$0.023/GB/month, approximately $23/TB/month) and doesn't consume Capacity Units
Free Mirroring storage included up to capacity SKU limit (e.g., F64 includes 64TB free Mirroring storage), with OneLake storage billed only when free Mirroring limits are exceeded or capacity is paused
Smoothing capabilities spread compute usage over time—background operations smoothed over 24 hours, interactive operations over 5-10 minutes
Capacity Planning Best Practices
Start with a Proof of Concept to gauge capacity needs relative to your data volume and processing frequency. Microsoft offers a 60-day free trial on F64 capacity (equivalent to Power BI Premium P1), but I recommend starting at the lowest estimated capacity and scaling up as needed.
Real-World Benchmarking: For perspective, one of our large healthcare clients processes approximately 350 million records total for one revenue cycle report. We're running at a little over 50% capacity utilization on an F16 (16 CUs). This gives you a baseline for planning your own capacity requirements.
Licensing Considerations:
F64 and larger capacities eliminate the need for individual Power BI Pro licenses for report consumers
Smaller SKUs (F2-F32) still require Power BI Pro or Premium Per User licenses for Power BI content sharing
Non-Power BI activities (pipelines, data warehouses, notebooks) don't require Pro licenses regardless of capacity size
Essential Monitoring with the Fabric Capacity Metrics App
Download and configure the Microsoft Fabric Capacity Metrics App immediately after capacity deployment. This app provides monitoring capabilities for capacity consumption and helps you make informed decisions about scaling, including identification of when to enable autoscale or scale up your capacity.
Key Metrics to Monitor: The compute page shows data for the last 14 days and includes three main visual components:
Cards showing key performance indicators: SKU information, average utilization percentage, and peak utilization percentage (calculated with respect to base capacity units, excluding autoscale)
Multimetric ribbon chart: Provides hourly view of capacity usage with four key values—CU processing time in seconds, duration in seconds, number of operations, and unique users (including service principals)
Matrix by item and operation: Displays metrics for each item on the capacity, sortable by CU consumption, duration, users, and performance delta with color-coded performance indicators
Utilization and Throttling Insights: The app shows both background operations (blue columns representing backend processes like data refreshes) and interactive operations (red columns for user-triggered activities) in 30-second periods. Critical thresholds include:
Interactive delay occurs when 10-minute smoothing crosses delay threshold
Interactive rejection happens when 60-minute smoothing crosses rejection threshold
Background rejection triggers when 24-hour smoothing crosses background threshold
Using Metrics for Data Ingestion Decisions: The capacity metrics help determine which Fabric data ingestion tool is most suitable for specific tables and pipelines. The matrix table allows you to sort by CU consumption to identify top consumers across workloads, with hover functionality providing operation-level details. Different operations have varying consumption rates, and understanding these patterns through the performance delta indicators (color-coded from no color for values above -10, to orange for -10 to -25, to red for below -25) enables you to optimize tool selection for cost efficiency.
Cost Optimization Strategies
Right-Size Your Capacity:
Plan for average requirements rather than peak loads, leveraging Fabric's bursting capabilities for occasional spikes
Proper right-sizing typically saves 20-30% or more on operational costs
For pay-as-you-go implementations, schedule automatic pause/resume cycles during off-hours to avoid idle time charges
Reserved vs. Pay-as-You-Go Decision Framework: Use reserved capacity if your workload runs more than ~60% of the time continuously. Reserved pricing offers ~40% savings but requires yearly commitment regardless of actual usage. Pay-as-you-go works better for sporadic usage patterns or development environments where you can pause capacity between work sessions.
5. Integration Ecosystem and best-in-class
The final thing worth noting is that we RARELY see people using only Fabric. It’s often used in conjunction with Databricks, or even vanilla Azure Data Factory (instead of ADF in Fabric which doesn’t have as many features). This means you need something that can integrate different tools such as Orchestra (or you can build out these integrations yourself in Fabric Notebooks and ADF Custom operators. Your choice.
Orchestra integrates with a bunch of stuff in Azure. You will hope the native integrations between Azure tools e.g. Fabric ADF are as good as these, but they likely won’t be e.g. if a Virtual Machine resource is required (there is no concept of “VM in Fabric”)
The second thing we very commonly see is a more distributed architecture with multiple non-Fabric and Fabric components across multiple teams. This necessitates having something that can orchestrate and monitor across teams (reference architecture here).
The final consideration is networking. It may be non-trivial to set-up networking patterns. For example, regional availability (see this article from Brent Ozar) can make networking more challenging if your operations must be locked down in a region-specific VPC.
Bonus
The community is very upset with Fabric. They don’t think it’s super production ready or well-priced. Why not go on reddit and see what the fuss is about?
Conclusion
Microsoft Fabric offers significant advantages for organizations seeking a unified data platform, but success requires careful planning across technical expertise, development practices, and security implementation. The platform's strength lies in its integration with the broader Microsoft ecosystem, but this also means you need deep Azure knowledge to get the most out of it.
Start your implementation by honestly assessing your team's expertise and budget for external support. Establish proper development lifecycle management from day one—it's much harder to retrofit these practices later. Finally, treat security and compliance as architectural decisions, not afterthoughts, especially if you're handling sensitive data in regulated industries.
When implemented thoughtfully with these considerations in mind, Fabric can indeed deliver on its promise of unified, scalable analytics that grows with your organization's needs.
Learn more
To learn more about Christian and Steinert Analytics check out their work here.
Solid advice, Hugo!
Especially the point about Compliance and Security. In regulated industries, 'default security' isn't enough; you need that deep understanding of RLS and Purview classification you mentioned.
Thanks for sharing that :)
Great points!