Many institutions struggle with a fragmented data ecosystem, where multiple platforms, services, and tools coexist—often using proprietary and incompatible formats. This fragmentation creates information silos, complicates governance, and slows down analytics and decision-making processes. And of course, generative AI has added yet another layer of complexity.

Other cloud providers have noticed the situation, as they have had to respond to market demands with seemingly unified solutions. However, in practice, most of these offerings are merely bundles of the same services that initially contributed to further fragmentation of the data ecosystem.

Analysis of Data Ecosystems (March 2025)

The idea of “platforms” is nothing new. We’ve seen it in numerous solutions we’ve migrated or decommissioned over the years—from licensed commercial options like SAS or Cloudera to platforms built on Hadoop and open-source technologies. But why is a new wave of migrations beginning? And why have hyperscalers like AWS and Azure launched their own new data platforms?

To answer both questions, we must first analyze why customers have started deprecating legacy platforms. Then, we should identify the key drivers—beyond purely commercial reasons— pushing the three major cloud providers to introduce new platform-based offerings.

The Decommissioning of Legacy Platforms

The answer to the first question is relatively straightforward, and in fact, we indirectly addressed it in this blog. The reality is that companies have been gradually phasing out these platforms primarily because they inherited numerous limitations from the on-premise deployments of their time—limitations that made them inefficient and expensive compared to modern cloud-based solutions.

This isn’t just about scalability constraints, challenges in operating under high availability scenarios, or inefficient resource utilization—where capacity was over-provisioned by default (leading to idle resources) and eventually became insufficient. More than anything, the biggest issue was the high cost of licensing.

Historically—and even today—licensing has been a major headache for users in areas like CRM, CBS, ERP, and beyond. Many platforms have adopted licensing models that, at times, have been outright abusive, creating significant frustration among customers—especially during critical moments like negotiations and renewals. How many companies have chosen to migrate after facing disputes with vendors over licensing? The answer: a lot.

And this is where, whether by coincidence or sheer market evolution, the paradigm shift in the cloud environment comes into play. The dawn of traditional licensing models, combined with pay-as-you-go pricing and the scalability and flexibility of the cloud, addressed most of the infrastructure and licensing challenges businesses were struggling with. This naturally led customers to massively and sensibly adopt cloud services—regardless of the fact that, in the early stages, some of the available options were far from perfect.

Cloud providers, in their race to dominate the market, have continuously added services—some of high quality, others with limitations—to address the diverse needs of businesses. Being the first to offer a solution provides a significant competitive edge. For example, Amazon Redshift quickly became the most widely adopted data warehouse solution (with around 6,500 implementations) after AWS acquired a company that had developed a PostgreSQL-based DWH—meaning it wasn’t a cloud-native solution and came with numerous limitations.

As new services were added and integrated into cloud platforms (grouped into categories), fragmentation increased, making governance and scalability more complex. While architects might already be used to these challenges—especially considering things were even worse in the past— building and managing a data ecosystem has become a major challenge. Each of the countless services follows its own roadmap, evolution, and set of incompatibilities, as well as inherited limitations that complicate integration and overall management.

Data fragmentation is not new—it existed before—but it was managed within the platforms that have since been decommissioned, and this is an important point to highlight. Those legacy platforms certainly carried their weight in terms of inefficiencies and drawbacks before the cloud revolution, but they also offered a higher degree of integration between components. Naturally, as time passed and technology advanced, companies ended up acquiring more software than they truly needed, further adding to the complexity of their ecosystems.

Next-Generation Platforms

We are witnessing a new boom in Unified Data & AI Platforms. In December of last year, AWS announced its unified platform, “Amazon SageMaker Lakehouse”, while the year before, Azure introduced “Fabric”. I believe it’s only a matter of time before Google follows suit—but the real question is: Why? And why now?

There are undoubtedly multiple reasons, and many of them are tied to strategic and commercial factors that I won’t dive into here to keep the focus of this blog clear. But one thing is obvious: it’s impossible to look at the conceptual diagram of Amazon SageMaker and not see a direct parallel to Databricks’ philosophy and approach. It’s the same pyramid—but flipped upside down.

Regardless of our technological preferences, one thing is undeniable: Databricks didn’t just introduce but rather adapted an existing “platform” concept to the cloud, evolving it to meet future market demands. By homogenizing and preparing the model for scalability, Databricks bridged the gap between the pre-cloud and cloud worlds, ultimately proving to be a successful and widely replicated approach—even by the cloud providers themselves.

In fact, we’re now seeing cloud-native customers being advised to migrate their ecosystems to this unified platform model. This is significant not just because it validates that “this was the way forward”, but also because it signals a clear trend: the future of data projects will revolve around integrated platforms, with competition centered around Databricks, Fabric, SageMaker, Palantir, and others (excluding Snowflake, given its lack of an end-to-end scope).

At the same time, it’s a bit frustrating—because once again, many companies will have to go through yet another migration cycle, leading to another round of disruption and challenges.

The good news for those betting on Databricks is that it’s not just the pioneer in this concept (being first doesn’t always mean being the best), but it also holds a significant technological advantage.

At this stage, it’s nearly impossible to recommend Amazon SageMaker to a client when it’s still in Preview and under development. A similar case can be made for Fabric, although to a lesser extent.

This makes the recommendation landscape quite clear: Databricks isn’t just the most advanced option—it’s also the most mature and reliable choice in this evolving scenario.

Databricks: Pioneers in Building a Cloud-Native Data Platform

From the very beginning, Databricks has aimed to reduce the complexity of data ecosystems, eliminating unnecessary bottlenecks and optimizing infrastructure to ensure greater interoperability, cost reduction, and efficient data management across the entire organization.

By nature, data is already inherently complex. Adding the burden of managing multiple services, orchestrations, and integrations between incompatible software—along with tools that partially complement each other while also overlapping excessively—only amplifies the challenge.

This reality, which most organizations face today, doesn’t just create friction, higher costs, and frustration—it also distracts companies from their true goal: transforming into a Data + AI-driven organization.

The Databricks Ecosystem

Companies need platforms that go beyond scalability and security—they require solutions that can optimize large-scale data processing, automate workflows, and enhance decision-making through artificial intelligence.

Databricks delivers an ecosystem specifically designed to meet these needs, integrating advanced storage, analytics, machine learning, and data governance into a single, unified platform.

With its Lakehouse architecture, Databricks enables organizations to reduce data fragmentation, consolidating information within a secure, high-performance environment.

The Lakehouse: Breaking Down Silos

Databricks has come a long way since introducing the lakehouse concept in 2019, a model that most modern architectures have now adopted in various forms. At its core, this architectural approach focuses on centralizing data storage in open formats within a data lake to eliminate silos.

By adopting open standards like Delta Lake and Apache Iceberg, Databricks enables seamless read and write capabilities across both, ensuring full interoperability and eliminating lock-in to proprietary formats. This approach not only simplifies architecture but also reduces costs by eliminating the need for multiple copies or redundant data replications.

Additionally, working with open formats enhances integration with other tools and services, improving flexibility and scalability as new analytics and AI demands emerge. By unifying data lakes and data warehouses, Databricks provides a modern, robust solution that fosters innovation while ensuring strong data governance across the entire data lifecycle.

Security & Privacy: Why Does Centralization Matter?

Security and privacy have become critical challenges in any data strategy. Since every stage of the data lifecycle—from ingestion to visualization—can be a potential vulnerability, it’s clear that security and privacy are fundamentally data problems. The proliferation of data sources, formats, and tools only amplifies this complexity.

Moreover, evolving regulatory requirements and increasingly sophisticated threats demand a unified governance approach—one that goes beyond data protection to encompass traceability, regulatory compliance, and cybersecurity threat prevention. Achieving this requires unified platforms that centralize security and privacy management while also enabling team collaboration and the rapid adoption of new technologies.

A strong data governance framework not only safeguards information but also builds trust among internal and external users—a key factor in driving innovation and ensuring data-driven decision-making in a secure and sustainable way.

Unity Catalog: Native Governance to Secure Your Data

Unity Catalog provides a unified, open governance solution that supports all of an organization’s data. It not only protects and manages structured data, but also unstructured data, files, notebooks, and AI assets.

Beyond governance and cataloging, Unity Catalog consolidates oversight, lineage, auditing, and other key capabilities, delivering end-to-end visibility and control—from data ingestion to dashboard generation and model deployment, covering everything in between.

Conclusions

Companies still relying on legacy platforms now have the perfect opportunity to migrate to consolidated solutions like Databricks, embracing a strategic approach that makes sense in today’s landscape. As we’ve mentioned before, data is already complex enough on its own—there’s no need to add more challenges with fragmented and outdated architectures.

Investing in unified platforms not only simplifies management and optimizes performance but also enables scalability, fosters innovation, and ensures long-term competitiveness in a world increasingly driven by data intelligence and automation.

The Chaos of Data: How Fragmentation is Stalling Innovation