Managing the Data

Much in the same way that software engineering teams transitioned to microservice architectures, the data mesh is the data platform version of microservices. Some are still confused as to its purpose, however. Silicon Republic brought this interesting information to our attention in their article, “What is data mesh and why is it important?“

The principles behind data mesh are not new. Organizations already model data, stand up data warehouses, master their data and ensure data quality. Data meshes leverage principles of domain-oriented design to deliver a self-serve data platform that allows users to abstract the technical complexity and focus on their individual data use cases.

Until recently, most companies leveraged a single data warehouse connected to myriad business intelligence platforms. Such solutions were maintained by a small group of specialists and frequently burdened by significant technical debt.

2020 brought us the data lake with real-time data availability and stream processing. This often fell short with disconnected data producers, impatient data consumers and a backlogged data team.

Data meshes provide a solution to the shortcomings of data lakes by allowing greater autonomy and flexibility for data owners, facilitating greater data experimentation and innovation while lessening the burden on data teams to field the needs of every data consumer through a single pipeline.

Melody K. Smith

Sponsored by Data Harmony, harmonizing knowledge for a better search experience.