What is a Distributed Data Mesh and why is it the future?

Centralised Data Lake

Enterprise-wide Data & Analytics calls for a properly managed, but scalable and predictable architecture. These Enterprise architectures remind us of a huge, centralised data lake, often augmented with a structured data store /database to store standardised data for faster retrieval and reporting needs. While the data lake exists to discover the unknown golden nuggets, the structured data store helps in supplying the discovered and known facts faster, and also help with re-usable golden master and reference data.

Large scale architectures call for standard operating procedures, coding standards, architecture standards, business glossaries, etc., which are often taken up by central teams. The division of work and responsibility works well in any manufacturing assembly operations. The same concepts work for Enterprise-Scale architectures as well. Distribution of data and pipelines enables the business domains and respective markets to have ownership of data processing, development with appropriate accessibility. It is this, that brings in the local knowledge associated with their own data.

The multiple facets of Enterprise-wide Data and Analytics cover storage, data classification, data extraction, data ingestion, networking, data security, role-based access management, data reconciliation, standardisation, and monitoring.

The Challenges of distributed responsibility, dealing with a monolithic data lake, and humongous structured data stores are best addressed by using a Distributed Data Mesh architecture. A mesh architecture consists of a data hub(s) with multiple data nodes connected to the hub(s).

The Data Hub(s) usually take on the central responsibilities and repeatable activities pertaining to metadata management, master data management, process monitoring, data quality, data protection tools, etc. The nodes cater to the individual business functional needs or individual markets/countries. Nodes also help in providing clarity on cost ownership, data ownership, and focus on local data processing knowledge.

Data Hub(s) in addition to focusing on the central responsibilities focus on having teams with specialist skills, which otherwise are expensive to the staff at all levels.

There are several advantages associated with this Hub and Node architectures, such as lesser storage space, data ownership along with data standardisation responsibility, well-delineated access management, centralisation of specialist tools and skills, leverages local data knowledge, cost distribution, etc. Most importantly it reduces the complexity of the architecture by leaps and bounds. While this architecture helps in the harmonisation of the tools, it also allows independence on the choice of tools at the node level.

At ACS we have experts who could help your organisation build a future-ready data platform. Complete the form for a free consultation and we shall be in touch to understand your specific requirements.