How DataOps can accelerate value creation for your business

Introduction to DataOps?

DataOps is a agile approach with a set of processes aimed at delivering analytics for an organisation. There are a set of key principles upon which DataOps has been developed which are published at https://www.dataopsmanifesto.org/. DataOps follows agile methodology which means all principles of agile will be applicable and will help us to accelerate value creation :

• Individuals and interactions over processes and tools
• Working analytics over comprehensive documentation
• Customer collaboration over contract negotiation
• Experimentation, iteration, and feedback over extensive upfront design
• Cross-functional ownership of operations over siloed responsibilities

DataOps comprises of both development and operations teams. Development team would generally comprise of different types of roles including

• Data Analysts
• Data Scientists
• Data Engineers
• Data Architects
• Developers

Operations team would comprise of both customers and support (for monitoring). There has been a detailed ebook published by datakitchen which should a more detailed insight into DataOps.

DataOps aligns with most of the phases of the DevOps methodology except it varies in 2 aspects

1. “Orchestrate” Phase
2. “Test” Phase

Orchestrate phase in itself is a set of processes allowing the team to look into making sense of the data before it can be available for deployment and general consumption. Orchestration allows DataOps team in creation of value pipeline and is a 5 step process including access, transform, model, visualise and report which is shown below:

Test phase is little more complicated than usual DevOps as it adds complexity for environments, data replications, security and governance. Some of challenges might include

• Size of the data generally various from gigabytes(GB), terabytes (TB) or even petabyte (PB)
• Data Cleansing
• Data anonymization
• Infrastructure scalability

All of should be well defined and a clear understanding is required of how to setup data in each of environments.

In order for DataOps to be successful and provide valuable insights, it should be treated as a shared goal for business functions.

In order for DataOps to generate value for each and every business function within the organisation, it is very critical for everyone to have a DataOps mindset. Everyone should look at this as a common goal. These are the key elements which will enable DataOps to become a shared goal and generate value :

• Standardisation of Metrics (to be agreed by all business functions)
• Meta Data Management (available at common place and accessible)
• Universal Access to Data at the outset (ensure controls / security in place)
• Data Centralisation (eliminate siloes or islands and agree on a common approach)
• Data Governance (define and implement policies, audits etc..)

Also an Information Governance Board should be established in order to promote DataOps allowing all business functions to collaborate and accelerate the journey of DataOps and create value. Data Analytics will generate more and more value over time as it gets more and more enriched.

Data Architecture should be revisited and should be adopted which will promote DataOps culture. In some cases Data Architecture might need some minor changes in its implementation and in other cases this might impose a restart from scratch. All changes can be made over a period of time as the organisation gain maturity. We have recently published an article “Moving into the future with a Distributed Data Mesh“, which should give you some pointers of how you define your Data Architecture.

We now call data as an asset and you can get value of your assets using DataOps. Based on a report produced by IDC only 2.5% of data has been used and analysed by organisations to get insights into ther data. The beauty of DataOps is it ensures the practice you establish is more streamlined and sustainable for longer term, allows you gain value of your data insights allowing you to have a competitive edge.

At ACS we have experts who could help your organisation build a DataOps team and implement all necessary capabilities for your data platform. Complete the form for a free consultation and we shall be in touch to understand your specific requirements.