How to create an enterprise data strategy that actually works
Creating an effective enterprise data strategy is easier said than done.
The sheer scale of a large organization’s data sources and the complex interplay between different departments and stakeholders make this one of the most challenging obstacles in your business’s AI transformation.
To help you find confidence in your data and build your data strategy, here are the three foundational pillars and the set of guiding principles our experts recommend.
The three pillars of data strategy
1. Technology
This encompasses everything from data sources, infrastructure, segmentation, and reporting tools. Like software, it requires routine maintenance and updates.
Here is an example of a data platform tech stack that we launched for MillerKnoll.
- Orchestration: Airflow (via Astronomer) orchestrated data pipelines by enabling the programmatic writing, scheduling, and monitoring of complex workflows using Python. SQL was the data transformation language.
- Compute: Snowflake was the SQL execution engine underlying the logical data warehouse.
- Storage: AWS S3 is used to store files and data.
- CI/CD: CircleCI provided the code deployment with two environments, testing, and prod.
- Source control: Github was the source control system, capturing Airflow source code, scheduling, the logic of the data pipelines and dependencies, data transformations, quality controls, and configurations.
- Data quality: SQL tests were embedded within the data pipelines to ensure data quality.
- Alerting and monitoring: Alerts from failures in airflow or notifications from Data Quality tests were integrated with Slack and Email notifications, ensuring the right people were notified to take action.
2. Processes
Process standardizes data management, governance, and analysis—as well as the protocols related to security and privacy.
When it comes to process, automate as much as you can. For example, we helped Douglas automate daily and weekly reporting so that C-levels had reports each morning, decreasing manual reporting time.
3. Organization
The third pillar of your data strategy defines the roles, responsibilities, and skills of the people (and platforms) supporting it. This involves establishing clear data ownership, fostering a data-driven culture, and providing training and development opportunities for your data team.
Of course, there are exceptions, but generally, a mature data organization should have the following roles in place:
Guiding principles for data engineering
To align each of these pillars in a way that creates the strongest possible data strategy, your data strategy also needs to be informed by a set of guiding engineering principles.
Throughout our experience helping enterprises assess, refine, and create data strategies, here’s what we’ve found works best:
Consolidate business logic
Business logic turns the raw data into new insights. It’s essential in helping the business understand how it’s performing and where gaps may exist. Ideally, business logic exists in a single language (SQL) and lives in a single system. An analyst should be able to find and understand the business logic easily.
Automate almost everything
Moving quickly requires that everything that can be automated is automated. Repeating manual steps creates room for error and eats up time that could be spent better elsewhere. Orchestration doesn’t just apply to the extract/load/transform process (ETL), it can also be used to test, deploy, alert, and other historically manual jobs.
Treat data like software
The process of building data platforms needs to be approached in the same way as software. Just like software, data platforms require constant iteration in terms of agile development, source control & code reviews, continuous deployment & integration, and the broader business’s changing needs.
Bring the internal team along
While a successful data platform is key, coordinating with internal teams is just as critical. Taking the time to onboard them sooner rather than later helps ensure long-term sustainability with the target processes, technologies, and tools.
Improving your enterprise’s data-driven future
An enterprise data strategy acts like an orchestra conductor, harmonizing data from different sources and departments to create a cohesive and beautiful symphony.
Remember, it’s not just about building a data platform; it’s about fostering a data-driven culture that empowers your team and drives your business forward. In a world where data is king, a well-crafted enterprise data strategy isn’t just an option—it’s an imperative.
