Why Strong Data Engineering Makes Your AI Project Scalable

Quick summary

Data engineering is the discipline that ensures data is reliable, structured, and readily available for AI models and Business Intelligence. Without that foundation, even the most advanced model will underperform. According to the Dutch Central Bureau of Statistics (CBS), by 2024 nearly a quarter of Dutch companies with ten or more employees were already using AI technology. But when it comes to scaling, the bottleneck is usually the data infrastructure, not the algorithm.

• Data quality determines model quality: incomplete or inconsistent data leads to unreliable AI output in real-world use, no matter how sophisticated the model is.
• Data engineers typically spend most of their time preparing data, not building models, which highlights the value of automated pipelines.
• The EU AI Act introduces mandatory data governance requirements for high-risk systems from 2 December 2027, including demonstrable data quality and a minimum ten-year retention period for technical documentation.
• The Dutch Data & AI consulting market is projected to grow to ten billion euros by 2027, with investment shifting from algorithms to the underlying data platform infrastructure.
• Twentynext brings Data Engineering, Data Science, and AI together in one integrated approach, always starting from the business challenge.

Introduction (Solutions)

"Better to spend one extra hour now, or have one difficult conversation, than deliver a half-finished solution."

— Martijn

Imagine an operations manager at a mid-sized manufacturer who wants to use predictive maintenance to reduce downtime. The model gets built, and the early results look promising in testing. But once the system goes live, the recommendations become inconsistent: sometimes it warns too late, sometimes not at all. The team starts looking for a better algorithm. That’s the wrong place to look.

In practice, the issue in cases like this almost always comes back to the data. Sensor feeds from three different systems use different timestamp formats. Measurements go missing during shift changes. Historical failures were labeled inconsistently. No model fixes those problems on its own.

Twentynext, a Dutch data and AI agency based in Eindhoven, sees this pattern time and again in organizations trying to scale AI. That is why its approach does not start with the model, but with the data foundation: the pipelines, transformations, quality checks, and architectural decisions that determine whether AI applications stay reliable in production.

What is data engineering, and why is it the critical link? (Services)

Data engineering is about designing, building, and managing the infrastructure that collects, transforms, and delivers data for analytics and AI models. Think ETL processes (Extract, Transform, Load), data pipelines, data warehouses, data lakes, and the quality controls that ensure downstream systems can rely on clean, usable input.

The difference between data engineering and data science

A common misconception is that data scientists and data engineers largely do the same job. In reality, they play complementary roles. The data scientist builds and evaluates models; the data engineer makes sure the right data is available, clean, and delivered at the right time. Without solid data engineering, data scientists often spend a large share of their time manually cleaning data instead of developing models. That is not just inefficient, it also makes results harder to reproduce.

Why scaling almost always hits a data infrastructure wall

A proof of concept often works on a limited dataset that has been manually cleaned. Moving into production means that same process has to run automatically, reliably, and continuously at much larger volumes, across multiple sources, and with fluctuating data quality. That is exactly where data engineering makes the difference. According to Erasmus University Rotterdam, building scalable data infrastructure requires specialist expertise in ETL processes, cloud-native solutions, big data technologies, and data governance.

The link between data quality and AI performance

AI models learn patterns from data. If that data is incomplete, inconsistent, or biased, the model inherits those flaws and may even amplify them. Research on the topic shows that even a small percentage of low-quality data can have a disproportionately large impact on model behavior, especially in edge cases. For production-grade AI applications such as the eye disease detection and tumor classification projects Twentynext works on, that is not a theoretical concern. It is a clinically relevant one.

Get started yourself:

• Map out the data sources for your AI use case: how many systems provide input, and who owns each one?
• Check whether timestamps, units, and category values are consistent across all sources.
• Test the pipeline for completeness: are there periods or segments where data is consistently missing?
• Assign an owner to each data source who is responsible for quality monitoring.

What does a strong data foundation look like in practice?

A scalable data foundation is not a single tool or platform. It is a combination of architectural choices, processes, and agreements. Twentynext starts with the business challenge: what does the system ultimately need to predict or decide, and what data is required to make that possible?

Architecture layers that make scaling possible

A typical data architecture for scalable AI includes at least three layers. The ingestion layer collects data from source systems through API integrations, batch uploads, or real-time streams. The transformation layer handles validation, normalization, and enrichment. The serving layer makes curated datasets available to models, dashboards, and reporting. Each layer requires its own decisions: which cloud platform, which orchestration tool, which validation rules.

Governance as part of the foundation, not an afterthought

As of 1 August 2024, the EU AI Act is in force. For high-risk AI systems, it introduces mandatory requirements around data governance: training data must be demonstrably high quality, representative, and free from bias. Technical documentation must be retained for at least ten years. In other words, governance is no longer an optional layer on top of the technology. It needs to be built into the architecture itself. Data lineage, audit logs, and quality reporting become requirements, not nice-to-haves.

Twentynext supports clients by treating AI governance as an integral part of architecture design. That helps organizations avoid expensive rework later on just to meet regulatory requirements. For a deeper look at what this governance framework involves, the article on the AI governance framework for Dutch organizations offers a practical overview of the six core elements.

Get started yourself:

• Identify which AI systems in your organization may fall under the AI Act’s high-risk category.
• Check whether you can demonstrate data lineage from source system to model input.
• Put a retention policy in place for technical documentation, with a minimum of ten years for high-risk systems.
• Make sure data quality reports are generated automatically rather than relying on manual work.

How does Twentynext apply data engineering in industrial and medical projects?

Twentynext uses data engineering as the backbone of a wide range of AI applications, from medical image analysis to manufacturing. The pattern is always the same: the business challenge determines what data is needed, and the data engineering approach determines whether the model will remain reliable in production.

Digital pathology and eye disease detection

To support the detection of more than thirty eye conditions at an early stage, Twentynext developed a self-learning application that analyzes retinal images at a microscopic level. The technical challenge here is not just the deep learning model. It also lies in consistently processing image data from different devices and labs. Variations in lighting, resolution, and color calibration need to be normalized in the data engineering pipeline before the model can work with the images reliably. The same principle applies to tumor classification in digital pathology, where adaptive color analysis in the pipeline compensates for differences in IHC staining between labs.

CAD/CAM and shipbuilding: mass customization through structured data

In manufacturing, Twentynext built an AI module for AutoCAD that generates production-ready stairlift designs based on individual staircase measurements. The data engineering challenge here is straightforward: every customer delivers input in a different format, and the module must interpret that data consistently before the design logic can be applied. For the AI Lightweight Construction project in collaboration with a shipbuilding software partner, Twentynext combines machine learning, genetic algorithms, and a rule-based inference engine. All three techniques depend on well-structured input data to produce useful output.

CRISP-DM as a methodological framework for data engineering

Twentynext uses CRISP-DM across all projects, giving explicit attention to the data understanding and data preparation phases before modeling begins. In practice, that means data engineers and data scientists design the data pipelines together at an early stage, rather than waiting until the model is finished. This significantly reduces the risk of unexpected issues during production rollout. If you want to learn more about how CRISP-DM fits into modern AI projects, take a look at the article on what CRISP-DM adds to modern data science projects.

Get started yourself:

• Bring data engineers and data scientists into the business analysis phase from the start, not only during modeling.
• Document the expected format, frequency, and responsible owner for every data source.
• Build validation rules into the ingestion layer so poor-quality data cannot silently enter the pipeline.

Checklist: best practices for data engineering as the foundation of AI

Best Practices Checklist for Data Engineering, Data Science, and AI:

• Define data sources and ownership early: For each source, decide who is responsible for quality and availability before engineering begins.
• Automate quality checks in the pipeline: Manual checks do not scale; build automated validation rules into both the ingestion and transformation layers.
• Design for reproducibility: Make sure every data transformation is documented, version-controlled, and repeatable so model retraining stays controlled.
• Implement data lineage from day one: Traceable data origin is required under the EU AI Act for high-risk systems and speeds up troubleshooting in practice.
• Plan for drift detection: Data patterns change over time; build monitoring that flags when input distributions drift away from the training distribution.
• Tie governance to architecture: Treat privacy rules (AVG), retention obligations, and access controls as architecture requirements, not as compliance paperwork after the fact.
• Test the pipeline for scalability: Validate whether the infrastructure can handle two to five times the current data volume without manual intervention.
• Put ISO-aligned management processes in place: Twentynext works with ISO-certified service and management processes for production environments, including drift detection, periodic retraining, and incident response.

What should you avoid when building a data foundation?

Starting with the model instead of the data

The most common mistake in AI initiatives is jumping straight into model selection and experimentation while the data pipelines are still unstable. The result is predictable: test results look good, but they cannot be reproduced in production. Twentynext works from the principle that the data infrastructure must be production-ready before a model can be seriously evaluated.

Treating data governance as a separate responsibility

Organizations that handle governance as a standalone compliance exercise usually run into trouble once AI systems begin to scale. Data governance works best when it is embedded in the day-to-day workflow of data engineers: in table naming conventions, in transformation documentation, in access rights across environments. If you try to bolt it onto an existing infrastructure later, the price is far higher than if you build it in from the start.

Letting hype drive technology choices instead of business needs

The market for data platforms and AI tooling is evolving fast. The Dutch Data & AI consulting market is projected to reach ten billion euros by 2027, and companies are putting more and more money into the underlying infrastructure. That makes technology selection both more urgent and more risky. A platform that is a perfect fit for a large enterprise data team may be oversized and unmanageable for a mid-sized organization. Twentynext always starts platform selection from business requirements: expected data volume, required latency, internal management capacity, and compliance obligations shape the architecture, not a vendor’s marketing promise.

Get started yourself:

• Test every tool choice against three concrete business requirements before moving forward.
• Ask your vendor for a reference case from an organization with similar scale and complexity.
• Check whether the chosen platform matches the management capacity of your internal team, or the service model of your implementation partner.

Frequently asked questions

What is the difference between data engineering and data science?

Data engineering focuses on building and managing the infrastructure that makes data available: pipelines, transformations, storage, and quality controls. Data science focuses on analyzing that data and building models. The two disciplines are highly complementary: without strong data engineering, data scientists cannot work reliably or at scale. In practice, the boundary can sometimes blur, but the core responsibilities are clearly different.

Why do AI projects so often fail in production when they performed well in testing?

Production environments are more complex and far less predictable than test setups. Data comes from multiple systems with varying quality, timestamps are not always aligned, and volumes are larger than in a test set. If the data pipelines are not designed to handle that variability, the model inherits those problems. Industry experts consistently point out that most AI projects that never make it to production fail because of data-related issues, not model complexity. That makes a robust data foundation the smartest investment if you want to improve success rates.

What does the EU AI Act require when it comes to data engineering?

The EU AI Act, which came into force on 1 August 2024, imposes mandatory data governance requirements on high-risk AI systems. Training data must be demonstrably high quality, representative, and free from systematic bias. Technical documentation, including data descriptions and quality reports, must be retained for at least ten years. These obligations take effect in phases: for high-risk applications in areas such as biometrics, employment, and education, they become applicable from 2 December 2027. Organizations building their data infrastructure now would be wise to bake governance into the architecture from the start.

How does Twentynext help build a scalable data foundation?

Twentynext combines Data Engineering, Data Science, Business Intelligence, and AI in one integrated service offering from the Brainport ecosystem in Eindhoven. The approach starts with the business challenge: what problem needs to be solved, what data is required, and how should the infrastructure be designed so the solution is scalable and manageable? With ISO-certified service and management processes, Twentynext helps ensure production models remain reliable after go-live, including drift detection, periodic retraining, and incident response. You can read more about the approach via Twentynext’s Data Science and AI solutions.

How do data engineering and Business Intelligence relate to each other?

Business Intelligence depends on the data structures provided by data engineering: curated datasets, consistent definitions, and reliable historical data are the building blocks of every trustworthy dashboard or report. Organizations that try to solve reporting issues by adding more visualizations without strengthening the underlying data engineering are treating the symptom, not the cause. The article on why your dashboard doesn’t make decisions explains why BI only becomes valuable when it is built on a dependable data foundation.

Conclusion

Scaling AI does not start with the model. It starts with the data infrastructure underneath it. Organizations that invest in well-designed data pipelines, automated quality checks, and thoughtful governance structures build a foundation on which multiple AI applications can grow, without reinventing the wheel every time. Regulatory pressure from legislation such as the EU AI Act means this is no longer optional. It is a requirement.

Twentynext takes an approach in which data engineering, Data Science, and AI are not treated as separate disciplines, but as one connected service built around the business challenge. For organizations in Eindhoven and across the Netherlands that want to take concrete next steps, the company’s career and project pages also offer insight into how data professionals work on these challenges in practice. If you want to know how strong your own data foundation really is, that is a question worth asking before the model is finished, not after.