Explore the fascinating relationship between Artificial Intelligence (AI) and data, and how this duo is revolutionising various sectors, from medicine to finance. Discover how data quality and availability directly impact the success of AI models, driving innovation and efficiency. Join us on this journey to understand the challenges and opportunities presented by this symbiosis, including success stories and failures in AI projects.

The impact of data quality on AI performance

AI, especially in its Machine Learning (ML) and Generative AI forms, relies on data for its learning and development. Essentially, the quality of this data is the fundamental pillar that determines the efficiency and accuracy of AI models. High-quality data, characterised by its accuracy, completeness, consistency and relevance, is crucial to the success of any AI project. On the other hand, low-quality data, which may be inaccurate, incomplete or biased, leads to inaccurate, unreliable and even harmful AI models.

To illustrate this point, let us imagine an AI system designed to predict credit risk. If the data used to train this system contains errors or is incomplete, the model could generate erroneous predictions, leading to incorrect credit decisions with negative financial consequences.

Machine Learning and Generative AI: Dependence on Data Quality

Both ML and Generative AI are highly sensitive to data quality. In ML, algorithms learn patterns and relationships from training data to make predictions or decisions. If the training data is erroneous or incomplete, the resulting model will be inaccurate. For example, an ML model trained to diagnose diseases from medical images, if based on low-quality data, could lead to incorrect diagnoses with serious implications for patient health.

Generative AI, on the other hand, uses data to create new content, such as images, text, or music. The quality of the training data determines the quality and originality of the generated content. Low-quality data can result in repetitive, unoriginal, or even inappropriate content. Imagine a generative AI model trained to write news articles. If the training data is of low quality, the model could generate articles with misinformation or inappropriate language, damaging the credibility of the source(^{1) .}

Delving deeper into the impact of data quality on different types of AI, we observe the following:

• Supervised learning: In this type of learning, the quality of the labelled data is crucial. If the labels are incorrect or inconsistent, the model will learn erroneous patterns, resulting in inaccurate predictions.
• Unsupervised learning: Data quality influences the model’s ability to identify meaningful patterns and clusters. Noisy or incomplete data can make it difficult to identify relevant patterns.
• Reinforcement learning: The quality of feedback data is critical for the model to learn to make optimal decisions. Erroneous or incomplete feedback data can lead to inefficient learning and poor performance.

Examples of failed AI models due to poor data quality

Throughout the development of AI, there have been cases where poor data quality has led to the failure of ambitious projects. These examples serve as reminders of the critical importance of data management in AI development.

• Amazon’s hiring bias: Amazon was forced to abandon a recruitment algorithm that showed bias against women. The system, trained with historical company data, learned to favour male candidates due to the predominance of men in technical roles in the past. This bias in historical data was reflected in the AI model, perpetuating gender inequality in the hiring process(^{1) .}
• Bias in Google ads: A study revealed that Google’s online advertising system showed higher-paying job ads to men than to women, perpetuating the gender pay gap. This bias originated in the data used to train the system, which reflected existing wage inequalities in the labour market¹.
• Bias in Midjourney: When Midjourney, an AI tool for image generation, was asked to create images of people in specialised professions, it was observed that the older people represented were always men, reinforcing gender bias in the workplace. This bias was due to a lack of diversity in the training data, which did not reflect the participation of older women in professional roles¹.

These cases illustrate how biased data can lead to discriminatory results, perpetuating existing inequalities. It is essential that AI developers are aware of these biases and take steps to mitigate them by using training data that is diverse and representative of reality.

Data manipulation attacks and AI

Data manipulation attacks pose a significant threat to AI systems. These attacks seek to alter or modify data to compromise the integrity and reliability of AI models.

Attackers can employ various techniques to manipulate data, including injecting false data, modifying existing data, or deleting crucial data. These actions can have a devastating impact on AI systems, leading to erroneous predictions, incorrect decisions, and even system failure.

An example of a data manipulation attack is the injection of false data into an AI system used for fraud detection. By introducing false data that simulates legitimate transactions, attackers can deceive the system and cause fraudulent transactions to go undetected.

A specific type of data manipulation attack is data poisoning, which targets the training process of AI models. In this type of attack, attackers introduce malicious data into the training dataset with the aim of corrupting the model and affecting its performance.

There are different types of data poisoning attacks, such as injecting random noise or introducing irrelevant data into the training set. These attacks can affect the model’s ability to generalise from the training data and lead to inaccurate or biased predictions.

Success Stories: Companies that Optimised Their AI Projects with Quality Data

Despite the challenges, many companies have recognised the importance of data quality and have managed to optimise their AI projects by improving their data. These success stories demonstrate the power of effective data management in AI development.

• Spotify: The music streaming giant uses the “Squad” model, where small cross-functional teams work independently on different aspects of the product. Each team has the autonomy to decide what to work on and how to do it, allowing for greater agility and efficiency in the development of new features. This decentralised model facilitates data management by allowing each team to focus on the data relevant to their area of work.
• Johnson & Johnson: Known for its decentralised structure, Johnson & Johnson has many units that operate autonomously. Some focus on specific product components, which requires cooperation between them. This structure allows for greater specialisation and a faster response to market needs. Decentralisation also facilitates data management by allowing each unit to manage the data relevant to its area of expertise.
• Illinois Tool Works: This decentralised company is divided into a series of units, each with a different function. The company further divides the units if they begin to outperform or fall behind the competition. This structure allows for precise identification of what works and what does not, based on the successes and failures of the various units. Data management in this model is based on the collection and analysis of performance data from each unit, enabling more informed decision-making.

These examples demonstrate how effective data management, including data collection, cleaning, organisation, and analysis, can significantly improve AI performance and lead to success in AI projects.

Failures due to poor data management in AI projects

Poor data management can be a major obstacle to the success of AI projects. Lack of data, poor data quality, or lack of access to data can lead to the failure of AI projects.

• Ford Pinto: Despite the ease with which the Pinto model caught fire due to its design, Ford refused to recall it until the US government forced it to do so. This is an example of poor business decision-making that prioritised economic profits over consumer safety. The lack of data analysis on vehicle safety and the lack of transparency in communicating risks contributed to this failure².
• Nestlé Lactogen: In the 1970s, Nestlé conducted an aggressive marketing campaign for its Lactogen powdered milk in countries with limited access to clean water. This ethically questionable decision ignored the needs and health of consumers. The lack of consideration of socio-economic and cultural factors in the marketing strategy contributed to this failure².

These cases demonstrate how a lack of consideration of the ethical and social implications of AI can lead to negative consequences. It is crucial that companies developing AI projects take into account not only the quality of the data, but also the social and ethical impact of their decisions.

Best practices for data management in AI projects

To ensure the success of AI projects, it is essential to implement best practices for data management. These practices include:

Best Practice	Description
Know the data	Understand the origin, nature, quality, and context of the data used in the AI project. This includes identifying potential biases, assessing the completeness and accuracy of the data, and understanding how the data was collected and processed.
Organise the data	Implement an organised and efficient data structure that facilitates data access, management, and analysis. This may include the use of databases, data warehouses, or data lakes, as well as the implementation of metadata schemas and data catalogues.
Maintain data integrity	Ensure the accuracy, consistency, and reliability of data throughout its lifecycle. This involves implementing data quality controls, data validation, and data version management.
Ensuring data privacy and security	Protect data from unauthorised access and misuse. This includes implementing security measures such as encryption, access control, and data anonymisation, as well as complying with data privacy regulations.
Obtaining company acceptance	Engage stakeholders in the data management process. This includes clearly communicating data policies, obtaining stakeholder approval for AI projects, and managing stakeholder expectations regarding data use.
Set objectives and metrics	Define clear and measurable objectives for data management and AI performance. This includes establishing key performance indicators (KPIs) for data quality, AI model efficiency, and the business impact of the AI project.

Tools and technologies to improve data quality

There are a variety of tools and technologies that can help improve data quality for AI projects. These include:

• Data discovery tools: These enable the identification and cataloguing of available data. These tools help businesses gain a comprehensive view of their data assets, making it easier to identify data relevant to AI projects.
• Data cleansing tools: These help identify and correct errors in the data. These tools can automate tasks such as detecting outliers, correcting inconsistent data, and removing duplicates.
• Data enrichment tools: These enable additional information to be added to existing data. These tools can be used to aggregate data from external sources, such as demographic data or geographic information, to improve the quality and usefulness of data for AI.
• Data analysis tools: These facilitate the exploration and analysis of data. These tools enable data scientists to visualise data, identify patterns, and gain insights that can be used to improve data quality and AI performance.
• Data management platforms: These provide a centralised environment for data management. These platforms offer a range of functionalities, such as data integration, data quality, data governance, and data security, to help organisations manage their data effectively.

Specific examples of tools that can be used to improve data quality for AI include:

• Nessus: A vulnerability scanning tool that can help identify and remediate security vulnerabilities in data systems.
• QualysGuard: A cloud-based vulnerability management platform that offers a range of functionalities for risk assessment, vulnerability detection, and patch management.
• OpenVAS: An open-source vulnerability scanner that can be used to detect and assess security vulnerabilities in systems and applications.

Data availability and its impact on AI

Data availability refers to the ease with which data can be accessed and used for AI projects. Greater data availability means that AI models have access to a wider range of information, which can improve their accuracy and performance.

Data lakes are an example of technology that facilitates the storage and analysis of large amounts of data, improving data availability for AI applications. Data lakes allow companies to store data in its original format, without the need for pre-structuring, which facilitates the ingestion of data from various sources.

Data availability is also affected by factors such as data infrastructure, data access policies, and data management tools. Companies seeking to improve data availability should invest in a robust data infrastructure, implement clear data access policies, and use data management tools that facilitate data access and use.

Data fabric: Weaving a unified data landscape

Data fabric is a data management approach that seeks to create a unified view of an organisation’s data. This is achieved by integrating data from various sources, creating a centralised data catalogue, and applying data governance policies.

Data Fabric uses a combination of technologies, such as data virtualisation, data integration, and metadata management, to create a layer of abstraction over data silos. This allows users to access data in a consistent manner, regardless of where it is stored or how it is structured.

The Data Fabric architecture consists of several key components, such as data connectors, a data catalogue, a policy engine, and an analytics engine. These components work together to provide a unified view of data, making it easier to access, manage, and analyse data.

Data Mesh: A decentralised approach to data management

Data Mesh is a data architecture paradigm that promotes the decentralisation of data ownership and management. Instead of centralising data in a single data warehouse or data lake, Data Mesh distributes data ownership to the business domains that know it best.

Each business domain is responsible for managing its own data, including data quality, data security, and data access. Business domains are also responsible for creating data products, which are sets of data made available to other domains and users within the organisation.

Data Mesh is based on four key principles:

• Domain-oriented architecture: Data is organised around business domains, enabling more agile and efficient data management.
• Data as a product: Business domains treat data as a product, which means they are responsible for data quality, security, and availability.
• Self-service data infrastructure: Business domains have access to a self-service data infrastructure that allows them to manage their data independently.
• Federated data governance: Data governance is distributed among business domains, allowing for greater flexibility and adaptability.

Cybersecurity measures for AI data

Data security is crucial to the success of AI projects. The data used to train and operate AI models must be protected against unauthorised access, manipulation and loss.

Organisations must implement a series of cybersecurity measures to protect AI data, including:

• Strong authentication: Implement strong authentication measures, such as multi-factor authentication, to prevent unauthorised access to data systems.
• Software updates: Keep software and systems up to date with the latest security patches to protect against known vulnerabilities.
• Employee training: Train employees on cybersecurity best practices and phishing awareness to prevent social engineering attacks.
• Firewalls: Implement firewalls to protect networks and data systems from unauthorised access.
• Data encryption: Encrypt sensitive data, both at rest and in transit, to protect it from unauthorised access.
• Data backups: Perform regular data backups to ensure recovery in the event of data loss or damage.

Data integrity in AI

Data integrity refers to the accuracy, consistency, and reliability of data. It is essential to the success of AI projects, as AI models rely on accurate and reliable data to learn and make decisions.

Data integrity can be affected by a number of factors, including human error, system errors, and malicious attacks. Organisations should implement measures to ensure data integrity, such as data validation, data cleansing, and data version control.

Data integrity is also closely related to data security. Security measures, such as access control and encryption, help protect data integrity by preventing unauthorised access and data manipulation.

Ethical implications of poor data quality in AI

Poor data quality can have significant ethical implications in AI applications. Biased or inaccurate data can lead to discriminatory outcomes, perpetuate existing inequalities, and erode trust in AI.

Companies developing AI projects must carefully consider the ethical implications of data quality. They should take steps to mitigate biases in data, ensure data privacy, and use AI responsibly and ethically.

Data governance plays a crucial role in mitigating the ethical risks of AI. Strong data governance practices, such as defining clear data policies, assigning roles and responsibilities, and implementing oversight mechanisms, can help ensure that AI is used ethically and responsibly.

The rise of data-centric AI

In recent years, there has been a shift towards data-centric AI development. This approach focuses on improving data quality rather than simply optimising AI models.

Data-centric AI development recognises that data quality is the most important factor for the success of AI projects. By improving data quality, organisations can improve the accuracy, reliability, and fairness of AI systems.

This approach involves a number of practices, such as feature engineering, data cleaning, data augmentation, and data validation. It also involves a cultural shift within organisations, where data quality becomes a priority for everyone involved in AI development.

Conclusion

Data quality and availability are crucial to the success of AI projects. High-quality data enables AI models to learn effectively, leading to better performance and more accurate results. Poor data management, on the other hand, can lead to the failure of AI projects, negative consequences, and even the perpetuation of existing biases.

Companies looking to harness the power of AI must prioritise data management. Implementing best practices, using the right tools, and considering the ethical implications of AI are key factors for success. By understanding and addressing the challenges of data management, companies can unlock the full potential of AI and gain a competitive advantage in today’s business landscape.

In the future, the importance of data quality and availability for AI will only increase. As AI becomes more sophisticated and is used in a wider range of applications, the need for high-quality data will be even greater. Companies that invest in data management will be better positioned to harness the power of AI and lead innovation in their respective industries.

Works cited

1. Examples of AI biases | IBM, accessed 12 February 2025, https://www.ibm.com/es-es/think/topics/shedding-light-on-ai-bias-with-real-world-examples

2. How to make good (bias-proof) decisions | IESE Insight, accessed 12 February 2025, https://www.iese.edu/es/insight/articulos/tomar-buenas-decisiones/