The Dark Side of Information: Dark Data in Companies

In the information age, companies face a paradox: while they accumulate vast amounts of data, much of it remains untapped. This “dark side” of information, known as dark data, represents both a challenge and an opportunity for organisations. This analysis delves into the concept of dark data, explores the reasons why companies accumulate it, analyses the challenges involved in leveraging it, and presents use cases that demonstrate its potential for decision-making.

What is Dark Data?

Dark Data is defined as the set of data that organisations accumulate during their normal operations but which is not used for any specific purpose, such as analysis or monetisation ¹. It is data that remains “hidden” or “in the shadows”, unstructured and untagged, located in data warehouses that are often difficult to access ². The rise of connected devices and the Internet of Things (IoT) has contributed significantly to the growth of this type of data (^{3) .}

To better understand the magnitude of Dark Data, we can use the analogy of the “Data-Berg” ⁴. Like an iceberg, where only a small part is visible above the surface of the water, most of a company’s data remains hidden and untapped.

Depending on its accessibility and organisation, Dark Data can be classified into different categories ⁵:

Category	Description
Inaccessible collected	Data that the company has collected but which is located in places that are difficult to access
Collected, accessible but disorganised	Data that is generated in large volumes and cannot be managed without proper categorisation
Accessible but unused data	Data that is well stored and categorised but not used

Why do companies accumulate dark data?

Several factors contribute to the accumulation of dark data in companies:

Cost-effective storage: The advent of low-cost data storage has made it easier for companies to store large amounts of information ‘just in case’, without a clear purpose ⁶.
Regulatory compliance: Companies often store data for long periods to comply with regulations and laws, even after it is no longer useful for day-to-day operations ⁵ .
Lack of knowledge: Many organisations are unaware of the existence or potential value of the data they accumulate ⁶ .
Data silos: Information can become “trapped” in different departments or systems, with other teams unable to access it ⁶.
Digital hoarding: Product teams often accumulate customer data, such as recordings, emails, and bug reports, which are not deleted after use ⁷.
Lack of tools and skills: The lack of adequate tools to process and analyse unstructured data, as well as the lack of personnel with the necessary skills to manage it, make it difficult to leverage Dark Data ² .

Challenges in leveraging Dark Data

Despite the potential of Dark Data, leveraging it presents significant challenges:

Identification and classification: It is essential to identify what data is held, where it is stored, what type of information it contains, and whether it is relevant to the business ¹ .
Security and privacy: Dark data may contain sensitive information that requires security and privacy protection measures ⁸. The loss of confidential information, such as intellectual property, can have negative consequences for the company ⁹.
Integration and structuring: In order to analyse dark data, it must be integrated with other systems and structured in such a way that it is accessible to analysis tools ¹⁰.
Costs and resources: Processing and analysing dark data requires investment in technology, infrastructure and qualified personnel ¹¹.
Data freshness: It is crucial to assess the relevance and timeliness of data, as outdated information may not be useful for decision-making ⁷.
Limited analysis: Companies only analyse 1% of the dark data they accumulate, indicating enormous untapped potential ³ .

Despite these challenges, some companies have managed to turn Dark Data into a valuable source of information, demonstrating its potential for decision-making.

Success stories: companies using dark data

Gas Natural: By installing devices that extract data from old electricity meters, Gas Natural has managed to personalise its offering and service for each customer ¹².
Reale Seguros: This company uses Dark Data to improve customer service, optimise processes and gain a more complete view of its customers’ needs ¹².
Hotels: Analysing Wi-Fi network data allows hotels to understand customer behaviour on their premises, such as peak times at the swimming pool or the most visited areas ⁴ .
Manufacturing companies: Analysing the routes taken by warehouse operators with mobile devices allows for the optimisation of logistics and inventory management ⁴ .
Ryanair: In late 2010, Ryanair used dark data to increase travel insurance sales. By analysing customer interaction data during the booking process, the company optimised the placement of insurance offers, leading to increased sales ¹³.
LinkedIn: LinkedIn has used “friend spam” to grow its user network. By analysing user data, the platform identifies connections and suggests potential contacts, helping users expand their professional network ¹³.

Technologies and strategies for structuring dark data

To convert Dark Data into useful information, companies can use various technologies and strategies:

Enterprise content management (ECM) software: This allows information to be stored, managed and accessed centrally, facilitating the classification and analysis of Dark Data ¹⁴.
ETL tools: These enable the extraction, transformation and loading of information from multiple sources, facilitating the integration and structuring of Dark Data ¹⁴.

The Role of AI and ML in Leveraging Dark Data

Machine learning (ML) and artificial intelligence (AI) play a fundamental role in leveraging dark data:

Analysis of unstructured data: ML and AI can help analyse large volumes of unstructured data, such as emails, text documents or images, to identify patterns and extract relevant information ¹⁰.
Process automation: ML can automate tasks such as data classification, anomaly detection, and event prediction, facilitating the management and analysis of dark data.
Data privacy compliance: AI can help comply with data privacy regulations by automatically redacting confidential information in stored data.

Strategies for managing dark data

To effectively manage dark data, organisations must adopt a comprehensive approach that involves people, processes, and tools ⁷:

People: Foster a data culture within the organisation, where employees understand the importance of data management and are committed to best practices in data hygiene.
Processes: Establish clear policies on data reuse, retention and classification, as well as approval workflows for data use.
Tools: Invest in data management technologies that enable data to be collected, organised, classified and identified, and provide options for its consumption.

Monetising Dark Data

Dark data can not only be used to improve internal decision-making, but can also be monetised in a variety of ways:

Sale of information: Once anonymised and aggregated, information extracted from dark data, such as demographic data or customer behaviour patterns, can be sold to other companies or institutions that can use it for market research, product development or marketing strategies ¹⁵.
Creation of new products and services: Analysis of dark data can reveal new business opportunities and enable the creation of innovative products and services. For example, a company could analyse its product usage data to identify new features or improvements that better meet customer needs ¹.
Improved operational efficiency: Process optimisation and cost reduction through Dark Data analysis can have a positive impact on a company’s profitability. For example, a company could analyse its logistics operations data to identify inefficiencies and optimise delivery routes, reducing transport costs and improving delivery times ¹⁶.

Use cases for dark data in decision-making

Dark data can be used for decision-making in different areas of the company:

Marketing and sales: Analysing customer interactions on social media, email and other channels can help personalise marketing campaigns, improve customer segmentation and increase sales ⁶.
Operations: Analysing data from machine sensors can help predict failures, optimise maintenance and improve production efficiency ¹² .
Human resources: Analysing employee data can help identify turnover patterns, improve recruitment and optimise human resources policies ⁶.
Finance: Analysing historical financial data can help identify trends, predict risks and improve financial decision-making ⁶.
Customer relations: Analysing customer interactions, such as emails, customer service chats and social media comments, can provide valuable insights into their preferences, needs and expectations. This enables companies to personalise their offerings, improve customer service and build stronger relationships ⁴.

Conclusions

Dark data represents a valuable but often untapped resource for businesses. More than half of an organisation’s data falls into this category (^17).While harnessing it presents challenges, organisations that overcome them can gain a significant competitive advantage (^18).The key is to implement appropriate strategies and technologies to identify, classify, structure and analyse dark data, turning it into a useful source of information for decision-making and value creation.

Recommendations for business leaders

Prioritise data governance: Implement clear data management policies, including data classification, retention and disposal, to minimise the accumulation of Dark Data and ensure information security and privacy.
Invest in technology: Acquire and implement tools that enable the analysis of dark data, such as ECM software, ETL tools, and ML and AI solutions.
Foster a data culture: Promote data literacy among employees and create an organisational culture that values the use of data for decision-making.
Explore relevant use cases: Identify areas of the business where dark data can have the greatest impact and develop strategies for leveraging it based on specific business objectives.

Works cited

1. Dark Data: What is it and what is its potential? – Coldview, accessed 13 February 2025, https://coldview.com/dark-data-que-es-y-cual-es-su-potencial/

2. Dark Data, the dark side of Big Data – Insight – Keyrus, accessed 13 February 2025, https://keyrus.com/sp/es/insights/dark-data-el-lado-oscuro-del-big-data

3. The dark side of Big Data, Dark Data, accessed 13 February 2025, https://www.campusbigdata.com/blog/la-parte-oscura-del-big-data-dark-data/

4. What is Dark Data? – Vandalytic – your blog on Power BI and more in Spanish, accessed on 13 February 2025, https://vandalytic.com/que-es-el-dark-data/

5. What is dark data and how is it managed | UE Blog – European University, accessed on 13 February 2025, https://universidadeuropea.com/blog/dark-data/

6. What is dark data? – IBM, accessed 13 February 2025, https://www.ibm.com/es-es/topics/dark-data

7. Dark Data: Concept, Challenges, and Ways to Manage It – Atlan, accessed 13 February 2025, https://atlan.com/dark-data-101/

8. Challenges of Dark Data Claudia Cardoso1 Catherine Collins2, accessed 13 February 2025, https://derecho.udd.cl/cdre/files/2024/06/2-6-claudia-cardoso_catherine-collins.pdf

9. What is dark data and how can it affect your business? – Avantika Blog, accessed 13 February 2025, https://blog.avantika.mx/que-es-dark-data-como-puede-afectar-empresa

10. What is Dark Data: Discover its potential for businesses – Capterra, accessed 13 February 2025, https://www.capterra.es/blog/1170/la-era-del-dark-data-sabes-lo-que-almacenas

11. Dark Data, a major obstacle to the real use of Big Data, accessed on 13 February 2025, https://haycanal.com/noticias/12827/dark-data-un-gran-obstaculo-para-el-aprovechamiento-real-del-big-data

12. 7 examples of how to use the hitherto unknown Dark Data – Insight – Keyrus, accessed on 13 February 2025, https://keyrus.com/sp/es/insights/7-ejemplos-de-como-utilizar-el-hasta-ahora-desconocido-dark-data

13. Dark patterns: use in UX design and examples – IONOS, accessed on 13 February 2025, https://www.ionos.com/es-us/digitalguide/paginas-web/desarrollo-web/que-son-los-dark-patterns/

14. Dark Data and its impact on information security – Coldview, accessed 13 February 2025, https://coldview.com/dark-data-seguridad-informacion-latam/

15. How Cybercriminals Monetise Ransomware Data – TrustDimension, accessed 13 February 2025, https://www.trustdimension.com/como-los-ciberdelincuentes-monetizan-los-datos-de-ransomware

16. How to monetise your data analytics offering? | Reveal BI, accessed 13 February 2025, https://www.revealbi.io/es/blog/how-to-monetize-data-analytics-offering

17. The State of Dark Data | Splunk, accessed 13 February 2025, https://www.splunk.com/en_us/form/the-state-of-dark-data.html18. Dark Data, in search of lost data – ITELLIGENT, accessed 13 February 2025, https://itelligent.es/dark-data/