top of page
Search

Risk assessment in the use of AI

  • oliverluerssen7
  • Apr 12
  • 4 min read

1. Introduction: The Paradox of AI Adoption

We are living in an era of rapid technological upheaval, where the pace of implementation often far outstrips the pace of safeguards. According to recent reports by McKinsey, 78% of all companies are already using generative AI (GenAI) to transform their value chains. Yet behind this boom lies a dangerous paradox: while usage is increasing massively, concerns among executives about cybersecurity, data protection, and the reliability of results are also growing.

In 2025, it is no longer enough to simply “introduce” AI. Companies face the challenge of securing competitive advantage without losing control over the accuracy of their systems or the integrity of their data. Those who focus only on the façade of implementation overlook the structural risks lurking beneath the surface.

 

2. The “Silent Decay”: Why 91% of All Models Have an Expiration Date

One of the most underestimated technical risks is “model drift” — the gradual degradation of model performance over time. Research findings from MIT, Harvard, the University of Monterrey, and Cambridge (2022) demonstrate the scale of this phenomenon:

91% of all machine learning models experience significant performance decline within a few years of deployment.

This “silent decay” occurs when real-world data streams diverge from the original training data or when environmental variables change. It is important to distinguish: while model drift describes performance degradation over time, bias is often already inherent in the training data and leads to systematic errors from the outset.

For companies making automated decisions, ignoring this decay can be fatal. A model that performs accurately today may cause financial losses tomorrow or damage reputation through unfair outcomes.

 

3. The Regulatory Sledgehammer: When Compliance Becomes Existential

With the introduction of the EU AI Act, the landscape of AI governance has fundamentally changed. What was once considered an ethical “nice-to-have” is now a strict legal requirement with severe consequences for violations.

“Compliance with regulatory requirements is non-negotiable if you want to avoid legal trouble.”

The EU AI Act imposes penalties of up to €35 million or 7% of global annual revenue — whichever is higher — for serious violations. For small and medium-sized enterprises (SMEs) and startups, a more lenient rule applies: fines are capped at the lower of the two amounts.

The regulation classifies systems into four risk categories:

  • Unacceptable Risk: Prohibited practices (e.g., social scoring)

  • High Risk: Strictly regulated areas such as recruitment, education, or critical infrastructure

  • Limited Risk: Transparency requirements for systems like chatbots or deepfakes

  • Minimal Risk: No specific regulation (e.g., spam filters)

Similar to GDPR, the AI Act has extraterritorial effect: any global company offering AI systems in the EU or serving EU users must comply with its requirements.

 

4. Shadow AI: The Invisible Threat in the Browser Bar

A major security risk arises from the gap between authorized and unauthorized AI usage. Two phenomena are at play:

  • Shadow AI: Employees using unauthorized browser-based tools

  • Model Sprawl: The uncontrolled proliferation of authorized applications

On average, companies today run 66 different GenAI apps, with around 10% classified as high-risk.

Traditional IT monitoring tools often fail here, as they cannot detect specific attack vectors such as:

  • Prompt Injection (manipulating AI through malicious inputs)

  • Data Poisoning (corrupting training data)

  • Jailbreaking

A real-world example is Samsung: in three separate incidents, engineers entered sensitive source code and proprietary semiconductor designs into ChatGPT to fix errors—unaware that this data could be used for public model training.

Blanket bans are often only short-term reactions; sustainable security requires clear usage policies or the deployment of local LLM instances.

 

5. Legal Liability: When the Chatbot Takes Your Company to Court

The Air Canada case marks a turning point in AI liability law. A chatbot from the airline falsely promised a customer a discount. In court, the company argued that the chatbot was a “separate legal entity” and that it should not be held responsible for its errors.

The court firmly rejected this argument and ruled that the airline is fully liable for promises made by its chatbot. AI-generated commitments are legally binding.

This ruling highlights the necessity of “human-in-the-loop” systems: for critical customer interactions or high-impact decisions, human oversight must be implemented to ensure that AI does not create unintended legal obligations.

 

6. Fundamental Rights: FRIA as the New Gold Standard

A key milestone in ethical and legal risk management is the Fundamental Rights Impact Assessment (FRIA), as required by Article 27 of the EU AI Act for high-risk systems.

Unlike traditional Data Protection Impact Assessments (DPIAs), FRIA goes far beyond data protection and examines impacts on:

  • Non-discrimination

  • Transparency

  • Human dignity

It is not merely a documentation exercise but an ex-ante evaluation that must be completed before procurement or development. A FRIA should play a decisive role in determining whether a system is deployed at all, by systematically analyzing both typical and worst-case scenarios.

 

7. Conclusion: A Living System Instead of Rigid Rules

AI risk management in 2026 is not a static project but a continuous lifecycle. The NIST AI Risk Management Framework defines four core functions:

  • Map: Understand context and identify risks

  • Measure: Assess risks quantitatively and qualitatively

  • Manage: Take prioritized actions to mitigate risks

  • Govern: The overarching function that integrates all others and establishes a culture of accountability

True AI governance is not a brake on innovation, but its essential foundation. Only those who understand, measure, and manage their systems can responsibly harness the transformative potential of the technology.

Are the companies ready to take full legal responsibility for a decision of their AI makes today without human oversight?


Following sources were used to create this content:


Editorial note: The content reflects my personal opinions and does not directly represent the views of the CANCOM Group.

This text was created with the help of AI.

 
 
 

Comments

Responsible for the content:

Oliver Lürssen

bottom of page