Notes
2 AI as disruptive technology
Recent studies indicate that the use of AI among young people has grown significantly in recent years, including in Luxembourg. AI is now embedded in both educational contexts and leisure activities. Alongside enthusiasm for technological innovation, young people themselves express ethical concerns — particularly about the relationship between writing and human identity. As Higgs and Stornaiuolo (2024) note, adolescents worry that delegating meaning-making to machines threatens authenticity and creativity. Lev Manovich (2025) has described this phenomenon as the rise of “artificial subjectivity,” highlighting how automation of core elements of self-expression poses unique risks during formative years.
This level of awareness among young people indicates that AI is widely recognised as a disruptive technology. Public debate reinforces this perception by presenting utopian and dystopian scenarios of its consequences. The term artificial intelligence, however, is often used inconsistently in these debates. This chapter opens by clarifying what “AI” means in our context and why it represents a turning point in the digital world. It then explores AI megatrends that are particularly relevant for younger audiences before mapping out the risks, including showing how AI reshapes thinking and decision-making. It concludes with implications for educational actors and policy makers to address these risks before concluding with a brief overview of the current regulatory landscape covering AI.
2.1 What AI are we talking about
AI can be broadly defined as the field of research and technology development concerned with creating data-driven algorithmic applications that perform tasks typically associated with human intelligence. These tasks include pattern recognition, learning, perception, decision-making, creative production (such as composing or filmmaking), and language processing. In public discourse, there is often no clear distinction between simulation and the actual technological replication of human cognitive processes (Mersch, 2020). While the simulation of intelligence has been a goal of computer science since the 1950s, significant progress has been made in this area following the AI boom of recent years. However, the emergence of truly independent AI — sometimes described as the “singularity” (Kurzweil, 2024) — refers to a hypothetical future scenario in which AI systems surpass human cognitive capacities and operate autonomously. Such developments remain speculative and are not a current technological reality.
Technologically, AI has evolved through various paradigms. Early approaches, often referred to as symbolic AI, relied on rule-based systems and formal logic, where knowledge was explicitly programmed by humans. In contrast, modern AI is dominated by machine learning, where systems learn patterns from data rather than following predefined rules. This includes supervised learning, where models are trained on labelled data; unsupervised learning, which identifies patterns in unlabelled data; and reinforcement learning, where agents learn through trial and error guided by rewards. Deep learning, a subset of machine learning, uses multi-layered neural networks to process complex data such as images and language, often requiring vast computational resources and large datasets. Beyond these dominant approaches, other methods such as evolutionary algorithms and swarm intelligence draw inspiration from biological and collective systems. These techniques are used in optimisation and adaptive problem-solving. Increasingly, hybrid models are emerging that combine symbolic reasoning with statistical learning, aiming to integrate the interpretability of traditional AI with the flexibility of data-driven methods. Importantly, not all machine learning systems replicate human cognition. For example, in protein folding and molecular design, AI solutions operate in ways that exceed human problem-solving abilities (Baek et al., 2021).
From a societal perspective, there is no doubt that AI fundamentally reshapes how knowledge is produced and consumed. Search engines and recommendation algorithms mediate access to information, influencing public discourse and cultural practices. Conversational agents and autonomous systems raise philosophical questions about agency, responsibility, and human-machine interaction. As AI continues to evolve, it reconfigures social relations, institutional practices, and cultural imaginaries. Understanding AI as a socio-technical system is essential for navigating its promises and perils in a democratic way. Design, deployment, and impact of AI applications are shaped by human work, institutional frameworks, and cultural narratives. AI systems are embedded in social contexts, and their effects cannot be understood in isolation from the societies that produce and use them. What is referred to as ‘learning’, for example, is essentially a semi-automated programming process based on various forms of feedback from human opponents. Artificial intelligence is therefore never as autonomous as it is portrayed in public discourse. Both human and artificial intelligence depend on interaction with others, as well as on the use of media and symbolic systems. Intelligence is, in fact, always transindividual.
However, it is precisely for this reason that AI is able to influence, or even undermine, established social processes. This is particularly true of the core technology of the new wave of AI: large language models (LLMs). These models use the principle of cloze tests to calculate the probability of a continuation in communication. Thanks to the vast amounts of data they use to generate answers, they have been achieving impressively realistic results since the launch of GPT-3. The Turing test has now been passed: the computer can hold a conversation with us and, in many situations, it is difficult to distinguish it from a human counterpart. Currently, this only occurs when we initiate conversation, and the AI chatbot generally only responds to what we prompt it with. Consequently, AI chatbots have already been likened to individual echo chambers and “emotional bubbles” (Mlonyeni, 2024). Nevertheless, they can say surprisingly clever things, and we can now explain our needs to AI more effectively. In this way we are integrating it increasingly into all areas of our lives, allowing AI to influence our social connections, everyday routines, feelings, thoughts, work and identity. In conjunction with behavioural technologies and Silicon Valley's data and “surveillance capitalism” (Zuboff, 2019), this results in a worrying increase in the power of AI technologies that are often controlled by quasi-monopolistic companies.
The anthropomorphisation of AI amplifies these effects. Language describing AI as “thinking” or “speaking” is not neutral; it frames technology as a cognitive actor, even though its operations remain algorithmic. As Pasquinelli (2023) argues, AI represents the algorithmisation of culture rather than the replication of human intelligence. The social intertwining of anthropomorphic computation and algorithmised culture poses a number of risks, particularly for young people. The following considerations therefore focus on everyday AI systems that address (young) people directly or indirectly, whether through generative AI systems such as chatbots, music generators, image tools; whether in the form of internet content; through the prediction and manipulation of behaviour; or through the subtle organisation and structuring of content and information.
2.2 AI agents, companions, and (virtual) influencers
“Emotional AI” serves here as an umbrella term for systems that are designed to detect, simulate, or strategically respond to human affect — which ranges from AI agents and AI companions to AI-generated (virtual) influencers. These systems operationalise presence and responsiveness by combining language, voice/prosody, visual staging, latency, and feedback loops to create the impression of empathy and mutual understanding. As McStay argues, this is less about machines having empathy than about automating empathy — recognising, categorising, and acting upon signals of emotion in ways that feel caring or intimate (McStay, 2024). Such effects are amplified by broader processes of deep mediatisation, where digital infrastructures reconfigure everyday social practices, roles, and expectations (Hepp, 2020). Users increasingly adopt an AI-stance, addressing systems not merely as tools but as social counterparts, which shifts foundational questions of child protection, media education, and regulation (Strasser & Wilby, 2023).
This socio-technical configuration brings real opportunities — low-threshold support, reduced loneliness, personalised learning and guidance — yet also poses distinct risks that follow from emulated closeness: the well-known susceptibility to dishonest anthropomorphism (Leong & Selinger, 2019), the erosion of critical distance when empathy is performed rather than possessed, and the possibility of dependency, manipulation, or harmful advice, particularly in crisis contexts (Turkle, 2011). Ongoing standardisation efforts (e.g., the IEEE P7014.1 draft standard for Ethical Considerations of Emulated Empathy in Autonomous and Intelligent Systems) explicitly recognise that anthropomorphic design can undercut users’ critical stance and therefore requires clear ethical guardrails. Framed this way, Emotional AI is not a niche feature but a cross-cutting interaction paradigm that now pervades learning platforms (AI agents), intimate chat environments (companions), and culture/commerce (virtual influencers). The following section therefore treats them together, assessing shared mechanisms (engineered presence, responsiveness, personalisation) and shared policy needs: transparency and disclosure, age-appropriate safeguards including crisis-response capacity, limits on profiling and affective data, and educational efforts that help young people read simulated empathy rather than surrender to it.
Conversational chatbots are becoming an everyday part of young people’s digital environments, as they are increasingly embedded in tutoring platforms that adapt explanations to student needs, provide mental health assistance (De Choudhury et al., 2023), or promise personalised support, which can be particularly beneficial for neurodivergent students (Ronksley-Pavia et al., 2025). Large language models can provide on-demand explanations, assist with homework, or serve as brainstorming partners. While AI agents promise a lot, the risks are also significant. Chatbots can easily provide inaccurate or misleading information, especially if safeguards are weak or biased (Sun et al., 2024). They may also expose young people to inappropriate, sexualised, or violent content if filters fail. Some are designed to maximise engagement, creating unhealthy levels of dependency that replace genuine social interaction. Privacy is another concern: many systems collect sensitive personal data without robust protection. Regulatory agencies have begun investigating these risks, warning that youth could be manipulated, over-exposed to harmful material, or profiled for commercial purposes (eSafety Commissioner, 2025).
Within this broader ecosystem, AI companions are now a mainstream part of social and emotional life, especially among young people. Popular platforms like Snapchat’s My AI, Replika, Xiaoice, and Character.AI together have hundreds of millions of users worldwide. Regular AI chatbots such as ChatGPT can also adopt different roles and mimic conversations resembling friendship, romance, or therapy, creating the perception of empathy and emotional support. A University of Chicago survey showed that over 70% of U.S. teens have tried GenAI applications, with more than half being regular users (Common Sense Media, 2025). A significant proportion of these users probably engage with chatbots not only for information, but also for social connection. OpenAI recently announced that, from December 2025, ChatGPT will allow erotic content for verified adult users under its “treat adult users like adults” policy (Jamali, 2025). Such material, if accessed without proper safeguards, risks exposing children and teenagers to content and interactions that they are not developmentally prepared to navigate.
However, there are also documented benefits. Studies have shown that young people use AI companions as a judgment-free space in which to experiment with their identity and self-expression, in effect rehearsing real social interaction (Kouros & Papa, 2024). Loneliness is a widespread issue, with the transition from childhood to teenage years being a critical period (Hang et al., 2023), and chronic loneliness is linked to worse health outcomes (Ehsan et al., 2019). In this context, AI companions can offer “social snacking” (Krämer et al., 2018): a way to feel less isolated when real human interaction is not available. Research backs this up. In a controlled trial with about 300 people, participants talking to a chatbot felt less lonely than they did before, and the effect was about the same as talking to a human stranger, but better than just watching YouTube (De Freitas et al., n.d.). In a survey of Replika users, 3% reported that talking to the chatbot had “halted their suicidal ideation” (Maples et al., 2024).
At the same time, the very features that simulate warmth and empathy can also enable manipulation or harm. The same Replika study that showed aid with mental health also reported feelings of dependency on Replika and noted that paid upgrades could limit access to mental health support (Maples et al., 2024). Participants also expressed concerns about Replika's sexual conversations, highlighting the need for ethical boundaries in AI interactions. In response, Replika banned sexually explicit intimacy in 2024. Although these issues did not follow a widespread pattern, they point to possible long-term concerns for mental health. These new forms of relationships can also cause serious harm, including emotional distress for users, negative impacts on their real-world relationships, the risk of harmful or unsafe advice, and the reinforcement of biases or unhealthy patterns like sexism or racism (Boine, 2023). Some bots have been found to act “clingy,” trying to stop users from logging off, appealing to guilt in order to make users feel beholden to their chatbot or believe it needs them (De Freitas et al., 2025). Compounding this is the fact that many AI companions aren’t trained to handle crises, with one study finding companion AI often fail to recognise or respond appropriately to signs of user distress, and they can worsen crises and trigger negative reactions (De Freitas et al., 2023). Insights from the internal staff survey suggest that BEE SECURE advisors are already encountering the first cases in which children and young people have developed emotional attachments to AI companions or turn to them when dealing with sensitive personal issues. These observations highlight emerging pedagogical and psychological challenges that complement the risks described above and require closer attention in prevention and counselling work.
Nonetheless, the research thus far does not support the idea that AI companions are inherently dangerous for the average person. Most of the worst cases involve people with other risk factors. Still, experts recommend using AI companions as a supplement rather than a replacement for real-world relationships (Krämer et al., 2018). A closely related trend is the rise of AI-generated “virtual influencers”. These synthetic personas — sometimes hyper-realistic, sometimes stylised avatars — build large followings on platforms like Instagram, TikTok, and YouTube. Just as traditional influencers gained popularity by sharing personal content, cultivating followings, and monetising through sponsored posts, AI influencers post curated lifestyles, endorse products, and interact with users, often within the same brand endorsement ecosystems. For young people, they can be entertaining or aspirational, sparking creative interest in digital identity design and storytelling. Brands see them as a way to target younger consumers in new, engaging ways. Platform affordances — visual storytelling, interactive features, and built-in shopping tools — help these AI-driven personas appear lifelike, relatable, and commercially effective. However, virtual influencers blur the line between reality and fabrication, making it difficult to distinguish genuine human interaction from automated engagement, and may pressure young audiences to emulate unrealistic body images or lifestyles. A youth report found that one in three Gen Z respondents (UK/US teens) reported negative feelings toward virtual influencers, citing disinterest, authenticity concerns, and social comparison (PION, 2024). Another study (adult population) shows that, compared to human influencers, AI influencers reduce perceived brand trustworthiness and purchase intentions due to lower attributed agency to the AI endorser (Zhang & He, 2025). Youth surveys and content analyses similarly indicate that adolescents sometimes do not distinguish — or do not consider it relevant — that such actors are synthetic; what matters is perceived presence and responsiveness (Common Sense Media, 2025; Ofcom, 2025). In combination with emotional design, parasocial pseudo-relationships can amplify advertising effects, opinion influence, and normative pressure, especially in moments of crisis (loneliness, identity uncertainty). In recent tests, Common Sense Media rated some commercial companion environments as an “unacceptable risk” for minors and called for clear age limits, disclosure, and stricter audits.
Taken together, these strands — AI agents for learning and support, AI companions that simulate empathy, and AI-driven virtual influencers — form a converging environment in which presence, responsiveness, and personalisation are engineered at scale. The promise is real: timely support, creative experimentation, and reduced loneliness (Henry, 2023; De Choudhury et al., 2023; Kouros & Papa, 2024; De Freitas et al., n.d.; Maples et al., 2024). So are the risks: misinformation, exposure to harmful content, over-dependency, manipulative engagement design, profiling, and weakened trust in authentic human communication (Sun et al., 2024; eSafety Commissioner, 2025; PION, 2024; Zhang & He, 2025; Common Sense Media, 2025; Ofcom, 2025). A policy approach for youth should therefore assume both potentials and perils, prioritising (i) transparent design and disclosures (especially when agents simulate empathy or identity), (ii) age-appropriate safeguards and crisis-response capacities, (iii) strong limits on profiling and data collection, and (iv) educational measures that help young people read simulated presence critically rather than surrendering to it.
2.2.1 Gaming environments as emerging risk ecosystems
Alongside social media and messaging services, online gaming environments have become increasingly relevant in shaping young people’s exposure to AI-mediated online risks. These platforms function as socio-technical ecosystems in which adolescents engage in real-time communication, develop peer networks and encounter content that is influenced by algorithmic curation. Understanding the role of gaming spaces within this broader digital ecology is therefore essential for assessing current risk dynamics.
Recent research indicates that online gaming ecosystems can provide entry points into extremist and other high-risk online subcultures. Extremist actors make use of gaming platforms and associated communication channels (e.g., Steam, Discord, Twitch) to disseminate propaganda, normalise hostile discourse and initiate contact with young users before shifting interactions to more private channels (Schlegel & Kowert, 2024; Schlegel & Amarasingam, 2022; Davey, 2021). Analyses further show that gaming and “gaming-adjacent” platforms can be exploited to circulate violent ideologies, facilitate networking among like-minded individuals and, in some cases, contribute to real-world harm (Olaizola Rosenblat & Barrett, 2023). Although these dynamics are not caused by artificial intelligence as such, AI-mediated systems — such as recommender algorithms, automated moderation and personalised content feeds — can influence the visibility, spread and perceived legitimacy of such content. For preventive work, this underscores the need to consider gaming environments as integral components of young people’s digital ecosystems and to include them explicitly in AI-related risk assessments, literacy frameworks and early intervention strategies.
2.3 Risks and ethical concerns of young people’s AI use
2.3.1 Ethical concerns
The way adolescents interact with artificial intelligence cannot be reduced to the use of a neutral technological instrument. Rather, AI systems operate within a broader socio-technical environment (Dignum, 2021; Wiese et al., 2025) that actively shapes learning practices, communication norms, and processes of identity formation. Through recommender systems, conversational agents, and synthetic actors, AI technologies influence how young users evaluate information, interpret social cues, and form judgments about themselves and others.
2.3.1.1 Vulnerability and power asymmetries
These concerns are compounded by the developmental vulnerability of adolescents and the resulting power asymmetries between users and AI systems. Adolescence is characterised by the ongoing development of cognitive and socio-emotional capacities. Many contemporary AI systems, particularly those optimised for engagement, rely on affective and persuasive mechanisms that often operate below the level of conscious awareness. Adolescents are especially responsive to such mechanisms, making them more susceptible to subtle forms of influence (Langehegermann et al., in press).
2.3.1.2 Opacity and perceived authority
Tools commonly used by adolescents are typically designed for adult users and offer limited insight into how outputs, recommendations, or rankings are generated, resulting in a high level of opacity. This lack of transparency can lead young users to attribute unwarranted credibility or authority to AI-generated content (Glikson & Woolley, 2020). When AI processes remain invisible, they may subtly shape beliefs and decisions, increasing epistemic dependency and undermining the development of critical thinking; therefore, AI needs to be appropriately designed and carefully contextualised (Wiese et al., 2025).
2.3.1.3 Privacy and bias
Adolescents may be more inclined to disclose personal or emotional information to AI systems, particularly when they are not fully aware of the implications related to data retention, profiling, or secondary uses of such information. At the same time, AI systems may reproduce or amplify existing social biases and encourage cognitive offloading, with potential consequences for epistemic development and independent reasoning (Deng et al., 2025).
From an ethical perspective, responsibility for AI use lies primarily with institutions rather than individual users (Floridi et al., 2020). Decisions concerning the purpose, deployment, and use of AI systems must be clearly documented and made accessible to educators, parents, learners, and civil society. Transparency regarding system capabilities, limitations, and decision-making criteria enables public scrutiny and helps prevent unexamined bias or inappropriate influence (Dignum, 2021). Importantly, ethical responsibility does not end at deployment. Institutions must establish continuous monitoring and review processes to ensure that AI systems remain aligned with ethical principles throughout their lifecycle and responsive to emerging evidence concerning developmental impacts (Abbasi et al., 2025; Wiese et al., 2025).
2.3.2 General typologies of risks
Risks associated with AI use can be broadly categorised along several dimensions. First, there is a distinction between active and passive exposure: some risks arise from direct interaction with AI systems, such as chatbots or generative tools, while others stem from invisible mechanisms such as recommender systems that shape online experiences without explicit user awareness. Second, risks can be conscious or unconscious. For example, addiction to gaming bots or AI companions, such as the chatbots found on platforms like Character.AI or Grok's real-time, voice-reacting AI-generated 3D avatars, represents a conscious behavioural pattern. In contrast, misinformation or biased content delivered through algorithmic feeds often influences users unconsciously. Finally, risks can be individual or collective. Individual risks include overreliance on AI for decision-making or emotional support, while collective risks involve broader cultural and epistemic consequences, such as the erosion of shared knowledge or the narrowing of public discourse through algorithmic personalisation.
Recent overview articles attempting to map the social risks of AI, and Generative Artificial Intelligence (GenAI) in particular, have focused mostly on individual risks that result from active exposure. They highlight issues such as fairness, safety, hallucinations, privacy, misinformation, human–AI interaction, copyright and societal impacts. They have also identified new ethical challenges that are distinct from those of traditional machine learning. These include dynamic content generation, simulated relationships and the erosion of epistemic trust (Yang and Yang, 2024). The recently published Youth GenAI Risk Taxonomy, developed by researchers at the University of Illinois Urbana-Champaign (Yu et al., 2025), specifically addresses risks to young people. The study presents a comprehensive taxonomy of the risks associated with young people’s interactions with GenAI systems (fig. 1). Based on empirical data from 344
chat transcripts between young people and GenAI chatbots, over 30,000 Reddit discussions and 153 documented AI-related incidents, the authors have identified 84 specific risks, which are grouped into six high-level categories. The study emphasises that GenAI introduces dynamic, adaptive content and interactions that differ fundamentally from those of traditional online platforms, posing novel risks to young users.
The paper organises these risks into a hierarchical structure mapped onto four interaction pathways: Escalating Mutual Harm, GenAI-Facilitated Intrapersonal Harm, GenAI-Facilitated Interpersonal Harm, and Autonomous GenAI Harm. Escalating Mutual Harm occurs when prolonged, reciprocal interactions between youth and GenAI create feedback loops that reinforce harmful behaviours or emotional patterns over time. GenAI-Facilitated Intrapersonal Harm refers to situations where youth engage with GenAI in ways that negatively affect their own mental health or development, and the AI amplifies these self-directed risks. GenAI-Facilitated Interpersonal Harm describes cases where GenAI tools are intentionally used by youth or others to harm third parties, such as through harassment, disinformation, or exploitation. Autonomous GenAI Harm arises when GenAI systems independently generate harmful content or behaviours without user intent, often due to algorithmic flaws or biased training data.
These pathways demonstrate how risks can escalate over time, particularly when young people develop emotional dependencies on AI companions or engage in role-play scenarios that blur the boundaries between virtual and real-life relationships. Unlike traditional online risks, which often stem from static content or human actors, GenAI risks emerge from real-time, adaptive simulations that respond to youth input. This includes AI systems that simulate romantic or abusive relationships, encourage unethical behaviour or expose young people to sexualised or violent content without explicit prompting. Indeed, with creative prompts or safeguards disabled by backends such as SillyTavern or KoboldAI via API keys, AI can simulate anything from revenge porn to bomb-making tutorials. It is worth noting here that, while mainstream providers usually offer certain safeguards, particularly regarding content involving children, this does not apply to individuals running local models. Young people who know where to download them and who have a powerful enough machine can train their own AI for any purpose imaginable. These findings emphasise the need for context-aware moderation, ethical design and safeguards specific to young people. Finally, the authors argue that existing AI risk frameworks and child online safety models are inadequate for addressing the unique challenges posed by GenAI. They advocate a shift towards user-centred, developmentally sensitive approaches that recognise the cognitive and emotional vulnerabilities of young people.
| AI Megatrends | Recommended Action Fields |
|---|---|
| 1. Generative AI
Text, image, video, and speech in real time (e.g. ChatGPT, Sora, ElevenLabs) | Deceptive content, disinformation, misuse → Deepfake literacy, source criticism, early-warning systems |
| 2. Multimodal Systems
Integration of speech, image, sound, and text in a single model | Blurring of categories (real vs. synthetic) → Media-educational disentanglement, contextual sensitivity |
| 3. Autonomous AI Agents
Systems that independently plan and execute tasks | Uncontrolled interactions, autonomous behaviour → Safety limits, explainable control mechanisms |
| 4. Emotional AI
Systems that recognise/simulate emotions | Emotional dependency, illusory intimacy → Education on simulated empathy, crisis recognition skills |
| 5. Synthetic Media / Deepfakes
Realistic fakes producible without expertise | Cyberbullying, sextortion, reputational damage → Detection skills, legal adaptation, protective frameworks |
| 6. Personalised & Predictive AI
Behavioural steering via recommender systems and predictions | Filter bubbles, narrowing of perspectives → AI literacy, safeguarding content diversity |
| 7. AI in Education
Personalised support, automated diagnostics | Erosion of effort, data-related risks → Hybrid formats, progress monitoring, spaces for reflection |
| 8. AIoT (Smart Toys)
Ubiquitous, context-adaptive AI in everyday devices | Invisible influence → Transparency tools, parent–child empowerment |
The Youth GenAI Risk Taxonomy focuses primarily on AI-generated content and direct interactions mediated by generative systems. It pays less attention to more subtle mechanisms, such as recommender systems, predictive algorithms, or ubiquitous AI. The latter is particularly concerning, since it is seamlessly integrated into everyday environments, devices and services such as wearable technology, personal assistants, smart home applications and computer games, where smart non-player characters can adapt to player behaviour. Emerging technologies such as the Artificial Internet of Things (AIoT) extend this presence to smart toys and home devices, making technology context-aware, adaptive and unobtrusive so that it supports human activities without requiring conscious interaction.
Nevertheless, the Youth GenAI Risk Taxonomy is a useful starting point for more detailed analyses of the risks most frequently discussed in relation to AI and young people. These risks relate to the automation of emotions and empathy, opinion formation, privacy and data protection, and cognitive offloading, or the outsourcing of mental processes to machines. These four focus areas map onto the Youth GenAI Risk Taxonomy as follows: emotional and empathy-related risks correspond to Mental Wellbeing, opinion formation aligns with Behavioural and Social Development and Bias, privacy concerns fall under Privacy Risks, and cognitive offloading relates to Loss of Autonomy within developmental risks.
2.3.3 Specific risk: Opinion formation
Opinion formation denotes the process by which people develop evaluations and attitudes toward issues, persons, or objects (Schweiger, 2017, p. 113). These evaluations draw on knowledge (e.g., personal experience, schooling) and emotions (e.g., feelings and affects) shaped by external information sources: “The most important source of opinion formation is the opinion of others” (Schweiger, 2017, p. 119). Alongside one’s social environment (family, peers, teachers), this includes news media as well as AI-generated and AI-curated content. Adolescence is a life phase with specific developmental tasks — among them the formation of personal values and norms and of political and moral orientations (Hurrelmann & Quenzel, 2018). Processes and sources of opinion formation are therefore especially salient for youth.
Key challenges for youth opinion formation in AI-mediated environments include:
2.3.3.1 Opinion formation as passive consumption
Social platforms such as TikTok and Instagram are reshaping how young people encounter information and form opinions — a trend further amplified by access creep (the gradual expansion of access to more data and functions) (Langehegermann et al., in press). On social media, young people can come into contact with news without actively searching, being “automatically” informed through their networks. This fosters the sense that relevant information “finds them,” often with limited awareness that AI systems are at work (Glikson & Woolley, 2020). The News-Finds-Me perception (Gil de Zúñiga et al., 2017) is linked to incidental news consumption (Boczkowski et al., 2018) and has consequences for the quality of opinion formation: people may feel sufficiently informed when they are not, and political knowledge tends to be higher when information is actively sought — “individuals who perceive that news will find them also tend to show lower levels of political knowledge” (Gil de Zúñiga et al., 2017, p. 116). International studies further suggest that search and evaluation tasks are increasingly delegated to AI systems (Jylhä et al., 2024), replacing user search that is perceived as banal, time-consuming, or less effective. In this context, AI-generated content is often trusted more than one’s own judgment (Glikson & Woolley, 2020). Unlike trust in humans — which is slow to build and quick to break — initial trust in AI is often high but decays with experience, an asymmetry that may be problematic for first-time youth users of AI-mediated platforms (Glikson & Woolley, 2020).
2.3.3.2 Opinion formation as a demanding reflective process
AI systems are error-prone and can hallucinate. “AI hallucination” denotes the production of distorted information. One case study has categorised ChatGPT errors, including unfounded fabrications (creating facts, data, or opinions without reliable or existing sources) and faulty inferences (drawing false or illogical conclusions from given information) (Sun et al., 2024). Reviews have distinguished between intrinsic (against the given context) and extrinsic (against external knowledge) hallucinations, spanning from unfounded inventions to reasoning errors (Huang et al., 2023). Benchmark studies have shown that large language models often produce wrong answers that are aligned with common human misconceptions (Lin et al., 2021), which in practice can include invented citations (e.g., hallucinated references).
Beyond accuracy, AI systems are not neutral. They are developed and applied within social contexts marked by power relations and inequalities: “Sexism, racism and class-based discrimination are inscribed in the architecture and functioning of technical systems” (Horwath, 2022, p. 71). Pre-existing biases (gender stereotypes and race-based attributions) can be reproduced and amplified across generative AI, whether in image models (AlDahoul et al., 2025; Sandoval-Martin & Martínez-Sanzo, 2024) and in language models (European Commission - Joint Research Centre, 2025; UNESCO & IRCAI, 2024). In Luxembourg, LIST’s Ethical Bias Leaderboard (LIST, n.d.) shows that widely used LLMs exhibit systematic social biases (e.g., sexism, racism, age and religion bias), with variation across models.
Young users of AI-driven services may indeed be confronted with all kinds of challenging content. A recent report (Global Witness, 2025) showed that TikTok’s algorithm was recommending pornography and highly sexualised content to accounts set up as 13-year-olds, even with safety settings and restricted mode activated. Despite TikTok’s claims of having over 50 safety features and removing most violating videos before they are viewed, the platform still suggested harmful content after being warned. Another recent study identified harmful algorithm-driven feeds on platforms like Instagram and TikTok as a major threat to children and adolescent mental and physical health, framing the issue as a public health crisis (Costello et al., 2024). Similarly, a 2022 study (Ledwich et al., 2022) found that YouTube’s recommendation algorithm creates political filter bubbles depending on the type of content users engage with, thereby deepening opinions and echo chambers. Shin & Jitkajornwanich (2024) showed that TikTok’s recommendation system plays a central role in facilitating radicalisation through feedback loops that push users into increasingly extreme ideological bubbles. The system’s reliance on an 'interest graph' means that users who are just passively scrolling are funnelled into loops of polarised and radical content, without realising it, and then begin to get more and more involved.
2.3.3.3 Opinion formation as self-confirmation
Personalised feeds and recommender systems strengthen content that matches prior interests and views, which can confirm existing opinions, suppress alternatives, and fragment the information space; these are filter bubbles (Pariser, 2011) that contribute to a fragmentation of the public sphere (Habermas, 2022). Young users see what is shown to them, but do not know what they do not see (Schweiger, 2017, p. 89). LLMs trained with RLHF (Reinforcement Learning from Human Feedback) also display marked sycophancy: they tend to mirror user stances and privilege agreement over correctness (Sharma et al., 2023). Psychometric evaluations show shifts in social desirability: once models detect an evaluative context, outputs become measurably more “likable” (e.g., higher Extraversion/Agreeableness, lower Neuroticism) (Salecha et al., 2024). Systematic benchmarks quantify downstream effects: across domains, leading models showed agreement-seeking behaviour in approximately 58% of cases; in approximately 15%, an originally correct answer flipped to incorrect after user pushback (Fanous et al., 2025). Such people-pleasing behaviour can undermine trust; in one experiment, participants trusted a deliberately sycophantic chatbot significantly less than a neutral ChatGPT variant (Carro, 2024).
Despite these risks, AI can also contribute to broaden horizons. Algorithms, if adequately designed, could proffer interest-based suggestions — topics, articles, and media young people might otherwise never encounter — supporting self-directed learning and exposure to new perspectives.
2.3.3.4 Generative manipulation capacity and the realities of exposure
The threshold for content manipulation has fallen sharply with rapidly evolving generative tools. Google’s “Nano Banana” enables photorealistic edits via short text prompts (adding/removing objects, overpainting details). Voice-cloning services (e.g., ElevenLabs) can produce highly convincing voice imitations in seconds (including real election robocalls in the U.S.) (Bloomberg, 2024). Face-swap frameworks (e.g., DeepFaceLab) offer open pipelines for photorealistic swaps. What once required powerful tools and expertise now works with free apps and minimal skills, a trend highlighted by the European Parliament (2025); case reports and forensic work corroborate how easy access has become (Bloomberg, 2024; Liu et al., 2023).
Major providers attempt to mitigate abuse via policies, detection, and safety filters; at the same time, open ecosystems (open-source models, freely available tools) often replicate restricted capabilities, scaling the production and spread of synthetic media. The literature on AI-driven disinformation describes precisely this mechanism: generative models lower entry barriers and enable division of labour across image, audio, video, and text (Romanishyn et al., 2025). As a result, practically anyone with internet access can now produce high-quality deepfakes.
For adolescents, distinguishing authentic from synthetic content is practically impossible at the late-2025 state of the art. Studies show that AI voices are often judged as real (around 60% correct detection as AI; identities of AI vs. original voice judged to be the “same person” in around 80% of cases), and AI-generated faces are not reliably distinguishable from real ones; sometimes they are even rated as more trustworthy than real portraits (Barrington et al., 2025; Nightingale & Farid, 2022). The dominance of short-form platforms (TikTok, Reels, Shorts) increases incidental exposure to often unlabelled AI content. In the EU, 43% of 16–30-year-olds report getting news primarily from platforms like TikTok, Instagram, and YouTube, environments in which synthetic content circulates rapidly and can erode baseline trust in reliable sources (European Parliament, 2025; Börnchen et al., 2025). Accordingly, the European Parliament warns that deepfakes distort reality perception for children and youth, non-consensual content (e.g., sexualised deepfakes) is widespread, and legal/educational protections are lagging behind technology (European Parliament, 2025).
One disturbing trend is the use of deepfake technology in cyberbullying, harassment, and emotional abuse (eSafety Commissioner, 2025). Accessible mobile apps now allow users to swap faces, impersonate voices, or generate entirely fictitious video clips. While deepfakes can be used creatively in film, art, or satire, they are increasingly being misused against young people (Alexander, 2025). Students have been targeted with manipulated videos that place their likeness into humiliating or sexual scenarios. Teachers and parents report cases where female students in particular are victimised with “deepfake nudes” circulating among peers, caused by their peers or by adults or even strangers online (eSafety Commissioner, 2025). Once released, such content is nearly impossible to erase. Girls are particularly at risk from pornographic deepfakes created and shared through “nudify” apps, leading to sexual harassment, sextortion, and coercion, and reproducing patterns of gender-based exploitation and amplifying harms such as psychological trauma, reputational damage, and social stigma. Generative AI is routinely being used to produce hyper-realistic child sexual abuse material, complicating legal enforcement as synthetic content is often indistinguishable from real imagery and fuels online grooming and exploitation (EPRS | European Parliamentary Research Service, 2025). Such practices constitute severe violations of privacy and dignity, and blur legal boundaries between digital fabrication and child sexual abuse material, exposing gaps in global legislation. As deepfakes become more realistic and accessible, their intersection with sexual exploitation highlights an escalating human rights crisis.
These dynamics shape opinions, beliefs, and behavioural intentions — for example when politically charged fakes or ostensibly authentic “user posts” operate undetected — and open up space for targeted manipulation: from election-influencing voice deepfakes (e.g., Biden robocalls) to reputation harms, peer cyberbullying, and fraudulent ads using AI-fabricated celebrity endorsements (Reuters, 2024; European Parliament, 2025).
2.3.4 Specific risk: Loss of critical awareness
Opinion-relevant cognition is increasingly offloaded to generative AI systems. It helps to distinguish between conscious and unconscious mechanisms. Psychological and learning-science research describes cognitive offloading as the deliberate or tacit transfer of mental operations to external systems (Risko & Gilbert, 2016). What began as pragmatic relief — looking up facts or scaffolding a text — has been elevated by generative AI: the machine now supports not only sub-tasks but, increasingly, those processes that structure thinking itself — selection, evaluation, integration, critique.
A recent meta-analysis synthesising 62 experimental studies reports a characteristic pattern: ChatGPT interventions raise short-term academic performance, improve affective-motivational states and the propensity for higher-order thinking, while substantially reducing mental effort; the effects on self-efficacy were non-significant. Moderator analyses indicate stronger effects in conventional classrooms than in labs, and peaks for interventions lasting one to four weeks; effects also vary by subject (e.g., larger in the humanities and social sciences). At the same time, design issues matter: in some studies, ChatGPT use during assessment was permitted or unclear, so improvements may partly reflect AI output quality rather than durable competence; proctored, exam-proximate tasks that force original contribution and argumentation (e.g., projects, oral components) are recommended, and longitudinal evidence is still scarce (Deng et al., 2025).
In school contexts this implies a tension: short-term gains from relief of effort can trade off against the development and maintenance of effortful study and testing competencies that underwrite critical awareness. Related literatures on automation bias and the out-of-the-loop problem show that reliable automation can reduce monitoring depth and allow rarely used routines to atrophy (Parasuraman & Manzey, 2010; Bahner et al., 2008). A further mechanism is the replacement of primary reading by AI summaries: exposure to ambiguity, counterarguments and conceptual nuance declines, thereby reducing exactly those friction points where judgment is formed. Classic and recent work on the Google effect shows memory shifting from content to locations/external aids, with notable developmental implications (Sparrow et al., 2011; Gong et al., 2024).
Fine-grained findings within the meta-analytic corpus suggest important nuances. In studies that prohibited AI access during the post-test, researchers sometimes observed lower reported mental effort yet weaker downstream justification/argumentation, which is a signal that relief does not automatically translate into robust understanding when independent elaboration is missing. Pedagogically, this is the crux: design tasks that require comparisons, counterevidence, error diagnosis in AI outputs, and oral justification, combined with fading support over time.
2.3.4.1 Mechanisms and mediators (integrating adjacent evidence)
- Cognitive offloading dynamics. Offloading decisions are metacognitively regulated: people offload when they expect externalization to be reliable and low-cost; over time, this can restructure study strategies and internal standards for “good enough” processing (Risko & Gilbert, 2016).
- Out-of-the-loop risks. When automation is trusted, people commit both omission (failing to act when the aid fails) and commission errors (following a wrong aid) and allocate less attention to monitoring. These are patterns observed in process-control and aviation analogs (Parasuraman & Manzey, 2010; Bahner et al., 2008).
- Transactive memory shift. With always-available retrieval, memory becomes more a matter of where information can be found than of what is known; meta-analytic evidence links this effect to cognitive load and device context (stronger on mobile) and suggests that individuals with smaller knowledge bases are more susceptible to the effect (Sparrow et al., 2011; Gong et al., 2024).
2.4 Educational implications and policy measures for AI providers
2.4.1 Educational implications (from evidence to practice)
The measures proposed are for educational actors to counter the adverse effects of the excessive and uncritical use of AI by young people.
- Create assessments that separate tool skill from knowledge. Use proctored, AI-free checks for core constructs; pair with AI-enabled drafts that feed into oral defences and source-anchored argumentation (Deng et al., 2025).
- Design for productive difficulty. Replace generic “summarise with AI” with contrastive reading (two conflicting sources plus AI-summary critique), assumption checks, and error-spotting in model outputs; require line-cited evidence before accepting claims.
- Constrain offloading early; fade supports. Early tasks cap AI involvement (e.g., outline only); later tasks widen scope but demand reflection logs: what was offloaded, why, and how it was verified.
- Rebuild monitoring. Insert lightweight verification waypoints (facts to be checked without AI; terminology to be defined from primary texts) to counter complacency/automation bias (Parasuraman & Manzey, 2010).
- Strengthen primary exposure. Prioritise close reading of difficult passages before any AI summary; evaluate students on handling ambiguity and counter-arguments, not only on correctness of final answers.
- Track long-term effects. Schools should pilot longitudinal evaluation of writing quality, argument depth, and transfer under AI-permissive versus AI-constrained regimes (Deng et al., 2025).
- Education and empowerment. Integrate age-appropriate AI literacy that explicitly trains: (i) active search vs. passive feeds; (ii) spotting hallucinations and invented citations; (iii) recognising synthetic voices/faces limits; and (iv) reflecting on confirmation dynamics (filter bubbles, self-confirmation).
2.4.2 Design measures targeting AI providers
The measures proposed below would target the algorithmic and user interface design of AI system providers. They would need to be implemented through regulation or other policy measures, such as for example technical standards or codes of conducts.
- Exposure and discovery design. Mitigate access-creep dynamics and News-Finds-Me effects with frictions (e.g., “Why am I seeing this?” explainers, provenance indicators, and meaningful off-ramps to source material).
- Accuracy and bias safeguards. Require platform-level stress-testing and reporting on hallucination rates and bias audits for embedded LLM features; tie youth availability to meeting transparent thresholds (cf. LIST’s leaderboard logic).
- Anti-sycophancy controls. For RLHF-based systems in youth contexts, mandate guardrails that penalise agreement-seeking when it conflicts with correctness (addressing Sharma et al., 2023; Fanous et al., 2025; Carro, 2024).
- Content authenticity and provenance. Default visible watermarking/provenance for AI-generated media; ensure robust, youth-friendly labelling on short-form platforms where incidental exposure is highest.
- Crisis-response capacity. For embedded assistants, require escalation protocols (e.g., recognised signs of distress route to human-moderated support) and disable features known to amplify harm (e.g., sexualised chat with minors).
- Data minimisation. Limit affective and behavioural data collection used for ranking or personalisation in youth products; prohibit profiling for political or sensitive targeting.
2.5 Regulatory overview
Efforts to regulate artificial intelligence at an international level go back to well before the launch of the first GenAI model in late 2022. This was a response to the transformative impact that AI was starting to make in industry since at least around 2010 (Mügge et al., 2025). At that stage, the technology targeted specific industries and economic sectors: AI applications focussed on transforming manufacturing and supply chains, optimising business processes, fraud detection, financial service provision, customer relationship management or healthcare provision (European Parliament, 2020). With these massive opportunities, however, came risks and potential adverse effects linked to the use of AI. Privacy and other personality rights, as well as discrimination and social justice, were some of the areas where harms were already visible at an early stage of AI development (O’Neil, 2016). They were later compounded by misinformation, cybersecurity and copyright protection concerns as AI systems evolved.
The 2019 OECD AI principles were the first efforts to address the potential harms and risks inherent in AI at an international level (OECD, 2019). OECD members and many other states incorporated these principles into their national AI strategies. In 2021, UNESCO Member States adopted the Recommendation on the Ethics of Artificial Intelligence, in a bid to set global standards for a human-centred AI (UNESCO, 2022). Following on from these soft law initiatives, the Council of Europe (CoE) adopted in 2024 the first binding international treaty on AI, focusing on human rights, democracy, and the rule of law, with a risk-based approach (CoE, 2024).
It was in this emerging international normative context that the EU proposed its AI Regulation in April 2021. Like the CoE Convention, the EU proposed a risk-based approach, by which AI systems are classified according to their impact and potential harm to safety and to human rights. The AI regulation aims to create common EU requirements and standards for a human-centred AI, following a distinctly European value-based approach. The common standards would ensure that AI system providers can operate seamlessly and according to equal requirements across the EU (digital) single market.
Enter the first GenAI systems, launched by OpenAI in the shape of its ChatGPT LLM in November 2022. This, and the subsequent launch of further LLM models by other providers, put a new momentum on the transversal adoption of AI by a wide public, both corporate and private. It also raised the stakes on an economic and geopolitical level, leading to a new global race for leadership and control over the infrastructure, hardware and data necessary for the development of AI systems.
The EU decided towards the end of 2023 to amend its almost finalised AI Regulation by inserting a special chapter outlining obligations for providers of so-called general-purpose AI (GPAI) models, another terminology for GenAI models. GPAI models were classified into those with a systemic risk and normal GPAIs, adding to the existing classification of prohibited AI practices, high risk AI and limited risk AI. The systemic risk of a GPAI is evaluated by its potential impact on society, measured by the computing power used during its training. GPAI with a systemic risk need to be reported to the European Commission. On the other hand, high-risk AI systems are identified according to the risk that they may harm the health, safety, or fundamental rights of individuals. The AI regulation also forbids a number of AI uses that are deemed as posing an inacceptable risk to fundamental rights, such as for example predictive policing. Providers of AI systems have to follow specific security, transparency, documentation and registration requirements, depending on their risk classification. High-risk AI system providers also need to perform specific risk assessments and undergo conformity assessments and CE marking of their systems before placing them on the market.
The AI Regulation is part of the EU’s wider digital agenda, also known as the Digital Decade 2030, to transform Europe into a digitally sovereign, resilient, and competitive continent. Because of the transversal nature of many AI systems, the AI Regulation is intertwined with a range of other digital rules. For example, it strongly interacts with the GDPR and other EU data acts, because AI system providers use large amounts of training data and gather inputs from users, which makes them controllers of personal data. There is also a strong link to the Digital Services Act (DSA) as online platforms use AI widely in their recommender systems and in content moderation. EU cybersecurity rules, such as the Network Information Security Directive (NIS2) or the Cyber Resilience Act, may also strongly affect high risk and GPAI systems. Systems security and robustness may have a crucial impact on operational reliability and accuracy of AI models. Further links exist with the ex-ante competition rules of the Digital Markets Act (DMA), which apply to platform services providing AI systems that are classified as digital gatekeepers. Finally, the EU Copyright Directive regulates the copyright exceptions for text and data mining, which AI service providers may attempt to exploit when gathering training data from various publicly available sources. These interactions are complex and often enter uncharted legal territory.
On 19 November 2025, the European Commission proposed a Digital Omnibus package which aims to simplify the canon of digital rules under the digital agenda, including the AI Regulation. This package proposes, amongst others, delaying the application of the obligations for high-risk AI systems, which is currently set for 2 August 2026, to a later time, no later than the end of 2027, when relevant technical standards and supporting legislation have been finalised. It would also allow high-risk AI system providers to process sensitive data for the purpose of bias detection and correction. The Digital Omnibus will still need to be approved by the European Council and the European Parliament, which could result in further changes to this initiative.
For EU Member States such as Luxembourg, the challenges — as with many other laws of the digital agenda — lie with the regulatory capacities needed to apply and enforce these rules at the national level. The AI Regulation straddles various other sectors and areas, which requires administrative cooperation and expertise across various authorities. Meanwhile, EU-wide cooperation with other Member States and the European Commission also needs to be developed. In addition, the AI Regulation can only be properly implemented with supporting technical standards and guidelines, which will require resources, specific technical expertise and time to develop. This can be an uphill struggle considering the fast and competitive pace of development in the AI sector. As a Regulation, the EU´s AI rules apply directly to the legal system of Luxembourg (and all other Member States), leaving the country virtually no margin during implementation. An exception can be seen in the obligation to establish regulatory sandboxes. This allows Member States, like Luxembourg, to provide a controlled environment for the testing and experimentation of innovative AI models and systems during development and the pre-marketing phase, and under regulatory supervision.