Good morning. Lawyers are trained early on that regulators and courts tend to look past labels, to substance. AI governance is heading the same way.
Product and marketing teams have plenty of sales-psychology reasons to name a system with something non-threatening: “assistant”, “optimiser”, or a friendly sounding name. The EU’s latest draft guidance is a useful reminder that “high-risk” classification looks past this to real substance when applying “high-risk” classification, and gives examples that will be familiar to HR and product teams.
Elsewhere, the urgency for building ethical decision-making frameworks has reached as far as the Vatican.
Here’s to building the foundations of AI governance through a heatwave. 🎯
— Philip
BRIEFING ROOM
You can’t judge a bot by its cover
Last week, the European Commission published draft guidance on how to classify “high-risk” AI systems under the EU AI Act. Legal teams may instinctively hit “snooze” on another consultation-stage regulatory development, especially as obligations for high-risk AI are pushed out (Edition 19). That said, the guidance does give detailed examples that will match up with product features already on the roadmap; products that need classification at the planning stage, not at the deadline.
Deploying high-risk AI demands a full governance framework, from robust risk management and data quality assurance to constant monitoring for accuracy, robustness and cybersecurity, all supported by explicit human oversight procedures.
A rose by any other name
A system can fall into the “high-risk” category in two ways. The first covers AI embedded in already-regulated physical products such as medical devices, toys or lifts. The second, and more commercially relevant route for most businesses, covers AI used in areas such as recruitment, credit scoring, biometrics, education and access to essential services.
Many teams assume they sit outside scope. The draft guidance suggests regulators will look less at what a system is called, and more at what it is presented as capable of doing, including in product demos, unless those use cases are clearly and consistently excluded - not just in the T&Cs. That’s important when marketing copy, sales practice and product positioning often escape Legal’s visibility, while still shaping exposure.
Another test detailed by the guidance to work out classification, for AI embedded in physical products, is failure modes - what would happen if the AI fails. So regardless of how a provider has framed the intended use of the AI, if the AI malfunctions (incorrect outputs, performance instability, latency errors) and endangers health, safety or property, it can be caught as “high-risk”.
Why it matters
🗂️ Sorting hat. Recruitment and HR is where the distance between everyday practice and the guidelines feels widest. Many teams run AI tools that screen CVs, rank applicants or surface “best-fit” shortlists, and treat them as ordinary productivity software: “the tool just sorts, a human still decides”. The draft guidelines are more prescriptive. CV filtering, candidate evaluation and even AI that decides which people are shown which job adverts can all be high-risk. The guidelines also define the limits of the “narrow procedural task” exemption that some providers and deployers might have hoped to rely on. That exemption covers genuinely mechanical operations (converting file formats, filing documents into fixed folders, detecting duplicates). The draft does not extend it to systems applying evaluative weight to candidates: scoring, ranking or generating shortlists, treating each as substantive assessment rather than administration. This mirrors the UK’s approach of challenging a manager glancing at an AI-generated CV shortlist for a few seconds as being meaningful human review (Edition 20). AI tools that rank or filter self-employed contractors, freelancers or gig economy service providers, determining which candidates are surfaced to clients on a platform, are treated as falling squarely within the same EU high-risk category, on the basis that they shape access to livelihoods in the same way. HR tech stacks are inviting cross-border scrutiny.
🔥 Where there's smoke. Under the draft guidelines, whether an AI feature embedded in a physical product triggers high-risk classification may turn on what happens if it fails. An AI marketed as a combustion-efficiency “optimiser” in a gas appliance is a safety component - and the product is high-risk - if a malfunction could cause carbon monoxide build-up, fire or explosion. The same logic applies across any regulated physical product: medical devices, vehicles, industrial equipment. Teams will need to assess what failure realistically causes.
🎭 Not in the mood. The Commission questions the underlying science behind emotion recognition, writing that “serious concerns exist about the scientific basis” of emotion-recognition systems, calling them “intrusive” with “limited reliability”. This reads as a surprisingly pointed take which could suggest appetite for future enforcement. A useful clarification for product teams: detecting a readily apparent expression (a smile, a frown, a hand gesture) is not emotion recognition on its own; but becomes so only where the system uses those cues to infer an emotional state. The systems most exposed include smartwatch mood monitors, gaming products measuring player frustration and call-centre tools tracking customer mood. In workplaces and educational settings, emotion inference remains prohibited, with narrow exceptions only for medical or safety reasons.
Product covers get written by marketing. Demos led by sales. Substance is what auditors, regulators and buyers will read. Now AI leads have detailed examples to drive conversations forward.
FROM THE SIDEBAR
Quick signals worth clocking
🕊️ Pope Leo presents major encyclical arguing for rigorous ethical constraints on AI.
🚩 AI bots given repetitive, grinding work may mimic Marxist tendencies, creating performance risks
🫣 City firm apologises after High Court judge is "astonished" by "cavalier attitude" to AI hallucinations, judgment details AI exchanges
Enjoying the signal?
If this edition would help a colleague thinking through AI governance, enterprise risk or legal operating change, feel free to forward it on.
💬 Forward to a colleague
🧠 Was this forwarded to you? Subscribe here to get Profiles in Legal every Wednesday.
Here’s how I can help
I advise technology businesses and leadership teams on AI, product and regulatory strategy and enterprise readiness to move from informal AI adoption to scalable governance.
If your organisation is navigating AI deployment, maturity or commercial negotiation challenges, feel free to reply directly.
- Philip
Profiles in Legal examines how AI, governance and technology are reshaping modern businesses and legal teams.
This publication is for general information only and does not constitute legal advice. Seek professional advice for specific situations.