Why AI security matters

Where AI already sits in a small company, and what can go wrong with it

By the end of this lesson you can

  • Describe at least four ways a small company already uses AI tools, and the data each one sees.
  • Sort an AI worry into one of three groups: attacks on AI systems, AI used by attackers, or AI misused by staff.
  • Explain which classic security ideas still apply to AI systems and what is genuinely new.
  • Name the main frameworks this course uses and say in one sentence what each is for.
  • Start an AI security log with a first inventory of AI tools.

01Monday at Alpenrose

Meet Alpenrose Travel, a fictional Swiss tour operator with about 80 staff in Lucerne and Bern. It sells group trips and coach tours, and it will be our example company for the whole course. You are in the shoes of Nora, who leads IT and security there.

On Monday morning, three things happen before coffee. Luca, the developer, proudly demos TripMate, a small internal assistant he built in a hurry: staff upload a supplier document and it writes a short summary. Then Nora notices that Sam in customer service has been answering complaints with the help of a public chatbot on his phone, copying in the customer's message. Finally, accounting forwards a supplier invoice that "reads oddly well": perfect grammar, the right names, a new bank account.

None of this is science fiction. Artificial intelligence (AI) here simply means computer systems that learn patterns from data instead of following fixed rules. The newer kind, generative AI, produces new text, images, audio, or code when you give it a request. Most of the text tools you meet today are built on a large language model (LLM), a model trained on huge amounts of text to predict what comes next. Lesson 2 explains how that works without any math.

Nora's first step is to write down where AI already sits in the company. Here is her first draft (all examples fictional):

Use at AlpenroseAI typeData involvedOwnerFirst risk Nora sees
TripMate, internal document assistantLLM app built in-houseSupplier documents, prices, contactsLuca (build), Nora (security)A supplier file contains text that steers the summary
Rosie, customer chat on help.example.comLLM app with an FAQ storeCustomer questions, booking numbersCustomer service leadA wrong answer about refunds is trusted by customers
Public chatbot on staff phonesGeneral chatbot, not approvedWhatever staff paste in, often customer namesNobodyPersonal data leaves the company unnoticed
Mail spam filterMachine learning classifierAll incoming mailITA fluent lure slips through and looks genuine
Translation feature in the office suiteBuilt-in AI featureBrochures, contracts, internal mailOffice suite adminConfidential text goes to a provider whose terms nobody read

Notice the "Owner" column. One tool has no owner at all. That is the first finding of almost every AI review, and we come back to it at the end of this lesson.

02Three faces of AI risk

"AI security" is used for very different worries. It helps to split them into three groups, because each group needs different people and different controls.

Face 1

Attacks on AI systems

Someone targets the AI system itself: a hidden instruction in a document fools TripMate, a crafted question makes Rosie reveal its internal notes, a tampered model file is installed. The fix lives in design, code, and operations.

Face 2

AI used by attackers

Criminals use AI as a tool against people: a cloned voice of the managing director, a deepfake video call, phishing mail without the spelling mistakes people used to look for. The fix lives in processes and verification habits.

Face 3

AI misused by staff

Well-meaning employees paste customer data into unapproved tools, or trust a confident but wrong answer. The fix lives in policy, training, and approved tools that actually help.

The diagram shows how the three faces overlap. Sam's phone chatbot is a staff issue, but if that chatbot were tricked by content it reads, it would also be an attack on an AI system. Around all three sits AI governance: the roles, rules, records, and reviews that decide how AI is chosen, used, monitored, and retired.

Three overlapping circles labeled Attacks on AI, AI used by attackers, and Staff use of AI, each with an Alpenrose example, inside a dashed frame labeled Governance GOVERNANCE Attacks on AI TRIPMATE FOOLED AI by attackers DEEPFAKE CALL Staff use of AI CHATBOT PASTE
Most real incidents touch more than one circle, and governance is the frame that holds all three together.

This course covers all three faces. Modules 2 to 4 focus on attacks on AI systems, module 5 on attackers using AI and on staff use, and module 6 on governance, testing and response.

03What is new and what is not

Much of AI security is ordinary security applied to a new kind of component. The classic goals still hold: keep data confidential, keep systems and data correct (integrity), and keep services available. Least privilege still matters: an AI feature should only reach the data and actions it needs. Patching, access control, logging, backups, and supplier checks all still apply. TripMate runs on a server, stores files, and writes logs like any other app, and every old weakness of servers, files, and logs is still there.

Two things are genuinely new, and they explain most of this course:

  • Instructions and data are mixed. In a normal program, code and data are kept apart. An LLM receives one long text that contains the developer's instructions, the user's question, and any documents, all in the same channel. The model has no reliable way to tell them apart. The UK National Cyber Security Centre (NCSC) wrote in December 2025 that prompt injection is not like SQL injection and may never be fully mitigated in the same way, so the focus must be on design that limits the damage. The OWASP Top 10 for LLM Applications 2026 says the same: no reliable prevention exists today.
  • Answers vary and can be wrong. The same question can get different answers, and a fluent answer can be false. You cannot test an LLM feature once and assume it will always behave the same way.

So the mindset is: keep every classic control, then add controls for the two new properties. Assume a model will sometimes be fooled and sometimes be wrong, and design so that this does little harm.

Myth

"AI security is a brand-new discipline that replaces normal security." It does not replace anything. The NCSC and CISA guidelines for secure AI system development are organized around secure design, secure development, secure deployment, and secure operation and maintenance: the same life cycle as any software. A model on an unpatched server with a shared admin password is insecure before anyone types a single prompt.

04The map of this course

You do not have to invent AI security yourself. Several free frameworks describe the risks and the controls, each from a different angle. You will meet them one by one. Here is the map, one line each.

FrameworkPublished byWhat it is forLesson
OWASP Top 10 for LLM Applications 2026OWASP GenAI Security ProjectThe ten most critical risks in apps built on LLMs, with mitigations6
MITRE ATLASMITREAttacker tactics and techniques against AI systems, modeled on ATT&CK7
NIST AI RMF 1.0 and AI 600-1US NISTHow an organization manages AI risk (Govern, Map, Measure, Manage), plus a profile for generative AI20
Guidelines for secure AI system developmentUK NCSC, US CISA, and partnersSecure design, development, deployment, and operation of AI systems20
Secure AI Framework (SAIF)GoogleCore elements and a risk map that says who owns each risk5, 20
EU AI Act and Swiss FADPEuropean Union and SwitzerlandLegal duties by risk level, and data protection rules that already apply to AI21

Two of them are easy to confuse. OWASP and ATLAS look at the system: what can go wrong in this app and how attackers do it. NIST AI RMF and the governance frameworks look at the organization: who decides, how risks are measured, how they are managed over time. You need both views.

The OWASP list is built from a community vote (weighted 75%) and evidence from real incidents (weighted 25%). It covers the model as a component inside your application. Once a model starts acting on its own with tools and memory, OWASP points to its separate Top 10 for Agentic Applications, which our follow-up course "Securing AI Agents and MCP" (Track 14) covers.

In the real world

In November 2023 the UK NCSC and the US CISA published "Guidelines for secure AI system development" together with 21 partner agencies from many countries, including the NSA, the FBI, and national security agencies of Australia, Canada, Germany, and France. It is free, short, and written for anyone who builds or deploys AI systems.

Source: UK NCSC, Guidelines for secure AI system development, November 27, 2023

05Hype check

AI attracts two opposite kinds of hype, and both make security worse.

The first says AI is magic: it understands everything, it will replace the security team, it cannot be wrong. People who believe this stop checking the output. The second says AI is "just autocomplete", so it is harmless. People who believe this connect it to customer data, email, and payment systems without a second thought. The truth sits in between: an LLM is a powerful text predictor that can be steered by its input, and it is as dangerous as the data and actions you connect it to.

A good example is false output. The OWASP 2026 list has an entry for misinformation (LLM07:2026). It is a security issue, not only a quality issue, because the core risk is that incorrect output is trusted and acted upon. If Rosie tells a customer that a cancellation is free when it is not, Alpenrose may face a dispute. If TripMate invents a new supplier contact, staff may send money to it. NIST uses the word "confabulation" for these confident inventions. Lesson 2 explains why they happen.

Notice what this means for defense: the fix is not a better model alone. It is a process where facts that matter are checked against a source before anyone acts on them.

06Your AI security log and the inventory

From today you keep an AI security log: a file or a paper notebook where you write threats, decisions, test results, and policy drafts. Every lesson adds an entry, and in the capstone (lesson 24) the log becomes the evidence for securing TripMate.

The first entry is an inventory: a list of every AI tool and AI feature in use. It sounds dull, but it is the base of everything else. You cannot assess, protect, test, or govern a tool you do not know about. The inventory also finds shadow AI: AI tools used for work without the organization knowing or approving them, often with company data pasted in. Sam's phone chatbot is shadow AI.

A simple inventory method that works on paper:

  1. Collect. List standalone tools (chatbots, assistants), AI features inside tools you already have (office suite, mail, phone keyboard, translation), and anything your own team built.
  2. Describe. For each entry, note what data it sees, who uses it, and who owns it.
  3. Mark status. Approved, tolerated, or unknown. "Unknown" is not a failure. It is the finding.
  4. Note a first worry. One sentence, sorted into one of the three faces of AI risk.

Do not try to rate everything precisely on day one. A rough, complete list beats a perfect list with half the tools missing.

Defender's view

Start with the inventory, not with a tool. Before buying any "AI security" product, know which AI systems you have, what data flows into them, and who is responsible. Ask teams what they use rather than punishing them for it: people only tell you about shadow AI if they expect help, not blame. Repeat the inventory regularly, because new AI features arrive in existing software all the time.

Practical exercise: your first AI inventory

Paper and pen, or a plain text file you will keep as your AI security log. Use your own work or home situation. If you prefer, use Alpenrose. Do not write real passwords or customer data into the log. Check off each step as you finish it. Your checkmarks are saved in this browser.

Key takeaways

  • AI is already in most small companies, often hidden inside tools they bought for something else.
  • AI risk has three faces: attacks on AI systems, AI used by attackers, and AI misused by staff. Each needs different controls.
  • Classic security still applies. What is new is that models mix instructions with data and that their answers vary and can be wrong.
  • OWASP and MITRE ATLAS describe what goes wrong in a system. NIST AI RMF and governance frameworks describe how an organization manages it.
  • Misinformation is a security issue because people act on trusted output.
  • Start with an inventory: you cannot protect what you do not know you have.