AI Glossary

An AI agent is a software system that carries out tasks independently: it is given a goal, plans the steps, uses tools (search, databases, APIs) and acts to reach the goal – without every single step being pre-defined.

How it works: An agent combines a language model (LLM) with memory and tools. It breaks a task into sub-steps, decides at each step which tool to call, evaluates the result and plans the next step until the goal is met.

Business example: A mid-sized company uses an agent in procurement: it reads quotation emails, compares prices against the ERP, builds a comparison table and prepares a purchase proposal for approval.

Application areas:

• Automating recurring office and administrative processes
• Researching and preparing information
• Customer service with access to internal systems
• Data maintenance and reconciliation across applications

Distinction: Unlike a bot, which follows fixed rules, an agent makes its own decisions and uses tools flexibly.

Related terms: Agentic AI, Bot

Agentic AI refers to AI systems made up of several cooperating agents that carry out complex projects largely on their own. Instead of a single request-and-response, they pursue an overarching goal across many steps.

How it works: Several specialised agents take on sub-roles (e.g. planner, researcher, reviewer) and coordinate with one another. They plan, execute, check their intermediate results and correct themselves until the overall result is ready.

Business example: For preparing quotes, three agents work together: one gathers requirements, one calculates based on the ERP, one drafts the quote – the employee only reviews the final result.

Application areas:

• End-to-end automation of multi-step business processes
• Creating and reviewing documents and reports
• Data analysis followed by a recommended action
• Orchestrating several specialist systems

Distinction: While a single agent handles one task, agentic AI coordinates multiple agents for larger, multi-step projects.

Related terms: Agent, Bot

AI governance is the set of rules, processes and responsibilities with which a company steers its use of artificial intelligence – from selecting and approving AI systems through risk assessment and documentation to ongoing monitoring in operation.

How it works: An AI governance framework defines who approves AI systems, which criteria are used to assess risks, and how models are documented, tested and monitored. Typical building blocks are an inventory of all AI systems in use, clear roles and approval processes, guidelines for data quality and transparency, and regular reviews. Regulations such as the EU AI Act set the outer frame.

Business example: An automotive supplier rolls out AI-based visual inspection and forecasting models across several plants. A central AI inventory records each system with its purpose, risk class and owner; new use cases pass a risk assessment before going live.

Application areas:

• Building AI inventories and approval processes
• Risk classification of AI use cases under the EU AI Act
• Guidelines for using generative AI in day-to-day work
• Documentation and evidence obligations towards customers and authorities

Distinction: Data Governance regulates how data is handled; AI governance builds on it and additionally covers models, their risks and their use. MLOps provides the technical implementation (versioning, monitoring) but does not replace the organisational rules.

Related terms: EU AI Act, Data Governance, MLOps, Explainable AI (XAI)

Anomaly detection is a machine learning and statistical technique for identifying data points that deviate markedly from expected normal behaviour. Unlike Classification, the cases of interest are typically rare and often unlabelled.

How it works: A model first learns what "normal" data looks like, using statistical distributions, density estimation, Clustering or autoencoders. New data points are then scored by how far they depart from this learned pattern; when the deviation exceeds a threshold, an alert is raised. Depending on the available labels, supervised, semi-supervised or unsupervised approaches are used.

Business example: An engineering company monitors vibration and temperature sensors across its production lines. An anomaly detection model flags unusual sensor patterns before a bearing fails, so maintenance can be scheduled instead of performed as an emergency repair.

Application areas:

• Predictive maintenance and machine condition monitoring
• Quality control in manufacturing
• Fraud detection in transaction data
• Spotting measurement errors in sensor and field data (agriculture, remote sensing)
• IT security and network monitoring

Related terms: Unsupervised Learning, Clustering, Pattern Recognition, Machine Learning

An API (Application Programming Interface) is a clearly defined interface through which different software applications communicate and exchange data. The API specifies which requests can be made, the format in which data is transferred and how errors are handled.

How it works: An API works on a request-and-response principle. An application sends a structured request to the API, which processes it and returns a response, often in JSON or XML format.

Business example: An online shop integrates its shipping provider via an API: as soon as an order comes in, the API automatically transmits the delivery data, with no manual step required.

Application areas:

• Connecting business intelligence tools to different data sources
• Integrating payment providers, shipping services or CRM systems
• Using external AI models (e.g. from OpenAI or Google) inside in-house applications
• Automating data pipelines between internal systems

Distinction: Unlike a direct database connection, an API abstracts the technical infrastructure; the caller does not need to know how the data is stored internally.

Related terms: Microservices, Data Engineer, ETL

ArcGIS is Esri's commercial geographic information system platform for capturing, managing, analysing and publishing geospatial data. It spans the ArcGIS Pro desktop application, server components for data services, and ArcGIS Online for web maps and dashboards.

How it works: Spatial data is organised as layers within projects and processed through geoprocessing tools for overlay, buffering, interpolation and network analysis. Workflows can be automated in Python via the ArcPy interface, and the arcgis.learn module packages deep learning models for remote sensing tasks such as object detection, image segmentation and point cloud classification, bringing PyTorch-based methods into the GIS environment. QGIS is the widely used open-source alternative, usually paired with libraries such as GDAL.

Business example: A public authority keeps its land and infrastructure records in an ArcGIS environment. A data science team trains a segmentation model for sealed surfaces outside the platform and publishes it as a geoprocessing tool, so domain staff can run the analysis on new aerial imagery without writing code.

Application areas:

• Management and maintenance of geospatial domain datasets
• Network, accessibility and catchment analysis
• Analysis of aerial and satellite imagery
• Maps, dashboards and field apps for business users
• Delivery of trained models to GIS practitioners

Related terms: Remote Sensing, Location Analysis, Point Cloud, Python

Artificial intelligence refers to computer systems that perform tasks which previously required human intelligence, such as language understanding, visual perception, decision-making or solving complex problems.

How it works: Modern AI is based almost entirely on data-driven machine learning: instead of explicitly programmed rules, models learn patterns from large datasets. The most capable current systems are large language models and multimodal models.

Business example: AI systems automate tasks across nearly every industry today, from automated document review in insurance and AI-assisted diagnostics in medicine to automated quality control in manufacturing.

Subfields of AI:

• Machine learning and deep learning
• Natural language processing (NLP)
• Computer vision / image recognition
• Robotics and autonomous systems
• Generative AI

Distinction: "AI" is often an umbrella term. Narrow AI can only handle one specific task (e.g. playing chess). Broad or general AI (AGI), which transfers human intelligence to arbitrary tasks, does not yet exist.

Related terms: Machine Learning, Large Language Model, Generative AI

AWS (Amazon Web Services) is Amazon's public cloud platform and, by market share, the largest cloud provider worldwide. It spans several hundred services, from compute and storage to databases, analytics and machine learning.

How it works: Infrastructure is provisioned on demand via API or console and billed by usage. Central to data projects are S3 for object storage (often the foundation of a Data Lake), EC2 and Lambda for compute, Redshift and Athena for analytics, and SageMaker for training, deploying and monitoring machine learning models; Bedrock provides foundation models as a managed service.

Business example: An agricultural company stores satellite and sensor data in S3, processes it into field-level metrics with serverless functions and trains yield-forecasting models in SageMaker. Compute-heavy training runs scale up by the hour, with no need to maintain in-house hardware.

Application areas:

• Scalable data and analytics platforms
• Training and operating ML models (SageMaker, Bedrock)
• Hosting applications, APIs and Microservices
• Archiving and processing large volumes of sensor and image data

Related terms: Microsoft Azure, Data Lake, MLOps

Bag of Words is a text representation method that describes a document as an unordered collection of its words together with their counts, discarding word order in the process.

How it works: A vocabulary is built from the whole corpus, and each document is turned into a vector of that length whose entries record how often each term occurs. Standard preprocessing includes tokenisation, lower-casing, stop-word removal and stemming or lemmatisation. Using n-grams instead of single words captures short phrases such as "no defect", which recovers a little of the lost word order.

Business example: An agricultural trading company routes incoming e-mails automatically into orders, complaints and price enquiries. A Bag-of-Words model trained on a few thousand historic messages provides a reliable first solution that runs on ordinary CPU hardware and is simple to retrain.

Application areas:

• Classification of e-mails, tickets and form submissions
• Topic modelling across document collections
• Baseline models before committing to heavier methods
• Simple search indexes
• Exploratory analysis of free-text fields

Distinction: Embeddings represent words as vectors that capture meaning, and contextual variants encode the word "bank" differently depending on the sentence. Bag of Words treats every occurrence of a word as identical.

Related terms: TF-IDF, Text Mining, Natural Language Processing (NLP), Token / Tokenisation

BERT (Bidirectional Encoder Representations from Transformers) is a language model released by Google in 2018 that encodes each word using context from both sides of the sentence and serves as a pre-trained basis for tasks such as text classification, entity recognition and semantic search.

How it works: BERT uses the encoder half of the Transformer Architecture. During pre-training, individual tokens are masked and reconstructed from the surrounding text, which forces the model to read left and right context at once. A specific task is then addressed through Fine-Tuning on a comparatively small labelled dataset. Alongside the general model there are domain-specific variants pre-trained on biomedical, scientific or non-English corpora.

Business example: A pharmaceutical company extracts substance names, dosages and adverse events from study and safety reports. A domain-adapted BERT model is fine-tuned on a few hundred annotated passages and reaches an accuracy that the previous rule-based extraction never achieved on varied report wording.

Application areas:

• Entity recognition in technical and regulatory text
• Semantic search and document retrieval
• Classification of support requests and reports
• Similarity matching for duplicate detection
• Vectorising text passages for Retrieval-Augmented Generation (RAG)

Distinction: BERT reads and encodes text; it does not generate longer answers on its own. Generative systems of the Large Language Model (LLM) family use the decoder side of the transformer for that purpose.

Related terms: Transformer Architecture, Fine-Tuning, Natural Language Processing (NLP), Embedding

BLAST (Basic Local Alignment Search Tool) is a bioinformatics search method that compares a DNA or protein sequence against the entries of a sequence database and returns locally similar stretches together with a statistical assessment.

How it works: BLAST breaks the query sequence into short words and first looks for exact matches in the database. From these seed points it extends the alignment in both directions for as long as the similarity score keeps improving. The reported E-value states how many hits of that quality would be expected by chance in a database of the same size, which gives a practical filtering threshold. Separate program variants cover the different comparison directions between nucleotide and protein sequences. The tool is provided by the US National Center for Biotechnology Information, is freely available, and runs as a command-line program that fits into automated analyses.

Business example: An agricultural laboratory sequences fungal isolates from field samples. An automated BLAST comparison against a curated reference database identifies the species and flags known resistance markers before the finding is passed on to the crop adviser.

Application areas:

• Species identification and taxonomic assignment
• Functional annotation of newly sequenced genes
• Finding related sequences in other organisms
• Checking primers and probes for unintended binding sites
• Contamination screening in sequencing data

Related terms: FASTQ, Pattern Recognition, Data Pipeline

A blockchain is a decentralised, tamper-resistant data structure that stores transactions or information in a chronological chain of blocks. No central actor controls the data; instead, the chain is stored on many computers at once and validated jointly by all participants.

How it works: Each block contains a cryptographic hash of the previous block, a timestamp and the actual transaction data. If a block is altered after the fact, its hash changes, which invalidates all subsequent blocks. This makes any later alteration immediately detectable.

Business example: A food producer uses blockchain to document its entire supply chain from farm to supermarket shelf. Every step is recorded immutably; in the event of a recall, a product's origin can be traced in seconds.

Application areas:

• Cryptocurrencies (Bitcoin, Ethereum)
• Supply-chain transparency
• Digital contracts (smart contracts)
• Proof of authenticity for documents and credentials
• Healthcare (secure patient records)

Distinction: A blockchain is not a database in the classic sense; it is optimised for immutability and trust, not for fast queries or flexible data structures.

Related terms: Data Warehouse, Structured Data

A bot is a program that automatically performs defined, recurring tasks – such as answering standard questions or filling in forms. Classic bots follow fixed rules or scripts.

How it works: A bot reacts to a trigger (e.g. a message or a schedule) and works through a predefined sequence. Modern chatbots combine this with a language model to understand more freely worded requests.

Business example: A chatbot on the website answers common customer questions about opening hours, shipping and prices, and only escalates complex cases to a human.

Application areas:

• Customer service and FAQ automation
• Appointment booking and form filling
• Monitoring and notifications
• Routine tasks in chat and messaging channels

Distinction: A bot follows fixed rules; an agent plans independently and uses tools flexibly to reach a goal.

Related terms: Agent, Agentic AI

Business intelligence refers to the systematic use of technologies, processes and methods to collect, prepare and visualise company data, with the goal of enabling sound, fact-based decisions.

How it works: BI systems draw on data from various sources (ERP, CRM, databases), consolidate it in a central data warehouse and present it clearly through dashboards and reports. Users can filter, drill into and analyse data interactively, without programming knowledge.

Business example: The head of sales opens the BI dashboard in the morning and sees immediately: which region missed its revenue targets yesterday? Which product is outperforming the plan? On that basis they make informed decisions before the first meeting.

Application areas:

• Revenue and cost analysis
• Customer and market segmentation
• Operational KPI monitoring
• Financial and budget planning

Distinction: BI explains what happened in the past (descriptive analytics). Predictive analytics goes a step further and forecasts what will happen next.

Related terms: Data Warehouse, Descriptive Analytics, Data Scientist

C++ is a compiled programming language for systems-level and performance-critical software, combining direct control over memory and hardware with high-level language features. In data science, C++ is the language under the hood: many core libraries of the Python ecosystem are implemented in it.

How it works: C++ code is compiled to machine code ahead of execution, which – together with manual memory control and compiler optimisations – yields runtimes interpreted languages cannot match. The compute kernels of PyTorch, TensorFlow and OpenCV are written in C++ and used from Python through bindings.

Business example: A surveying company processes laser-scan point clouds with billions of points. The filtering and registration algorithms are implemented in C++ so processing takes hours instead of days, while the analysis workflows on top are driven from Python.

Application areas:

• Performance-critical processing of large data (point clouds, images, signals)
• Implementing the cores of ML frameworks and databases
• Embedded and real-time systems, robotics
• Speeding up individual Python bottlenecks via native extensions

Related terms: Python, Point Cloud, Signal Processing

CAD (Computer-Aided Design) is the computer-based creation and editing of geometric models of components, plants and buildings. CAD models describe geometry parametrically and precisely, and serve as the authoritative reference for manufacturing, assembly and documentation.

How it works: A CAD system holds solid, surface and curve geometry along with dimensions, tolerances and material properties; changing one parameter propagates through the design history to dependent features. The reverse direction matters wherever measurement meets design: in reverse engineering, a Point Cloud from a laser scan is first fitted to surfaces and regular primitives, from which a CAD model is built. Exchange between systems runs through formats such as STEP and IGES, or IFC in the building domain.

Business example: A mechanical engineering company operates a legacy plant for which complete drawings no longer exist. The plant is captured by laser scanner, the point cloud is segmented automatically into pipes, vessels and steelwork, and a CAD model is reconstructed from it as the basis for planning a retrofit.

Application areas:

• Design of components, tooling and industrial plant
• Reverse engineering from scan data
• As-designed versus as-built comparison
• Digital twins and simulation models
• As-built documentation of buildings and infrastructure

Related terms: Point Cloud, Industry 4.0, PDAL, Signal Processing

Cellpose is an open-source deep learning tool for segmenting cells and nuclei in microscopy images. It is built as a generalist model: a single trained network segments cells of varying shape, size and imaging modality, so a bespoke model is not required for every new dataset.

How it works: Rather than predicting a class for each pixel directly, the model predicts a vector field pointing towards the centre of the cell that each pixel belongs to. Pixels are then grouped by following those vectors, which keeps touching and irregularly shaped cells separate. Cellpose ships with a graphical interface, a command line and a Python API, and a handful of manually corrected images is usually enough to fine-tune the model for difficult preparations.

Business example: A pharmaceutical company analyses cell culture plates from a compound screen. Cellpose segments individual cells across several thousand images per run, after which area, shape and fluorescence intensity are measured per cell and compared across concentration series – an analysis that manual counting could not deliver in a workable time frame.

Application areas:

• Cell counting and morphometry in microscopy
• High-throughput screening in drug discovery
• Analysis of histology and immunofluorescence preparations
• Cell tracking in time-lapse imaging
• Generating annotations for downstream analysis

Distinction: StarDist represents objects as star-convex polygons, which makes it strong on round, densely packed nuclei; Cellpose is more flexible with highly irregular cell shapes. In practice both tools are benchmarked on the same data before one is chosen.

Related terms: StarDist, Image Segmentation, U-Net, Image Recognition

CI/CD (Continuous Integration / Continuous Delivery or Deployment) is a software engineering practice in which code changes are built, tested and released automatically. The goal is small, frequent, low-risk releases instead of rare big-bang deployments.

How it works: With continuous integration, every committed change is automatically compiled and run through test suites, so defects surface early. Continuous delivery keeps every successfully tested state ready to release; continuous deployment pushes it to production without a manual gate. Pipelines are implemented with tools such as GitHub Actions, GitLab CI or Azure DevOps.

Business example: A software team at an agricultural company maintains a field-management application. Every change automatically passes formatting, test and security checks and is first deployed to a staging environment; only after passing review does it reach production – several times a week rather than once a quarter.

Application areas:

• Automated quality assurance for every code change
• Delivery of applications, APIs and Microservices
• Deployment of data pipelines and ML models (as part of MLOps)
• Infrastructure changes via infrastructure as code

Related terms: MLOps, Microservices, Application Programming Interface (API)

Classification is a supervised learning method in which a model learns to assign new data points to one of several predefined categories.

How it works: The model is trained on labelled data (e.g. emails marked "spam" or "not spam"). It learns which features are typical of which class. For new, unknown data points it then makes an assignment based on these learned patterns.

Business example: A bank trains a classification model on historical loan applications. The model learns which combination of income, debt and payment history led to defaults, and classifies new applicants as "low risk" or "high risk".

Common algorithms:

• Logistic regression
• Decision trees and random forests
• Support vector machines (SVM)
• Gradient boosting (XGBoost, LightGBM)
• Neural networks

Distinction: In classification the possible outputs are discrete categories (e.g. "yes/no", "A/B/C"). In regression a continuous value is predicted (e.g. a price or a temperature).

Related terms: Supervised Learning, Clustering, Feature Engineering

Clustering is an unsupervised learning method in which an algorithm groups data points by similarity, without the groups being defined in advance.

How it works: The algorithm measures the similarity between data points (e.g. by distance) and groups similar points together. The methods differ in how they do this: K-Means splits data into a predefined number of clusters, hierarchical clustering builds a tree structure step by step, and DBSCAN also detects irregularly shaped clusters and filters out outliers.

Business example: An e-commerce company clusters its customers by purchasing behaviour and interests. The result: clusters such as "bargain hunters", "brand fans" and "occasional buyers", each targeted with different marketing.

Application areas:

• Customer segmentation in marketing
• Anomaly and fraud detection
• Grouping documents by topic similarity
• Gene expression analysis in bioinformatics

Distinction: In clustering there are no predefined categories; the algorithm discovers the structure itself. In classification, by contrast, the categories are known and the model learns to assign new data to them.

Related terms: Unsupervised Learning, Classification, Feature Engineering

CNN (Convolutional Neural Network) is a deep neural network architecture that scans image data with learnable filters and builds up features layer by layer, from simple edges through textures to complete objects.

How it works: In a convolutional layer, small filters slide across the image and compute a response at every position. Because the same filter weights apply everywhere, a CNN needs far fewer parameters than a fully connected network and recognises a pattern regardless of where it appears. Pooling layers reduce resolution so that deeper layers see larger portions of the image. Derived architectures such as U-Net, Fast R-CNN and Mask R-CNN reuse this feature extractor and add output heads for segmentation or detection.

Business example: An automotive supplier inspects painted surfaces for inclusions and scratches. A CNN pre-trained on general image data is fine-tuned on a few thousand line-scan camera images and takes over the pre-sorting of inspection images, leaving only borderline cases for human review.

Application areas:

• Image classification in optical quality inspection
• Backbone for object detection and segmentation
• Land-cover classification in remote sensing
• Analysis of microscopy images
• Analysis of sensor data rendered as images, such as spectrograms

Related terms: Deep Learning, Image Recognition, U-Net, Object Detection

Conformal prediction is a statistical framework for uncertainty quantification that augments the point predictions of any machine learning model with prediction intervals or prediction sets carrying a guaranteed coverage rate. The guarantee is distribution-free: it assumes no particular data distribution, only that calibration and new data are exchangeable.

How it works: On a held-out calibration set, the method measures how far the model's predictions typically deviate from the true values (nonconformity scores). A quantile of these scores then widens new predictions into intervals (Regression) or class sets (Classification). At a 90% confidence level, the intervals contain the true value in at least 90% of cases – regardless of which model produced the predictions.

Business example: An agricultural company adds conformal prediction intervals to its yield forecasts. Instead of "7.2 t/ha", the procurement team receives "6.5 to 7.9 t/ha with 90% coverage" and can align storage and logistics decisions with the actual risk.

Application areas:

• Reliable uncertainty estimates for forecasts in planning and procurement
• Safeguarding models in regulated domains (pharma, medical devices)
• Triage: routing uncertain cases to human experts automatically
• Retrofitting uncertainty onto already trained models

Related terms: Statistical Inference, Machine Learning, Regression, Classification

A confusion matrix is a table for evaluating classification models that contrasts predicted classes with actual classes. All common quality metrics of a classifier can be read directly from it.

How it works: For a binary problem the matrix has four cells: true positives, true negatives, false positives and false negatives. From these, metrics such as accuracy, precision, recall and F1 score are derived. With more than two classes, the matrix additionally reveals which classes the model systematically confuses – hence the name.

Business example: An engineering company evaluates a model for automatic defect detection. The confusion matrix shows that although 97% of parts are judged correctly, a substantial share of true defects goes undetected – the team then shifts the decision threshold in favour of catching defects.

Application areas:

• Evaluating Classification models
• Choosing the decision threshold for imbalanced classes
• Quality evidence before deploying a model to production
• Error analysis: which classes get confused?

Distinction: Accuracy alone is misleading when classes are rare (e.g. 1% defects); the confusion matrix exposes such distortions by reporting all four outcome types separately.

Related terms: Classification, Supervised Learning, Machine Learning

Dagster is an open-source orchestration tool for data platforms that puts data assets, rather than individual tasks, at the centre: tables, files and models are declared as assets with dependencies and updated automatically in the correct order.

How it works: Developers describe in Python which assets exist, what they are derived from and when they count as stale. Dagster infers the dependency graph, schedules runs (time- or event-driven), logs every materialisation and surfaces data lineage and freshness in a web interface. Integrations for dbt, Spark and cloud storage are built in.

Business example: An agricultural company orchestrates its daily data processing with Dagster: weather data and satellite imagery are ingested, computed into field-level metrics and fed into forecasting models. If an intermediate step fails, Dagster shows exactly which downstream assets are stale and recomputes only those.

Application areas:

• Orchestrating ETL and ELT workflows
• Coordinating model training and feature pipelines
• Data lineage and freshness monitoring
• Managing heterogeneous tools (dbt, Spark, Python) under one interface

Distinction: Dagster orchestrates when and in what order transformations run; tools such as dbt perform the transformation itself. In practice, Dagster frequently triggers dbt runs.

Related terms: Data Pipeline, ETL (Extract, Transform, Load), dbt

Data engineers are the architects of the data infrastructure. They build, operate and optimise the data pipelines that capture raw data from various sources, transform it and make it available for analysis. Their work is the foundation data scientists build on.

How it works: Data engineers develop and maintain ETL processes (extract, transform, load), implement databases and data warehouses, manage cloud infrastructure and make sure data is complete, correct and available on time.

Business example: A retail group has sales data in five different regional systems. The data engineer builds a pipeline that consolidates, cleans and loads this data into a central data warehouse each night, so the BI team finds current, consistent data each morning.

Core tasks:

• Building and maintaining data pipelines
• Data cleaning and transformation
• Implementing and managing databases
• Cloud platform management (AWS, Azure, GCP)
• Ensuring data quality and availability
• Developing APIs for data distribution

Distinction: Data engineers build the infrastructure; data scientists use it. A data scientist analyses data and develops models; a data engineer makes sure the right data arrives in the right quality.

Related terms: Data Scientist, Data Warehouse, Data Pipeline

Extraction is the first step in the ETL process. Data is read from one or more source systems and made available for further processing.

How it works: Extraction must place as little load as possible on the source systems (e.g. through incremental extraction – only new or changed records are loaded) while still ensuring completeness.

Business example: For its quality reporting, a pharmaceutical company reads batch records from the ERP system, measurement series from the laboratory systems and cleanroom environmental readings every day. So that live operations are not slowed down, extraction works against a replica of the production database and transfers only the records changed since the previous run.

Typical sources: databases, ERP systems, web APIs, log files, external data feeds.

Related terms: Data Transformation, Data Loading, Data Pipeline

Data governance is the set of rules, responsibilities and processes with which a company ensures the quality, security and usability of its data. It defines who owns which data, what quality standards apply and who may access what – the foundation that analytics and AI projects can rely on.

How it works: Data governance assigns an owner to each data domain (data ownership), defines standards for data quality, access rights and retention, and anchors them in processes and tools such as data catalogues. Quality metrics are measured continuously; changes to central datasets follow defined approval paths.

Business example: A machinery manufacturer wants to use sensor data from several plants for Predictive Maintenance. Only once uniform signal definitions, clear responsibilities and documented quality rules are in place do the datasets become comparable – and the models reliable.

Application areas:

• Data quality standards for analytics and AI projects
• Access and authorisation concepts for sensitive data
• Data catalogues and data ownership
• Prerequisite for compliance (GDPR, EU AI Act)

Distinction: Data governance sets the rules; data management and tools such as a Data Warehouse or Data Pipeline implement them. AI Governance carries the same idea over from data to AI systems.

Related terms: AI Governance, Data Warehouse, Data Pipeline, Data Engineer

Data ingestion describes the first step of a data pipeline: importing data from various sources into a target system for immediate processing or storage.

How it works: A distinction is made between batch ingestion (data is collected at regular intervals, e.g. daily) and streaming ingestion (data is transferred continuously in real time). During ingestion, data is checked for completeness and basic consistency.

Business example: A wind farm operator streams readings from several thousand sensors continuously into a central platform, while maintenance reports and settlement data are taken in once a day as files. On arrival, both paths check only whether a timestamp and a turbine identifier are present; substantive processing follows in later steps of the pipeline.

Typical sources:

• Relational databases (SQL)
• REST APIs and web services
• File-based sources (CSV, JSON, Parquet)
• Streaming systems (Kafka, Kinesis)
• IoT sensors and device data

Distinction: Data ingestion is the entry point of the data pipeline, before data wrangling. It transfers data but changes it as little as possible.

Related terms: Data Pipeline, Data Wrangling, ETL

A data lake is a central storage location where large volumes of raw data are kept in their original format – structured, semi-structured or unstructured. Unlike a data warehouse, a data lake does not enforce a predefined data structure.

How it works: Data is first stored unchanged ("schema on read"). The structure is defined only when the data is read and analysed, depending on the question. Modern data lakes are usually built on cloud storage systems such as AWS S3, Azure Data Lake Storage or Google Cloud Storage.

Business example: A pharmaceutical company stores clinical trial data, lab results, patient records and sensor data from production facilities in a data lake. Depending on the analysis need – efficacy, safety or production efficiency – different subsets of this data are retrieved and analysed.

Advantages:

• Maximum flexibility in data storage
• Cost-effective for very large volumes of data
• Suitable for exploratory analysis and machine learning

Disadvantages:

• Without careful management it quickly becomes an unmanageable "data swamp"
• Slower queries than optimised data warehouses

Distinction: A data warehouse stores structured, already-prepared data for defined analyses. A data lake stores everything, raw and unprocessed, for as-yet-unknown future questions.

Related terms: Data Warehouse, Data Engineer, Unstructured Data

Loading is the final step in the ETL process: the transformed data is written to the target system, typically a data warehouse, a database or a data lake.

How it works: The load step writes the prepared data into the target structure, for example the fact tables of a data warehouse or the file layout of a data lake. The target system's schema, key relationships and integrity rules have to be respected, so loads usually run transactionally: an aborted run leaves no half-filled tables behind. Large volumes are written in bulk rather than record by record, which is orders of magnitude faster. Repeatability matters just as much – a load that runs a second time must not create duplicates.

Business example: A mechanical engineering company loads the previous day's production data from its shop-floor control system into the data warehouse every night. Only the shift logs added since the last run are transferred rather than the full set (incremental load), while quality-control results corrected after the fact overwrite the records already present (upsert). By the morning, departments have up-to-date reports.

Variants:

• Full load: a complete rewrite of all data (simple, but resource-intensive)
• Incremental load: only new or changed records are loaded (more efficient)
• Upsert: new records are inserted, existing ones updated

Related terms: Data Extraction, Data Transformation, Data Warehouse

A data pipeline is an automated process that captures, transforms and moves data from one or more sources into a target system. It is the technical infrastructure on which data-driven applications are based.

How it works: A pipeline consists of a sequence of steps: data is extracted (e.g. from an API or database), transformed (cleaned, enriched, aggregated) and loaded (into a data warehouse, a data lake or directly into an application). Modern pipelines are event-driven or scheduled and run fully automatically.

Business example: A fintech company has a real-time data pipeline that checks transaction data for signs of fraud in under a second and automatically triggers an alert on suspicion.

Application areas:

• Real-time analytics (streaming pipelines)
• Nightly batch processing for BI systems
• ML feature pipelines for model training and inference

Distinction: A data pipeline is the route the data travels. ETL is the pattern many pipelines are built on. A data warehouse is the target system a pipeline often delivers to.

Related terms: Data Engineer, ETL, Data Lake

Data science is an interdisciplinary field that combines methods from statistics, computer science and domain knowledge to gain insights from data and enable data-driven decisions. It links theory and practice: from data collection through model development to communicating results.

How it works: The starting point is a business question that is translated into an analytical task – a forecast, a classification, or the search for relationships in the data. The data needed is then sourced, cleaned and turned into meaningful variables (feature engineering). Models are trained on that basis, tested against held-out data and validated together with the domain experts. The sequence is iterative: what one round shows changes the question, the data basis or the choice of model for the next.

Core areas:

• Statistical analysis and probability
• Machine learning and modelling
• Data wrangling and data preparation
• Data visualisation and storytelling
• Deployment and monitoring of models (MLOps)

Business example: A hospital uses data science to reduce readmission rates: models analyse patient data to identify at-risk patients early and provide targeted follow-up care.

Distinction: Data science is broader than business intelligence (which mainly analyses historical data) and broader than machine learning alone (which covers only one part of the methods used). Data science spans the entire process from question to decision.

Related terms: Data Scientist, Machine Learning, Business Intelligence

Data scientists combine statistical expertise, programming skills and domain knowledge to turn large volumes of data into actionable insights. They develop models that recognise patterns in data and enable predictions.

How it works: A typical data science project follows an iterative process: problem definition → data collection → data wrangling → exploratory analysis → model development → validation → deployment → monitoring.

Business example: An insurance company commissions a data scientist to develop a fraud-detection model. The model analyses historical claims, identifies suspicious patterns and automatically flags new cases for manual review, which noticeably eases the workload on claims handlers.

Core tasks:

• Exploratory data analysis (EDA)
• Developing and validating machine learning models
• Statistical evaluation and hypothesis testing
• Communicating results to non-technical stakeholders
• Working with data engineers to bring models into production

Distinction: A data scientist develops models and gains insights. A data engineer builds the infrastructure that work depends on. A BI analyst creates reports and dashboards on existing structures, without model development.

Related terms: Data Engineer, Machine Learning, Feature Engineering

A data staging area is a temporary buffer in the ETL process. Data from various source systems is collected here before it is transformed and loaded into the target system (e.g. a data warehouse).

How it works: The staging area isolates the load process from the production system. Raw data is first stored unchanged, checked, and only processed further after successful validation. If errors occur, the process can be repeated without affecting the target system.

Business example: A remote sensing provider collects scenes from several satellite missions, along with reference measurements from the field, in a staging area each night. Tile boundaries, acquisition times and coordinate reference systems are checked there, and only validated scenes pass on to the analysis system. When one mission temporarily delivered incomplete metadata, the problem stopped in the staging area instead of corrupting the existing time series.

Advantages:

• Improved data quality and integrity
• Fault tolerance: problems are caught before they reach the data warehouse
• Enables fast re-runs of failed loads

Disadvantages:

• Additional storage and maintenance effort
• Increased complexity of the data pipeline

Related terms: Data Warehouse, Data Extraction, Data Engineer

Transformation is the second step in the ETL process. Raw data is brought into a uniform, analysable format.

How it works: Transformation sits between extraction and loading: it brings data of different origins onto a common schema, common units and common keys. The rules are defined once and applied identically on every run, so that analyses remain comparable across reporting periods. Whether transformation happens before loading (ETL) or only inside the target system (ELT) depends mainly on the target's compute capacity – modern cloud data warehouses often take on the step themselves.

Business example: A farming business combines yield data from combine harvester telemetry, soil samples from a laboratory and weather data from an external provider. Transformation resolves the differing field labels used by each source to the internal field number, converts yields to a common unit and aggregates readings into daily means. Only then can the three sources be analysed together.

Typical transformation steps:

• Removing duplicates and inconsistent values
• Standardising formats and data types
• Enriching with data from other sources
• Aggregation (e.g. building daily totals from individual transactions)
• Calculating derived metrics

Related terms: Data Extraction, Data Loading, Data Wrangling

A data warehouse is a central, structured database that consolidates data from various source systems, standardises it and makes it available in a form optimised for analysis and reporting. It is the heart of many business intelligence architectures.

How it works: Data is extracted from operational systems via ETL processes, transformed (standardised, cleaned) and loaded into the data warehouse. The data structure is defined in advance ("schema on write"), which enables fast, consistent queries.

Business example: A retail group consolidates sales data from 200 stores, its online shop and its ERP system into a central data warehouse. The BI team can then retrieve cross-region analyses on demand.

Common solutions:

• Cloud-based: AWS Redshift, Google BigQuery, Snowflake, Azure Synapse
• On-premises: SAP BW, Oracle Exadata, IBM Db2 Warehouse

Distinction: A data warehouse is structured and optimised for known analyses. A data lake stores raw data flexibly for unknown, future questions.

Related terms: Data Lake, Business Intelligence, ETL

Data wrangling (also: data munging) is the process of cleaning, transforming and standardising raw data so it can be used for analysis or in machine learning models. In practice this step often takes up the largest share of a data scientist's time.

How it works: Real-world raw data is rarely clean: values are missing, formats are inconsistent, duplicates exist, units do not match. Data wrangling addresses all of this systematically before the actual analysis begins.

Typical steps:

• Merging data from various sources
• Handling missing values (delete, interpolate, replace)
• Removing duplicates and outliers
• Standardising formats (date, currency, spelling)
• Type conversions and unit conversions
• Creating new, derived variables

Business example: A logistics company captures delivery data from three systems in different formats. Before routes can be optimised, addresses must be normalised, missing timestamps estimated and duplicate entries removed – that is data wrangling.

Distinction: Data wrangling is a preparatory step before the actual analysis. It is not about gaining insights but about ensuring data quality. "Garbage in, garbage out" – without careful wrangling, every downstream analysis is unreliable.

Related terms: Data Engineer, Feature Engineering, Data Ingestion

Dataiku is a commercial end-to-end platform for data science and machine learning that brings data preparation, model development, deployment and monitoring together in one collaborative environment. It serves both data scientists (code) and business users (visual workflows).

How it works: Users build flows in Dataiku: visual data pipelines made up of data sources, preparation steps (recipes) and models. Code steps in Python, R or SQL mix freely with no-code building blocks; trained models are versioned, deployed and monitored through the platform.

Business example: A pharmaceutical company standardises its analytics projects on Dataiku: lab teams prepare study data visually, while data scientists develop predictive models in Python within the same environment. Governance features document which data and model versions fed into each analysis.

Application areas:

• Collaborative data science projects across teams and skill levels
• Building and operating recurring data pipelines
• Model deployment and monitoring (MLOps)
• Self-service analytics for business departments

Related terms: Machine Learning, MLOps, Data Pipeline

dbt (data build tool) is an open-source tool for transforming data directly inside the data warehouse: transformation logic is defined as versioned SQL models that dbt runs in the correct dependency order, tests and documents. Alongside the open-source core (dbt Core), dbt Labs offers a commercial cloud service.

How it works: Each dbt model is a SELECT statement that produces a table or view in the data warehouse. Models reference one another, from which dbt derives the dependency graph. Built-in tests (such as uniqueness, not-null and referential integrity) validate the results on every run; documentation and lineage are generated from the code.

Business example: A pharmaceutical company consolidates sales, production and quality data in a cloud warehouse. The data team uses dbt to define validated metric tables on top; every change to the logic goes through code review and automated tests before it reaches the reports used by business departments.

Application areas:

• Building validated metric and reporting layers in the warehouse
• ELT workflows: transformation after loading the raw data
• Data quality tests and auto-generated documentation
• Versioned transformation logic delivered via CI/CD

Distinction: dbt covers only the "T" in ETL/ELT – extraction and loading are handled by other tools, and scheduling usually falls to an orchestrator such as Dagster.

Related terms: SQL, Data Warehouse, ETL (Extract, Transform, Load), Dagster

Deep learning is a subfield of machine learning based on multi-layer neural networks. Thanks to their large number of layers, these models can learn extremely complex patterns and representations, without manual feature engineering steps.

How it works: Each layer of the network learns increasingly abstract representations of the input data. In image recognition, for example: layer 1 detects edges, layer 2 shapes, layer 5 faces. Training requires large volumes of data and considerable computing power (typically GPUs).

Business example: A radiologist is supported by a deep learning model trained on millions of X-ray images that flags anomalies in new scans with high accuracy, as a second opinion, not a replacement.

Key architectures:

• Convolutional neural networks (CNN): image recognition
• Recurrent neural networks (RNN / LSTM): time series, text
• Transformers: NLP, LLMs
• Generative adversarial networks (GAN): image synthesis

Distinction: Deep learning is a subset of machine learning that relies specifically on deep neural networks. Classic ML (e.g. random forest, linear regression) needs neither deep networks nor necessarily large datasets.

Related terms: Neural Network, Machine Learning, Transformer Architecture

Descriptive analytics answers the question: "What happened?" It analyses historical data and summarises it in understandable reports, metrics and visualisations.

How it works: Based on data aggregation and statistical measures (mean, median, distributions), trends, patterns and anomalies are made visible, often in dashboards or standardised reports.

Business example: A retailer analyses the past Christmas season: which products sold best? In which regions? Which discount campaigns boosted revenue the most?

Typical visualisations:

• Bar and line charts
• Heatmaps and geographic maps
• Pivot tables and KPI tiles

Distinction: Descriptive analytics looks back. Predictive analytics looks ahead. Prescriptive analytics recommends actions.

Related terms: Business Intelligence, Predictive Analytics, Prescriptive Analytics

Dimensionality reduction refers to techniques that reduce the number of features in a dataset while preserving as much of the essential information as possible. It makes high-dimensional data easier to visualise, speeds up model training and mitigates problems such as the "curse of dimensionality".

How it works: Linear methods such as principal component analysis (PCA) project the data onto a few axes of maximum variance. Non-linear methods such as t-SNE and UMAP preserve local neighbourhoods and are particularly suited to 2D visualisation; autoencoders learn compressed representations with neural networks. Alternatively, feature selection picks a subset of the original features rather than constructing new ones.

Business example: A pharmaceutical company analyses gene expression data with over 20,000 features per sample. After PCA reduces these to a few dozen components, sample groups become visible in Clustering, and downstream models train faster and more stably.

Application areas:

• Visualising high-dimensional data (omics, spectra, sensor streams)
• Preprocessing before clustering and classification
• Noise reduction and data compression
• Speeding up training and inference

Distinction: Feature Engineering deliberately constructs new, domain-motivated features; dimensionality reduction compresses existing features in a data-driven way. The two are often combined.

Related terms: Unsupervised Learning, Clustering, Feature Engineering, Self-Organizing Map (SOM)

Edge AI means running AI models directly on devices in the field – machine controllers, cameras, vehicles or industrial PCs – instead of in the cloud. Data is analysed where it is generated.

How it works: The model is trained centrally (cloud or data centre) and then optimised for the target hardware: reduced in size through quantisation and pruning and converted into a runtime-optimised format. Inference then runs locally with low latency, works without a stable internet connection – and raw data never leaves the device.

Business example: A supplier inspects parts directly on the production line: a camera and an industrial PC evaluate every part within the cycle time. The decision is made in milliseconds on the device – no image has to travel to the cloud.

Application areas:

• Visual quality inspection at production speed
• Condition monitoring of machines and vehicles
• Autonomous mobile machinery (e.g. agricultural equipment)
• Privacy-critical analyses where data must stay on site

Distinction: With cloud AI, data travels to a central service that returns the result – flexible, but dependent on bandwidth and latency. Edge AI trades that flexibility for speed, autonomy and data sovereignty; the two are often combined (training in the cloud, inference on the device).

Related terms: Image Recognition, Industry 4.0, Predictive Maintenance, CNN (Convolutional Neural Network)

The Elastic Stack is a software suite from Elastic built around the Elasticsearch search engine, complemented by Kibana (visualisation), Logstash and Beats (data collection), for storing, searching and analysing large volumes of text, log and event data. The core components are available under open-source licences, with advanced features reserved for commercial subscriptions.

How it works: Elasticsearch stores documents in inverted indices and answers full-text and aggregation queries in near real time, scaling across distributed clusters. Logstash and Beats handle Data Ingestion from log files, databases or sensors; Kibana presents the results in interactive dashboards.

Business example: A machinery manufacturer consolidates the log data from its production lines in the Elastic Stack. Maintenance engineers search error messages across all lines and spot recurring fault patterns in Kibana dashboards before they lead to downtime.

Application areas:

• Log and event analysis (observability) for IT and production
• Full-text search across document and product repositories
• Security monitoring (SIEM)
• Monitoring dashboards for sensor and time-series data

Related terms: Unstructured Data, Data Ingestion, Text Mining

An embedding is a numerical representation of objects, typically text, images or other data, as a vector in a high-dimensional space. Similar objects sit close together, dissimilar ones further apart.

How it works: An embedding model (e.g. Word2Vec, BERT or OpenAI's text-embedding-3) processes an input value and returns a vector of numbers, e.g. 1,536 numbers for a piece of text. This vector encodes the semantic meaning. When "king" minus "man" plus "woman" roughly equals "queen", embeddings are working.

Business example: A customer service system stores all previous support requests as embeddings. When a new request comes in, the system immediately finds the semantically most similar past requests and suggests proven answers.

Application areas:

• Semantic search (search by meaning, not just keywords)
• Recommendation systems
• Duplicate detection in large text collections
• The foundation for RAG systems
• Similarity analysis in product catalogues

Distinction: An embedding is not a classification, it does not assign anything to a category but represents objects as a point in a semantic space. Statements about similarity only emerge from comparing embeddings (e.g. via cosine similarity).

Related terms: Large Language Model, RAG, Tokenisation

ETL (Extract, Transform, Load) is a fundamental data integration pattern in which data is extracted from source systems, transformed into a target schema and then loaded into a destination such as a Data Warehouse. For decades it has been the standard way of making scattered operational data available for analysis.

How it works:

1. Extract: Read data from sources such as ERP systems, databases, files or APIs (Data Extraction).
2. Transform: Clean, standardise, enrich and map the data to the target schema (Data Transformation).
3. Load: Write the result to the target system (Data Loading), usually on a schedule or incrementally.

Business example: A pharmaceutical company runs nightly ETL jobs that move data from production equipment, lab systems and the ERP into a central warehouse. Only there can batch quality, equipment utilisation and delivery data be analysed together.

Application areas:

• Populating data warehouses and reporting layers
• Consolidating heterogeneous source systems after mergers or system changes
• Preparing training data for machine learning models
• Migrating legacy systems

Distinction: In the more recent ELT pattern, raw data is loaded first and transformed inside the target system (for example with dbt) – a good fit when the warehouse provides enough compute.

Related terms: Data Extraction, Data Transformation, Data Loading, Data Pipeline

The EU AI Act is the European Union's regulation on artificial intelligence. It follows a risk-based approach: the higher the risk an AI system poses to health, safety or fundamental rights, the stricter the obligations – from transparency notices up to comprehensive requirements for high-risk systems; certain practices are banned outright. It applies in its current form as amended by the Digital Omnibus package.

How it works: The regulation assigns AI systems to risk classes (prohibited practices, high risk, limited risk, minimal risk) and distinguishes roles: providers, who develop an AI system or place it on the market under their own name, carry the most extensive obligations; deployers, who use a system professionally, have their own, lighter set of duties. Under Art. 4, companies must also ensure sufficient AI literacy among staff working with AI systems.

Business example: A machinery manufacturer uses AI in three places: visual inspection in production, an internal LLM tool for documents and a forecasting model in service. An inventory maps each system to a risk class and a role (provider or deployer) and derives where documentation, human oversight and training are required.

Application areas:

• Risk classification of a company's AI portfolio
• Clarifying one's role as provider or deployer
• Building AI literacy in the workforce (Art. 4)
• Documentation and transparency duties in AI projects

Distinction: The GDPR governs the handling of personal data, the EU AI Act the use of AI systems – both can apply to the same project at the same time. AI Governance is the company-internal framework for implementing such requirements.

Related terms: AI Governance, Artificial Intelligence (AI), Explainable AI (XAI)

Explainable AI (XAI) is a subfield of artificial intelligence that provides methods for making the decisions of machine learning models understandable to humans. XAI is the umbrella term for concrete explanation techniques such as SHAP and LIME.

How it works: There are two routes to explainable systems: using inherently interpretable models from the start (such as Logistic Regression or decision trees), or explaining complex models such as Deep Learning networks after the fact (post-hoc) – locally for individual predictions or globally for overall behaviour. Typical tools include feature attributions, surrogate models, partial dependence analysis and saliency maps for images.

Business example: A medical device manufacturer must demonstrate to its notified body how its diagnostic support model arrives at recommendations. Using XAI methods, the team documents the decisive factors behind each recommendation and meets the transparency requirements.

Application areas:

• Regulatory requirements (EU AI Act, medical devices, finance)
• Building user trust in AI systems among domain experts
• Model debugging and detecting bias
• Scientific insight: what has the model actually learned?

Related terms: SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), Artificial Intelligence (AI), Deep Learning

Fast R-CNN is a region-based object detection method that passes an image through a convolutional neural network once and then evaluates all object proposals on the shared feature map.

How it works: The earlier R-CNN ran a separate network pass for every candidate region, which made it slow. Fast R-CNN reverses the order: it first computes a feature map for the entire image, then uses RoI pooling to cut a fixed-size feature patch for each proposal. Two output heads predict the class and refined box coordinates, and both are trained jointly. The proposals themselves still come from a separate upstream algorithm; the successor Faster R-CNN folds that step into the network as a region proposal network.

Business example: An industrial manufacturer detects several component types simultaneously in test-rig photographs and records their position in the frame. The two-stage design copes well with partially occluded parts.

Application areas:

• Object detection with bounding boxes
• Foundation for instance segmentation with Mask R-CNN
• Reference baseline during model selection
• Detection tasks with few, clearly separated object classes

Distinction: Two-stage detectors of this family were long the more accurate but slower option next to single-stage methods such as YOLO or SSD. Real-time applications usually settle on a single-stage model.

Related terms: Object Detection, Mask R-CNN, CNN (Convolutional Neural Network), Image Recognition

FASTQ is a text-based file format used in DNA sequencing that stores, for every sequenced read, both the sequence of bases and a quality score for each individual base.

How it works: One entry occupies four lines: an identifier, the base sequence, a separator line and a string whose characters encode the quality score of each base. Those scores express the estimated probability that the base was called incorrectly and come from the sequencing instrument. Since a single run easily produces hundreds of millions of reads, the files are stored compressed and still reach several gigabytes. Analysis begins with quality control and the trimming of low-quality ends, followed by alignment to a reference or de novo assembly.

Business example: A life science company receives sequencing runs from an external service provider every day. A data pipeline checks the quality scores automatically, discards unusable reads and passes the cleaned data on to downstream analysis, including BLAST searches against reference databases.

Application areas:

• Exchange format for raw sequencing data
• Quality control and filtering of reads
• Input format for bioinformatics analysis pipelines
• Archiving and traceability of sequencing runs
• Metagenomics and microbiome studies

Distinction: The older FASTA format stores sequences only. FASTQ adds the per-base quality scores, which is what makes reliable filtering of raw reads possible.

Related terms: BLAST (Basic Local Alignment Search Tool), Data Pipeline, Unstructured Data

Feature engineering is the process of extracting, transforming or creating the most relevant and informative features from raw data for a machine learning model. Good feature engineering is often more decisive for model quality than the choice of algorithm.

How it works: A feature is a single measurable property of a data point. From a transaction timestamp, for example, one can derive weekday, time of day, days since last purchase or season – all potentially informative features. Feature selection then picks the most relevant ones to avoid overfitting.

Business example: For a credit risk model the raw data (account number, date of birth, transactions) says little. Feature engineering turns it into average monthly income, spending volatility and number of payment defaults in the last 12 months – features the model can actually use.

Key techniques:

• Normalisation and standardisation of numerical values
• One-hot encoding of categorical variables
• Deriving time-based features from timestamps
• Interaction features (product or ratio of two features)
• Dimensionality reduction (PCA)

Distinction: Feature engineering is manual and domain-knowledge-driven. Deep learning models can learn many features automatically (automatic feature learning), which displaces manual feature engineering in some areas but does not replace it everywhere.

Related terms: Data Wrangling, Machine Learning, Supervised Learning

Fine-tuning is the process of further training a pre-trained model, e.g. a large language model, on a specific, smaller dataset to adapt it to a particular task or domain.

How it works: A pre-trained model already has broad general knowledge. In fine-tuning, this model is further optimised with domain-specific examples (e.g. internal documents, specific writing styles, technical terminology). The model keeps its general knowledge but learns the specific requirements of the new task.

Business example: A pharmaceutical company takes a general language model and fine-tunes it with internal research reports. The resulting model answers questions about internal studies more precisely and uses the company's own terminology correctly.

Variants:

• Full fine-tuning: all model parameters are adjusted (compute-intensive, expensive)
• Parameter-efficient fine-tuning (PEFT): only a small share of parameters is adjusted (e.g. LoRA – more efficient and cheaper)
• Instruction tuning: the model is trained to follow instructions better
• RLHF (reinforcement learning from human feedback): the model is optimised against human ratings

Distinction: Fine-tuning changes the model itself. Prompt engineering changes only the input; the model stays unchanged. RAG enriches the input with external knowledge without changing the model.

Related terms: Large Language Model, Prompt Engineering, Transfer Learning

A Gaussian process is a statistical modelling technique that delivers not only a prediction but also its uncertainty. Instead of a single function it describes a probability distribution over plausible functions – which makes it particularly suitable for small, expensive-to-collect datasets.

How it works: The core is a kernel function that defines how similar predictions at neighbouring input points should be. From the training points, the model derives a normal distribution for every new point: the mean is the prediction, the spread the confidence interval. Close to measured points the model is confident, far away it becomes honestly uncertain – a property that makes Gaussian processes the standard tool in Bayesian optimisation and for Surrogate Models.

Business example: A development team calibrates an engine test bench with only 60 expensive test runs. A Gaussian process predicts the characteristic map between the measured points and shows at the same time where uncertainty is largest – the next test run is placed exactly there, which markedly reduces the number of runs required.

Application areas:

• Surrogate models for expensive simulations and physical tests
• Bayesian optimisation of process and design parameters
• Calibration of sensors and test benches
• Interpolation of spatial measurements (kriging)

Distinction: A Neural Network needs many data points and does not quantify uncertainty by default; a Gaussian process excels with few data points but scales poorly to very large datasets.

Related terms: Surrogate Model, Regression, Statistical Inference, Conformal Prediction

Generative AI refers to AI systems that can create new content – text, images, audio, video, code or synthetic data. Unlike classic AI models that only classify or predict, generative models produce original outputs.

How it works: Generative models learn the statistical structure of their training data so well that they can produce new data matching that structure. The best-known architectures are transformer-based large language models for text and diffusion models for images (e.g. Stable Diffusion, DALL·E).

Business example: A marketing team uses generative AI to create dozens of text variants for A/B tests in minutes. A software company speeds up development by having developers generate code snippets from natural-language requests.

Application areas:

• Text creation and summarisation
• Code generation and completion
• Image and video synthesis
• Synthesising training data for other models
• Automated report generation
• Personalised customer communication

Distinction: Classic AI recognises and classifies (e.g. "Is this spam?"). Generative AI creates (e.g. "Write an email"). Generative AI also carries risks: hallucinations, deepfakes or copyright questions.

Related terms: Large Language Model, Transformer Architecture, Hallucination

Gradient boosting is an ensemble machine learning method that trains many simple models – usually shallow decision trees – in sequence, with each new model correcting the errors of the ones before it. On structured, tabular data it remains one of the strongest standard methods available.

How it works: The procedure starts with a simple prediction and computes the remaining errors (residuals). A new tree is trained to predict exactly those errors and added to the ensemble with a small weight (the learning rate). This step repeats hundreds to thousands of times until prediction quality stops improving. Widely used implementations include XGBoost, LightGBM and CatBoost.

Business example: An agricultural company forecasts field-level yields from soil data, weather history and management records. A gradient boosting model handles the mixed tabular features directly and captures the non-linear relationships that linear models miss.

Application areas:

• Yield and demand forecasting on tabular data
• Risk assessment and scoring
• Classification and Regression in industrial data projects
• Ranking tasks, for instance in search and recommendation systems

Distinction: A Random Forest trains many trees independently and averages them; gradient boosting builds trees sequentially on top of each other, which often yields higher accuracy but demands more careful tuning.

Related terms: XGBoost, Random Forest, Supervised Learning

A graph database is a database system that stores data as a graph: nodes represent entities (such as customers, products or genes), while edges represent the relationships between them. The data model treats connections as first-class citizens alongside the objects themselves, making it well suited to highly connected data.

How it works: Rather than assembling relationships at query time through joins, as relational databases do, the graph model stores them directly. Queries navigate (traverse) from node to node; the cost depends on the number of edges visited, hardly on the overall size of the database. Common query languages are Cypher, Gremlin and the ISO standard GQL.

Business example: An agricultural company links varieties, field trials, locations and environmental conditions in a graph database. Breeders can query how two varieties are related through past trial series and under which conditions related lines have already been tested.

Application areas:

• Knowledge graphs for research and product data
• Network and dependency analysis (supply chains, IT infrastructure)
• Recommendation and fraud-detection systems
• Linking heterogeneous data sources through shared entities

Distinction: A Vector Database finds similar items via numerical distances; a graph database answers questions about explicitly modelled relationships. The two are increasingly combined, for instance in knowledge graphs with embedding search.

Related terms: Neo4j, Ontologies, Structured Data

Hallucination is the phenomenon where an AI language model generates statements that sound plausible but are factually wrong or entirely made up, without the model signalling this.

How it works: Language models generate text by computing statistically likely continuations. They do not "know" whether a statement is true – they produce what reads as linguistically coherent. In areas where the model has little training data, or when asked for specific facts, the risk of hallucination rises.

Business example: An employee asks an AI system about an internal company standard. The model confidently cites an internal document – one it has invented. Without verification the wrong information is passed on.

Typical types of hallucination:

• Invented facts (wrong figures, names, dates)
• Non-existent sources or citations
• Misinterpretation of context
• Seemingly logical but factually wrong conclusions

Countermeasures:

• Retrieval-augmented generation (RAG): the model is given verified sources as context
• Fine-tuning on verified data
• Human review of critical outputs
• Using models that cite their sources

Distinction: Hallucination is not an "error" in the classic sense; the model works technically correctly. It is a structural property of generative models that can be reduced through suitable system architecture but not fully eliminated.

Related terms: Large Language Model, RAG, Generative AI

Image recognition is a subfield of computer vision: the ability of software systems to automatically identify, classify and interpret content in images or videos. It is based on neural networks, in particular convolutional neural networks (CNNs).

How it works: The model is trained on large, labelled image datasets. It gradually learns to assemble simple features (edges, colours) into more complex structures (shapes, objects). After training it can classify new, unknown images.

Business example: A machine manufacturer uses image recognition in quality control: a camera photographs each component and the model detects cracks or deviations in milliseconds, faster and more consistently than a manual visual inspection.

Application areas:

• Quality assurance in production
• Face recognition and access control
• Medical image analysis (X-ray, MRI)
• Autonomous driving (object detection in traffic)
• Plant disease or pest detection in agriculture

Distinction: Image recognition classifies images (e.g. "cat" or "dog"). Object detection goes a step further and locates multiple objects within an image at once using bounding boxes.

Related terms: Neural Network, Deep Learning, Supervised Learning

Image segmentation is the pixel-level partitioning of an image into coherent regions, assigning every pixel a class label or an object instance.

How it works: Semantic segmentation labels pixels by class without separating individual objects; instance segmentation distinguishes each object in its own right; panoptic segmentation combines both. Encoder-decoder models such as U-Net and region-based models such as Mask R-CNN are the common architectures. Training relies on manually drawn masks, and results are scored with metrics such as intersection over union or the Dice coefficient. Images too large for GPU memory are processed with tiled inference.

Business example: A life science laboratory determines cell counts and cell areas from microscopy images. A pre-trained specialist model such as Cellpose segments the cells and replaces manual counting, which previously occupied a technician for several hours per experimental series.

Application areas:

• Area measurement in remote sensing
• Cell and tissue segmentation in microscopy and diagnostics
• Measuring defect areas in materials testing
• Analysis of medical imaging
• Isolating objects for downstream measurements

Distinction: Object Detection marks objects with rectangles, whereas segmentation follows the actual object outline. Any task that measures area, contour or proportion needs segmentation rather than boxes.

Related terms: Object Detection, U-Net, Mask R-CNN, Tiled Inference

Industry 4.0 refers to the digital transformation of industrial production through the networking of machines, equipment and processes, also known as the fourth industrial revolution. At its core is the integration of cyber-physical systems, the Internet of Things (IoT) and data-driven algorithms.

How it works: Sensors continuously capture machine data (temperature, vibration, utilisation). This data is transmitted in real time, analysed and used to control processes, either automatically through algorithms or as a basis for operators' decisions.

Business example: A factory monitors all production lines via sensors. An algorithm detects from vibration data that a machine is likely to fail within the next 48 hours, and automatically triggers a maintenance order before an unplanned outage occurs (predictive maintenance).

Core technologies:

• Internet of Things (IoT)
• Edge computing
• Cloud platforms
• Machine learning for predictive maintenance
• Digital twins

Related terms: Machine Learning, Time Series Analysis, Pattern Recognition

Knowledge discovery in databases (KDD) describes the complete, multi-stage process of extracting useful knowledge from large datasets, from selecting raw data to the interpreted insight.

How it works: KDD arranges work on large datasets into a fixed sequence of steps that leads from raw data to a defensible statement. The steps are not strictly linear – what the evaluation shows regularly sends the work back to data selection or preparation. In practice, most of the effort goes into the preparatory steps rather than into the algorithm itself.

Process steps:

1. Data selection: select relevant data from existing sources
2. Pre-processing: cleaning, handling missing values, consistency checks
3. Transformation: convert data into suitable formats and representations
4. Data mining: apply algorithms (clustering, classification, regression, etc.)
5. Interpretation & evaluation: check results for domain relevance and communicate them

Business example: A machinery manufacturer wants to know why individual assemblies fail more often than the rest. Service reports, sensor readings and bills of material are merged into one cleaned dataset; the pattern analysis that follows shows the failures are tied to a particular combination of supplier batch and operating temperature. Engineering reviews the finding for plausibility, and the result feeds into incoming goods inspection.

Distinction: Data mining is only one step within the KDD process, namely the algorithmic search for patterns. KDD describes the entire knowledge-generation process.

Related terms: Data Wrangling, Data Science

A knowledge graph is a structured knowledge base that represents entities (products, components, customers, standards) and their relationships as a graph. It connects information from scattered sources into one queryable picture of a company's knowledge.

How it works: Domain knowledge is modelled as triples – for instance: component A is used in assembly B, assembly B is governed by standard C. The graph is typically stored in a Graph Database; Ontologies define the permitted types and relationships. Queries follow the edges across several hops, and combined with RAG the graph supplies language models with verifiable factual knowledge (GraphRAG).

Business example: An automotive supplier links bills of materials, change states, complaints and suppliers in a knowledge graph. When a defect appears in series production, one query shows within seconds which other assemblies contain the same component from the same supplier in the same period.

Application areas:

• Connecting product, process and supplier data
• Traceability and root-cause analysis for quality issues
• Knowledge management: making expert knowledge queryable
• Fact base for RAG systems and company chatbots

Distinction: A Graph Database is the storage technology; the knowledge graph is the semantically modelled knowledge built with it. A Vector Database finds similar items via numerical distances, a knowledge graph answers questions via explicit relationships.

Related terms: Graph Database, Ontologies, Retrieval-Augmented Generation (RAG), Neo4j

A large language model (LLM) is an AI model trained on very large amounts of text that understands, generates and processes natural language. LLMs are the foundation behind systems such as ChatGPT, Claude or Gemini.

How it works: LLMs are based on the transformer architecture. They are first trained in a self-supervised way on enormous text corpora running to trillions of tokens, in order to learn language structures. They are then optimised for helpful, safe conversations through fine-tuning and RLHF (reinforcement learning from human feedback).

Business example: An insurance group uses an internally hosted LLM that helps claims handlers analyse and summarise long claim reports. Handling time per case drops noticeably.

Application areas:

• Text summarisation and extraction
• Automated customer service (chatbots)
• Code assistance for developers
• Document analysis and classification
• Internal knowledge search (RAG)
• Automated report generation

Well-known models: GPT (OpenAI), Claude (Anthropic), Gemini (Google), Llama (Meta), Mistral (Mistral AI).

Distinction: LLMs are a subfield of generative AI specialised in language. Multimodal models can additionally process images, audio or video. Classic NLP models (e.g. named entity recognition) are smaller and more task-specific.

Related terms: Transformer Architecture, Prompt Engineering, RAG, Fine-Tuning, Hallucination

LIME (Local Interpretable Model-agnostic Explanations) is an Explainable AI (XAI) method that explains individual predictions of any machine learning model by approximating the model around the data point in question with a simple, interpretable surrogate model. "Model-agnostic" means it works with any model type, from a Random Forest to a deep neural network.

How it works: LIME generates many slightly perturbed variants of the data point to be explained, has the original model score them, and fits a weighted linear model to the results, giving nearby variants more weight. The surrogate's coefficients show which features push the specific prediction up or down locally. The principle works for tabular data, text and images.

Business example: An agricultural service provider uses an image model to detect leaf diseases. LIME highlights the image regions that drove each diagnosis, letting agronomists verify that the model responds to actual lesions rather than image artefacts.

Application areas:

• Spot-check explanations of individual model decisions
• Plausibility checks for image and text classifiers
• Communicating model behaviour to domain experts
• Uncovering spurious correlations picked up in training

Distinction: LIME explains locally and approximately; SHAP attributes contributions consistently on game-theoretic grounds, at higher computational cost. In practice the two complement each other.

Related terms: Explainable AI (XAI), SHAP (SHapley Additive exPlanations), Machine Learning

Location analysis is the data-driven assessment of geographic sites for a specific use, such as a retail branch, a plant or a service location. It combines geospatial, market and movement data into a comparable score across candidate sites.

How it works: Features are assembled for every candidate location: population and purchasing power, competitor sites, accessibility by road and public transport, land use and footfall. A statistical or Machine Learning model then learns the relationship between those features and an outcome such as revenue, customer count or utilisation from existing locations and applies it to the candidates. Results are delivered as maps and as a ranked shortlist.

Business example: A retailer with 80 existing branches trains a model on their revenues and surrounding characteristics. For 300 possible new locations the model predicts expected revenue and flags which candidates would mainly draw customers away from branches the company already operates.

Application areas:

• Expansion and branch network planning in retail
• Geomarketing and catchment area analysis
• Site selection for production and service facilities
• Valuation of commercial property and development land
• Public-sector service coverage planning

Related terms: Spatial Optimization, Predictive Analytics, ArcGIS, Smart City

Logistic regression is a statistical supervised learning method that models the probability of class membership and, despite its name, is used for Classification. Its simplicity and interpretability make it a common first model in classification projects.

How it works: The model computes a weighted sum of the input features and passes it through the logistic (sigmoid) function to obtain a probability between 0 and 1; above a chosen threshold (e.g. 0.5) the positive class is assigned. The learned coefficients show directly how strongly, and in which direction, each feature influences the outcome – valuable wherever decisions must be justified.

Business example: A pharmaceutical company estimates from process parameters the probability that a production batch will fail quality inspection. The model's coefficients simultaneously show the process engineers which parameters drive the risk.

Application areas:

• Binary classification (pass/fail, failure/no failure)
• Risk scoring in medicine and quality management
• Baseline model before applying more complex methods
• A/B test evaluation and effect analysis

Distinction: Unlike linear Regression, logistic regression predicts a class probability rather than a continuous quantity.

Related terms: Regression, Classification, Supervised Learning

LSTM (Long Short-Term Memory) is a recurrent neural network variant that uses gating mechanisms to control which parts of a sequence are stored, overwritten or passed on, allowing it to capture dependencies over long stretches of data.

How it works: Each LSTM cell carries a memory state governed by an input gate, a forget gate and an output gate. These gates are learned functions that decide at every time step how much new information enters the memory and how much of the existing memory is discarded. This design avoids the vanishing-gradient problem that prevents simple recurrent networks from learning long-range dependencies.

Business example: A plant operator monitors vibration and temperature time series from a production line. An LSTM learns the normal signature across several operating cycles and flags deviations that indicate early bearing wear, often days before the machine would fail.

Application areas:

• Forecasting demand, yield and sensor time series
• Predictive maintenance
• Anomaly detection in measurement series
• Sequence modelling for speech and handwriting
• Analysis of continuous signals in Signal Processing

Distinction: Transformer models have largely replaced LSTMs in language processing. For industrial time series with limited training data and constrained compute, LSTMs remain a practical choice.

Related terms: Neural Network, Deep Learning, Transformer Architecture, Anomaly Detection

Machine learning (ML) is a subfield of artificial intelligence in which algorithms learn from data to recognise patterns and make predictions, without being explicitly programmed for every situation.

How it works: An ML model is optimised on a training dataset by iteratively adjusting its internal parameters to minimise a defined error. The result is a model that generalises to new, unseen data.

Learning paradigms:

• Supervised learning: learning from labelled examples
• Unsupervised learning: finding patterns without labels
• Reinforcement learning: learning through reward and penalty

Business example: A utility company uses ML to forecast electricity consumption for the next 24 hours, based on weather data, time of day and historical consumption patterns. This enables more efficient grid management.

Distinction: ML is a tool within the broader field of AI. Not all AI systems use ML (rule-based systems, for example, do not). Deep learning is a subfield of ML based on multi-layer neural networks.

Related terms: Deep Learning, Feature Engineering, Supervised Learning

Mask R-CNN is a deep learning architecture for instance segmentation that outputs a class, a bounding box and a pixel-accurate mask for every detected object.

How it works: The architecture extends the Faster R-CNN detector with a third, parallel output head that predicts a binary mask for each object proposal. To keep the mask aligned with the image, an interpolating feature lookup called RoIAlign replaces the coarser RoI pooling used by earlier models. A CNN backbone provides the features, usually combined with a feature pyramid so that objects of very different sizes are handled equally well.

Business example: A seed breeder photographs trial plots by drone. Mask R-CNN separates individual plants from one another, so stand density and leaf area can be evaluated per plant rather than only as an aggregate coverage figure for the whole plot.

Application areas:

• Counting and measuring individual objects
• Instance segmentation in microscopy and lab diagnostics
• Detecting and delineating components in manufacturing
• Capturing vehicles and buildings in aerial imagery
• Automatic pre-annotation of large image collections

Distinction: Semantic segmentation, for example with U-Net, marks all pixels of a class as one undifferentiated area. Mask R-CNN additionally separates the individual objects inside that area, which is what counting requires.

Related terms: Image Segmentation, Object Detection, Fast R-CNN, CNN (Convolutional Neural Network)

Microservices are a software architecture pattern in which an application consists of a collection of small, independently deployable services, each fulfilling a clearly defined function and communicating through defined interfaces (APIs).

How it works: Instead of a monolithic application in which all functions are tightly interwoven, a microservice architecture consists of services such as "authentication", "order management" or "recommendation engine", which can be developed, tested, scaled and updated independently of one another.

Business example: On Black Friday, an e-commerce company can scale up just the "checkout" microservice, rather than upgrading the entire application. If one service fails, all the others keep running.

Advantages:

• Independent scalability of individual components
• Faster development through parallel work by multiple teams
• Technology flexibility (each service can be written in the best-suited language)
• Better fault tolerance

Distinction: Unlike a monolith (everything in one application) or a service-oriented architecture (SOA), microservices are smaller, more autonomous and more strongly geared towards independent deployment.

Related terms: API, Data Pipeline

Microsoft Azure is Microsoft's public cloud platform, with globally distributed data centres and services for compute, storage, databases, analytics and machine learning. Alongside AWS (Amazon Web Services), Azure is one of the two largest cloud providers and is especially widespread in organisations that already run Microsoft software.

How it works: Resources such as virtual machines, Kubernetes clusters, databases and data lake storage are provisioned on demand and billed by usage. For data projects, the most relevant services are Azure Machine Learning (model training and deployment), Microsoft Fabric and Synapse (analytics), Azure OpenAI (LLM services), plus the close integration with Microsoft 365 and Power BI.

Business example: A machinery manufacturer runs its IoT data platform on Azure: machines send telemetry to an event hub, the data lands in a data lake, anomaly detection models run in Azure Machine Learning, and the results reach plant management as a Power BI dashboard.

Application areas:

• Cloud data platforms (data lake, warehouse, streaming)
• Training and operating machine learning models
• Hosting applications and Microservices
• Generative AI services via Azure OpenAI

Related terms: AWS (Amazon Web Services), Power BI, MLOps

MLOps (Machine Learning Operations) refers to the practices and tools used to develop, deliver and monitor machine learning models reliably in production. The concept carries DevOps principles such as automation and CI/CD over to the model lifecycle, which additionally involves data and trained artefacts.

How it works: Typical building blocks are versioned data and models, reproducible training pipelines, automated tests before deployment, and monitoring in production. If model quality degrades – for instance because the input data shifts (data drift) – the pipeline triggers retraining or an alert. Widely used tools include MLflow, Kubeflow and the ML services of the major cloud providers.

Business example: A machinery manufacturer runs a scrap-prediction model on its shop floor. An MLOps pipeline tests every new model version against historical data, rolls it out gradually and monitors it in production; if prediction quality drifts, the model is automatically retrained on current production data.

Application areas:

• Automated training, deployment and rollback of models
• Monitoring model quality, data drift and latency in production
• Versioning and traceability of data, code and models
• Governance and compliance requirements for production AI

Distinction: DevOps deals with code only; MLOps adds data and models as separate, evolving artefacts with their own testing and monitoring needs.

Related terms: CI/CD, Machine Learning, Data Pipeline

A multimodal model is an AI system that can jointly process and understand several data modalities, typically text, images, audio and video.

How it works: Multimodal models have separate encoders for different modalities, whose outputs are merged into a shared representation space. The model learns to understand relationships between modalities, e.g. describing the content of an image in text, or generating an image from a text description.

Business example: An insurance company has damage photos analysed automatically by a multimodal model: the model describes the damage, estimates the repair cost and generates a first draft of the claim report, all in one step.

Application areas:

• Automatic image captioning
• Visual document analysis (invoices, forms)
• Video analysis with language understanding
• Medical diagnostic support (image + text)

Well-known models: GPT-4o (OpenAI), Gemini (Google), Claude (Anthropic).

Distinction: A pure LLM processes text only. A multimodal model extends this to further input types. Generative multimodal models can also output images or audio (e.g. DALL·E, Sora).

Related terms: Large Language Model, Generative AI, Image Recognition

Natural language processing (NLP) is a subfield of AI concerned with the understanding, analysis and generation of human language by computer systems.

How it works: NLP systems process text in several steps: tokenisation (breaking text into units), parsing (recognising grammatical structures), semantic analysis (understanding meaning) and pragmatic analysis (taking context into account). Modern NLP systems are based on transformer architectures and far exceed the capabilities of earlier rule-based approaches.

Business example: An energy group uses NLP to analyse thousands of customer complaints from emails and chats every day: topics are recognised, urgency is assessed and tickets are automatically routed to the right teams.

Application areas:

• Automatic text summarisation
• Sentiment analysis
• Chatbots and virtual assistants
• Machine translation
• Information extraction from unstructured documents

Distinction: Classic NLP used rule-based and statistical methods for specific subtasks. Modern LLMs can handle almost all NLP tasks in a single model, at a considerably higher level.

Related terms: Large Language Model, Token / Tokenisation, Text Mining

NDVI (Normalized Difference Vegetation Index) is a vegetation index computed from remote sensing data that combines reflectance in the near-infrared and red bands to describe the density and vigour of vegetation. It is the longest-established and most widely used index in Remote Sensing.

How it works: The index is calculated as (NIR − Red) / (NIR + Red) and therefore ranges from −1 to +1. Healthy plants reflect near-infrared strongly and absorb red light for photosynthesis, giving values around 0.3 to 0.9; bare soil sits near 0.1 to 0.2 and open water falls below zero. Over dense canopies the index saturates and stops resolving differences, which is where indices such as EVI or red-edge variants perform better.

Business example: A farming operation has NDVI trajectories computed for each field from weekly satellite passes. Sections with unusually low values point early to compaction, nutrient deficiency or pest pressure and are carried over as management zones into the variable-rate fertiliser map.

Application areas:

• Crop monitoring and variable-rate fertilisation
• Yield estimation and harvest scheduling
• Detection of drought stress and damaged areas
• Delineation of vegetation and land use classes
• Index time series as inputs during Feature Engineering for yield models

Related terms: Remote Sensing, Spatio-temporal Statistics, Predictive Analytics, Feature Engineering

Neo4j is a native graph database that stores data as nodes and relationships with properties and exposes it through the Cypher query language. It is the most widely used system of its kind, available as an open-source community edition and in commercial variants, including the managed cloud service AuraDB.

How it works: Instead of computing relationships at query time via foreign keys and joins, Neo4j stores them directly as references (index-free adjacency). Queries such as "all suppliers of a supplier" traverse the graph along its edges, keeping multi-hop relationship queries fast even at scale. The Graph Data Science library adds algorithms such as community detection and pathfinding.

Business example: A machinery manufacturer models its supply chain as a graph in Neo4j: parts, suppliers, plants and transport routes as nodes and relationships. When a supplier fails, a single Cypher query reveals which end products are affected and through how many tiers.

Application areas:

• Knowledge graphs and connected master data
• Supply-chain and dependency analysis
• Fraud detection through relationship patterns
• Recommendation systems

Related terms: Graph Database, Ontologies, Vector Database

An artificial neural network (ANN) is a computational model loosely inspired by the structure of the human brain. It consists of layers of interconnected nodes (neurons) that together can learn complex patterns in data.

How it works: Information flows from the input layer through several hidden layers to the output layer. Each connection has a weight that is adjusted during training via backpropagation to minimise the prediction error. Adding more layers produces deep neural networks (deep learning).

Business example: A credit card provider uses a neural network for fraud detection. It analyses over 100 features of a transaction in real time and decides in milliseconds whether a fraud attempt is taking place.

Application areas:

• Image and speech recognition
• Predictive models
• Recommendation systems
• Medical diagnosis

Distinction: Shallow neural networks have few layers and suit simpler tasks. Deep learning refers to networks with many layers that can model more complex relationships.

Related terms: Deep Learning, Machine Learning, Transformer Architecture

A neural operator is a neural network that maps whole functions onto one another rather than individual values – for example, a component's geometry and boundary conditions onto the complete stress or flow field. This makes it possible to learn physical simulations and accelerate them drastically.

How it works: Classical neural networks learn mappings between fixed-size vectors. Neural operators (e.g. Fourier Neural Operators) instead learn mappings between function spaces: trained on many simulation runs, they predict a simulation's full result field directly – independently of the resolution of the computational mesh. Once trained, a prediction takes milliseconds instead of hours.

Business example: A vehicle manufacturer trains a neural operator on hundreds of CFD runs of the airflow around body variants. In the early design phase, engineers see the drag of a new variant almost in real time instead of waiting days for simulation results.

Application areas:

• Accelerating CFD and FEM simulations
• Interactive design exploration in product development
• Real-time models for digital twins
• Weather and flow forecasting in research and industry

Distinction: A classical Surrogate Model usually predicts individual key figures for a fixed simulation setup; a neural operator predicts entire fields and generalises across geometries and resolutions.

Related terms: Surrogate Model, Neural Network, Deep Learning

Object Detection is an image analysis task in which a model locates every object of interest in an image and assigns each one a class and a surrounding rectangle, known as a bounding box.

How it works: Training uses images in which objects have been manually annotated with boxes and classes. Two-stage methods such as Fast R-CNN generate candidate regions first and classify them afterwards, while single-stage methods such as YOLO predict boxes and classes in a single pass. Every prediction carries a confidence score, and non-maximum suppression discards duplicate detections of the same object. Models are evaluated by how far predicted and reference boxes overlap, summarised as mean average precision.

Business example: An agricultural contractor flies fields with a drone and locates weed patches within the crop. The box coordinates are converted into an application map so the sprayer treats only the affected sub-areas, which cuts herbicide use across the field.

Application areas:

• Defect and foreign-object detection in quality inspection
• Counting tasks in production, logistics and agriculture
• Capturing objects in aerial and satellite imagery
• Monitoring of access points and traffic areas
• First stage for tracking moving objects

Distinction: Object detection returns a rectangle per object, which is sufficient for counting and positioning. Image Segmentation goes further and labels every individual pixel, which is what area and shape measurements require.

Related terms: Image Segmentation, Fast R-CNN, Mask R-CNN, Image Recognition

An ontology is a formal, machine-readable model of a knowledge domain that explicitly defines its concepts (classes), their properties and the permissible relationships between them. It pins down what terms such as "active ingredient", "batch" or "field trial" mean in a dataset and how they relate to each other.

How it works: Ontologies are expressed in standards such as OWL or RDF Schema and define class hierarchies (for example "wheat is a cereal"), properties and logical rules. On this basis, systems can map data from different sources to the same concepts unambiguously and derive new facts through inference. In practice, an ontology often serves as the schema of a knowledge graph in a Graph Database.

Business example: A pharmaceutical company harmonises research data from several sites through a shared ontology: substances, targets and study types are defined once, so searches hit the same concepts across sites – regardless of how each source system labels them.

Application areas:

• Schema and terminology control for knowledge graphs
• Data integration across heterogeneous systems and domain vocabularies
• Domain terminologies in life science (such as the Gene Ontology) and industry
• Foundation for semantic search and Retrieval-Augmented Generation (RAG)

Distinction: A taxonomy only arranges terms hierarchically; an ontology additionally describes properties, relationship types and rules.

Related terms: Graph Database, Neo4j, Structured Data

Open data is data that anyone can freely access, use and redistribute for any purpose, at most subject to a requirement to credit the source and to share adaptations on the same terms.

How it works: Open data is typically published by governments, research institutions or companies under open-data licences (e.g. Creative Commons). It is available in standardised, machine-readable formats (CSV, JSON, XML).

Business example: A logistics start-up uses open geodata from the Federal Agency for Cartography and Geodesy and weather data from the German Weather Service free of charge for its route-optimisation model, without collecting any data of its own.

Important sources:

• GovData (Germany): open administrative data
• Eurostat: statistical data of the EU
• OpenStreetMap: map data
• UCI Machine Learning Repository: datasets for ML research
• Destatis: Federal Statistical Office

Distinction: Open data is not the same as anonymised data. Data subject to data-protection law (e.g. personal data) may not be published even when it is technically available.

Related terms: Data Lake, Data Ingestion

Pandas is an open-source library for the Python programming language, used to analyse and process tabular data. Its central object, the DataFrame, is the de facto standard for table-shaped data in the Python ecosystem.

How it works: A DataFrame holds data column-wise in memory and provides operations for filtering, grouping, joining and reshaping, along with readers and writers for CSV, Excel, Parquet and databases. Its compute kernels build on NumPy and work in a vectorised way – column by column rather than looping over rows.

Business example: An agricultural company merges weather data, soil samples and yield records from three systems with Pandas: read them in, standardise units, join on field IDs, handle missing values – the result feeds a yield-forecasting model.

Application areas:

• Exploratory data analysis in notebooks
• Data cleaning and Data Wrangling
• Feature preparation for machine learning models
• Smaller ETL jobs and format conversions

Distinction: Pandas works within the memory of a single machine; for substantially larger datasets, teams turn to warehouses (SQL) or engines built for larger-than-memory data such as Spark or Polars.

Related terms: Python, Data Wrangling, Structured Data

Pattern recognition deals with the automatic identification of regularities, structures and hidden relationships in data, regardless of whether they come as images, text, time series or measurements.

How it works: Algorithms analyse data points for shared properties or recurring structures. Depending on the task, rule-based methods, statistical methods or neural networks are used.

Business example: A telecommunications provider uses pattern recognition on network data to detect the typical early signs of a device failure, and can intervene preventively before customers are affected.

Application areas:

• Anomaly detection in production processes
• Fraud detection in financial transactions
• Speech recognition and processing
• Biometric authentication

Related terms: Machine Learning, Image Recognition, Signal Processing

PDAL (Point Data Abstraction Library) is an open-source library for processing point cloud data, providing readers, writers and filters for formats such as LAS, LAZ, COPC and database tables. It plays the role for 3D point data that GDAL plays for raster and vector data.

How it works: Processing steps are declared as a pipeline in a JSON file: one or more readers, a chain of filters, and a writer. Typical filters reproject coordinates, thin the data, remove noise, classify ground points, compute height above ground, or rasterise the cloud into a terrain model. Pipelines run from the command line or through the Python bindings, which makes them straightforward to embed in scheduled processing.

Business example: An engineering services firm receives weekly construction-site scans in varying coordinate systems. A PDAL pipeline standardises projection and point density, filters noise, and writes both a cleaned LAZ file and a derived terrain model – running unattended as part of the overnight batch.

Application areas:

• Pre-processing of LiDAR and laser scan data
• Format conversion and reprojection of large point datasets
• Derivation of terrain and surface models
• Preparation of training data for point cloud classification
• Automated geospatial processing inside a Data Pipeline

Related terms: Point Cloud, Data Pipeline, Python, Remote Sensing

A point cloud is a set of measured points in three-dimensional space, each carrying coordinates and often additional attributes such as intensity, colour or class label. It is the native output format of LiDAR systems, terrestrial laser scanners and photogrammetric reconstruction.

How it works: A laser scanner measures the return time of emitted pulses to derive ranges; combined with the sensor's position and orientation these become absolute 3D coordinates. The resulting datasets routinely run into billions of points and are stored in formats such as LAS and LAZ. Analysis involves filtering, thinning and classifying the points into categories such as ground, vegetation, buildings or power lines, increasingly with neural networks that operate directly on the points rather than on a rasterised version.

Business example: A grid operator surveys its overhead lines by helicopter-mounted LiDAR. The classified point cloud separates conductors from vegetation and computes the remaining clearances, so that trimming crews are scheduled only where the clearance limits are actually breached.

Application areas:

• Surveying and as-built documentation of plants and buildings
• Vegetation clearance along power lines, tracks and roads
• Generation of digital terrain and surface models
• Conversion of as-built scans into CAD (Computer-Aided Design) models
• Dimensional inspection in manufacturing

Related terms: PDAL, CAD (Computer-Aided Design), Remote Sensing, Image Segmentation

Power BI is Microsoft's business intelligence platform for building interactive reports and dashboards, tightly integrated with Microsoft 365, Microsoft Azure and Microsoft Fabric. Reports are developed in the desktop application and shared across the organisation through the cloud service.

How it works: Power BI loads data from files, databases and cloud services, prepares it with Power Query and models metrics with the DAX formula language. The visualisations within a report are interactively linked; published reports refresh on a schedule or query the source directly.

Business example: A machinery manufacturer consolidates sales and service data in a Power BI report: management sees order intake, service cases and spare-parts revenue by region in one dashboard that refreshes daily from the Data Warehouse.

Application areas:

• Standard reporting and management dashboards
• Self-service analytics in business departments
• Distributing KPIs across Microsoft 365 workplaces
• Visualisation layer on top of data warehouses and lakehouses

Distinction: Power BI visualises and aggregates; large-scale data preparation and model development happen upstream, for example with SQL, dbt or Python.

Related terms: Business Intelligence (BI), Tibco Spotfire, Microsoft Azure

Predictive analytics answers the question: "What will happen?" It uses historical data, statistical models and machine learning algorithms to calculate the probabilities of future events or developments.

How it works: Based on historical patterns, the model learns which factors influence future events. The model is optimised on training data and then applied to new data to make predictions.

Business example: A telecommunications provider analyses usage patterns, complaint history and contract data to predict the probability that a customer will cancel within the next 30 days (churn prediction). High-risk customers proactively receive an offer.

Application areas:

• Churn prediction (customer retention)
• Predictive maintenance (forecasting machine failures)
• Demand forecasting and inventory management
• Credit risk assessment
• Weather forecasting

Distinction: Descriptive analytics describes the past. Predictive analytics forecasts the future. Prescriptive analytics goes a step further and recommends concrete actions.

Related terms: Descriptive Analytics, Prescriptive Analytics, Machine Learning

Predictive maintenance uses sensor and operating data to assess the condition of machines and predict failures before they occur. Maintenance happens not on a fixed calendar, but when the data calls for it.

How it works: Sensors continuously record quantities such as vibration, temperature, current or pressure. Machine Learning models detect deviations from normal behaviour (Anomaly Detection) and estimate the remaining useful life – often with methods from Survival Analysis, which correctly handle censored data (machines that have never failed).

Business example: A commercial vehicle manufacturer analyses telemetry data from its fleet. A model estimates each component's failure probability over the coming weeks; service schedules workshop visits so that repairs fall into planned downtime and unplanned breakdowns drop sharply.

Application areas:

• Condition monitoring of production equipment and fleets
• Predicting remaining useful life
• Spare parts and maintenance planning
• Reducing unplanned downtime

Distinction: Predictive Analytics is the umbrella term for data-based forecasting of any kind; predictive maintenance applies it to machine condition. Pure Anomaly Detection flags unusual behaviour but does not yet predict when a failure will occur.

Related terms: Predictive Analytics, Survival Analysis, Anomaly Detection, Time Series Analysis

Prescriptive analytics answers the question: "What should we do?" It goes beyond description and prediction and recommends concrete options for action by simulating and evaluating their effects.

How it works: Based on predictive models and optimisation algorithms, various decision scenarios are played through. The system calculates which measure leads to the best results, taking defined goals and constraints into account.

Business example: An airline uses prescriptive analytics for dynamic pricing: the system calculates in real time the ticket price that best balances capacity utilisation against revenue per flight, taking demand, booking status and competitor prices into account.

Application areas:

• Dynamic pricing
• Supply chain optimisation
• Capacity planning in healthcare
• Portfolio optimisation in finance

Distinction: Predictive analytics forecasts what will happen. Prescriptive analytics says what should be done. The latter builds on predictive models and adds optimisation logic.

Related terms: Predictive Analytics, Descriptive Analytics, Machine Learning

Process mining reconstructs and analyses business processes from the digital traces they leave in IT systems. From the event logs of ERP, MES or ticketing systems emerges an objective picture of how processes actually run – not how the manual says they should.

How it works: Every process step leaves a log entry with a case ID, an activity and a timestamp. Process mining tools stitch these entries into process paths and visualise every variant that actually occurred. On this basis, cycle times can be measured, loops and bottlenecks identified, and deviations from the target process (conformance checking) and their causes analysed statistically.

Business example: An automotive supplier analyses its order flow from receipt to delivery. Process mining reveals that most delays trace back to a manual approval loop between sales and production planning – a step that did not exist in the official process at all.

Application areas:

• Cycle time and bottleneck analysis in production and administration
• Root-cause analysis of quality deviations and delays
• Conformance checking against target processes
• Preparing automation and digitalisation projects

Distinction: Classical process consulting relies on interviews and workshops; process mining measures the as-is process completely from system data. Business Intelligence (BI) aggregates KPIs but does not reveal the process paths behind them.

Related terms: Business Intelligence (BI), Data Pipeline, Statistical Process Control (SPC), Pattern Recognition

A prompt is the input given to an AI language model – a question, instruction or context on which the model generates an output. Prompt engineering is the systematic optimisation of these inputs to maximise the quality of the outputs.

How it works: Language models are highly context-dependent. Small changes in how a prompt is phrased can fundamentally change the output. Prompt engineering uses this effect deliberately: through clear role instructions, examples, format specifications or step-by-step reasoning (chain-of-thought).

Business example: A legal team uses an LLM daily for contract review. With a well-designed prompt ("Analyse this contract as an experienced lawyer. List every clause that could be disadvantageous to us, with a reason and page number.") it gets structured, usable outputs instead of generic summaries.

Key techniques:

• Zero-shot: pose the task directly, without examples
• Few-shot: provide a few examples as a template
• Chain-of-thought: let the model reason step by step
• System prompt: define the model's basic role and behaviour
• Structured output: specify the output format (JSON, table, list)

Distinction: Prompt engineering changes only the input; the model itself stays unchanged. Fine-tuning changes the model through further training. For many use cases, prompt engineering is the faster, cheaper first approach.

Related terms: Large Language Model, Fine-Tuning, RAG

Python is a general-purpose, interpreted programming language and the most widely used language in data science. Its readable syntax and extensive library ecosystem make it the standard tool for data analysis, machine learning and automation.

How it works: Python code runs in an interpreter and delegates compute-heavy work to highly optimised libraries implemented in C++ or C under the hood – NumPy for numerical computing, Pandas for tabular data, PyTorch for deep learning. The language thus combines fast development with high computational performance.

Business example: A pharmaceutical company automates the analysis of its clinical data exports with Python: a script reads the raw data, runs plausibility checks, computes metrics and produces standardised reports – reproducibly, instead of manually in spreadsheets.

Application areas:

• Data analysis and preparation
• Building and training machine learning models
• Data pipelines, automation and APIs
• Scientific computing and prototyping

Related terms: Pandas, PyTorch, Data Science

PyTorch is an open-source framework for Deep Learning, originally developed at Meta and now governed by the PyTorch Foundation under the Linux Foundation. It is the most widely used framework for building and training neural networks in research, and increasingly in industry.

How it works: PyTorch computes with tensors (multi-dimensional arrays) on CPUs and GPUs and builds its computation graph dynamically at runtime – models are written as ordinary Python code, which makes them easy to debug. Automatic differentiation (autograd) supplies the gradients for training; libraries such as torchvision and Hugging Face Transformers build on top of it.

Business example: An agricultural company trains a PyTorch model to detect crop diseases in drone imagery. The team starts from a pre-trained vision model in the PyTorch ecosystem and adapts it to its own annotated field images via Transfer Learning.

Application areas:

• Building and training neural networks for images, text and time series
• Research and prototyping of new model architectures
• Fine-tuning pre-trained models (computer vision, LLMs)
• Deployment via TorchScript, ONNX or serving frameworks

Distinction: TensorFlow is the other major deep learning framework; PyTorch is generally seen as more flexible and research-oriented, TensorFlow as traditionally strong in production serving. Today the choice often comes down to team experience and ecosystem.

Related terms: TensorFlow, Deep Learning, Neural Network

R is a programming language and environment for statistical computing and data visualisation, built by statisticians for statisticians. In biostatistics, agricultural research and pharma, R is frequently the standard for methodologically demanding analyses.

How it works: R ships statistical methods – from regression models and the analysis of designed experiments to survival analysis – as core functionality; the CRAN archive adds over 20,000 contributed packages, and the tidyverse provides modern tooling for data preparation and graphics. Analyses are typically written as scripts or reproducible reports (R Markdown/Quarto) in the RStudio development environment.

Business example: An agricultural company analyses its field trials in R: mixed models matched to the trial designs, standardised plots per site and an automatically generated trial report. The same scripts run again each year on the new season's data.

Application areas:

• Statistical analysis of trials and studies (including regulatory work in pharma)
• Exploratory data analysis and visualisation
• Reproducible reports and interactive applications with R Shiny
• Biostatistics and bioinformatics

Distinction: Compared with Python, R's strength lies in statistics and visualisation; Python dominates deep learning and software engineering. Many teams use both side by side.

Related terms: R Shiny, Python, Statistical Inference

R Shiny is a web framework for the R programming language that turns R code into interactive data applications and dashboards – no HTML, CSS or JavaScript skills required. It is developed by Posit (formerly RStudio), and a sibling framework, Shiny for Python, now exists as well.

How it works: A Shiny app consists of a UI definition (input controls, charts, tables) and a server function holding the computation logic. The framework links the two reactively: when a user changes a filter or parameter, only the outputs that depend on it are recomputed. Apps are deployed via Posit Connect, Shiny Server or containers.

Business example: An agricultural company gives its trial managers a Shiny app in which they filter field-trial data by location, variety and year and retrieve statistical analyses as interactive charts. The statisticians maintain only their R code – no separate web development is needed.

Application areas:

• Interactive analysis dashboards for domain experts
• Exposing statistical models as usable applications
• Prototyping data products before full-scale development
• Companion apps for studies and trial series

Related terms: R (programming language), Business Intelligence (BI), Data Science

A random forest is an ensemble machine learning method that trains many decision trees on random subsets of the data and features, then combines their predictions by averaging or majority vote. It is easy to apply and resistant to overfitting, and it delivers solid results without extensive tuning.

How it works: Each tree is trained on a bootstrap sample of the training data, and at every split only a random subset of features is considered. This double randomness produces trees that differ from one another, and their averaged prediction is more stable than that of any single tree. As a by-product, the method reports how important each feature is to the predictions.

Business example: A forestry service provider classifies tree species from remote sensing and terrain data. A random forest handles the heterogeneous features reliably and also shows which spectral bands drive the Classification most strongly.

Application areas:

• Land use and vegetation classification from remote sensing data
• Failure and quality prediction in manufacturing
• Regression on tabular data with many features
• Feature importance as a first step towards model interpretation

Distinction: Unlike sequential Gradient Boosting, the trees in a random forest are grown independently – this makes it more tolerant of noisy data and hyperparameter choices, though carefully tuned boosting often wins on accuracy.

Related terms: Gradient Boosting, Supervised Learning, Classification

Regression is a family of supervised learning and statistical methods that model the relationship between input variables and a continuous target in order to predict numeric values. Typical targets include quantities, prices, temperatures or durations.

How it works: A regression model learns a function from historical input-output pairs that maps inputs to the target – from classical linear regression through polynomial and regularised variants (ridge, lasso) to non-linear methods such as Random Forest or Gradient Boosting. Model quality is assessed with error metrics such as MAE or RMSE on held-out test data.

Business example: An agricultural company estimates the nitrogen demand of individual field zones from soil samples, satellite indices and weather data. The regression model returns a concrete fertiliser rate per zone for variable-rate application.

Application areas:

• Yield, sales and demand forecasting
• Calibration of sensor and measurement data
• Estimating process parameters in manufacturing
• Price and cost modelling

Distinction: Classification predicts a discrete category, regression a continuous numeric value. Despite its name, Logistic Regression is a classification method.

Related terms: Supervised Learning, Predictive Analytics, Logistic Regression

Reinforcement learning (RL) is a machine learning paradigm in which an agent learns to make decisions that maximise a long-term reward through interaction with an environment.

How it works: The agent observes the state of the environment, chooses an action, receives a reward (positive or negative) and updates its strategy (policy) accordingly. Over many iterations it learns which sequences of actions lead to the best results.

Business example: A robot in a warehouse learns through reinforcement learning how to grip and transport packages efficiently, without explicit programming of every movement. Through millions of simulated attempts it optimises the use of its gripper arm.

Application areas:

• Robot control and autonomous systems
• Autonomous driving
• Game optimisation (AlphaGo, AlphaZero)
• Optimising supply chains and resource planning
• Training LLMs (RLHF)

Distinction: In supervised learning, the model learns from labelled examples. In reinforcement learning it learns through its own actions and feedback, without being shown beforehand what is correct.

Related terms: Machine Learning, Fine-Tuning, Neural Network

Remote sensing is the acquisition of information about the Earth's surface using sensors that are not in physical contact with the target, typically mounted on satellites, aircraft or drones. The sensors record reflected or emitted radiation across several wavelength bands, from which properties of soil, vegetation, water and built structures can be derived.

How it works:

1. Acquisition: optical, multispectral, hyperspectral, thermal or radar sensors capture the scene; LiDAR systems add elevation.
2. Pre-processing: geometric correction, atmospheric correction and calibration to physical reflectance values.
3. Analysis: computing indices such as NDVI (Normalized Difference Vegetation Index) or classifying pixels, increasingly with Deep Learning models.
4. Output: maps, time series or area statistics for downstream analysis.

Business example: An agricultural trading company processes free Sentinel-2 imagery for every contracted field in a region. The seasonal trajectory of vegetation indices produces crop-condition maps that tell field advisors which parcels to inspect first.

Application areas:

• Crop condition and yield monitoring
• Land use and land cover classification
• Forestry, water and environmental monitoring
• Detection of buildings and infrastructure change
• Damage and risk assessment after extreme weather

Related terms: NDVI (Normalized Difference Vegetation Index), Image Segmentation, ArcGIS, Point Cloud

RAG is an architecture that combines large language models with an external knowledge base. Before the model generates an answer, it searches an indexed knowledge base for relevant information and uses it as context.

How it works:

1. The user's query is converted into an embedding.
2. The vector database searches for the semantically most similar documents or passages.
3. These documents are passed to the LLM together with the original query as context.
4. The model generates an answer based on the supplied information.

Business example: A mechanical engineering company has thousands of internal maintenance manuals, service reports and technical specifications. A RAG system lets technicians ask questions in natural language ("What torque is used for bolt M12 on model X?") and receive precise answers from the actual documents, with a source reference.

Advantages:

• Significantly reduces hallucinations
• No retraining needed for new information
• Answers are traceable to verifiable sources
• Cheaper than fine-tuning for knowledge-based applications

Distinction: RAG supplements the model with external knowledge at runtime. Fine-tuning anchors knowledge permanently in the model through training. RAG is more flexible and current; fine-tuning is better suited to adjusting style or behaviour.

Related terms: Large Language Model, Embedding, Hallucination, Vector Database

A self-organizing map (SOM) is an unsupervised neural network that projects high-dimensional data onto a – usually two-dimensional – grid while preserving the neighbourhood structure of the data. Similar data points end up close together on the map, making clusters and patterns visually accessible.

How it works: Each grid node carries a weight vector in the dimension of the input data. During training, the most similar node (best matching unit) is found for each data point, and this node and its grid neighbours are nudged towards the data point. Over many iterations the map organises itself so that the topology of the data is retained in the grid.

Business example: An engineering company maps thousands of operating states from its machine fleet onto a SOM. Regions for normal operation, start-up phases and atypical states emerge on the map, so irregularities can be located at a glance.

Application areas:

• Visualising high-dimensional sensor and process data
• Exploratory data analysis and Clustering
• Anomaly detection via distance to the map
• Customer and product segmentation

Distinction: Unlike classical Dimensionality Reduction methods such as PCA, a SOM produces a discrete, grid-based mapping geared primarily towards visual exploration.

Related terms: Unsupervised Learning, Clustering, Neural Network, Dimensionality Reduction

SHAP (SHapley Additive exPlanations) is an Explainable AI (XAI) method that fairly distributes a machine learning model's prediction across the contributions of its individual input features. It is grounded in Shapley values from cooperative game theory.

How it works: For each individual prediction, SHAP computes how much each feature contributes to the deviation from the average outcome, averaged over all possible feature combinations. The contributions add up exactly to the model's prediction, which makes the explanation internally consistent. For tree-based methods such as Gradient Boosting, TreeSHAP provides an exact and fast computation; the widely used open-source implementation is the Python library shap.

Business example: A pharmaceutical company runs a model predicting batch failures. SHAP analyses show, per batch, which process parameters drove the failure risk – giving process engineering concrete levers instead of a black-box number.

Application areas:

• Explaining individual model decisions to domain teams and auditors
• Global feature importance and model debugging
• Plausibility checks before production deployment
• Documentation requirements in regulated industries

Distinction: LIME approximates the model locally with a simple surrogate; SHAP's game-theoretic foundation yields more consistent but computationally more expensive explanations.

Related terms: Explainable AI (XAI), LIME (Local Interpretable Model-agnostic Explanations), Gradient Boosting

Signal processing refers to methods for processing and analysing data that comes as continuous or discrete signals, e.g. time series, audio data, images or sensor measurements.

How it works: Signals are filtered (noise removed), transformed (e.g. Fourier transform for frequency analysis), segmented and reduced to meaningful features, which are then used for further analysis or ML models.

Business example: A wind farm operator analyses vibration signals from the turbine bearings in real time. Signal processing algorithms detect frequency changes that indicate impending wear, before a human technician notices anything.

Application areas:

• Predictive maintenance through vibration analysis
• Speech recognition (audio processing)
• ECG and EEG analysis in medicine
• Radar and sonar signal processing

Related terms: Pattern Recognition, Time Series Analysis, Industry 4.0

A smart city uses digital technologies and data analysis to make urban infrastructure more efficient, conserve resources and improve residents' quality of life.

How it works: Sensor networks, IoT devices and connected systems continuously capture data on traffic, energy, water, waste and the environment. This data feeds into analysis platforms that enable real-time decisions or automated control interventions.

Application areas:

• Adaptive traffic control to reduce congestion
• Smart street lighting (active only when needed)
• Demand-driven waste collection (sensors report full bins)
• Real-time monitoring of air quality and noise
• Digital citizen services and participation platforms

Business example: A mid-sized municipality links the counting data from its traffic lights with timetable and weather data in order to match the green wave on the main routes to actual load. Urban planning uses the same data to spot bottlenecks and to extend cycle lanes where they take noticeable pressure off car traffic. The analysis runs on a central platform to which further sensors – air quality stations, for instance – are connected step by step.

Related terms: Industry 4.0, Internet of Things (IoT), Time Series Analysis

Spatial optimization is the mathematical optimisation of decisions with a geographic component, such as choosing sites, dividing territories or planning routes. The goal is a solution that minimises or maximises a measure such as cost, travel time or coverage subject to spatial constraints.

How it works: The problem is written as an optimisation model with decision variables, an objective function and constraints. Common families are facility location, set covering, districting and vehicle routing. Small instances are solved exactly with mixed-integer programming; large networks call for heuristics and metaheuristics. Distances and travel times come from road networks or GIS data rather than straight-line geometry.

Business example: A logistics provider needs to know how many transhipment points a region requires and where to put them. The optimisation model picks, from several hundred candidate sites, the combination that reaches every customer within a promised delivery window while minimising the sum of site and transport costs.

Application areas:

• Network and site planning for warehouses, branches and service points
• Sales and field-service territory design
• Route planning and fleet dispatching
• Land-use and cropping plans in agriculture
• Layout of sensor and monitoring networks

Distinction: Location Analysis describes and scores candidate sites from market data; spatial optimization turns those scores into a selection decision under an explicit objective.

Related terms: Prescriptive Analytics, Location Analysis, Reinforcement Learning, Spatio-temporal Statistics

Spatio-temporal statistics is the branch of statistics that analyses data carrying both a spatial and a temporal dimension, such as readings from fixed sensor networks or satellite imagery of the same area across several years. It models dependence in both dimensions jointly rather than treating location and time separately.

How it works: The starting assumption is that nearby locations and consecutive points in time carry similar values, an effect known as spatial and temporal autocorrelation. Methods such as space-time kriging, spatio-temporal Gaussian processes and hierarchical Bayesian models use that structure to estimate values for unobserved places and dates, together with an uncertainty range. Ignoring the autocorrelation makes standard methods understate variance and report confidence intervals that are too narrow.

Business example: An agricultural business records soil moisture through a network of field sensors and supplements it with weekly satellite indices. A spatio-temporal model interpolates a continuous moisture map for each calendar week, including for field sections without a sensor, which then feeds irrigation planning.

Application areas:

• Crop development and yield monitoring across farmland
• Environmental monitoring of air, water and soil
• Spread analysis in epidemiology and plant protection
• Demand and utilisation forecasting in urban areas
• Quality control for distributed sensor networks

Distinction: Time Series Analysis treats each measurement site in isolation; spatio-temporal statistics additionally borrows information from neighbouring sites, which yields usable estimates even where local data is sparse.

Related terms: Time Series Analysis, Statistical Inference, Remote Sensing, Spatial Optimization

SQL (Structured Query Language) is the standard language for querying and managing relational databases. Developed at IBM in the 1970s and standardised by ANSI and ISO in the 1980s, SQL remains the baseline skill of all data work – from reporting to preparing training data.

How it works: SQL is declarative: the query describes the desired result (filters, joins, aggregations), and the database determines the efficient execution path itself. Core operations are SELECT queries with JOINs across tables, plus statements for creating and modifying tables and data. Modern warehouses and query engines run SQL over very large, distributed datasets as well.

Business example: A machinery manufacturer answers questions like "which spare parts sold most per region last quarter?" with a single SQL query over order and article tables – seconds instead of manually combining several exports.

Application areas:

• Querying and reporting on structured data
• Data preparation in the Data Warehouse (for example with dbt)
• Ad-hoc analysis by business users and data scientists
• Data quality checks and data migration

Related terms: Data Warehouse, dbt, Structured Data

StarDist is an open-source deep learning method for segmenting cell nuclei and similarly shaped objects in microscopy images by describing every object as a star-convex polygon. That shape assumption makes it particularly reliable on densely packed, roughly round nuclei.

How it works: A network based on the U-Net architecture predicts two quantities per pixel: the probability of lying inside an object, and the distances to the object boundary along a fixed set of radial directions. Each pixel thus proposes a candidate polygon, and non-maximum suppression selects the final objects. Because every object exists as its own polygon, touching nuclei stay separate, where purely pixel-wise segmentation classically merges them. StarDist covers 2D and 3D data and is integrated into tools such as Fiji/ImageJ, napari and QuPath.

Business example: A contract research laboratory quantifies nuclei in tissue sections for a preclinical study. StarDist resolves individual nuclei even in crowded regions, so nuclear density per area and marker intensity per nucleus can be measured reproducibly across every section in the study.

Application areas:

• Nucleus segmentation in histology and fluorescence microscopy
• Quantitative readouts from tissue sections in preclinical work
• 3D segmentation in volumetric datasets
• Pre-processing for cell tracking and classification
• Creation of training and reference annotations

Related terms: Cellpose, Image Segmentation, U-Net, CNN (Convolutional Neural Network)

Statistical inference refers to methods used to draw conclusions about a larger population from a sample, taking uncertainty and variability into account.

How it works: Based on sample data, estimators (e.g. mean, variance) are calculated, hypotheses are tested (e.g. "Does measure A have an effect?") and confidence intervals are determined, indicating the range in which the true value is expected at the chosen confidence level.

Business example: A pharmaceutical company tests a new drug on 500 patients. Statistical inference enables the statement: "At the 95% level, the drug is significantly more effective than the placebo", based on the sample, not the entire population.

Key concepts:

• Hypothesis testing and p-value
• Confidence intervals
• Bayesian vs. frequentist statistics
• A/B tests in marketing and product development

Related terms: Descriptive Analytics, Data Science, Predictive Analytics

Statistical process control (SPC) monitors manufacturing processes with statistical methods to detect deviations early and keep quality stable. Its core tool is the control chart, which plots measurements against statistically derived control limits.

How it works: Samples are measured from the running process (e.g. dimensions, forces, temperatures). As long as the values scatter randomly within the control limits, the process counts as in control. Systematic patterns – trends, jumps, outliers – point to special causes (such as tool wear or a material batch) and trigger a root-cause analysis. Capability indices such as cp/cpk describe how reliably the process stays within its tolerances.

Business example: An automotive supplier monitors bore diameters in machining. Control charts per production line detect a creeping drift caused by tool wear days before any scrap is produced – the tool is replaced during a planned maintenance window rather than after a quality incident.

Application areas:

• Monitoring critical characteristics in series production
• Process capability studies (cp/cpk) at production ramp-up
• Root-cause analysis of quality deviations
• Foundation for data-driven quality control loops

Distinction: SPC works with a few deliberately chosen characteristics and transparent statistical rules; Anomaly Detection with machine learning can monitor many signals at once and complements SPC, but does not replace its methodology anchored in quality standards.

Related terms: Statistical Inference, Anomaly Detection, Time Series Analysis, Process Mining

Structured data is data organised in a clearly defined schema, typically in tabular form with fixed columns and data types. It is easy to store, query and analyse.

How it works: Structured data can be stored directly in relational databases (SQL). Each row represents a record, each column an attribute with a defined type (text, number, date, etc.).

Business example: A crop protection manufacturer brings production batch records, quality control lab results and delivery notes together in a relational database. Because all three sources use the same schema for batch number and timestamp, a conspicuous lab sample can be traced back by SQL query to the production shift and the raw material used. Analyses that were previously pieced together by hand from separate spreadsheets become a repeatable query.

Examples: customer databases, transaction data, product catalogues, measurements from production facilities.

Distinction: Unstructured data such as emails, images or social media posts follows no fixed schema and requires special processing. Semi-structured data (JSON, XML) has a flexible but recognisable structure.

Related terms: Unstructured Data, Data Warehouse, Data Wrangling

Supervised learning is the most widely used learning paradigm in machine learning. The model learns a mapping from input to output by training on examples where the correct output is known.

How it works: The training dataset contains pairs of input (features) and known output (label). The model iteratively adjusts its parameters to minimise the difference between prediction and actual output. After training it generalises to new, unseen data.

Business example: An email provider trains a spam-filter model on millions of labelled emails ("spam" / "not spam"). The model learns which features (sender, subject, content) are characteristic of spam and reliably filters new emails.

Main tasks:

• Classification: the output is a category
• Regression: the output is a continuous value

Distinction: Supervised learning needs labelled data, which is often expensive and time-consuming to create. Unsupervised learning works without labels but delivers results that are less directly controllable.

Related terms: Classification, Unsupervised Learning, Feature Engineering

A support vector machine (SVM) is a supervised learning method that separates data points with a decision boundary maximising the distance to the closest points of each class. Those closest points – the support vectors – give the method its name.

How it works: The SVM finds the hyperplane that maximises the margin between classes; only the support vectors determine its position. Data that is not linearly separable is mapped implicitly into a higher-dimensional space via the kernel trick, where a linear separation becomes possible – common kernels are RBF and polynomial. Variants exist for Regression (SVR) and outlier detection (one-class SVM).

Business example: A laboratory automation vendor classifies spectral data from samples with only a few hundred labelled measurements. An SVM with an RBF kernel achieves reliable separation on the small, high-dimensional dataset where data-hungrier methods overfit.

Application areas:

• Classification of small, high-dimensional datasets (spectra, gene expression data)
• Text categorisation after TF-IDF vectorisation
• Outlier and novelty detection (one-class SVM)
• Quality classification in laboratory and manufacturing settings

Distinction: Compared with neural networks, SVMs need far less training data and compute, but they scale poorly to very large datasets and do not output class probabilities directly.

Related terms: Supervised Learning, Classification, Anomaly Detection

A surrogate model is a fast, data-driven approximation of an expensive computation or physical test – typically of simulations such as FEM (finite element method) or CFD (computational fluid dynamics). It delivers in milliseconds results for which the original simulation needs hours.

How it works: First, the original simulation is run for a limited number of parameter combinations. A machine learning model is then trained on these results – depending on the data situation a Gaussian Process, Gradient Boosting or a Neural Operator – to learn the relationship between input parameters and simulation outcome. The trained surrogate then replaces the simulation in optimisation loops, sensitivity analyses or interactive design tools.

Business example: An automotive supplier designs a structural component. Instead of starting a multi-hour FEM run for every geometry variant, a surrogate model screens thousands of variants in minutes and proposes the most promising ones for the final simulation – the development loop shrinks from weeks to days.

Application areas:

• Accelerating FEM- and CFD-based product development
• Parameter optimisation and sensitivity analyses
• Real-time capable models for digital twins
• Test planning with fewer physical prototypes and test runs

Distinction: A surrogate model approximates the results of one specific simulation setup, usually as individual key figures; a Neural Operator predicts entire result fields and generalises across geometries and resolutions.

Related terms: Gaussian Process, Neural Operator, CAD (Computer-Aided Design), Machine Learning

Survival analysis is a family of statistical methods for modelling the time until an event occurs – such as machine failure, treatment failure or customer churn. Its defining strength is the correct handling of censored data: observations where the event has not yet occurred by the end of the observation period.

How it works: The central quantities are the survival function (the probability that the event has not occurred by time t) and the hazard function (the instantaneous event risk at time t, given survival so far). Classical methods include the Kaplan-Meier estimator and Cox regression, which quantifies how covariates affect the risk; machine learning variants such as random survival forests exist as well.

Business example: An engineering company models the time to failure of hydraulic pumps from operating hours, load profiles and maintenance history. Many pumps are still running at the analysis date – survival analysis makes use of these censored cases too and delivers dependable remaining-useful-life estimates for maintenance planning.

Application areas:

• Predictive maintenance and remaining useful life (RUL) estimation
• Clinical trials and pharmaceutical research (time to event)
• Customer churn over time
• Reliability analysis of components and systems

Distinction: An ordinary regression on the event time would have to discard censored cases or treat them with bias; survival analysis is built precisely for this data situation.

Related terms: Statistical Inference, Predictive Analytics, Time Series Analysis

Tabular foundation models are pre-trained machine learning models based on the Transformer Architecture that make predictions on tabular data without task-specific training. The best-known representatives belong to the TabPFN model family.

How it works: The model is pre-trained once on a large number of synthetic tabular prediction tasks. For a new task, it receives the training rows and the rows to be predicted together as input and predicts the targets in a single forward pass (in-context learning) – no gradient training, no hyperparameter tuning. This currently works best on small to medium-sized tables.

Business example: An engineering consultancy needs to estimate the failure risk of an assembly from only 400 documented test runs. A tabular foundation model delivers a competitive result on this small dataset within seconds, where training and tuning a conventional model would have taken considerably longer.

Application areas:

• Prediction on small and medium-sized tabular datasets
• Fast baselines and feasibility studies in data science projects
• Use cases with expensive data collection (lab experiments, field trials)
• AutoML pipelines, as a strong default component

Distinction: Unlike Gradient Boosting or Random Forest, no model is trained per task; the knowledge resides in the pre-training. For very large tables, classical methods usually remain the first choice for now.

Related terms: Transformer Architecture, Structured Data, Transfer Learning, Gradient Boosting

TensorFlow is an open-source machine learning library originally developed by Google's Brain team. It enables the development, training and deployment of ML models across various platforms, from mobile devices to cloud clusters.

How it works: TensorFlow represents computations as directed graphs in which nodes are mathematical operations and edges are data tensors. The library automatically optimises these computations for the available hardware (CPU, GPU, TPU).

Ecosystem:

• TensorFlow Core: fundamental ML operations
• Keras: high-level API for fast model prototyping
• TensorFlow Lite (now LiteRT): optimised for mobile and edge devices
• TensorFlow.js: ML in the browser via JavaScript
• TensorFlow Serving: deploying models in production

Business example: An agricultural machinery manufacturer trains an image model that tells weeds from crop plants and runs it directly on the control unit of a field sprayer. Training happens on GPU servers through the Keras API; for use on the device the model is converted to LiteRT and quantised so that it stays within the available memory and time budget. The choice between TensorFlow and PyTorch came down to the deployment chain here, not to the model architecture – both frameworks express the networks required.

Distinction: TensorFlow is one of several leading ML frameworks. PyTorch (Meta) is especially popular in research. Scikit-learn suits classic ML. Keras abstracts TensorFlow for easier use.

Related terms: Deep Learning, Neural Network, Machine Learning

Text mining (text analytics) refers to the automatic extraction of structured information from unstructured text using NLP and data-mining methods.

How it works: Texts are tokenised, analysed linguistically and examined for patterns. Results can be classifications (e.g. sentiment: positive/negative), extracted entities (names, places, dates) or thematic clusters.

Business example: A consumer goods manufacturer analyses thousands of online reviews and social media comments each day. Text mining automatically identifies which product features are rated positively and which negatively, without manual reading.

Application areas:

• Customer feedback analysis
• Market research and competitive monitoring
• Automatic document classification
• Forensics and compliance monitoring
• News monitoring

Distinction: Text mining extracts information from existing text. Generative AI creates new text. NLP is the technical foundation text mining builds on.

Related terms: NLP, Unstructured Data

TF-IDF (Term Frequency – Inverse Document Frequency) is a statistical weighting scheme that scores how characteristic a word is for one document within a larger collection of documents.

How it works: Term frequency counts how often a term appears in a document. Inverse document frequency reduces the weight of terms that occur across almost every document in the corpus, such as function words and boilerplate. The product of the two is highest for terms that are frequent in one document and rare across the collection. Each document becomes a sparse numeric vector that standard machine learning methods can consume directly.

Business example: An engineering firm analyses several tens of thousands of maintenance reports. TF-IDF surfaces the distinctive fault vocabulary in each report while generic words such as "machine" or "checked" carry little weight. Reports can then be grouped by fault pattern, which makes recurring weak points visible across product lines.

Application areas:

• Document search and relevance ranking
• Automatic keyword extraction
• Text classification and topic assignment
• Detection of near-duplicate documents
• Feature generation for downstream models

Distinction: Bag of Words counts word occurrences without weighting them. TF-IDF adds a measure of how much a term distinguishes one document from the rest of the corpus.

Related terms: Bag of Words, Text Mining, Natural Language Processing (NLP), Feature Engineering

Tibco Spotfire – marketed simply as Spotfire since the product moved to Cloud Software Group – is a commercial business intelligence and analytics platform for the interactive visualisation and exploration of large datasets. It is particularly widespread in data-intensive industries such as life science, chemicals and energy.

How it works: Spotfire connects to databases, files and streaming sources and presents the data in linked, interactive visualisations: a selection in one chart filters all the others (brushing and linking). For deeper analysis, R and Python scripts can be embedded, making statistical models directly usable inside dashboards.

Business example: A pharmaceutical company analyses high-throughput screening data in Spotfire: researchers filter millions of measurements interactively by compound class and spot activity patterns visually, while embedded R scripts fit dose-response curves.

Application areas:

• Exploratory analysis of large research and lab datasets
• Self-service dashboards for business departments
• Process and quality monitoring in manufacturing
• Combining visual analysis with embedded statistics (R/Python)

Distinction: Compared with Power BI, Spotfire leans towards exploratory, science-oriented analysis; Power BI is stronger in standard reporting and Microsoft ecosystem integration.

Related terms: Business Intelligence (BI), Power BI, Descriptive Analytics

Tiled Inference is a technique for applying a model to very large images by splitting them into overlapping tiles, processing each tile separately and stitching the partial results back into a single output.

How it works:

1. The image is divided into tiles of a size the model was trained for, with a defined overlap at the edges. Without that overlap, objects sitting on a tile boundary are cut in half and end up counted twice or missed entirely.
2. The model processes the tiles individually or in batches, so GPU memory use stays constant regardless of the size of the source image.
3. The partial results are merged: segmentation masks are blended across the overlap, and bounding boxes are de-duplicated with non-maximum suppression.
4. Coordinates are mapped back into the full image and, for geospatial data, into the reference coordinate system.

Business example: A geodata provider analyses satellite scenes tens of thousands of pixels across. Rather than downscaling the scene and losing small structures, the segmentation runs tile by tile at full resolution and the results are recombined into one georeferenced layer.

Application areas:

• Analysis of satellite and aerial imagery
• Whole-slide images in pathology and microscopy
• High-resolution inspection photography in manufacturing
• Orthomosaics from drone surveys

Related terms: Image Segmentation, Object Detection, Remote Sensing, U-Net

Time series analysis covers statistical and algorithmic methods for examining data that is ordered in time and collected at regular intervals. The goal is to understand patterns, detect anomalies and predict future values.

How it works: Time series data has typical structures: trend (long-term direction), seasonality (recurring patterns), cycles and noise. Common methods such as ARIMA, SARIMA or neural approaches (LSTM, Transformer) model these structures and use them for forecasts.

Business example: A utility company forecasts hourly electricity demand based on historical consumption data, temperature and calendar events. Time series analysis enables precise production planning and prevents costly overcapacity.

Application areas:

• Demand forecasting in retail
• Anomaly detection in production processes
• Financial market and stock analysis
• Forecasting maintenance needs (predictive maintenance)
• Epidemiological modelling

Distinction: Time series analysis is a specialised form of data analysis that explicitly models the temporal dependence between observations, unlike classic ML models, which treat data points as independent.

Related terms: Predictive Analytics, Signal Processing, Machine Learning

A token is the smallest processing unit of a language model. Tokenisation is the process of breaking a text into these units. Tokens are neither necessarily words nor individual letters – they typically lie somewhere in between.

How it works: Modern tokenisers (e.g. byte pair encoding, BPE) break text into frequently occurring character sequences. The word "tokenisation", for example, might be split into "token", "is", "ation". Each token is assigned a number (ID) that the model processes.

Why tokens matter:

• The context length of an LLM is measured in tokens (e.g. 128,000 tokens ≈ 100,000 words)
• API costs are calculated per token
• Tokens influence how a model "sees" text – typos or unusual spellings can disrupt tokenisation

Business example: A company plans to have long contracts (30 pages ≈ 15,000 tokens) analysed by an LLM. Understanding tokens helps with cost planning and choosing the right model.

Distinction: Tokens are the model's "vocabulary", not to be confused with semantic embeddings, which encode the meaning of a text as vectors.

Related terms: Large Language Model, Embedding, NLP

Transfer learning is the method of using an already pre-trained model as the starting point for a new task, instead of training a model from scratch.

How it works: A large model is first trained on a general, extensive dataset (pre-training). The learned representations are then transferred to a more specific task, either through fine-tuning (the model is further adapted) or feature extraction (the representations are used directly).

Business example: A start-up wants to develop a model for medical image recognition. Instead of needing millions of training images, it uses a pre-trained image recognition model (e.g. ResNet) and adapts it with a few thousand X-ray images. Transfer learning reduces the effort considerably.

Advantages:

• Drastically reduced data requirement for new tasks
• Considerably lower training costs
• A faster route to high-quality models

Distinction: Transfer learning is the principle of transferring existing knowledge. Fine-tuning is one concrete method by which transfer learning is implemented.

Related terms: Fine-Tuning, Large Language Model, Deep Learning

The transformer architecture is the technical foundation of most modern AI language models, including GPT, Claude, BERT and many others. It was introduced in 2017 by Google in the landmark paper "Attention Is All You Need" and has displaced earlier architectures such as RNNs and LSTMs in most NLP tasks.

How it works: The core mechanism is the self-attention mechanism: for each part of a text, the model learns which other parts are relevant, regardless of their position in the text. This enables it to capture long-range dependencies (e.g. a pronoun and its associated noun at the start of a long sentence). Transformers can also be trained in a highly parallel way, which is what makes training on huge datasets practical in the first place.

Business example: The transformer is the architecture behind every modern LLM-powered assistant. Without transformers there would be no GPT, no Claude and no Gemini, and so none of the modern AI applications companies use today.

Key variants:

• Encoder-only (e.g. BERT): good for text understanding and classification
• Decoder-only (e.g. GPT, Claude): good for text generation
• Encoder-decoder (e.g. T5, BART): good for translation and summarisation

Distinction: The transformer architecture is the blueprint. A large language model is the concrete implementation, a transformer trained on huge amounts of text.

Related terms: Large Language Model, Deep Learning, NLP

U-Net is a CNN architecture for semantic image segmentation that pairs a contracting encoder path with an expanding decoder path and connects the two through skip connections.

How it works: The encoder progressively downsamples the image and captures context, while the decoder upsamples the feature maps back to the original resolution. Skip connections pass fine spatial detail from the early layers straight to the decoder, which keeps object boundaries sharp. The output is a per-pixel probability map for each class. The architecture originated in biomedical imaging and was published in 2015 by a research group at the University of Freiburg; combined with data augmentation, it trains usefully on a small number of annotated images.

Business example: An agricultural service provider derives field boundaries and tramlines from satellite imagery. A U-Net segments the areas pixel by pixel, and the very large scenes are processed through tiled inference so that fine structures survive at full resolution.

Application areas:

• Area segmentation in satellite and drone imagery
• Cell and tissue segmentation in microscopy
• Crack and defect area detection in materials testing
• Extraction of roads, buildings and water bodies
• Preprocessing for area-based statistics

Related terms: Image Segmentation, CNN (Convolutional Neural Network), Tiled Inference, Remote Sensing

Unstructured data is data without a fixed schema or database format. It makes up the bulk of all data generated daily and often contains valuable information, but requires special processing methods.

Examples: emails, documents, social media posts, images, videos, audio files, raw sensor signals.

How it works: To make unstructured data analysable, it is converted into structured or vectorised formats through NLP (for text), image recognition (for images) or signal processing (for audio/sensor data).

Business example: A large share of the information in a pharmaceutical company sits in unstructured research reports, emails and clinical notes. Only through NLP and text mining does this information become systematically usable.

Distinction: Structured data fits into tables and is directly queryable. Unstructured data needs pre-processing but often offers richer information.

Related terms: Structured Data, Text Mining, NLP

Unsupervised learning refers to ML methods in which the model recognises patterns and structures in data without those being labelled. The model discovers the structure of the data on its own.

How it works: Without predefined answers, the algorithm searches for inherent structures – similarities, groupings, redundancies or dependencies in the data.

Key methods:

• Clustering: grouping similar data points
• Dimensionality reduction (PCA, t-SNE, UMAP): making high-dimensional data visualisable
• Autoencoders: learning compressed representations
• Association analysis: finding frequent item combinations (e.g. "those who buy X often also buy Y")

Business example: A retailer analyses purchasing data without predefined customer segments. Unsupervised learning independently discovers five distinct customer groups, which are then targeted with marketing.

Distinction: In supervised learning there is a known answer for every training data point. In unsupervised learning there are no such answers – the model derives the structure itself.

Related terms: Clustering, Supervised Learning, Reinforcement Learning

A vector database is a specialised database system designed to efficiently store high-dimensional vectors (e.g. embeddings) and search them by similarity.

How it works: Instead of searching for exact matches (like SQL), a vector database performs a similarity search (approximate nearest neighbour search). It finds the vectors most similar to a query vector, in milliseconds, even with millions of entries.

Business example: A media company stores all articles as embeddings in a vector database. When a reader reads an article, the recommendation system immediately finds the ten most similar articles by content, without keyword matching.

Well-known solutions: Pinecone, Weaviate, Qdrant, Milvus, pgvector (PostgreSQL extension).

Application areas:

• Semantic search
• RAG systems
• Recommendation systems
• Duplicate detection in large datasets

Distinction: Classic databases search for exact values. Vector databases search for semantic similarity. They are not a replacement for relational databases but a complement for AI applications.

Related terms: Embedding, RAG, Large Language Model

Vision foundation models are large image models pre-trained on vast image collections (e.g. the DINO model family) that have learned general visual features and can be adapted to new tasks with few labelled examples.

How it works: The model is pre-trained self-supervised on millions of unlabelled images – it learns which visual features matter without any labels. For a concrete task (defect detection, segmentation), the pre-trained model serves as a feature extractor or is adapted through Fine-Tuning with a few hundred labelled examples. The labelling effort drops by orders of magnitude compared with a model trained from scratch.

Business example: A supplier introduces visual quality inspection for a new component. Instead of collecting tens of thousands of defect images, the team adapts a vision foundation model with a few hundred example images – the inspection reaches production maturity in weeks instead of months and reliably detects even rare defect patterns.

Application areas:

• Visual quality inspection with few training examples
• Image Segmentation and Object Detection in industry and remote sensing
• Anomaly and defect detection on image data
• Image search and similarity comparison in large image archives

Distinction: A classical CNN is trained per task from scratch or via Transfer Learning and needs considerably more labelled data. A Vision Language Model (VLM) additionally combines visual understanding with language.

Related terms: Vision Language Model (VLM), Fine-Tuning, Transfer Learning, Image Recognition

A vision language model (VLM) is an AI model that processes images and text together: it can describe image content, answer questions about images and extract information from visually structured documents such as drawings, forms or tables.

How it works: An image encoder converts the image into Embeddings, which a Large Language Model (LLM) processes together with the text prompt. The model thus understands layout, text and image content in context and answers in natural language – for instance with a structured extraction of all line items from a scanned delivery note.

Business example: A machinery manufacturer digitises purchase orders, delivery notes and technical data sheets that arrive as PDFs or scans. A VLM extracts line items, dimensions and tolerances directly from the document image – including tables and embedded drawings where classical text recognition fails.

Application areas:

• Automated document extraction (orders, delivery notes, data sheets)
• Question answering on images and technical drawings
• Describing and tagging large image collections
• Visual inspection reports with reasoning in natural language

Distinction: A Multimodal Model is the umbrella term for models with several input types; VLM denotes the combination of image and language. A Vision Foundation Model delivers visual features without language understanding.

Related terms: Multimodal Model, Large Language Model (LLM), Vision Foundation Models, Natural Language Processing (NLP)

Web Crawling is the automated, systematic retrieval of web pages by a program that starts from a set of seed addresses, follows the links it finds and stores the retrieved content for later analysis.

How it works: A crawler maintains a queue of pending addresses, fetches pages, extracts new links from the HTML and returns them to the queue while filtering out addresses it has already visited. Considerate behaviour is standard practice: honouring robots.txt, limiting the request rate and identifying the crawler through its user agent. Pages whose content is assembled by JavaScript require a headless browser. Copyright, the source's terms of use and data protection law should be clarified before a crawl is set up.

Business example: An engineering company tracks competitor technical data sheets. A weekly crawl covers the relevant product pages, detects which documents have changed and hands the differences to product management in structured form instead of as a folder of downloads.

Application areas:

• Building text corpora for language models and Text Mining
• Market, assortment and price monitoring
• Watching regulatory publications for changes
• Enriching internal master data from public sources
• Indexing for company-internal search services

Distinction: Web crawling covers the collection of content, while Web Mining covers the analysis that follows. Scraping is the narrower activity of pulling specific fields out of a page whose structure is already known.

Related terms: Web Mining, Text Mining, Data Extraction, Unstructured Data

Web mining is the application of data-mining methods to data from the World Wide Web in order to extract patterns, structures and insights.

How it works: Web mining is divided into three areas: web content mining (analysing the content of web pages), web structure mining (evaluating link structures) and web usage mining (analysing user behaviour on web pages).

Business example: An e-commerce company uses web usage mining to analyse the paths users take through the online shop before they buy or drop off. The insights feed directly into UX optimisations.

Application areas:

• Search engine optimisation (SEO)
• Competitive and price monitoring
• Trend analysis from social media
• User behaviour and conversion optimisation

Related terms: Text Mining

XGBoost (Extreme Gradient Boosting) is an open-source software library that implements Gradient Boosting with a focus on speed, memory efficiency and built-in regularisation. For years it has been among the most widely used tools for machine learning on tabular data.

How it works: XGBoost trains decision trees sequentially on the errors of the current ensemble while optimising a regularised objective that penalises overly complex trees and curbs overfitting. On the engineering side, the library offers parallelised training, native handling of missing values and GPU support, with interfaces for Python and R, among others.

Business example: A pharmaceutical company prioritises drug candidates based on tabular molecular and assay data. An XGBoost model ranks thousands of candidates by likelihood of success, reducing the number of expensive laboratory experiments.

Application areas:

• Forecasting models on structured data (yield, failure, demand)
• Scoring and ranking systems
• Machine learning competitions and rapid prototyping
• Baseline models in data science projects

Distinction: XGBoost is one specific implementation; LightGBM and CatBoost apply the same core idea with different emphases (speed on very large datasets and handling of categorical features, respectively).

Related terms: Gradient Boosting, Random Forest, Machine Learning

Last updated: 2026 | Supper & Supper GmbH, Berlin