Zendesk
Freshdesk
Gorgias
Front
Shopify
Stripe
Slack
Notion
Google Workspace
Confluence
Transformer Model
Definition
A transformer model is a neural network architecture that processes an entire sequence of text at once using an attention mechanism to weigh how much each word relates to every other word, and it's the foundation of nearly every modern large language model.
How it works
Transformers were introduced in the 2017 paper "Attention Is All You Need." Their key innovation is self-attention: for each token, the model computes how relevant every other token in the input is, letting it capture long-range context (like which noun a pronoun refers to) without processing words strictly one at a time.
Text is first broken into tokens and turned into numeric vectors (embeddings). Stacked attention layers then transform those vectors, and the model predicts the next token. Because attention runs in parallel, transformers train efficiently on huge datasets — which is what made today's large language models possible.
Why it matters for support
Almost every AI support agent you'll encounter is built on a transformer under the hood. Understanding the architecture explains both its strengths and its limits.
- It reads a whole conversation in context, not word by word
- Its fixed context window caps how much history it can consider at once
- It predicts likely text, which is why grounding with your own data (RAG) matters for accuracy
Frequently asked
What is the transformer in a large language model?
The transformer is the underlying neural-network architecture. A large language model is a very large transformer trained on huge text corpora to predict and generate language.
What does the attention mechanism do?
Attention lets the model weigh how relevant each word in the input is to every other word, so it can understand context and relationships across a long passage rather than reading in isolation.
Related terms
Large Language Model (LLM)
A large language model (LLM) is a neural network trained on vast amounts of text to predict and generate language, enabling it to understand questions, summarize, classify, and write human-like responses..
Embeddings
Embeddings are numeric vector representations of text (or images, code, etc.) that capture meaning, so that pieces of content with similar meaning sit close together in a high-dimensional space..
Context Window
The context window is the maximum amount of text — measured in tokens — that a language model can consider at once, including both the input you give it and the output it generates..
Tokens (LLM)
Tokens are the small chunks of text — words, parts of words, or characters — that a language model reads and generates; they're the fundamental unit models use to measure input, output, context limits, and usually billing..
Foundation Model
A foundation model is a large AI model trained on broad, general-purpose data at scale that can be adapted to many downstream tasks — through prompting, retrieval, or fine-tuning — rather than being built for one narrow job..
Put these ideas to work
Macha is an AI agent layer that sits on top of the help desk you already run — Zendesk, Freshdesk, Front, Intercom, or Gorgias.
Start Trial