Voice AI in a contact centre is a pair of systems reading the same audio. One transcribes what was said. The other measures how it was said, turning pitch, pace and loudness into an estimate of the caller’s state. That second layer is speech emotion recognition, and it produces a probability, never a fact.
Quick Overview
A contact centre without this layer is structurally reactive. It learns that a call went badly when the customer says so, when the complaint arrives, or when a manager listens back to a sample of recordings days later. Every one of those routes shares the same defect: the information arrives after the moment in which it could still have been used.
Emotion analysis moves that information forward in time. Frustration has audible precursors — the sentence gets shorter, the pitch climbs, the pauses shrink — and those precursors appear before the customer names the problem. A system that reads them gives the agent something to act on while the call is still recoverable. That is the whole shift, and it is narrower than the marketing around it suggests: the technology does not make the organisation empathetic, it makes one specific piece of information available earlier.
This is a different question from tailoring what an organisation offers to whom. Personalisation at scale decides the content of an interaction from what is already known about a customer; emotion analysis decides the handling of an interaction from what is happening inside it right now. One optimises the offer, the other optimises the next sixty seconds. They also sit on top of different foundations, and the customer-relationship strategy underneath both of them decides whether either is worth building — a point developed in CRM is not just technology.
What the System Actually Measures
The transcription layer is the mature half. Speech-to-text converts the audio stream to text, and natural language processing then works on that text: intent classification, keyword detection for terms that carry commercial or legal weight, and verification that mandatory disclosures were actually read out. Commodity cloud services do this competently in Polish and in English, and the vendor documentation is explicit about what the service returns — the Azure engine, for instance, is described in What is Azure Speech? as a set of transcription, translation and synthesis capabilities, not as a system that understands a speaker.
The second layer works on the parts of speech that survive translation. Pitch and its variation over a sentence. Tempo, including the length of pauses. Intensity, meaning loudness relative to that speaker’s own baseline. Voice quality, the roughness or breathiness that changes under stress. None of these are words, which is why the layer behaves differently from the first one: it degrades with line quality, with background noise, and with any speaker whose baseline the model has not seen.
Language matters more here than the vendor material admits, and it matters asymmetrically. Transcription quality in Polish is now close enough to English that the gap rarely changes a deployment decision. Emotion scoring is not in that position: the acoustic markers of irritation are partly conventional, they differ between speech communities, and models trained predominantly on English-language corpora carry those conventions with them. An organisation running Polish-language traffic through an English-trained emotion model is not getting a slightly worse version of the same thing. It is getting a measurement whose calibration nobody has checked.
The two layers answer different questions and fail differently. A transcription error produces a wrong word, which a human reader can usually spot. An emotion-scoring error produces a plausible number attached to a real conversation, which nobody spots unless the system is built to be doubted.
What It Can Infer, and What It Only Estimates
The reliable end of the range is sentiment: whether the arc of a conversation is trending positive, negative or flat. Coarser categories — irritation, satisfaction, confusion — are harder but usable, provided they are read as trends across a call rather than as labels on a sentence. The hardest cases are the ones the sales material likes most: sarcasm, resignation, a decision to leave that has already been made and is not being announced.
Everything in that range is probabilistic. The system observes acoustic patterns and estimates a state; it has no access to the state itself. This is not a defect that better models will remove, because the underlying inference — from how a voice sounds to what a person feels — is uncertain by construction. Risk frameworks for machine learning treat this explicitly: the Artificial Intelligence Risk Management Framework (AI RMF 1.0) organises its guidance around measuring and managing the uncertainty of a system’s outputs rather than around eliminating it.
The practical consequence is a design rule. A signal that is right most of the time is useful as a prompt to a human and dangerous as an input to a decision. Prompt the agent, and a wrong signal costs a moment of misplaced sympathy. Feed the same signal into routing, prioritisation or a scorecard, and a wrong signal becomes a decision nobody can trace back — the same distinction that separates a helpful personalisation engine from an intrusive one.
Live Assistance and Post-Call Review Are Different Products
Organisations tend to buy these as one thing. They are not. Post-call analysis runs over recordings, has no latency budget, and can afford heavy models and human review of edge cases. Its value is coverage: instead of a manager sampling a handful of conversations a week, every conversation gets scored, and the ones worth a human ear surface by themselves. That is a quality-assurance change, and it is the cheaper half.
Live assistance is an engineering problem of a different class. The signal has to reach the agent’s screen while the sentence that produced it is still in the air, which puts transcription, scoring, aggregation and display inside a budget measured in hundreds of milliseconds. The scoring model has to be small enough to run in that window. The events it emits — one score per utterance, per call, across every seat — arrive as a continuous stream that has to be aggregated and queried while it is still being written, which is the same architectural problem that streaming analytics platforms exist to solve; teams that have to build this layer themselves generally need the ingestion-and-query patterns taught in Apache Druid: real-time data analytics before they need a better emotion model.
Buying the live half first is the common mistake. Post-call analysis tells an organisation what its conversations actually contain, and that knowledge is what makes the live prompts worth writing.
Where the Regulation Draws the Line
European law treats emotion inference as two different things depending on whose emotions are being inferred, and the difference is not a matter of degree. Under the Artificial Intelligence Act, the use of AI systems to infer emotions of a natural person in the areas of workplace and education institutions is a prohibited practice under Article 5, outside narrow medical and safety exceptions. An agent sitting in a contact centre is at work. A system that scores that agent’s emotional state is therefore not a compliance project with extra paperwork; it is a practice the regulation forbids.
Scoring the caller is a different case. Emotion recognition appears in Annex III of the Artificial Intelligence Act among the high-risk uses, which brings the full high-risk regime — conformity assessment, data governance, technical documentation, human oversight. Article 50 adds a transparency duty owed to the people exposed to the system: they are to be informed that an emotion recognition system is operating.
Underneath both sits the General Data Protection Regulation, because a voice recording processed to infer characteristics of the speaker is personal data with a biometric dimension, requiring a lawful basis, a retention limit and a completed impact assessment before the first call is scored. Read together, these instruments produce one design constraint that is easy to state and expensive to retrofit: build the system to help the person on the headset, never to measure them.
Where This Goes Wrong
The first failure is trusting the number. An agent who is told the caller is angry, and who therefore starts apologising to a caller who is merely tired, has been made worse at the job by the tool. Prompts have to be written so that a false positive costs nothing, which usually means phrasing them as questions rather than instructions.
The second failure is the one the regulation already forbids, arriving through the side door. Nobody proposes scoring agents on empathy; someone proposes an aggregate view of “call quality” that happens to be computed from emotional signal, and the aggregate ends up in a performance conversation. Agents work out what is being measured, adjust their voices to satisfy it, and the data degrades to the point where it stops describing anything.
The third failure is quieter. Emotion analysis surfaces where customers get frustrated, and those places are frequently not in the contact centre at all — a confusing invoice, a process with a dead end, a product that fails in a predictable way. An organisation that treats the output as a coaching input for agents will keep coaching people for a defect they did not create. The signal is a diagnostic for the whole operation, and reading it as an agent metric wastes it. Teams building conversational systems hit the same boundary from the other side, which is why the design work in building chatbots for developers spends so much of its time on what to escalate rather than on what to answer.
Sequencing an Implementation
Start with transcription and post-call scoring, and spend the first months reading what comes out. The categories that matter will not be the ones chosen in advance, and the analysis team has to learn the vocabulary of its own conversations before it can automate anything about them.
Add live assistance next, beginning with knowledge lookups rather than emotion prompts. Agents accept a system that hands them the right answer faster; that acceptance is what makes them willing to look at the screen when a harder prompt appears later.
Introduce the emotion layer only after both of those are working, with the notification obligations satisfied and the agent-scoring question closed in writing before deployment rather than after. And connect the output to the systems where customer history already lives, because a frustration score with no context is a number without a subject — the integration discipline covered in customer relationship management is what turns the signal into something an organisation can act on twice.
What the Agent Has to Learn
The technology arrives as a screen change, and it is almost always introduced as one. That framing is where most of the value leaks out, because a prompt is not information until somebody knows what to do with it, and knowing what to do with it is a skill the job did not previously require.
Agents need three things that no vendor supplies. They need a working model of what the prompt means — that it reports an acoustic pattern, not a diagnosis, and that the correct response to an uncertain signal is to check rather than to assume. They need permission to disagree with it, stated by a manager and visible in how the team talks about the system, because an agent who believes the machine outranks their own judgement will follow it into obvious mistakes. And they need practice under conditions where being wrong is free, which means simulated calls rather than live ones.
Managers need a narrower and harder thing. The output of an emotion layer looks exactly like a performance dataset, and the discipline of refusing to use it as one has to survive the first quarter in which someone asks why it is not being used that way. The legal answer is available and blunt; the cultural answer has to be built. A supervisor who reads the aggregate as evidence about the operation — this queue frustrates people, this script fails at this point, this product generates the same call every week — is using it correctly. A supervisor who reads it as evidence about individuals has converted a diagnostic instrument into a surveillance one, and the team will notice within days.
Frequently Asked Questions
Can this system tell what a customer is feeling?
No. It estimates a state from acoustic patterns and reports a probability. Treating that probability as knowledge of a person’s emotions is the single most common implementation error, and it is the reason prompts should be written to be harmless when they are wrong.
Is it legal to analyse the emotions of our own agents?
Not in the European Union. The Artificial Intelligence Act lists emotion inference in the workplace among prohibited practices, with narrow medical and safety exceptions that a contact centre does not meet. This applies regardless of consent or of how the output is labelled internally.
Do we have to tell customers that emotion analysis is running?
Yes. The transparency duty in the same regulation requires informing people exposed to an emotion recognition system, and data-protection law separately requires a lawful basis and a retention policy for the voice data itself.
Where should the first deployment start?
With recordings rather than live calls. Post-call scoring reveals what the conversations actually contain, and that vocabulary is what makes live prompts worth writing. Organisations that reverse the order end up prompting agents about categories chosen in a workshop rather than observed in their own traffic.
Why does this article not quote accuracy figures for emotion detection?
Because published accuracy numbers depend on the dataset, the language, the recording conditions and the emotion set, and none of those match a specific contact centre. A figure that cannot be reproduced in your environment is not information about your environment.