Anatomy of an AI collections call

A production AI collections call is a pipeline under a strict clock: the reply must begin within roughly 700 milliseconds of the borrower finishing, or the call feels like a machine and the borrower hangs up. Everything below happens inside that budget, in a loop, for every turn.

The loop, step by step

• Listen: speech recognition streams the borrower's words, in their language, as they speak.
• Understand: a language model reads the transcript so far, the account state and the strategy, and decides what to say - or which tool to use.
• Act: tools do real work mid-call - record a promise with a date, send a payment link, schedule a callback, mark do-not-call, hand off to a human.
• Speak: the reply is synthesised in a natural voice in the borrower's language, with the numbers - amounts, dates - spoken the way people actually say them.
• Listen again: if the borrower interrupts, the agent stops and yields. Barge-in is what separates a conversation from an announcement.

What makes it collections-grade rather than a chatbot with a phone number

• The compliance layer sits inside the loop: quiet hours, script boundaries and disclosure rules constrain what the agent may say before it says it.
• Every turn is recorded, transcribed and attributable afterwards.
• The call ends in a state the flow understands - a disposition - so the next contact is decided by strategy, not by chance.

Where the 700 milliseconds actually goes

The budget is tight because it is shared. Speech recognition needs a moment after the borrower stops to be confident they have stopped rather than paused for breath. The language model needs time to read the conversation so far and decide. Speech synthesis needs time to produce the first audible syllable, though not the whole sentence, because audio can stream. Network transit takes its cut twice. Nothing in that chain can be slow, and the usual failure is not one slow component but four merely acceptable ones.

Language is not a setting, it is the conversation

A borrower who starts in Hindi may finish in English, and many will mix the two inside a single sentence. Treating language as a flag chosen at dial time produces an agent that mishears its own customers. It has to be detected, it has to be allowed to change mid-call, and the numbers have to follow: an amount is spoken differently in Hindi and in Tamil, and a rupee figure read out as bare digits is the fastest way to sound like a machine.

What happens when it goes wrong

It will go wrong, and the design question is what the borrower experiences when it does. Speech recognition returning nothing usable twice in a row, a tool call failing, a borrower who is angry, confused, or says something the script has no branch for: each needs a defined path, and the good path is almost always a human. An agent that cannot hand off gracefully is worse than no agent, because it turns a recoverable conversation into a complaint.

The handoff

A warm transfer is the difference between help and a runaround. The person who picks up should arrive holding the transcript, the account state and the reason for the transfer, so the borrower is not asked to repeat what they have just finished saying. When nobody is free, the honest move is to say so, take a callback time, and keep it.

What to measure

• Time to first response at the ninety-fifth percentile rather than the average. Averages hide exactly the calls that failed.
• Interruption handling: how often the borrower had to talk over the agent to be heard.
• Containment and escalation read together. Containment alone rewards an agent that refuses to hand off.
• Disposition accuracy, sampled against the recording. Everything downstream assumes this number is honest.
• Complaint rate per thousand calls, tracked separately from human calls so that a rise is attributable.

The result is not a human impersonation. It is a competent, endlessly patient caller that always follows the script boundaries, never has a bad day, and writes perfect notes.