DUOLABS
Work

Real-time video and voice for an AI bank teller

For a service in which an on-screen AI bank teller in branches understands what customers say and explains products such as deposits, savings, and loans by voice and video, we built the app-side segment that sends the customer's voice to the server and receives and plays the AI-generated video response of a teller. If the video starts late after the customer finishes speaking, the exchange stops feeling like a conversation and becomes waiting, so the focus was on cutting the delay between sending voice and receiving video. We designed each stage separately: capturing and compressing microphone input, streaming it to the server over a live connection, and playing the returned video right away. Speech recognition and video synthesis run on the server; our scope was the app.

Launch
2022
Client
A major Korean bank
Type
Banking & Identity
01 / 04

Voice input gathered without gaps while the customer speaks

The AI bank teller app captures what the customer says into the microphone in real time and turns it into raw audio data suited for sending to the server. We tuned the input buffer and the timing of sends so the audio does not break up or lag while the customer is speaking. Speech recognition happens on the server, so any sound the app misses, the AI never hears.

Voice input gathered without gaps while the customer speaks
02 / 04

Streaming that shrinks the voice and sends it over a live connection

The AI bank teller app compresses the raw audio with a speech-specific codec before sending it to the server. We organized the compression step and the flow of data packets to cut the amount sent over the network while keeping the quality speech recognition needs. The compressed voice streams continuously over a live connection held open with the server. Exceptions such as connection status, failed sends, drops, and reconnecting were built into the flow.

Streaming that shrinks the voice and sends it over a live connection
03 / 04

Response playback that picks up the server-made teller video at once

The AI bank teller's answer comes back as a synthesized video made on the server. The app receives this video through a low-latency real-time video protocol and plays it. The teller in the video is a virtual person modeled on a real one, speaking in response to the customer's question through voice and video synthesis. The app handles video reception, playback state, and what is shown on screen so the response follows naturally once the customer stops speaking. The focus was on reducing needless waits and state mismatches between the side that sends voice and the side that receives video.

Response playback that picks up the server-made teller video at once
04 / 04

A seven-stage pipeline from microphone to video

Voice and video handling for the AI bank teller runs through seven stages: microphone input, raw audio conversion, speech compression, streaming over a live connection, server processing, video reception, and on-screen playback. Six of them, all except server processing, are the app's work, and we kept them separate so that when a problem occurs, the stage it came from can be traced on its own. In a service where response delay turns directly into frustration, cutting needless waits between stages was the design standard.

A seven-stage pipeline from microphone to video

Highlights

01

Live streaming that compresses microphone audio and sends it to the server without gaps

02

Low-latency receipt and immediate playback of the teller video the server generates

03

A pipeline split into stages from microphone to video, so problems can be traced

Insights

Read the blog