Real-time voice agents that turn conversations into pipeline

Cosellr

Cosellr needed voice agents that feel human in conversation while still behaving like reliable software—fast responses, safe tool usage, and consistent outcomes. We implemented a Pipecat-based real-time pipeline on top of Daily.co for voice transport, enabling low-latency turn-taking and multi-step workflows (qualification, scheduling, follow-ups, and CRM updates). The system includes guardrails for compliance and tone, plus tracing and evaluation foundations so conversational quality improves over time instead of drifting.

Real-Time Voice Agents

Agent Platforms & Orchestration

Conversation Intelligence

Tooling Integration (CRM + scheduling + enrichment)

Reliability Engineering (latency, retries, fallbacks)

Security & PII Controls

Observability & Evaluation Frameworks

Production Readiness & Runbooks

The Problem

Traditional sales automation stops at email and chat. Cosellr wanted voice agents that can hold real conversations, execute tool-backed actions (like scheduling and CRM updates), and maintain consistent behavior across campaigns. Early prototypes often struggled with latency, awkward turn-taking, inconsistent reasoning, and missing operational visibility—making it hard to scale responsibly.

Constraints

Low-latency requirements for natural turn-taking (avoid long silences and interruptions).
Tool correctness and safety (no hallucinated facts, controlled side effects).
PII handling constraints and guardrails suitable for real customer/prospect conversations.
Campaign/persona scalability (multiple scripts, tones, and workflows without bespoke engineering each time).
Operational observability: trace what happened on a call, why it happened, and what tools were invoked.

Goals

Build real-time voice agents using Pipecat + Daily.co with production-grade reliability.
Enable end-to-end workflows: qualify → handle objections → schedule → follow up → update CRM.
Add guardrails for policy, tone, and PII handling with auditable behavior.
Implement observability and evaluation foundations to drive iterative improvement.
Create a scalable architecture to support multiple personas and campaigns safely.

The Solution

We built the voice agent system as a real-time pipeline: Daily.co for audio transport, Pipecat for orchestration and streaming control, and a tool layer for business actions (CRM, calendar, enrichment, follow-up automation). We focused heavily on latency, turn-taking ergonomics, safe tool usage, and operational visibility—so the system could scale beyond a prototype into a dependable product capability.

Implemented Daily.co as the real-time audio transport layer.
Built Pipecat pipelines for streaming ASR/TTS and conversational turn management.
Established targets for latency and conversational pacing (silence handling, interruption policy, barge-in behavior).

Designed structured conversation flows for qualification, discovery, objections, and next-step commitment.
Integrated scheduling flows (calendar availability checks, booking, confirmations).
Integrated CRM writebacks (notes, outcomes, structured fields, follow-up tasks).

Implemented policy constraints for what the agent can claim, promise, or do.
Added tone controls per persona/campaign (confidence, friendliness, formality).
Introduced PII handling rules: redaction where needed, restricted logging, and safe storage conventions.

Captured call event timelines (audio start/stop, interruptions, tool calls, failures, retries).
Logged tool traces and structured outcomes (qualified/not qualified, next step, scheduled meeting, objections).
Built evaluation datasets and regression checks for conversational quality over time.

Standardized deployment patterns for multiple personas/campaigns.
Introduced concurrency controls and cost controls (tool budgets, retry budgets).
Created runbooks for incident response and behavior drift.

The Timeline

12 Weeks; Foundation-first build with rapid iteration loops and staged rollout across personas/campaigns.

Weeks 1–2

Voice pipeline MVP (Daily + Pipecat)

Audio transport established via Daily.co. Pipecat pipeline integrated for streaming ASR/TTS and basic conversational turns. Initial latency targets and interruption rules defined.

Weeks 3–5

Tool-driven workflows: qualify → schedule → CRM writeback

Qualification and objection flows implemented. Scheduling flow integrated and tested end-to-end. CRM updates implemented (notes, structured fields, follow-up tasks).

Weeks 6–8

Guardrails, PII controls, and handoff paths

Policy constraints and tone controls per persona/campaign. PII handling conventions and logging restrictions added. Escalation/handoff workflows implemented for edge cases.

Weeks 9–10

Observability + evaluation foundations

Call timelines, tool traces, and structured outcomes instrumented. Baseline datasets created for regression testing and quality scoring.

Weeks 11–12

Production hardening + rollout expansion

Concurrency controls and cost controls added. Runbooks, dashboards, and alerting implemented. Scaled to multiple personas/campaigns with consistent release patterns.

The Outcome

Cosellr’s voice agents evolved from “talking demos” to production-grade systems capable of completing real workflows. The platform supports natural real-time conversation, tool-backed actions, and measurable outcomes—while remaining governable and observable. This created a scalable foundation for voice-driven outbound and customer engagement.

Median voice response latency (end-to-end)

steady-state target • Estimate for natural turn-taking; varies by ASR/TTS and tool usage. Replace with telemetry (median/P95).

0.9–1.8s

P95 voice response latency (non-tool turns)

steady-state target • Estimate excluding long tool calls; replace with real P95 once instrumented.

2.5–4.0s

Call-to-meeting conversion lift

within 6–10 weeks • Estimated improvement vs baseline outbound processes when using consistent qualification + scheduling flows.

15–35%

Agent-driven scheduling success rate

per scheduling attempt • Estimate for calls where scheduling is attempted and prospect is willing; depends heavily on ICP and script.

60–80%

Reduction in “dead-end calls” (no next step captured)

after flow + guardrail tuning • Estimated reduction from structured next-step capture and fallback behaviors (handoff, recap, scheduling).

30–55%

Decrease in hallucinated/unsupported claims

after policy + tool grounding • Estimate driven by policy constraints, tool-based grounding, and strict refusal behaviors.

70–90%

Join the Team

Looking for your next gig?

Get Unlimited PTO, 401(k), Health & Dental Insurance, and many more perks when you join our team.

View Open Positions

Have questions? Our community is here to help. Learn more about our community

Employees

and Counting

We Are Proud to be an
International Employer