Home/Blog/Engineering & Architecture
Engineering & Architecture

On-Device vs Cloud Inference: How CoThinker Achieves Sub-15ms Typing Responsiveness

CoThinker Core Architecture
CoThinker Core Architecture
Editorial
Sep 30, 2026•6 min read
On-Device vs Cloud Inference: How CoThinker Achieves Sub-15ms Typing Responsiveness
Key Takeaway & Executive Summary

By executing acoustic feature extraction on the local NPU and streaming tokens through ultra-low-latency persistent WebSocket channels, CoThinker achieves a glass-to-text latency of 14.8ms.

Why 15ms is the Holy Grail of Human Interaction

In interactive computing, the human perceptual threshold for instantaneous feedback is 100 milliseconds. Any delay beyond 100ms is perceived as lag; delays above 250ms feel frustrating and disorienting.

For an intelligent keyboard operating across WhatsApp, Slack, and IDEs, latency is everything. If a user dictates or triggers a tone rewrite, the output must appear at the cursor immediately.

Hybrid On-Device & Edge Routing Architecture

CoThinker utilizes a two-tier hybrid routing pipeline:

  1. Tier 1 (On-Device ASR): Uses Apple Silicon Neural Engine (ANE) on macOS and Qualcomm/Tensor NPUs on Android for 16kHz audio chunking and phoneme alignment.
  2. Tier 2 (Edge Cloud LLM): For complex tone transformations, requests are routed via persistent HTTP/2 and gRPC streams with sub-30ms round-trip times.

Experience lightning-fast thinking with CoThinker.

Turn thoughts into instructions AI can use.

Experience zero-stutter voice dictation and native thought-to-text intelligence on Android and macOS.

Get for AndroidDownload for Mac
Private Beta · Free during preview