Relaydesk
A support chat widget that answers from a help center, links the article it quoted, and opens a ticket in a staff inbox when it can't help.
- result
- Hybrid retrieval improved first-result hits from 30/36 to 34/36, with the right result in the top three on all 36 evaluation cases. A separate 23-question evaluation recorded zero unnecessary tickets.
- my contribution
- Built the retrieval and conversation loop, human handoff, multi-tenant help centers, and evaluation harness.
- 2026–Present
- ai
how it works
Relaydesk is a support widget and staff inbox, demoed on Nimbus, a made-up analytics company. A visitor asks a question and a LangGraph loop retrieves help articles, answers by quoting the best match (or with an LLM when a key is set), and searches again with a rewritten query when the first pass is weak. Questions the help center can't answer, or a tap on "This didn't help", open a ticket with the full transcript.
Most of the work went into the second message. Follow-ups like "and Starter?" or "can I export them first?" are resolved against the conversation, "thanks" gets a reply instead of a ticket, and a price question with no price in the articles goes to a person. A 43-conversation eval and Playwright tests in CI check that it stays that way.
engineering details
- Combined BM25 and MiniLM embeddings with reciprocal rank fusion, improving hit@1 from 30/36 to 34/36 and reaching 36/36 at hit@3 on the retrieval evaluation set.
- Rejected chunking after context recall fell by 0.098, exceeding the 0.05 regression gate.
- Recorded zero unnecessary tickets (precision 1.0) on 23 labeled questions, including prompt-injection and Taglish cases.
- Built multi-tenant help centers, self-serve signup, a reply co-pilot, an MCP server, and redacted Langfuse traces; covered by 120 tests.
- Built the LangGraph loop: retrieve, answer by quoting or generating, and retry with a rewritten query when retrieval is weak.
- Resolved follow-up questions against the conversation, checked by a 43-conversation multi-turn eval. Failures found in review went in as test cases before each fix.
- Traced a failing CI quality gate to a metric that scored correct refusals near zero, and changed my eval harness to score refused answers separately.
- Covered the widget with Playwright tests that run against a production build in CI.
in the product
Answers With a Source
Each reply quotes the help article it came from and links it under "Answer based on".
Handoff to a Staff Inbox
Unanswerable questions and "This didn't help" open a ticket. Staff see the full transcript, with bot replies marked.
The Help Center It Searches
Articles are grouped by topic, and each ends with a "Still stuck?" box that opens the chat.
