The Challenge
The client was a seed-stage B2B SaaS company in the project management space. Their product was genuinely good — their NPS among power users was excellent. But they had a serious activation problem.
55% of new signups were churning before they reached their first meaningful outcome (what the team called their "aha moment" — successfully completing their first automated workflow). Support was drowning in repetitive onboarding questions. The team was considering hiring a dedicated onboarding specialist, which at seed stage meant a significant runway commitment.
When they came to us, they'd already tried a few things: a 12-step product tour (skip rate: 83%), a knowledge base (used by fewer than 15% of new users), and an email drip sequence (open rate: 18%).
The insight we brought was simple: onboarding fails when it's passive. Users need to ask questions and get answers in context — not hunt through docs.
Our Approach
We had one constraint from the start: the solution had to feel like part of the product, not a chatbot bolted on. A floating "chat with us" button wasn't going to solve this. We needed something embedded in the product flow itself.
Discovery Work
Before writing a line of code, we did two things:
-
We analyzed the support ticket history — 300+ tickets from the previous 90 days — and categorized them. 70% fell into 8 distinct question patterns. That was the target.
-
We interviewed 8 churned users in the 14-day window. The most common phrase: "I couldn't figure out how to get started with [specific feature]." Not "the product is bad." Confusion, not rejection.
What We Built
We built a context-aware AI assistant embedded directly in the application sidebar. The key design decisions:
Context injection: The assistant knows where the user is in the product, what they've completed so far, and what's blocking them. Every conversation starts with a structured context payload: current page, last action, account type, days since signup, incomplete setup steps.
Knowledge grounding: The assistant is grounded on the product's documentation, help center, and a curated set of "getting started" workflows we wrote specifically for common use cases. We used RAG (retrieval-augmented generation) to pull the right context for each question.
Handoff logic: When the assistant detects a question it can't confidently answer, or when it identifies a genuine bug report, it creates a structured support ticket automatically and surfaces a human handoff — without interrupting the user flow.
Conversation design: We worked with Claude's prompt architecture extensively to ensure the assistant never confabulates. If it doesn't know, it says so and routes to documentation or a human. No hallucinated features. No incorrect instructions.
Technical Architecture
The assistant runs as a server-side streaming API built on Next.js API routes. Each message goes through:
- Context retrieval from the product database (current user state)
- Relevant documentation retrieval via vector search
- Claude API call with the assembled context
- Response streaming to the client
- Structured logging for conversation analytics
We kept the client-side implementation minimal — a React component that manages streaming state and renders markdown responses. No heavy chat library required.
Results
We ran a 30-day A/B test: 50% of new signups got the assistant, 50% got the existing experience.
Support ticket volume: Down 45% in the assistant group. The 8 common question patterns we targeted? Almost entirely eliminated.
14-day activation rate: Up 38% — from 45% to 62% of users reaching their first completed workflow.
Time to first aha moment: Reduced from a median of 4.2 days to 1.8 days.
NPS (activated users): Up 22 points. Users who activated with the assistant's help reported feeling more confident in the product.
Support team capacity: The team went from reactive firefighting to proactive work — improving docs, building better templates, doing customer success outreach rather than answering the same questions repeatedly.
What We Learned
Context is everything. A generic chatbot would have helped marginally. A context-aware assistant that knows exactly where the user is and what they're trying to do is qualitatively different. The investment in structured context injection was the single highest-leverage design decision.
Grounding matters more than model capability. The biggest risk with LLM-powered onboarding is hallucination — the assistant confidently telling a user to click a button that doesn't exist. We spent significant time on the grounding architecture and implemented strict guards. The assistant's instruction accuracy rate in testing was 99.1%.
Conversation logging is product gold. The assistant conversations became one of the company's best product research tools — a direct window into where users are confused and why.