0-to-1: Building & Scaling a RAG System for Social Good

The Problem: Crashing on Long Conversations

I identified that the core function was crashing with a 400 Bad Request error. I realized that my most engaged users—those with long, detailed conversations—were the most likely to experience a total app failure, breaking trust and halting their journey.

What Was Wrong

I dove into the logs and saw that the text-embedding-004 model has a small 36,000-byte limit. My code was sending the entire chat history for vectorization, which was inefficient and, for long chats, fatal.

The Fix: A Dual-History Approach

I identified two different needs: RAG only needs recent context to find relevant documents, while the AI needs full context to understand the user's journey. I implemented a solution by creating two history variables:

1. recentHistoryText: A small, truncated history sent to the embedding model for efficient document retrieval.
2. fullHistoryText: The complete history sent to the final chat model (gemini-2.5-flash) to maintain conversational context.

The Problem: User Frustration from Quickfire Questions

I noticed that users who were asked for details (e.g., "how long has the mould been present?") would try to type a multi-line answer. When they hit Enter for a new line, the UI submitted their partial, incomplete thought, confusing the AI and forcing it to ask the same questions again.

What Was Wrong

The UI was fighting the user's intent. A single-line <input> box that submitted on Enter was preventing users from providing the detailed, multi-line answers the AI needed.

The Fix: Aligning the UI with User Intent

My solution was to re-align the UI to match the user's natural workflow. I implemented this by:

1. Replacing the single-line <input> with a multi-line, auto-resizing <textarea>.
2. Changing the submit event from "Enter" to "Ctrl+Enter" (or "Cmd+Enter").
3. Updating the placeholder text to teach this new, more deliberate interaction.

The Challenge: Solving for Both System and User

This project highlighted a core product management tension: how do you balance non-negotiable technical constraints with intuitive, user-centric design? Two critical issues brought this into sharp focus.

What Was Wrong: A Two-Front Problem

The initial build faced failures on two fronts:

• System Failure: A technical limitation (the 36k-byte context window for the embedding model) was causing the app to crash for our most engaged users, directly punishing them for using the product correctly.
• User Failure: A usability flaw (an input field that submitted on "Enter") fought against natural user behavior, leading to incomplete submissions, frustrating loops, and poor data quality for the AI.

The Solution: A Holistic Approach

Fixing these required a holistic view. For stability, I engineered a dual-history system to manage context efficiently without sacrificing conversational quality. For usability, I redesigned the input field to align with user expectations, enabling detailed, multi-line responses.

Key Learning: True User-Centricity is Full-Stack

This experience underscored that a successful product must solve for both technical stability and user intuition. Optimizing one at the expense of the other leads to failure. A truly user-centric product manager must understand the entire stack—from API limitations to user psychology—and design solutions that respect both.