Lyssin's Mission

Lyssin, a Portland-based performance optimization company, helps organizations capture and elevate frontline employee insights into actionable strategic intelligence. With a patent-pending approach to employee feedback, Lyssin had a bold vision: enable company leadership to hear directly from their teams in a way that was meaningful, prioritized, and actionable. They needed a technical partner who could determine whether a large language model could actually solve this problem, and then build their version 1 application.

The Business Challenge: predictably categorize large amounts of employee feedback into actionable insights

Leadership needs to understand what's happening on the front lines of their organizations, but traditional employee feedback mechanisms are broken. Comment boxes overflow with unstructured feedback ranging from cafeteria complaints to serious operational concerns. Without context or prioritization, this valuable intelligence gets lost in noise.

Organizations that can't surface critical employee insights risk missing early warning signs of operational problems, talent retention issues, and cultural challenges that directly impact the bottom line.
— Josh Frantz, CEO and co-founder, Lyssin

Lyssin envisioned a system that could automatically categorize, score, and prioritize employee feedback, surfacing what matters most to leadership. But they faced fundamental questions: Could AI reliably extract meaning from unstructured employee comments? Could it distinguish between urgent issues and everyday gripes? Could it understand context well enough to group related feedback without creating chaos?

Adding to the challenge, obtaining real employee feedback data for testing proved difficult during the discovery phase. The team would need creative solutions just to validate the concept.

The Technical Challenge: Engineering Reliable AI Classification at Scale

As Olio Apps began building Lyssin's system, we focused on the inherent challenge of working with large language models: optimizing their performance for consistent, reliable results in a production environment. LLMs require careful prompt engineering and systematic testing to ensure they deliver accurate classifications that meet specific business requirements.

The technical requirements were complex and interconnected. The system needed to:

• Categorize feedback dynamically as messages flowed in, without creating topic proliferation (where "cafeteria donuts," "cafeteria coffee," and "cafeteria pizza" become three separate topics instead of grouping under "cafeteria")
• Score feedback across multiple dimensions: importance, urgency, frequency, and relevance to CEO preferences
• Match similarity to identify which new feedback resembled topics the CEO had previously engaged with
• Handle scale efficiently without throwing 10,000+ feedback items into a single prompt

Traditional approaches wouldn't work. We couldn't process massive datasets in single prompts as the cost and processing time would be prohibitive. Instead, we combined strategic prompt engineering with intelligent data retrieval to process only the most relevant feedback items. This hybrid approach balanced accuracy with performance while keeping costs manageable.

But the bigger challenge was iteration speed and collaboration. How do you accelerate the prompt refinement process? How do you enable non-technical stakeholders to participate meaningfully in optimizing AI behavior?

The Solution: A Collaborative Approach to Prompt Engineering

The breakthrough came from creating a custom prompt engineering workbench early in the project that accelerated iteration and enabled collaborative refinement of prompts.

The Workbench gave the Lyssin team the ability to:

Reduce iteration time from days to hours, accelerating time-to-production

Upload sample data and run it through custom prompts

Observe the structured outputs the LLM generated

Adjust parameters, such as topic priorities

Test prompts against multiple LLMs side-by-side

Iterate independently through a tune → observe → repeat cycle

This tool transformed prompt engineering from an opaque, developer-only activity into a transparent, collaborative process. When real sample data was scarce, we used LLMs to generate fabricated feedback with controlled parameters (specific tones, urgency levels, topics), then tested our classification system against this synthetic dataset.

The toolbench also allowed us to evaluate different LLM providers. After testing several options, we selected Claude for its superior accuracy in classification tasks.

What once required endless meetings and emailed spreadsheets became an efficient feedback loop. The Lyssin team could experiment with prompt variations and immediately see the results. We estimate this approach saved at least 80 hours of development and testing time per prompt.

A Collaborative Partnership

This wasn't a traditional vendor-client relationship. The project required deep collaboration between Olio's technical team (two developers, a fractional CTO, and a part-time project manager), and Lyssin's product team, including CEO Josh Frantz, a product manager, and a designer.

Together, these cross-functional teams worked through product design, sprint planning, and workflow refinement. The Lyssin team brought domain expertise about employee feedback and CEO needs. Olio brought expertise in AI implementation and system architecture. The Prompt Workbench served as the bridge, enabling non-technical stakeholders to participate meaningfully in prompt engineering decisions.

Results & Impact

The project delivered significant value across multiple dimensions:

The project validated that AI could reliably solve a complex, real-world problem: turning unstructured employee feedback into strategic intelligence. Lyssin is now successfully serving paying customers with a production system that consistently delivers accurate categorization and prioritization.
— Josh Frantz, CEO and co-founder, Lyssin