
**Adapted from the <a href="https://www.youtube.com/watch?v=NuoszZNeTHI"><u>Code & Cognition</u> Podcast</a> by Olio Apps**


At Olio Apps, we’re always exploring how real-world AI applications are built, deployed, and scaled. In a recent episode of the Code and Cognition podcast, we sat down with Zach, a software engineer at Backrs, to unpack how his team built and scaled an LLM-powered platform that delivers hyper-personalized career guidance to students and how AWS Lambda made it possible.

From the architectural choices to the streaming gotchas and database bottlenecks, this post explores what it really takes to build a serverless GenAI app at scale.

---

## The Problem: Usage Spikes alongside Personalized Output

Backrs’ mission is to help students and young professionals figure out what they want to do and how to get there. With its mission to help support students and democratize their access to opportunities, Backrs created an AI-driven success coach. Think career coaching, resume feedback, and college recommendations, all delivered in a personalized and context-aware interface.

However, this presented two core challenges:


<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li>
    <p><strong>Usage Spikes</strong></p>
    <p>Whenever large user cohorts would onboard, the application's perforance would degrade and costs would unexpectedly increase.</p>
  </li>
  <div style="height: 2em;"></div>
  <li>
    <p><strong>Consistant Personalized Experience</strong></p>
    <p>Each student needed to receive individualized LLM powered recommendations and content generation.</p>
    <div style="height: 1em;"></div>
  </li>
</ol>

To deliver this at scale without incurring cloud costs from idle compute, the Backrs team turned to **AWS Lambda**.

---

## Why AWS Lambda Was the Right Fit

Initially, the prototype ran on a single EC2 instance. This worked fine until more users began to onboard. With these new users, performance tanked under load and the team knew they needed a scalable solution.

**Lambda was the answer.**

AWS Lambda gave Backrs instant horizontal scalability and cost-efficiency. Most importantly, they didn't have to pre-provision a fleet of EC2 instances just for them to sit idle between school hours.

Lambda’s ability to spin up hundreds of instances in parallel, then spin them back down to zero, was ideal for Backrs' high-burst, low-persistence usage patterns.

---

## AI Personalization with LLMs

LLMs are at the core of Backrs' personalization engine. When a student logs in, they provide some personal information by


<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    Uploading their resume
  </li>
  <li style="margin-bottom: 1em;">
   Answering questions about their interests and location
  </li>
  <li style="margin-bottom: 1em;">
   Choosing desired job fields, majors, and goals
  </li>
</ul>

The platform then uses OpenAI's GPT API to generate personalized content like:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    A college short list with reach, target, and safety schools
  </li>
  <li style="margin-bottom: 1em;">
    Custom essay prompts
  </li>
  <li style="margin-bottom: 1em;">
    Tailored advice for learning paths and internships
  </li>
</ul>

These are all presented within a polished, notebook-style UI that feels educational and trustworthy, perfectly aligned for their user.

---

## The Challenge of Streaming Responses from Lambda

LLMs like GPT-4 take time to generate structured outputs, especially when multiple steps (agents, context fetching, summarization) are involved. For responsiveness, streaming is critical.

But streaming from AWS Lambda is not straightforward for several reasons. 

For starters, API Gateway doesn't support long lived connections well, as it has a 30 second timeout. 

Also, AWS Lambda's default behavior is to buffer output, where it has certain payload requirements to meet before it sends its output. 

Structured outputs (like JSON with chatbot replies + suggested follow-ups) arrive in chunks that must be reassembled on the frontend.

To get around this, the team decided to utilize Lambda response streaming by progressively streaming them through Lambda function URLs. They also replaced their typical Python Lambda adapter, Magnum, with AWS Lambda Web Adapter, a little known but powerful AWS maintained tool. They also dockerized their Python Lambda using FastAPI + the Web Adapter, enabling real-time streaming of LLM outputs.

---

## OpenAI API Rate Limits: Why Backrs Spent $200 on Fake Stories

Even once the infrastructure was sound, there were API rate limits to contend with.

OpenAI doesn’t let you pay to unlock higher throughput. Instead, you have to use the API enough to earn higher tiers.

So Zach ran a script that generated **thousands of silly stories about his coworkers overnight** just to bump their usage volume.

“We had to spend $200 generating completely useless content just to hit the threshold.”

It worked, it was fun, and Backrs finally unlocked the next usage tier and scaled up without rate limit failures.

---

## Database Bottlenecks and the RDS Proxy Fix

Once concurrency increased to 100+ users, a new problem emerged: the database couldn’t handle the load.

Each Lambda function was initiating a fresh DB connection and opening a pool of 20. Multiply that by 100 concurrent Lambdas and... the RDS instance fell over.
The fix? AWS RDS Proxy.

The proxy sits in front of the database and manages connection pooling across Lambdas. Now every Lambda borrows from a shared pool, and the DB remains stable, even under 100 concurrent sessions.

## Configuring Lambda for Long-Running AI Tasks

By default, Lambda is meant for short-lived jobs, yet LLM-powered agents often need 20+ seconds to do complete their tasks.

To support this:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    The Lambda timeout was extended to the 15-minute max (via function settings).
  </li>
  <li style="margin-bottom: 1em;">
    A heartbeat job was created to ping Lambdas periodically to keep them warm before anticipated traffic (e.g., classroom sessions).
  </li>
  <li style="margin-bottom: 1em;">
    Reserved concurrency was configured for critical paths (like logging in), ensuring they’re always responsive.
  </li>
</ul>

---

## Streaming LLM Responses in the Frontend

The Backrs app parses streamed JSON chunks into structured UI elements:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
Chatbot responses  </li>
  <li style="margin-bottom: 1em;">
Suggested next questions </li>
  <li style="margin-bottom: 1em;">
Embedded videos and articles  </li>
</ul>

This makes the product feel fast and intelligent, even when the backend is doing heavy lifting.
They also run multiple AI agents concurrently, fetching context and resources from internal knowledge bases and user history.
This way a seamless user experience is created with the LLM responses tailored to each student’s profile, leveraging their real data as context.

---

## Real-World GenAI Architecture: What We Used

In summary, this was the backend and frontend stack that made Backrs' Personalization Engine Possible

**Backend Stack**
<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
AWS Lambda with function URLs  </li>
  <li style="margin-bottom: 1em;">
FastAPI + AWS Lambda Web Adapter (Python) </li>
  <li style="margin-bottom: 1em;">
Dockerized Lambdas for dependency-heavy apps  </li>
<li style="margin-bottom: 1em;">
OpenAI GPT-4 API for LLM inference  </li>
<li style="margin-bottom: 1em;">
AWS RDS + RDS Proxy for database operations  </li>
</ul>


**Frontend**
<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
Custom-built UI using React</li>
  <li style="margin-bottom: 1em;">
JSON chunk parser for streaming responses </li>
  <li style="margin-bottom: 1em;">
Component architecture for chatbot, resource cards, and context-aware updates  </li>
</ul>

Now Backrs' personalized coach is able to more directly guide, mentor, and problem solve with students who are looking to make the most of their education and career journies. 

---


## Final Thoughts: Lessons from Scaling GenAI with AWS at Backrs

<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li>
  <p><strong>Don't underestimate cold starts and concurrency limits</strong></p>
  <p>These are show stoppers and require planning ahead to make sure your application can continue working smoothly.</p>
  </li>
    <div style="height: 1em;"></div>
   <li>
  <p><strong>LLM usage quotas are real</strong></p>
  <p>OpenAI requires you to "earn" throughput with real use, you can't just request a rate increase like a model on Amazon Bedrock. So factor this in if you plan to use OpenAI's models.</p>
<div style="height: 1em;"></div>
  </li>
  <li>
  <p><strong>Structured streaming from Lambda is possible</strong></p>
  <p>Though it requires it's own set of architectural considerations and the adaptor pattern utilizing function URLs.</p>
<div style="height: 1em;"></div>
  </li>
  <li>
  <p><strong>An RDS Proxy is essential</strong></p>
  <p>This allows you to connect AWS Lambda to databases at scale.</p>
<div style="height: 1em;"></div>
  </li>
  <li>
  <p><strong>GenAI is more software engineering than data science</strong></p>
  <p>The 'magic' in an LLM powered application lies in its orchestration. These applications need to be properly operationalized.</p>
<div style="height: 1em;"></div>
  </li>
</ol>

These have been our major takaways from listening how Backrs created a personalized LLM Mentor on AWS. If you're trying to build something similar, we hope this can serve as a blueprint and save you some time. If you have any specific questions or would like any hellp with your LLM applicaiton, please don't hesidate to <a href="https://www.olioapps.com/contact-us"><u>reach out</u></a>.


Backrs’ mission is to help students and young professionals figure out what they want to do and how to get there, now supported with actionable, AI-driven guidance. Think career coaching, resume feedback, and college recommendations, all delivered in a personalized and context-aware interface.

Scaling Personalized AI at Backrs: How We Build A Serverless LLM App With AWS Lambda

**From the Code & Cognition Podcast by Olio Apps**

Artificial Intelligence is changing how software is written and who writes it. Thanks to large language models (LLMs) like ChatGPT, DeepSeek, and Claude, anyone with a clear idea can now generate code, learn new frameworks, and prototype apps. This is all done by collaborating with an AI assistant and this emerging approach has a name: **vibe coding.**

Coined by AI researcher Andrej Karpathy, vibe coding describes a new, conversation-driven way of programming. Instead of manually writing every line of code, developers (or non-developers) **“vibe” with the AI, guiding it via natural language prompts and iteratively improving results.**

In this post, we’ll break down what vibe coding is, how it works, and share 11 practical tips to get better results when using AI to build software, including tips on code generation, debugging, documentation, and prototyping.

---


## What Is Vibe Coding?

**Vibe coding is the process of building software by communicating with an AI model using natural language.** You describe what you want and the AI returns working code, often faster than a human could (hopefully!).

Unlike traditional programming, where coding knowledge is a prerequisite, vibe coding lowers the barrier to entry. Designers, entrepreneurs, product managers, and junior developers are now building MVPs, games, and tools without deep knowledge of frameworks or syntax.

But make no mistake: vibe coding is still a skill. Just like knowing how to Google efficiently helps you debug faster, knowing how to communicate with an LLM can dramatically improve your output quality.

---

## Why This Matters

Search interest for tools like **ChatGPT**, **Cursor**, and **GitHub Copilot** has exploded. As of 2024, over 50% of professional developers report using AI coding assistants weekly and that trend has only increased in throughout 2025.

What this means:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    <strong>Prototypes are being built in minutes instead of days.</strong>
  </li>
  <li style="margin-bottom: 1em;">
   <strong>AI is changing how the industry is learning how to code, prototyping applications, and launching products.</strong> 
  </li>
  <li style="margin-bottom: 1em;">
   <strong>The bottleneck around shipping products becoming less contingent on knowing syntax, but now includes one's ability to instruct and communicate with LLMs like one would talk to a junior developer.</strong> 
  </li>
</ul>


---

## 11 Practical Tips for Better AI-Powered Coding
Here’s what we’ve learned building real tools and workflows using LLMs like GPT-4 and Claude.

<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li>
  <p><strong>Treat the AI Like a Teammate</strong></p>
  <p>Imagine you're talking to a junior dev who never gets tired but can sometimes misinterpret instructions. LLMs are like this. LLMs are powerful, but they need structure and clarity just like a person would.</p>
<div style="height: 1em;"></div>
  <p>Don’t just say: “Build me an accounting app.” Insead say “Build a simple accounting dashboard with login, user roles, and an expense entry form using React and Firebase.”</p>
  </li>
    <div style="height: 2em;"></div>
   <li>
  <p><strong>Break Big Projects into Small Tasks</strong></p>
  <p>Just like a real-world software team, AIs do better with modular, sequential instructions.</p>
<div style="height: 1em;"></div>
  <p>Instead of saying, “Write the whole app,” instead try saying, “Let’s start with the login form UI.” Then, “Now write the Firebase auth logic.” Then, “Now wire them together.”</p>
  <div style="height: 1em;"></div>
<p>This modular, step by step instruction will reduce the chance your LLM misunderstands your direction or jumps to conclusions regarding what you want. 
</p>
  </li>
      <div style="height: 2em;"></div>
   <li>
  <p><strong>Provide Background and Style Preferences</strong></p>
  <p>Before starting, give your AI some context, like what language or framework you’re using, your folder structure preferences, or your current coding conventions. This will help contextualize it’s answer based on your exact project, leaving it less opportunity to hallucinate. </p>
<div style="height: 1em;"></div>
  <p>For example, you could tell your LLM, “This project uses Next.js, Tailwind CSS, and stores API calls in /lib/api. Please follow those conventions in your response.”</p>
  </li>
  <div style="height: 2em;"></div>
   <li>
  <p><strong>Explain Why The Output is Wrong </strong></p>
  <p>When the AI makes a mistake, explain the issue and ask for a fix. Tell it, “This breaks because useEffect needs a dependency array. Can you fix that?” </p>
<div style="height: 1em;"></div>
  <p>Over time, this improves both your result and your ability to communicate technical expectations to the LLM.</p>
  </li>
  <div style="height: 2em;"></div>
   <li>
  <p><strong>Choose Popular Frameworks When Possible </strong></p>
  <p> Since LLMs have been trained on the most available data present on the internet, frameworks with longer usage histories and available examples are better candidates to "vibe" with.</p>
<div style="height: 1em;"></div>
  <p>That means if your application uses lots of React, you should be good to vibe code! However if you’re asking the LLM to help you with ClosureScript edge cases, you may be better off skipping the vibe coding session. </p>
  <div style="height: 1em;"></div>
  <p>Use this principle to your advantage by not expecting your LLM to perform like a niche expert of esoteric frameworks. Let it do the undifferentiated heavy lifting instead of putting in charge of niche expertise.
</p>
  </li>
  <div style="height: 2em;"></div>
   <li>
  <p><strong>Use AI to Learn New Languages or Frameworks </strong></p>
  <p>     Want to learn Rust, Go, or Svelte? Ask the AI to build a sample CRUD app in that stack. Then ask it to explain each file, function, or line of code.</p>
<div style="height: 1em;"></div>
  <p>This works like personalized tutoring on demand and can accelerate your learning in ways that are tailored to your unique learning style. If you know the way you like to learn, you can have the LLM teach you exactly like that! </p>
  </li>
  <div style="height: 2em;"></div>
   <li>
  <p><strong>Rapid Prototyping = Less Attachment </strong></p>
  <p>         With vibe coding, it’s possible you can create 3 or 4 design variations quickly. If you don’t like them, throw them away. You haven’t lost weeks of work! You’ve only gained insight and perspective on how you don’t want to build your application. </p>
<div style="height: 1em;"></div>
  <p> Being able to iterate prototypes quickly like this helps you avoid sunk cost bias and encourages iteration.</p>
  </li>

   <div style="height: 2em;"></div>
   <li>
  <p><strong>Manage Large Codebases with Context Windows </strong></p>
  <p> Modern LLMs can process more context (100k+ tokens), but still can’t “see” your entire codebase. Use code assist tools like Cursor to give it targeted files or examples to reference instead of having the LLM take its best guess at the design patterns inherent in your code base. </p>
  </li>
  
 <div style="height: 2em;"></div>
   <li>
  <p><strong>Use Version Control Aggressively </strong></p>
  <p>       Vibe coding can quickly take your project from working to broken. Use Git (or any VCS) to commit early and often. Replit and Cursor have built-in checkpoint systems, but don’t rely on autosave alone. If you haven't already developed good version control habits, now is the time!
 </p>
  </li>
<div style="height: 2em;"></div>
   <li>
  <p><strong>Have It Write Docs and Tests </strong></p>
  <p>To speed up workflows, you can manage LLMs to consistently generate README.md setup instructions, runbooks for production environments, and unit/integration tests for key functions. This is another area where LLMs can shine as these tasks can often be systematized and feel repetitive. 
 </p>
  </li>

  <div style="height: 2em;"></div>
   <li>
  <p><strong>Don't Sleep on Bash Scripts </strong></p>
  <p>Need to spin up a dev environment, deploy to AWS, or notify Slack on success? Let LLMs write your Bash scripts with flags, logging, and dry-run options.
 </p>
 <div style="height: 1em;"></div>
<p>You’ll find yourself writing and refining automation scripts in 10 minutes that you previously would’ve avoided for a week.
</p>
  </li>
</ol>

---

## Use Case: Vibe Coding with AWS Bedrock and App Sync

In a recent project, we used Amazon Bedrock and AWS AppSync to build a real-time LLM interface using WebSockets. However, a new AppSync Real-Time Websockets feature was released mid project, (released in March 2025), and seemed like the ideal solution, but we were also learning the stack in real-time.

We had to decide:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
Use the “faster to implement” old method with higher cost, but with supportive “vibes” from the LLM? </li>
  <li style="margin-bottom: 1em;">
Or use the “optimal but untested” new method with unknown quirks?  </li>
</ul>

This is where vibe coding met long term engineering decision making. Vibe coding may be fast and powerful, but it isn't a substitute for judgment, experience, and experimentation.

In this scenario, the time to value trade of vibe coding vs. utilizing a less supported but optimal stack were the critical points to consider.

---

## So, Is Vibe Coding Cheating?

Not at all.

Just like calculators didn’t stop people from learning math, vibe coding won’t stop people from learning to program. In fact, we’ve seen the opposite.

Some of our team members attest that their understanding of programming has improved faster using AI than it did with bootcamps or video courses.

LLMs give you instant feedback, endless examples, and real-time explanations. When skillfully utilizing LLMs, your learning is limited only by your curiosity.

---

## Final Thoughts: Vibe Coding Is a New Methodology of Software Development

Whether you're an experienced engineer or a curious creative, we are seeing that AI coding tools and "vibe coding" unlocks a new kind of collaboration between humans and machines.

Vibe Coding allows us to accomplish a myriad of tasks, including:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
Building Prototypes</li>
  <li style="margin-bottom: 1em;">
Learning new technologies </li>
  <li style="margin-bottom: 1em;">
Shipping products faster  </li>
  <li style="margin-bottom: 1em;">
Building our communication skills </li>
</ul>

And most importlanly, *have fun building software!*

---

## Resources Mentioned

<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li>
    <a href="https://open.spotify.com/show/1AqPzt76zBbTfBOhZRyqPz?si=cf858e26906c430c" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Code and Cognition Podcast</a>
  </li>
  <li>
    <a href="https://aws.amazon.com/bedrock/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Amazon Bedrock</a>
  </li>
  <li>
    <a href="https://www.cursor.com/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Cursor IDE</a>
  </li>
  <li>
    <a href="https://x.com/karpathy/status/1886192184808149383?lang=en" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Vibe Coding</a>
  </li>
</ol>



Artificial Intelligence is changing how software is written—and who writes it. Thanks to large language models (LLMs) like ChatGPT, DeepSeek, and Claude, anyone with a clear idea can now generate code, learn new frameworks, and prototype apps—all by collaborating with an AI assistant. This emerging approach has a name: vibe coding.

Vibe Coding: How To Build Software With AI (Even If You're Not A Developer)

The journey of integrating generative AI (GenAI) into enterprise operations and business operations in general continues to evolve. A recent discussion inspired by an insightful MIT Technology Review report highlighted significant GenAI development challenges that businesses faced with Large Language Model (LLM) applications in 2024.

Drawing from these industry observations and our own development experience from developing and refining LLM solutions, this post offers a roadmap and tangible LLM development strategies for navigating the complexities of enterprise AI and overcoming common GenAI roadblocks in 2025.

**Top 6 Enterprise GenAI Challenges in 2024**

The MIT report identified six major obstacles hindering widespread, successful GenAI deployment in enterprises, significant roadblocks that demand clear solutions for 2025:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    <strong>LLM Hallucinations and Toxicity:</strong> A striking 72% of companies reported ongoing problems with the quality and reliability of AI-generated outputs, including factual inaccuracies (hallucinations) and inappropriate content (toxicity). Addressing this is a key strategy for building trust in <strong>LLM</strong> solutions.
  </li>
  <li style="margin-bottom: 1em;">
   <strong>Difficult GenAI Integration with Existing Infrastructure:</strong> 62% of organizations encountered significant difficulties connecting LLMs with their organization’s broader IT infrastructure and business operations, a critical step for <strong>deploying LLMs</strong> at scale. An effective <strong>LLM development strategy</strong> must prioritize seamless integration.
  </li>
  <li style="margin-bottom: 1em;">
   <strong>Prohibitive Costs of Fine-Tuning LLMs:</strong> Many enterprises are shifting away from resource-intensive custom model fine-tuning. Instead, they're exploring more cost-effective approaches like Retrieval-Augmented Generation (RAG) or leveraging smaller, specialized fine-tuned language models to manage the <strong>cost of fine-tuning LLMs.</strong>
  </li>
  <li style="margin-bottom: 1em;">
   <strong>LLM Latency Issues:</strong> Slow model response times were a hurdle for 56% of companies, impacting user experience and the feasibility of real-time <strong>LLM applications.</strong> Reducing latency is vital for practical <strong>GenAI solutions.</strong>
  </li>
   <li style="margin-bottom: 1em;">
  <strong>Scalability Problems with GenAI Applications:</strong> 51% struggled to scale their GenAI applications effectively to handle increasing user demands and larger datasets, a key factor in <strong>scaling GenAI solutions.</strong>
  </li>
   <li style="margin-bottom: 1em;">
   <strong>Low Production Adoption of GenAI:</strong> Despite considerable buzz and bold predictions for <strong>GenAI in business,</strong> only a mere 5% of companies had achieved real production deployments by mid-2024. Overcoming this <strong>GenAI roadblock</strong> is paramount for realizing ROI.
  </li>
</ul>

This data strongly resonates with our experiences on the ground. The initial wave of enthusiasm for "productionizing" GenAI apps has undeniably met the complex realities of real-world implementation, tempering expectations for immediate, widespread GenAI production adoption. Effective LLM development strategies are needed to bridge this gap.


---

## Core Strategy - The Critical Role System Prompts Play in LLM Performance

One of the foundational lessons from our work in LLM development and a core strategy for 2025 success has been the use of system prompts. Learning how to properly implement system prompts has been essential to how we approach using LLMs.  A system prompt acts like a set of guardrails for an LLM, telling it:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    What <strong>role</strong> it should adopt (e.g., "You are a biomedical researcher")
  </li>
  <li style="margin-bottom: 1em;">
    What <strong>response style</strong> it should use (e.g., "Respond in formal academic tone")
  </li>
  <li style="margin-bottom: 1em;">
    Any <strong>rules</strong> it should follow (e.g., "Only answer based on the provided document")
  </li>
</ul>

Without a clear system prompt, LLMs like Claude or ChatGPT can start exhibiting undesirable behaviors. They can start "self-talking" - misinterpreting their context, spiraling into incoherent responses, or even crashing.
 Our experience has shown that incorrectly handling the system prompt, like passing it as a user message instead of a system role, led to serious breakdowns during live LLM interactions, negatively impacting LLM output Quality.
 
**Key Takeaway for LLM Success**

Always implement a system prompt from the beginning as part of your LLM development strategy. Continuously iterate and refine it to ensure it reliably guides the LLM's behavior, aligning its responses with your specific enterprise AI use case or business operation use case. 

---

## Structured Output: A Key Strategy for Scaling LLM Applications Reliably

Another vital lesson for successful LLM deployment, and a crucial development strategy for 2025, is the necessity of structured output. Moving beyond simple free-form text responses is essential if you aim for your LLM applications to scale, integrate seamlessly with your existing IT infrastructure, and perform reliably.

**Examples of Structured Output Include**

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    <strong>JSON</strong> (JavaScript Object Notation)
  </li>
  <li style="margin-bottom: 1em;">
    <strong>XML</strong> (Extensible Markup Language)
  </li>
  <li style="margin-bottom: 1em;">
    <strong>CSV</strong> (Comma Separated Values)
  </li>
</ul>

Why is **structured output from LLMs** so important for an effective **GenAI strategy?**
<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    <strong>It allows for easier parsing and validation.</strong> Structured responses simplify the process of programmatically parsing and validating the AI’s answers.
  </li>
  <li style="margin-bottom: 1em;">
    <strong>It streamlines downstream automation.</strong> Structed output enables smoother automation of subsequent processes and workflows that depend on the AI output.
  </li>
  <li style="margin-bottom: 1em;">
    <strong>It improves output quality analysis.</strong> More effective analysis and systematic improvement of the LLM's response quality is enabled over time.
  </li>
  <li style="margin-bottom: 1em;">
    <strong>It enables a consistent user experience.</strong> Structued output ensures there is structured consistency in how information is presented to end-users of your GenAI applications.
  </li>
</ul>

## A Critical Warning for Complex LLM Tasks

Attempting to combine complex reasoning tasks with strict structured formatting instructions within a single LLM prompt can be counterproductive. The model might overly focus on adhering to the formatting rules at the expense of reasoning accuracy, potentially leading to an increase in LLM hallucinations or nonsensical structured data.

**The Solution?** Decouple the tasks. Allow the LLM to perform the complex reasoning first, then instruct it to structure the reasoned output in a subsequent step or with a dedicated formatting prompt. We have seen this separation leading to more reliable and accurate structured AI responses.

---

## Introducing Data Pilot: The LLM Workbench for Reliable Development

In direct response to these prevalent GenAI development challenges and to support robust LLM development strategies, we developed 
[Data Pilot](https://www.yourdatapilot.com/), a comprehensive LLM development platform designed to empower developers to build superior LLM prompts and measure their effectiveness across various models and use cases.

**Data Pilot** helps your team to effectively implement GenAI best practices by:


**Tips to maximize productivity while minimizing tech debt:**

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    Developing more effective prompts through a collaborative, chat-based process, centralizing prompt engineering efforts.
  </li>
  <li style="margin-bottom: 1em;">
    Generating synthetic datasets, crucial for training and robustly testing LLMs against diverse scenarios.
  </li>
  <li style="margin-bottom: 1em;">
    Collect, organize, and structure data to systematically evaluate LLM output quality and benchmark performance.
  </li>
  <li style="margin-bottom: 1em;">
    Allowing you to efficiently compare the outputs of different leading LLM models (such as Claude, GPT-4, Meta Llama, and others) to select the best fit for your use case.
  </li>
</ul>

Essentially, Data Pilot functions as a copilot for your entire LLM development lifecycle. It helps your team move faster and more intelligently, significantly improving output reliability and optimizing time spent on prompt engineering.

**Example Workflow with Data Pilot**

<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li>
    <strong>First describe your goal.</strong> State your objective conversationally, like, "I need to accurately categorize incoming customer service tickets based on urgency and topic."
  </li>
  <li>
    <strong>Second, refine your prompt and suggest structured output.</strong> Data Pilot can assist in refining your initial prompt and automatically suggests an optimal structured output format. Test different prompts to see is the output serves your usecase.
  </li>
  <li>
    <strong>Third, test structured output with synthetic test case data:</strong> When prompted, the platform synthesizes realistic test cases to challenge and evaluate the LLM's categorization capabilities.
  </li>
  <li>
    <strong>Finally, evaluate and optimize!:</strong> Now evaluate the LLM's performance using real and synthetic data, and your refined prompt. Continue tweaking the prompt and model choices within Data Pilot to achieve desired accuracy and reliability for your use case. This workflow is the bases for informed and data-driven LLM optimization and these processes form the cornerstone of a successful LLM strategy.
  </li>
</ol>

---

## Final Thoughts: 2024 Forged Hard Lessons, 2025 Demands Applied Wisdom & Strategy

The gap between the initial hype surrounding GenAI and consistent, valuable GenAI production remains, but it is steadily narrowing. The enterprises succeeding in the enterprise AI landscape today, and those poised to conquer GenAI roadblocks in 2025, are characterized by their commitment to clear LLM development strategies that include:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
Investing early and consistently in disciplined prompt engineering and robust system prompt design.  </li>
  <li style="margin-bottom: 1em;">
Prioritizing the generation of structured, measurable, and verifiable outputs from their LLM applications.  </li>
  <li style="margin-bottom: 1em;">
Building or adopting comprehensive LLM development tools (like Data Pilot) to accelerate the development lifecycle, enhance reliability, and enable GenAI governance.  </li>
  <li style="margin-bottom: 1em;">
Understanding that successful AI development is a nuanced blend of psychological insight (understanding model behavior) and rigorous engineering.
  </li>
</ul>
---


The future of GenAI in business belongs to teams who treat LLMs not as inscrutable black boxes, but as exceptionally powerful (yet sometimes quirky) tools. These tools demand clear operational boundaries, well-defined structures for their outputs, and continuous, vigilant supervision as part of an overarching strategy to unlock their full potential and overcome the challenges of AI in production.

---

## Listen to the full conversation on Code & Cognition

Listen to the full conversation on the **[Code & Cognition Podcast](https://open.spotify.com/episode/1ZaScZqZcHjen24mu1i7r5?si=3de4ed3fcd73457b)**, and follow along as we continue building the future of AI-powered software.

---

## Resources Mentioned

<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li>
    <a href="https://open.spotify.com/episode/1ZaScZqZcHjen24mu1i7r5?si=3de4ed3fcd73457b" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Code and Cognition Podcast</a>
  </li>
  <li>
    <a href="https://aws.amazon.com/bedrock/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Amazon Bedrock</a>
  </li>
  <li>
    <a href="https://www.cursor.com/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Cursor IDE</a>
  </li>
  <li>
    <a href="https://www.yourdatapilot.com/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Data Pilot</a>
  </li>
  <li>
    <a href="https://www.anthropic.com/claude" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Claude AI</a>
  </li>
</ol>


The journey of integrating generative AI (GenAI) into enterprise operations and business operations in general continues to evolve. A recent discussion inspired by an insightful MIT Technology Review report, highlighted significant GenAI development challenges that businesses faced with Large Language Model (LLM) applications in 2024.

LLM Development Strategies for 2025

**From the Code & Cognition Podcast by Olio Apps**

At Olio Apps we have been steadily exploring the field of Gen AI development and building software with the assistance of Gen AI tools. From building MVPs to benchmarking LLMs head-to-head with internal developer tools, we’ve been hands-on with testing the bleeding edge of generative AI.

In this post, we wanted to share our firsthand experience using Gen AI to speed up prototyping, write production-quality code, and even generate infrastructure diagrams for cloud-native systems.

Whether you're scaling a dev team or experimenting solo, we wanted to show how using Gen AI in software development can looks like in practice.


---

## Prototyping Software with GenAI: What Actually Works (and What Doesn’t)

We started with a challenge: Could we build usable software prototypes for apps, games, and UI using only natural language prompts and GenAI tools?

We tested tools like **Bolt.new**, **v0.dev**, and **Claude Sonnet** by asking them to create:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    Classic games like Snake and Centipede
  </li>
  <li style="margin-bottom: 1em;">
    A bizarre Pac-Man/Snake hybrid with real-time ghost movement
  </li>
  <li style="margin-bottom: 1em;">
    A Reddit-style message board with image uploads, nested comments, and voting
  </li>
</ul>

Some prompts worked out of the gate. Others fell apart - usually when backend integration introduced complexity. But what we learned was this: prompting GenAI is like managing a junior engineer. You don’t drop a wall of specs. You guide it step-by-step, providing context and corrections as you go.

It’s fast. It’s messy. But it’s getting better every week.

---

## How We Prompt, Collaborate, and Ship with GenAI

We've tried out "vibe coding" for ourselves and after giving it a shot, we have determined some rules of thumb that help make it a usable methodology. 

It’s less about writing perfect specs, and more about co-creating with AI in a fluid, conversational way.

Some principles that emerged from our sessions were:

<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li style="margin-bottom: 1em;">
    <strong>Treat the AI like a team member</strong>: ask clarifying questions, give feedback, iterate
  </li>
  <li style="margin-bottom: 1em;">
    <strong>Don’t front-load everything</strong>: break tasks into smaller units, step by step.
  </li>
  <li style="margin-bottom: 1em;">
    <strong>Include background and context</strong>: we got better results by naming the tech stack, conventions, and expected file structure
  </li>
  <li style="margin-bottom: 1em;">
    <strong>Version control everything</strong>: make frequent commits. One wrong suggestion can undo hours of work.
  </li>
</ol>

Following these principles helped elevate our experiments beyond mere vibes into successful Proofs of Concepts and MVPs. 

---

## We Tried Benchmarking Claude 3.5 Sonnet vs. Gemini 1.5: Here’s What Happened

We were curious which LLM is actually better at coding, so we used [WebDevArena] (https://web.lmarena.ai/), a head-to-head testing ground, to compare **Claude 3.5 Sonnet**, **Gemini 1.5 Pro**, and **DeepSeek** on real coding tasks.

We ran each model through the same prompts - mostly game development and UI logic.

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    <strong>Claude 3.5 Sonnet</strong> came out ahead, producing clean, readable code with far fewer hallucinations.
  </li>
  <li style="margin-bottom: 1em;">
    <strong>Gemini 1.5</strong> struggled with complex logic in games like Centipede and Snake.
  </li>
  <li style="margin-bottom: 1em;">
    <strong>DeepSeek</strong> (open-source) surprised us. It wasn’t perfect, but showed promise as a lightweight assistant.
  </li>
</ul>

These kinds of direct, practical benchmarks gave us confidence when evaluating which models to use in different parts of our stack, whether as coding copilots or agents for internal developer tooling.

---

## AI Generated Architecture: How We Design Cloud Systems in Minutes

We’re heavy users of Amazon Web Services and one of our pain points has always been visualizing and communicating infrastructure, especially when iterating quickly.

That’s where **DiagramGPT** and **Eraser.io** came in.

> API Gateway triggers a Lambda → writes to DynamoDB → sends a message to an SQS queue → triggers a second Lambda that sends email via SES

And within seconds, we had a polished AWS diagram, complete with icons and connections.

Compared to Miro or Lucidchart, this process saved considerable time and made it easier to collaborate with clients and team members during architecture reviews. We’re now integrating it into our solution design workflows for both consulting and internal projects.

---

## Final LLM Dev Tip: Don't Break the Illusion

**System prompts matter**  

If your app’s identity relies on LLM behavior, run forensic checks and ensure your system prompt is structured, clear, and hard to break.

You may find some LLMs have an easier time holding on to your system prompt so it is good to test and compare.

---

## Additional Insights: Database Modeling and Cursor System Prompts

Two other tools we explored that deserve a shoutout are:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    <strong>database.build</strong> Generated SQL schemas and ER diagrams from casual prompts like “a message board with users, posts, comments, votes, and favorites.” Surprisingly accurate.
  </li>
  <li style="margin-bottom: 1em;">
    <strong>Cursor</strong> We dug into its **system prompt** (yes, you can view it) to understand how it balances user intent with safe code editing. The result? Our ability to customize our dev environment so we can prompt edits with confidence.
  </li>
</ul>

For full-stack prototyping, the combo of schema generation + AI-assisted editing is starting to feel like the future of scaffolding software.

## Final Take: Gen AI Software Development Is About Velocity and Direction

Here’s the real shift from our perspective: **Gen AI doesn’t just make coding faster, it changes how we think about building.**

We’re no longer just writing code line-by-line. We’re **directing the AI**, setting the tone, choosing the structure, guiding the outcome and filling in gaps as necessary. Developers are becoming more like systems architects, especially in the prototyping phase.

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    Building prototypes in hours instead of days
  </li>
  <li style="margin-bottom: 1em;">
    Testing new architectures without spinning up infrastructure
  </li>
  <li style="margin-bottom: 1em;">
    Onboarding junior devs faster by pairing them with GenAI copilots
  </li>
</ul>

If you're in software consulting, running a SaaS, or managing engineering teams—this is the shift to start practicing.

Listen to the full conversation on the [Code & Cognition Podcast](https://open.spotify.com/show/1AqPzt76zBbTfBOhZRyqPz?si=599227791fa34b93), and follow along as we continue building the future of AI-powered software.

---

## Resources Mentioned

<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li>
    <a href="https://open.spotify.com/show/1AqPzt76zBbTfBOhZRyqPz?si=cf858e26906c430c" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Code and Cognition Podcast</a>
  </li>
  <li>
    <a href="https://aws.amazon.com/bedrock/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Amazon Bedrock</a>
  </li>
  <li>
    <a href="https://www.cursor.com/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Cursor IDE</a>
  </li>
  <li>
    <a href="https://database.build/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Database.Build</a>
  </li>
  <li>
    <a href="https://www.anthropic.com/claude" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Claude AI</a>
  </li>
  <li>
    <a href="https://web.lmarena.ai/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">WebDevArena</a>
  </li>
  <li>
    <a href="https://bolt.new/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Bolt.new</a>
  </li>
</ol>


Insights from the Code & Cognition Podcast by Olio Apps on building agentic LLM systems, improving workflows, and managing tech debt in 2025.

AI-Powered Prototyping: How We Are Using GenAI Tools to Build Apps and Architectures Faster

**From the Code & Cognition Podcast by Olio Apps**

In a recent episode of *Code & Cognition*, the Olio Apps team sat down to talk shop on the evolving world of Large Language Models (LLMs), agentic workflows, and how these tools are changing how we design, scope, and implement software projects in 2025.

Here's a quick recap of key insights from that conversation, plus takeaways for developers and teams experimenting with generative AI app development.

---

## Agents vs. Workflows: What’s the Difference?

There’s a lot of buzz around “AI agents,” but what distinguishes them from traditional LLM workflows?

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    <strong>Workflows</strong>: Think of these as on-the-rails sequences. They follow predefined steps like “if this, then that” logic.
  </li>
  <li style="margin-bottom: 1em;">
    <strong>Agents</strong>: More autonomous. They dynamically decide what steps to take and how to use tools, making decisions in real time. This allows for more flexible, adaptive systems.
  </li>
</ul>

<a href="/assets/blog/2025-05-19-agents-workflow-and-llm-productivity-in-2025/AgentsvsWorkflows.png" target="_blank" rel="noopener noreferrer">
  <img src="/assets/blog/2025-05-19-agents-workflow-and-llm-productivity-in-2025/AgentsvsWorkflows.png" alt="Agents vs Workflows diagram" style=" display:block; margin: 0 auto; max-width: 70%; height: auto; border-radius: 10px">
</a>



> Tools like [Amazon Bedrock](https://aws.amazon.com/bedrock/) and [Anthropic’s AI agent frameworks](https://www.anthropic.com/engineering/building-effective-agents) are pushing this forward, enabling developers to define tools, provide schema access, and let LLMs decide how to achieve outcomes—like dynamic project status reports pulled from multiple sources.

---

## Real-Time Evaluation & Human-in-the-Loop

One emerging pattern we are interested in exploring more: **LLM as judge**. This is where a more expensive model is used to evaluate output from a cheaper one. This cost-effective technique can be used live within systems, not just during prompt engineering.

For example:

<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li style="margin-bottom: 1em;">
    A cheaper model drafts responses.
  </li>
  <li style="margin-bottom: 1em;">
    A more advanced LLM evaluates output against success criteria.
  </li>
  <li style="margin-bottom: 1em;">
    Feedback is looped back until the result meets expectations.
  </li>
</ol>

This principle also plays out in human-in-the-loop workflows, where users review, refine, or approve outputs at strategic points in the flow. This further improves reliability while still leveraging automation.

---

## Cursor, Codegen, and Managing Tech Debt

[Cursor](https://www.cursor.com/en) has become a favorite among the Olio team for AI-assisted programming. Used well, it has boosted developer velocity by at least 40% for our teams. But with great speed comes great responsibility!

**Tips to maximize productivity while minimizing tech debt:**

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    <strong>System Prompts & Cursor Rules</strong>: Use project-specific guidance files to maintain consistent output and enforce patterns.
  </li>
  <li style="margin-bottom: 1em;">
    <strong>Refactor Often</strong>: Don’t just copy/paste generated code. Commit to reviewing and improving your code base according to your existing processes. 
  </li>
  <li style="margin-bottom: 1em;">
    <strong>Name @mentioning</strong>: Reference existing code and structures when asking Cursor to generate similar components. Giving Cursor this context is key for optimal outputs.
  </li>
</ul>

---

## What’s Next in Q2?

As we move deeper in Q2, our team is eagar to explore:

<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    Agentic workflows using Bedrock’s orchestration tools
  </li>
  <li style="margin-bottom: 1em;">
    Cross-source LLM synthesis: merging data from time trackers, Slack, and GitHub
  </li>
  <li style="margin-bottom: 1em;">
    Natural language end-to-end testing powered by LLMs + Playwright
  </li>
  <li style="margin-bottom: 1em;">
    More human-in-the-loop flows for approvals, refinements, and efficient collaboration between team members and projects
  </li>
</ul>

---

## Final LLM Dev Tip: Don't Break the Illusion

**System prompts matter**  

In testing, Claude consistently “stayed in character” while GPT-4 often broke with minimal pressure. If your app’s identity relies on LLM behavior, run forensic checks and ensure your system prompt is structured, clear, and hard to break.

You may find some LLMs have an easier time holding on to your system prompt. It may be more advantageious to stick with those LLMs going forward!

---

## Final Thought

Whether you’re just starting with LLMs or deep in building intelligent systems, the landscape is shifting fast. Stay agile, experiment, and don’t forget: **great AI starts with great context**.

Listen to the full conversation on the [Code & Cognition Podcast](https://open.spotify.com/show/1AqPzt76zBbTfB0hZRYqPz?si=e9ad58d720bd4f59), and follow along as we continue building the future of AI-powered software.

---

## Resources Mentioned

<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li>
    <a href="https://open.spotify.com/show/1AqPzt76zBbTfBOhZRyqPz?si=cf858e26906c430c" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Code and Cognition Podcast</a>
  </li>
  <li>
    <a href="https://aws.amazon.com/bedrock/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Amazon Bedrock</a>
  </li>
  <li>
    <a href="https://www.cursor.com/" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Cursor IDE</a>
  </li>
  <li>
    <a href="https://medium.com/@dtoka/create-test-automation-with-playwright-using-langchain-agents-62a136ce6071" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Playwright End-To-End Testing</a>
  </li>
  <li>
    <a href="https://www.anthropic.com/claude" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Claude AI</a>
  </li>
</ol>


AI Agents, Workflow, and LLM Productivity in 2025

When your application grows quickly, so do the demands on your infrastructure. AWS Lambda offers automatic scaling by design, but without the right configuration, it can lead to cold start latency, manual overhead, and escalating costs. As part of our Fractional CTO Service, we’ve worked with clients to address challenges like cold start latency and scaling in AWS Lambda, so we wanted to highlight an example where we were able to successfully implement Lambda auto scaling.

In this guide, we’ll walk through how our team tackled these challenges by implementing **Lambda auto scaling with provisioned concurrency**—a move that significantly improved both performance and cost-efficiency.

------

## The Problem with AWS Lambda Cold Starts and Manual Scaling

Our client’s application was built using AWS Lambda, but their serverless architecture had a major bottleneck: As more users joined, Lambda functions experienced cold starts, leading to sluggish response times. To compensate, we manually increased provisioned concurrency before major events. This fixed performance—but introduced human error and high costs when concurrency was left high after traffic dropped.

---


## Understanding Lambda Auto Scaling with Provisioned Concurrency

A common misconception is that AWS Lambda always scales instantly. While it can spin up new instances as requests come in, each new instance introduces **cold start latency**.

**Provisioned concurrency** solves this by keeping a set number of Lambda instances warm and ready. But provisioning too many functions 24/7 can be expensive.

<a href="/assets/blog/2025-04-30-how-we-solved-aws-lambda-scaling-with-auto-scaling-and-provisioned-concurrency/ProvisionedvsColdStart.png" target="_blank" rel="noopener noreferrer">
  <img src="/assets/blog/2025-04-30-how-we-solved-aws-lambda-scaling-with-auto-scaling-and-provisioned-concurrency/ProvisionedvsColdStart.png" 
       alt="Provisioned vs Unprovisioned Lambda Concurrency Chart" 
       style="max-width: 100%; height: auto; border-radius: 10px;" />
</a>

That’s where **AWS Application Auto Scaling** comes in. It dynamically adjusts provisioned concurrency based on real-time metrics—scaling up during traffic spikes, and scaling down when demand slows. This gives us the performance benefits of warm Lambdas without the cost of over-provisioning.



## Real-World Lambda Scaling: Our Configuration in Practice
Here’s how we implemented it using the Serverless Framework:
<ul style="list-style: initial; padding-left: 1.2em;">
  <li style="margin-bottom: 0.5em;"><strong>Provisioned concurrency</strong> is configured for high-traffic endpoints.</li>
  <li style="margin-bottom: 0.5em;"><strong>CloudWatch alarms</strong> monitor usage and trigger scaling actions.</li>
  <li style="margin-bottom: 0.5em;">We maintain a buffer by targeting 70% utilization—this keeps 30% of provisioned Lambdas in reserve for surges.</li>
  <li style="margin-bottom: 0.5em;">Scaling thresholds are evaluated over a 3-minute trailing window to smooth out short-term spikes.</li>
</ul>



Instead of manually maintaining 100 Lambdas 24/7, we now start with a baseline and **let the system scale intelligently**. This configuration is version-controlled, reproducible, and deployable through code.

## Key Benefits of Lambda Auto Scaling

This technical refactor of our client's serverless architecture solved real business problems and alleviated serious developer painpoints, all while enabling their application to scale further.

In summary, the core benefits were that now their application had:

<ol style="list-style-type: decimal; padding-left: 1.2em;">
  <li style="margin-bottom: 1.2em;">
    <strong>Reduced Cold Start Latency</strong><br>
    With warm Lambdas available before traffic arrives, users no longer experienced delays during onboarding events or high-concurrency scenarios.
  </li>

  <li style="margin-bottom: 1.2em;">
    <strong>Lower AWS Costs Without Sacrificing Performance</strong><br>
    We eliminated the need to keep high concurrency levels running 24/7. The system automatically scales down during quiet periods, cutting unnecessary spend.
  </li>

  <li style="margin-bottom: 1.2em;">
    <strong>Automation Replaced Human Guesswork</strong><br>
    No more manual adjustments to concurrency settings. CloudWatch and Application Auto Scaling now handle scaling based on real usage, reducing operational overhead and human error.
  </li>

  <li style="margin-bottom: 1.2em;">
    <strong>Infrastructure That Grows with Demand</strong><br>
    The system now scales with actual traffic patterns. Whether we have 10 users or 10,000, Lambda concurrency adjusts dynamically to meet demand.
  </li>
</ol>

<a href="/assets/blog/2025-04-30-how-we-solved-aws-lambda-scaling-with-auto-scaling-and-provisioned-concurrency/PerformanceComparison.png" target="_blank" rel="noopener noreferrer">
  <img src="/assets/blog/2025-04-30-how-we-solved-aws-lambda-scaling-with-auto-scaling-and-provisioned-concurrency/PerformanceComparison.png" alt="Lambda Scaling: Before vs After Auto Scaling" style="max-width: 100%; height: auto; border-radius: 10px;" />
</a>


## Tips for Implementing Auto Scaling with the Serverless Framework

If you’re looking to implement something similar, here are some key recommendations:
<ul style="list-style: disc; padding-left: 1.2em;">
  <li style="margin-bottom: 1em;">
    <strong>Use the Serverless Framework</strong> with the <code>serverless-provisioned-concurrency-autoscaling</code> plugin.
  </li>

  <li style="margin-bottom: 1em;">
    <strong>Define your min/max concurrency values</strong> clearly to prevent over- or under-scaling.
  </li>

  <li style="margin-bottom: 1em;">
    <strong>Monitor with CloudWatch</strong> and adjust based on real traffic, not estimates.
  </li>

  <li style="margin-bottom: 1em;">
    <strong>Tune your buffer and utilization targets</strong> based on observed load patterns.
  </li>
</ul>



## Final Thoughts: Smarter Scaling, Lower Costs, and Better Performance

By combining **AWS Lambda provisioned concurrency** with **auto scaling**, and isolating our environments, we created a system that is more resilient, cost-effective, and scalable.

If your team is manually adjusting concurrency or struggling with cold start latency, this pattern could save you time, money, and headaches.

Want a more detailed walkthrough or help applying this in your stack? Get in touch <a href="https://www.olioapps.com/contact-us" style="color: blue; text-decoration: underline;" target="_blank" rel="noopener noreferrer">here.</a>

## Resources Mentioned
<ol style="list-style-type: decimal; padding-left: 1.5em; margin: 1em 0;">
  <li><a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">AWS Lambda</a></li>
  <li><a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Lambda Provisioned Concurrency</a></li>
  <li><a href="https://docs.aws.amazon.com/autoscaling/application/userguide/what-is-application-auto-scaling.html" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">AWS Application Autoscaling</a></li>
  <li><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">CloudWatch Alarms</a></li>
  <li><a href="https://www.serverless.com/plugins/serverless-plugin-optimize" style="text-decoration: underline;" target="_blank" rel="noopener noreferrer">Serverless Provisioned Concurrency Plugin</a></li>
</ol>


In this guide, we’ll walk through how our team tackled these challenges by restructuring our AWS environments and implementing Lambda auto scaling with provisioned concurrency—a move that significantly improved both performance and cost-efficiency.

How We Solved AWS Lambda Scalaing with Auto Scaling and Provisioned Concurrency


As a software developer with little background in design and color theory, you might find yourself tasked with creating visually appealing web pages. Without a solid foundation in design principles, especially color theory, this can seem daunting. You might wonder how to quickly and effectively choose colors that create a cohesive and aesthetically pleasing design. I'll explore how I used AI, specifically ChatGPT-4o, to solve this problem by leveraging color theory principles to generate color schemes mathematically.

As a developer with a background in art, I've always relied on traditional watercolor theory to create visually striking color palettes. The theory is simple: take your primary color, find its contrasting color, and mix the two to get a neutral tone. This neutral tone helps other colors pop on the canvas or page. However, applying this theory to website design can be challenging, especially for those without an art background. I wondered if there was a mathematical way to reproduce these watercolor results using hex codes, making the process more accessible for everyone.

I started with a watercolor study, using an orange as my primary color and blue as its contrasting color. By mixing these two, I obtained a neutral tone. I scanned these watercolor samples and used an eye dropper tool to extract their hex codes. With these hex codes, I styled a dummy app, incorporating the primary color, complementary color, and neutral tones for text and drop shadows.
<img src="/assets/blog/2024-09-09-utilizing-LLMs-to-mathematically-determine-color-themes-for-websites/Pallette.png" alt="Colors" style="display: block; max-width: 100%;">
<img src="/assets/blog/2024-09-09-utilizing-LLMs-to-mathematically-determine-color-themes-for-websites/Headers.png" alt="Headers" style="display: block; max-width: 100%;">

Next, I turned to ChatGPT. I provided it with the hex codes: the starting color (orange), the contrasting color (blue), and the neutral tone obtained from my watercolor study. My goal was to see if ChatGPT could replicate this process mathematically.

Initially, ChatGPT calculated the contrasting color and the neutral tone. However, the neutral tone was always a perfect gray (hex code: #888888), which didn't match the nuanced neutral tones I observed in my watercolor samples. I realized that the mathematical approach needed refinement to account for the blending of colors in a more realistic manner, since with this mathematical approach the exact contrasting color was achieved every time, which, in color theory, means that the neutral tone will always be a perfectly neutral gray.

I queried ChatGPT to explore if using light filters or different blending techniques could achieve a more accurate neutral tone. This led to the discovery of alpha blending, which involves overlaying colors with varying transparency levels. By applying a 60% filter for the primary color and a 40% filter for the contrasting color, I achieved a neutral tone that leaned more towards the primary color, mimicking the watercolor effect.

<img src="/assets/blog/2024-09-09-utilizing-LLMs-to-mathematically-determine-color-themes-for-websites/equation.png" alt="equation" style="display: block; max-width: 100%;">

With this refined approach, I developed an equation where I input a single color, and ChatGPT calculates the complementary and neutral tones. This method allows me to create harmonious color palettes mathematically, making the process accessible to those without an art background. For example, in a scenario where a client's logo color is orange, I can now generate a visually appealing color scheme for their website with ease.

<img src="/assets/blog/2024-09-09-utilizing-LLMs-to-mathematically-determine-color-themes-for-websites/colors.png" alt="equation" style="display: block; max-width: 50%;">
<img src="/assets/blog/2024-09-09-utilizing-LLMs-to-mathematically-determine-color-themes-for-websites/Headers2.png" alt="Headers2" style="display: block; max-width: 100%;">

Traditionally, determining color themes relied heavily on artistic intuition and manual blending techniques. While effective, this approach can be challenging for those without a strong art background. By leveraging AI and mathematical principles, we can automate and simplify this process, making it accessible to developers without design teams or time to get a Master's in Fine Art. This new approach not only saves time but also ensures consistency and accuracy in color theme generation for websites. And it's fun!



You might wonder how to quickly and effectively choose colors that create a cohesive and aesthetically pleasing design. I'll explore how I used AI, specifically ChatGPT-4o, to solve this problem by leveraging color theory principles to generate color schemes mathematically.

Using AI To Create Website Color Themes


At Olio Apps we understand the challenges organizations adopting cloud for the first time or experimenting with serverless development run into when it comes to designing and deploying scalable and cost effective architecture on AWS.

Seldom does an organization have the time and resources to commit to learning a new tech stack while developing features.

At Olio Apps we regularly support clients who are in need of talent specialized in specific AWS services who can disperse that knowledge throughout the organization.

This is why we support our team members in developing their knowledge of cloud and serverless best practices when it comes to AWS Infrastructure.

Today, we are excited to announce and celebrate Aaron Whitaker’s recent AWS Solutions Architect Certification.

The AWS Solutions Architect Certification validates proficiency in designing and deploying scalable and cost-efficient cloud solutions on AWS. Individuals with this expertise are able to tailor AWS infrastructure to an organization’s needs while optimizing performance and cost-effectiveness. Additionally, those who achieve AWS Certification in general can be assured they are up to date with Amazon’s best practices and innovation standards.

## AWS Solutions Architect Associate Bolsters Technical Abilities

Aaron found that the process of studying and learning for the AWS Solutions Architect Certification exam reinforced his practical knowledge while exposing him to additional best practices:

> Getting the solutions architect certification helped to reinforce the knowledge I’ve gained through practical experience working in AWS for the past two years. It also gave me more insight into best practices and patterns I hadn’t previously known about. It's given me more confidence to design well-architected cloud systems

Aaron is a Fullstack Engineer at Olio Apps who specializes in fullstack and serverless applications. You can learn more about the team at Olio Apps <a href="https://www.olioapps.com/team">here.</a>

Congratulations again Aaron, we are excited to see how this new knowledge from the AWS Solutions Architect Certification encourages your further learning.


At Olio Apps we acknowledge that Amazon Web Services (AWS) is an industry leader when it comes to providing reliable and scalable cloud services for companies across all domains.

Celebrating Aaron Whitaker’s AWS Solutions Architect Certification


At Olio Apps EventBridge has become a huge topic of conversation ever since we’ve been using it with one of our clients. Initially, our client’s architecture followed a synchronous inline model where you give the server a request and then it does A, B, C, D, and E.

As the app continues to grow, it becomes harder to manage all of these different threads and if-thens. So we've shifted to utilize AWS event-driven architecture, a first for this client. Initially, we integrated with SQS queues (Amazon Simple Queuing Service) to manage asynchronous job processing. Since then we have begun to shift more and more to event driven architecture patterns.

## Facilitating Event Driven Architecture Evolution using Amazon EventBridge

Some of you may recognize this paradigm. Initially, we needed to trigger notifications based on specific factors like the day of the week, how long it had been since a certain event happened, or a user’s relationship to one another.

So we began to utilize a serverless event bus, Amazon EventBridge, to listen, schedule, & trigger these events. EventBridge Listener would wait for specific events to enter the event bus. Once they did, this would trigger a corresponding lambda function to take the event to an SQS queue where it would then be processed asynchronously.

The above description is essentially the base use case described by AWS’ official illustrations. This image shows the event bus and the different events being pumped into it from specific sources.

<img src="/assets/blog/2024-09-04-facilitating-event-driven-architecture-evolution-using-amazon-eventbridge/eventbridgediagram.png" alt="EventBridge Diagram" style="display: block; max-width: 100%;">

And as stated in <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-service-event-list.html">AWS’ documentation</a> EventBridge can handle input from a variety of AWS sources or from a custom event bus.

In particular, we provisioned our AWS resources using the serverless framework and used a combination of EventBridge’s rules, event matching, and routing capabilities to enable a user experience informed by the context of these specific events.

## Value of Eventbridge

In summary this is how EventBridge functions as we have events flowing into the pipeline with the following orchestration.

**Rules**: We define rules within EventBridge that act as event filters. These rules can listen for specific events, like "user created" or "user updated."

**Event Matching**: Each rule is configured with an event pattern. This pattern specifies the data characteristics and the rule it is looking for. For instance, a rule might match events where the "detail-type" property equals "user-updated" and the "source" property equals "staging-API."

**Routing Events**: When an event matches a rule's pattern, EventBridge routes it to a designated target. This target can be a Lambda function that we've previously defined to handle the event.

Imagine a rule named "user-updated-event-bridge-rule." This rule intercepts events flowing through the pipeline, searching for events that match its criteria (detail-type="user-updated" and source="staging-API").
If an event doesn't match, it's simply passed along. However, if a match occurs, the event is directed to the Lambda function associated with that rule. This Lambda function executes the desired logic based on the received event data.

A major reason EventBridge has been so valuable for our client’s event driven architecture is because of EventBridge’s robust flexibility, including:

**Multiple Rules per Event**: A single event can be matched by multiple rules, allowing for diverse processing paths.

**Multiple Targets per Rule**: Each rule can route events to up to five different targets, enabling parallel processing or sending events to various services.

**Event Data**: We can include any relevant data within the event itself. This allows Lambda functions to perform actions based on the differences between old and new data (e.g., detecting a user's change in profile information).

This ability to define custom rules and targets empowered us to make intricate event workflows that fit our client’s needs. Currently, we have Lambda functions that trigger additional events based on the processed data, sending them back into the EventBridge pipeline for further processing of 3rd party APIs. This is one example where we are able to leverage the vast capabilities of EventBridge for enabling event driven architecture. for building sophisticated event-driven architectures using EventBridge.

Overall, this EventBridge architecture has allowed us to offload all these semi-heavy computational side effects when there is a profile update to the cloud. So we're not taking a bunch of performance hits by running everything in line on individual requests.

We hope you enjoyed our discussion on how Olio Apps is utilizing EventBridge to meet customer needs using event driven architecture on AWS. If you have any questions about using EventBridge yourself or have any questions about how your application may be able to utilize EventBridge, please don’t hesitate to contact us.


At Olio Apps, EventBridge has become a huge topic of conversation ever since we’ve been using it with one of our clients. As the app continues to grow in complexity and perform more complex functions, it becomes harder to manage all of these different threads and if-thens

Reinforced Learning