Advent Calendar Day 7: How AI Agents Handle Unavailable Colors Gracefully

This December, I'm showing how naive chatbots fail and how AI agents make shopping easier.

We've seen naive chatbots fail at clarifying questions, context shifts, numerical requirements, multiple requests in one query, price filters and style suggestions.

Today, the customer wants shoes in burgundy or maroon. The naive chatbot finds nothing and just says no. The AI agent, however, asks for details, then shows similar colors that might appeal to the customer.

Source Code

For a deeper dive into this example, check out my GitHub repository for a detailed, step-by-step implementation in Jupyter Notebooks, breaking down the code and methodology used by the AI agent in today's scenario.

Introducing SoleMates

SoleMates is our fictional online shoe store we'll use to illustrate these concepts:

SoleMates is our fictional online shoe store

Today's Challenge: Unavailable Colors

Scenario

Customer: "I'm looking for shoes in burgundy or maroon":

A customer initiates a chat with SoleMates and asks for shoes matching a blue dress

Naive Chatbot Response

The naive chatbot turns the query into a vector and searches:

The naive chatbot tries to pull maroon and burgundy shoes from the vector database

Unfortunately, it cannot find burgundy or maroon because these colors do not exist in the product database:

Neither burgundy nor maroon colors are available in the product database

It ends up saying:

Naive Chatbot: "There are no shoes available in burgundy or maroon":

Naive chatbot replying with "There are no shoes available in burgundy or maroon" and showing no suggestions

This leaves the customer with no help. The chatbot does not offer any alternatives or ask any questions.

Why Did the Naive Chatbot Fail?

• Treats the request as a strict color match only
• Does not attempt to suggest similar colors or styles
• No follow-up to understand what the customer might consider as a close alternative

Limitations Highlighted

• No reasoning about color variants
• No willingness to help beyond exact matches
• No guided fallback to related products

AI Agent Solution

The AI agent tries a different approach.

Instead of immediately look for burgundy or maroon shoes, it asks for more details first (like gender, style, or occasion) to refine the search:

The AI agent asking the user to clarify: "Could you please specify if you are looking for men's or women's shoes, and if you have any specific style or occasion in mind?"

Agent's Interaction:

1. Customer: "I'm looking for shoes in burgundy or maroon"
2. Agent: "Could you please specify if you are looking for men's or women's shoes, and if you have any specific style or occasion in mind?"
3. Customer: "Women's shoes"
4. Agent: Sees there are no burgundy or maroon women's shoes
5. Agent suggests brown or red shoes that might appeal to the customer:

The AI agent listing red and brown shoes as alternatives when burgundy/maroon are unavailable

After learning the customer wants women's shoes, it checks the database for similar options, like red or brown, and suggests them as possible alternatives.

How Did the AI Agent Succeed?

• Asks clarifying questions to gather more info
• When exact match is not found, suggests similar colors
• Tries to help the user find something they might like, instead of simply giving up

Key Takeaways

Naive Chatbot Limitation

• Says "no" and ends the conversation

AI Agent Advantages

• Provides a fallback solution when exact requests are unavailable
• Asks follow-up questions to understand the customer's needs
• Suggests related products and encourages further interaction

Conclusion

When a customer wants something not available, a naive chatbot offers no help.

The AI agent, on the other hand, tries to guide the customer to similar products, ensuring a better experience.

This shows how AI agents can handle tricky situations gracefully, improving customer satisfaction.

About This Series

Each day in December, I show a scenario where naive chatbots fail, and how AI agents solve the problem.

Follow along to learn how to build chatbots that understand users and offer helpful solutions.

Coming Up: Understanding Context Shifts

In tomorrow's issue, we'll explore How AI Agents Handle Negations in Queries

Learn to Build Your Own AI Agent Chatbot

I'm preparing a course to teach you how to build and deploy your own AI agent chatbot. Sign up here!

---

Behind the Scenes: Code Snippets

Here's a simplified illustration of how the AI agent processes the query.

We're giving the agent access to two tools it can use freely:

1. Vector database metadata filtering
2. Vector database query

1. Vector database metadata filtering tool

This tool lets the agent break down user queries into multiple filters:

def create_metadata_filter(filter_string):
    # For "burgundy or maroon women's shoes":
    # Since burgundy or maroon are not known colors, the filter might just return 
    # [{"key": "gender", "value": "women", "operator": "=="}]
    filters = parse_filters(filter_string)
    return filters

2. Vector database query

def search_footwear_database(query_str, filters_json):
    embedded_query = embed_query_aws_titan(query_str)
    results = vector_db.search(embedded_query, filters=filters_json)
    return results

AWS Titan Multimodal Embeddings in Action

I use AWS Titan, a multimodal embedding model that converts both product texts and images into vectors, integrated as the function embed_query_aws_titan into the AI agent tool search_footwear_database.

This means the AI agent can process a query like "red heels" and match it to not only product descriptions but also actual images of red heels in the database.

By combining text and image data, the model helps the AI agent provide more relevant and visually aligned recommendations based on the customer's input.

Agent workflow

Construct agent worker with access to the two tools and initiate the agent:

from llama_index.core.agent import AgentRunner, FunctionCallingAgentWorker

agent_worker = FunctionCallingAgentWorker.from_tools(
    [
        create_metadata_filters_tool,
        query_vector_database_tool,
    ])

agent = AgentRunner(
    agent_worker,
)

Initiate conversation with agent:

agent.chat("I'm looking for shoes in burgundy or maroon")

Agent workflow:

1. Agent realizes it needs more details and asks:
   • "Could you please specify if you are looking for men's or women's shoes, and if you have any specific style or occasion in mind?"
2. User says: "Women's shoes"
3. Agent calls create_metadata_filter("burgundy or maroon women's shoes")
4. Gets filter:

[
    {"key": "gender", "value": "women", "operator": "=="}
]

5. Agent calls search_footwear_database("burgundy or maroon shoes", filters)
6. No exact match for burgundy/maroon
7. Agent suggests brown or red women's shoes:

The AI agent listing red and brown shoes as close matches to burgundy or maroon

Here's the full agent workflow output:

Added user message to memory: I'm looking for shoes in burgundy or maroon
=== LLM Response ===
Could you please specify if you are looking for men's or women's shoes, and if you have any specific style or occasion in mind? This will help me narrow down the options for you.
Added user message to memory: Women's shoes
=== Calling Function ===
Calling function: create_metadata_filter with args: {"filter_string": "burgundy or maroon women's shoes"}
=== Function Output ===
[{"key": "gender", "value": "women", "operator": "=="}]
=== Calling Function ===
Calling function: search_footwear_database with args: {"query_str": "burgundy or maroon shoes", "filters_json": [{"key": "gender", "value": "women", "operator": "=="}]}
=== Function Output ===
[{'color': 'brown', 'text': 'Hm women brown shoes', 'gender': 'women', 'product_type': 'flats', 'product_id': 81, 'usage': 'casual', 'price': 155, 'similarity_score': 1.74298072}, {'color': 'red', 'text': 'Catwalk women red shoes', 'gender': 'women', 'product_type': 'casual shoes', 'product_id': 54, 'usage': 'casual', 'price': 75, 'similarity_score': 1.71854472}, {'color': 'brown', 'text': 'Gliders women brown shoes', 'gender': 'women', 'product_type': 'casual shoes', 'product_id': 80, 'usage': 'casual', 'price': 75, 'similarity_score': 1.71579671}, {'color': 'red', 'text': 'Carlton london women casual red casual shoes', 'gender': 'women', 'product_type': 'casual shoes', 'product_id': 53, 'usage': 'casual', 'price': 90, 'similarity_score': 1.71017599}, {'color': 'multi', 'text': 'Rocia women multi- coloured shoes', 'gender': 'women', 'product_type': 'flats', 'product_id': 69, 'usage': 'casual', 'price': 130, 'similarity_score': 1.70238113}, {'color': 'black', 'text': 'Skechers women black casual shoes', 'gender': 'women', 'product_type': 'casual shoes', 'product_id': 11, 'usage': 'casual', 'price': 185, 'similarity_score': 1.68744576}, {'color': 'black', 'text': 'Reebok women black casual shoes', 'gender': 'women', 'product_type': 'flats', 'product_id': 13, 'usage': 'casual', 'price': 85, 'similarity_score': 1.68109059}, {'color': 'white', 'text': 'Adidas women white bolt sports shoes', 'gender': 'women', 'product_type': 'sports shoes', 'product_id': 29, 'usage': 'sports', 'price': 185, 'similarity_score': 1.67744803}, {'color': 'turquoise blue', 'text': 'Catwalk women turquoise casual shoes', 'gender': 'women', 'product_type': 'casual shoes', 'product_id': 40, 'usage': 'casual', 'price': 65, 'similarity_score': 1.66859019}, {'color': 'black', 'text': 'Nike women flyclave black casual shoes', 'gender': 'women', 'product_type': 'casual shoes', 'product_id': 12, 'usage': 'casual', 'price': 175, 'similarity_score': 1.66677618}]
=== LLM Response ===
I couldn't find any specific burgundy or maroon women's shoes in the database. However, here are some options that might interest you:

1. **Hm Women Brown Shoes**
   - Color: Brown
   - Type: Flats
   - Usage: Casual
   - Price: $155

2. **Catwalk Women Red Shoes**
   - Color: Red
   - Type: Casual Shoes
   - Usage: Casual
   - Price: $75

3. **Gliders Women Brown Shoes**
   - Color: Brown
   - Type: Casual Shoes
   - Usage: Casual
   - Price: $75

4. **Carlton London Women Casual Red Shoes**
   - Color: Red
   - Type: Casual Shoes
   - Usage: Casual
   - Price: $90

If you have any other preferences or need further assistance, feel free to let me know!

Additional Resources

For a deeper dive into this example, check out my GitHub repository where I break down the code and methodology used by the AI agent in today's scenario.

Learn to Build Your Own AI Agent Chatbot

Want to build your own AI agent that applies price filters? Sign up here and learn how to create smarter chatbots.

Why use AI agents instead of simple chatbots?

If you're building AI chatbots for e-commerce, consider AI agents that handle unavailable requests with grace.

By using retrieval augmented generation (RAG) and metadata filtering, AI agents can suggest alternatives when exact matches are not found.

This improves customer satisfaction, reduces frustration, and increases the chance of a sale.