Advent Calendar Day 6: How AI Agents Offer Style Suggestions

This December, I'm showing how naive chatbots fail and how AI agents make shopping easier.

We've seen naive chatbots fail at clarifying questions, context shifts, numerical requirements, multiple requests in one query and price filters.

Today, the customer wants shoes that match a blue dress. The naive chatbot just returns random blue shoes. The AI agent asks a clarifying question, then finds shoes that fit the style request.

Source Code

For a deeper dive into this example, check out my GitHub repository for a detailed, step-by-step implementation in Jupyter Notebooks, breaking down the code and methodology used by the AI agent in today's scenario.

Introducing SoleMates

SoleMates is our fictional online shoe store we'll use to illustrate these concepts:

SoleMates is our fictional online shoe store

Today's Challenge: Style Coordination

Scenario

A customer asks our SoleMates chatbot:

Customer: "I need shoes to match a blue dress":

A customer initiates a chat with SoleMates and asks for shoes matching a blue dress

Naive Chatbot Response

The naive chatbot turns the query into a vector and searches for shoes mentioning "blue."

It returns shoes in various styles, even men's shoes:

The naive chatbot pulls random blue shoes from the vector database, including men's sports shoes and unrelated options

Because it doesn't ask about style or shoe type, it just pulls anything blue and replies.

Naive Chatbot Says: "For a blue dress, you might consider Skechers men blue shoes or Nike women zoo blue shoe."

The naive chatbot pulls anything blue and replies

This does not consider that a customer might want women's flats or heels to go with a dress, or might care about style. The chatbot just picks items with the color blue.

Why Did the Naive Chatbot Fail?

• No clarifying question
• No style reasoning
• Treats user request as a simple color match

Limitations Highlighted

• No attempt to understand the type of shoes desired
• No follow-up to narrow down style or occasion
• Results that do not meet the customer's needs

AI Agent Solution

The AI agent sees "shoes to match a blue dress" and realizes this might need clarification.

Therefore, it asks the user what kind of shoes they prefer:

The AI agent asking the user to clarify: "Could you please specify if you are looking for a particular style or type of shoe?"

Agent's Interaction:

1. Customer: "I need shoes to match a blue dress"
2. Agent: "Could you please specify if you are looking for a particular style or type of shoe, such as heels, flats, or sandals? This will help me find the best options for you."
3. Customer: "I like flats"
4. Agent: "Here are some blue flats that would match your dress:

The AI agent proactively asks the customer for more information to ensure accurate recommendations

How Did the AI Agent Succeed?

• Asks clarifying question to understand the user's style preference
• Applies metadata filtering after learning the style (flats) and color (blue)
• Delivers options that actually match the user's outfit and preferences

Key Takeaways

Naive Chatbot Limitation

• Offers random matches without clarification

AI Agent Advantages

• Asks follow-up questions
• Personalizes the search based on user feedback
• Provides relevant options that suit the requested style

Conclusion

By asking a simple clarifying question, the AI agent turns a vague request into a precise one.

Instead of random blue shoes, it finds blue flats that match a dress. Naive chatbots fail because they treat the request as static text, while AI agents adapt and refine their search.

About This Series

Each day in December, I show a scenario where naive chatbots fall short, and how AI agents fix the issue. Follow along to learn how to build better, more user-friendly AI systems.

Coming Up: Understanding Context Shifts

In tomorrow's issue, we'll explore How AI Agents Handle Unavailable Colors Gracefully

Learn to Build Your Own AI Agent Chatbot

I'm preparing a course to teach you how to build and deploy your own AI agent chatbot. Sign up here!

---

Behind the Scenes: Code Snippets

Here's a simplified illustration of how the AI agent processes the query.

We're giving the agent access to two tools it can use freely:

1. Vector database metadata filtering
2. Vector database query

1. Vector database metadata filtering tool

This tool lets the agent break down user queries into multiple filters:

def create_metadata_filter(filter_string):
    
    # For "blue flats":
    # Returns: 
    # [{"key": "color", "value": "blue", "operator": "=="}]

    filters = parse_filters(filter_string)
    return filters

2. Vector database query

def search_footwear_database(query_str, filters_json):
    embedded_query = embed_query_aws_titan(query_str)
    results = vector_db.search(embedded_query, filters=filters_json)
    return results

AWS Titan Multimodal Embeddings in Action

I use AWS Titan, a multimodal embedding model that converts both product texts and images into vectors, integrated as the function embed_query_aws_titan into the AI agent tool search_footwear_database.

This means the AI agent can process a query like "red heels" and match it to not only product descriptions but also actual images of red heels in the database.

By combining text and image data, the model helps the AI agent provide more relevant and visually aligned recommendations based on the customer's input.

Agent workflow

Construct agent worker with access to the two tools and initiate the agent:

from llama_index.core.agent import AgentRunner, FunctionCallingAgentWorker

agent_worker = FunctionCallingAgentWorker.from_tools(
    [
        create_metadata_filters_tool,
        query_vector_database_tool,
    ])

agent = AgentRunner(
    agent_worker,
)

Initiate conversation with agent:

agent.chat("I need shoes to match a blue dress")

Agent workflow:

1. Agent realizes it needs more details and asks:

• "Could you please specify if you are looking for a particular style?"

2. User says: "I like flats"
3. Agent calls create_metadata_filter("blue flats")
4. Gets filters:

[
    {"key": "color", "value": "blue", "operator": "=="}
]

5. Agent calls search_footwear_database("flats", filters)
6. Finds blue flats that match the dress
7. Returns them to user:

The AI agent proactively asks the customer for more information to ensure accurate recommendations

Here's the full agent output:

Added user message to memory: I need shoes to match a blue dress
=== LLM Response ===
Could you please specify if you are looking for a particular style or type of shoe, such as heels, flats, or sandals? This will help me find the best options for you.
Added user message to memory: I like flats
=== Calling Function ===
Calling function: create_metadata_filter with args: {"filter_string": "blue flats"}
=== Function Output ===
[{"key": "color", "value": "blue", "operator": "=="}]
=== Calling Function ===
Calling function: search_footwear_database with args: {"query_str": "flats", "filters_json": [{"key": "color", "value": "blue", "operator": "=="}]}
=== Function Output ===
[{'color': 'blue', 'text': 'Catwalk women casual blue flats', 'gender': 'women', 'product_type': 'flats', 'product_id': 37, 'usage': 'casual', 'price': 90, 'similarity_score': 1.68500006}, {'color': 'blue', 'text': 'Catwalk women blue casual flats', 'gender': 'women', 'product_type': 'flats', 'product_id': 39, 'usage': 'casual', 'price': 180, 'similarity_score': 1.68423748}, {'color': 'blue', 'text': 'Catwalk women blue casual flats', 'gender': 'women', 'product_type': 'flats', 'product_id': 41, 'usage': 'casual', 'price': 95, 'similarity_score': 1.68423748}, {'color': 'blue', 'text': 'Nike women ten blue white shoe', 'gender': 'women', 'product_type': 'casual shoes', 'product_id': 44, 'usage': 'sports', 'price': 65, 'similarity_score': 1.20576775}, {'color': 'blue', 'text': "Adidas men's canon dialect blue yellow shoe", 'gender': 'men', 'product_type': 'sports shoes', 'product_id': 31, 'usage': 'sports', 'price': 55, 'similarity_score': 1.19341826}, {'color': 'blue', 'text': 'Nike women zoo blue shoe', 'gender': 'women', 'product_type': 'sports shoes', 'product_id': 43, 'usage': 'sports', 'price': 85, 'similarity_score': 1.1900245}, {'color': 'blue', 'text': 'Skechers men blue shoes', 'gender': 'men', 'product_type': 'casual shoes', 'product_id': 34, 'usage': 'casual', 'price': 195, 'similarity_score': 1.17973208}, {'color': 'blue', 'text': 'Cobblerz blue womens wedges', 'gender': 'women', 'product_type': 'heels', 'product_id': 38, 'usage': 'casual', 'price': 110, 'similarity_score': 1.17023802}, {'color': 'blue', 'text': 'Adidas women clima motionwave blue sports shoes', 'gender': 'women', 'product_type': 'sports shoes', 'product_id': 42, 'usage': 'sports', 'price': 200, 'similarity_score': 1.16630864}, {'color': 'blue', 'text': 'Puma men faas 300 blue sports shoes', 'gender': 'men', 'product_type': 'sports shoes', 'product_id': 32, 'usage': 'sports', 'price': 200, 'similarity_score': 1.16353559}]
=== LLM Response ===
Here are some blue flats that would match your dress:

1. **Catwalk Women Casual Blue Flats**
   - Price: $90
   - Style: Casual

2. **Catwalk Women Blue Casual Flats**
   - Price: $180
   - Style: Casual

3. **Catwalk Women Blue Casual Flats**
   - Price: $95
   - Style: Casual

These options should complement your blue dress nicely. Let me know if you need more information or if you'd like to explore other styles!

Additional Resources

For a deeper dive into this example, check out my GitHub repository where I break down the code and methodology used by the AI agent in today's scenario.

Learn to Build Your Own AI Agent Chatbot

Want to build your own AI agent that applies price filters? Sign up here and learn how to create smarter chatbots.

Why use AI agents instead of simple chatbots?

If you need an AI chatbot that can handle price filters, numeric reasoning, and metadata-based product searches, AI agents can help.

By using retrieval augmented generation and product metadata, you can build chatbots that understand budget limits and return products that fit your price range.

This improves the shopping experience and helps users find what they want quickly.