Advent Calendar Day 3: How AI Agents Handle Requests for Specific Measurements

This December, I'm showing how naive chatbots fail when customers ask for details and how AI agents step up.

Previously, we saw how naive chatbots fail at clarifying questions and context shifts.

Today, we show how they struggle with requests involving numbers. A customer might ask for shoes with a heel height over a certain number. A naive chatbot will ignore the number and just show all heels, while an AI agent will apply the correct filter.

Source Code

For a deeper dive into this example, check out my GitHub repository for a detailed, step-by-step implementation in Jupyter Notebooks, breaking down the code and methodology used by the AI agent in today's scenario.

Introducing SoleMates

SoleMates is our fictional online shoe store we'll use to illustrate these concepts:

SoleMates is our fictional online shoe store

Today's Challenge: Failure to Adapt After Context Shift

Scenario

A customer asks our Solemates chatbot:

Customer: "I need women's heels with a heel height over 2 inches":

A customer initiates a chat with SoleMates and asks about casual women's shoes

Naive Chatbot Response

The naive chatbot just turns the user's request into a vector and searches without any numeric filtering.

While it finds a list of women's heels, there's no guarantee it will include those over 2 inches because it doesn't apply any numerical constraints - only semantic similarity.

So the vector database pull consists of various heels, but none with heel height over 2 inches:

The naive chatbot's search doesn't enforce a heel height filter, so it's not guaranteed to return heels over 2 inches

So the simple chatbot ends up replying "There are no women's heels with a heel height higher than 2 inches available":

Because a simple chatbot can't handle the numerical request, it ends up saying that there are none with heel height over 2 inches

Why Did the Naive Chatbot Fail?

• Ignores Numerical Details: Doesn't understand the "over 2 inches" part
• No Fine-Grained Search: Treats the entire query as text and doesn't filter on heel height
• Inaccurate Results: Misses products that actually meet the requirement

Limitations Highlighted

• No Numerical Reasoning: Fails to apply filters based on numbers
• No Metadata Usage: Doesn't know how to use product metadata like heel height to refine results

AI Agent Solution

The AI agent keeps track of all product metadata. When the user asks for women's heels over 2 inches, the agent extracts "women," "heels," and "heel height > 2 inches" from the request. It then applies a filter to the vector database query.

Agent's Interaction:

1. Customer: "I need women's heels with a heel height over 2 inches"
2. Agent: "Understood. Let me apply a filter for women's heels over 2 inches"

The agent finds a product that matches exactly:

The agent decided to apply a filter to the vector database query and found red stilettos with a 3-inch heel

How Did the AI Agent Succeed?

• Metadata Filtering: Uses product metadata for gender, product type, and heel height
• Numeric Reasoning: Correctly applies "greater than 2" to find the right product
• Relevant Results: Returns the perfect match the user asked for

Key Takeaways

Naive Chatbot Limitation

• Treats the user's message as plain text and ignores numbers

AI Agent Advantages

• Understands Numeric Requests: Applies numeric filters to refine results
• Uses Metadata: Leverages product attributes like heel height
• Delivers Targeted Results: Finds products that match the user's exact needs

Conclusion

This example shows the importance of handling numeric details. A naive chatbot fails to recognize and apply a simple numeric filter. An AI agent, however, uses metadata and numeric reasoning to deliver accurate results.

Stay tuned for tomorrow's issue, where we'll explore another scenario in which AI agents excel over naive chatbots.

About This Series

Each day in December, I show a different weakness of naive RAG chatbots and how AI agents solve it. By understanding these differences, you can build smarter, more helpful AI systems.

Learn to Build Your Own AI Agent Chatbot

I'm preparing a course to teach you how to build and deploy your own AI agent chatbot. Sign up here!

Coming Up: Handle Multiple Product Requests

In tomorrow's issue, we'll explore how AI Agents Handle Multiple Product Requests in One Query

---

Behind the Scenes: Code Snippets

Here's a simplified illustration of how the AI agent processes the query.

We're giving the agent access to two tools it can use freely:

1. Vector database metadata filtering
2. Vector database query

1. Vector database metadata filtering tool

Allows the agent to create filters based on available metadata (e.g., gender, product type, usage).

The agent uses a tool to create metadata filters. Now it also handles numeric comparisons:

def create_metadata_filter(filter_string):
    # Example filter string: "women's heels heel height higher than 2 inches"
    # Parses and returns:
    # [{"key": "gender", "value": "women", "operator": "=="}, {"key": "heels_height", "value": 2, "operator": ">"}]

    filters = parse_filters(filter_string)
    return filters

2. Vector database query

Once filters are established, the agent queries the vector database using the numeric constraint:

def search_footwear_database(query_str, filters_json):

    # Embeds the query string and searches the vector database with filters
    embedded_query = embed_query_aws_titan(query_str)
    results = vector_db.search(embedded_query, filters=filters_json)

    return results

AWS Titan Multimodal Embeddings in Action

I use AWS Titan, a multimodal embedding model that converts both product texts and images into vectors, integrated as the function embed_query_aws_titan into the AI agent tool search_footwear_database.

This means the AI agent can process a query like "red heels" and match it to not only product descriptions but also actual images of red heels in the database.

By combining text and image data, the model helps the AI agent provide more relevant and visually aligned recommendations based on the customer's input.

Agent workflow

Construct agent worker with access to the two tools and initiate the agent:

from llama_index.core.agent import AgentRunner, FunctionCallingAgentWorker

agent_worker = FunctionCallingAgentWorker.from_tools(
    [
        create_metadata_filters_tool,
        query_vector_database_tool,
    ])

agent = AgentRunner(
    agent_worker,
)

Initiate conversation with agent:

agent.chat("I need women's heels with a heel height over 2 inches")

Agent:

1. Calls create_metadata_filter with "women's heels heel height higher than 2 inches"
2. Returns filters:

[
    {"key": "gender", "value": "women", "operator": "=="},
    {"key": "heels_height", "value": 2, "operator": ">"}
]

3. Calls search_footwear_database(query="shoes", filters=above_filters)
4. Finds Carlton London Women Casual Red Stilettos with a 3-inch heel
5. Replies with the correct product:

The agent decided to apply a filter to the vector database query and found red stilettos with a 3-inch heel

Here's the full agent output:

Added user message to memory: I need women's heels with a heel height higher than 2 inches
=== Calling Function ===
Calling function: create_metadata_filter with args: {"filter_string": "women's heels heel height higher than 2 inches"}
=== Function Output ===
[{"key": "gender", "value": "women", "operator": "=="}, {"key": "heels_height", "value": 2, "operator": ">"}]
=== Calling Function ===
Calling function: search_footwear_database with args: {"query_str": "heels", "filters_json": [{"key": "gender", "value": "women", "operator": "=="}, {"key": "heels_height", "value": 2, "operator": ">"}]}
=== Function Output ===
[{'color': 'red', 'text': 'Carlton london women casual red stilettos', 'gender': 'women', 'product_type': 'heels', 'product_id': 56, 'usage': 'casual', 'price': 175, 'similarity_score': 1.18981349}]
=== LLM Response ===
I found a pair of women's heels for you: the **Carlton London Women Casual Red Stilettos**. They have a heel height higher than 2 inches and are priced at $175. Would you like more information or are you interested in purchasing them?

Additional Resources

For a deeper dive into this example, check out my GitHub repository where I break down the code and methodology used by the AI agent in today's scenario.

Learn to Build Your Own AI Agent Chatbot

I'm preparing a course to teach you how to build and deploy your own AI agent chatbot. Sign up here!