Welcome back, future AI-powered frontend wizard! In the previous chapter, we mastered the art of receiving and beautifully displaying streaming AI responses. You learned how to make your UI feel alive by showing AI’s thoughts as they unfold, character by character. That’s a huge step towards a dynamic user experience!

Now, let’s unlock the next level of AI interaction: UI-driven tool calling. Imagine your AI assistant isn’t just talking, but doing things. It can look up real-time information, interact with external systems, or even perform actions within your application, all initiated by its own reasoning. This capability transforms a conversational AI into a truly agentic AI, making your applications incredibly powerful and interactive.

In this chapter, we’ll dive deep into how your frontend application can seamlessly integrate with AI agents that perform “tool calls.” We’ll explore how these calls are communicated to the UI, how to interpret them, and most importantly, how to execute the associated actions and update your user interface accordingly. By the end of this chapter, you’ll be able to build React and React Native applications where the AI doesn’t just respond, but actively engages with the world through your UI. Let’s make your AI agents truly empower your users!

What is “Tool Calling” in AI Agents?

At its heart, “tool calling” (often referred to as “function calling” by some LLM providers) is an AI model’s ability to identify when it needs to use a specific external function or “tool” to fulfill a user’s request. Instead of just generating text, the AI generates a structured output indicating which tool to call, along with the arguments for that tool.

Think of it this way:

  • User: “What’s the weather like in New York?”
  • AI (without tool calling): “I don’t have real-time weather data.”
  • AI (with tool calling): Recognizes it needs a getWeather tool. It then outputs: call_tool(name="getWeather", arguments={"location": "New York"}).

The magic happens when your application intercepts this call_tool instruction, executes the getWeather function, and then feeds the result back to the AI (or directly displays it to the user). This allows the AI to overcome its knowledge limitations and interact with dynamic, real-world data.

The Frontend’s Role in Tool Calling

While the AI (typically an LLM running on a backend service) decides when and which tool to call, your frontend application plays a crucial role in the entire flow:

  1. Receiving Tool Call Instructions: The AI’s streaming response will contain special tokens or structured data indicating a tool call. Your UI needs to be able to parse this.
  2. Displaying Intent (Optional but Recommended): Inform the user that the AI is about to perform an action (e.g., “Thinking…”, “Checking weather…”). This enhances transparency and user experience.
  3. Executing the Tool: This is where the frontend shines! It can trigger local functions, make API calls to your own backend, or even interact with third-party services.
  4. Displaying Tool Output: Once the tool has executed, your UI needs to render the results in a meaningful way. This might involve showing a weather forecast, updating a calendar, or displaying data from a search.
  5. Feeding Results Back (Optional, but common for complex agents): For multi-turn agentic flows, the result of a tool call might be sent back to the LLM to inform its next response. However, in many UI-centric scenarios, the frontend might simply display the result and continue the interaction.

Let’s visualize this flow:

flowchart TD User["User Input"] --> Frontend["React/RN Frontend"] Frontend -->|"Send Prompt"| Backend_LLM["Backend AI Service"] Backend_LLM -->|"Stream Response (Text/Tool Call)"| Frontend Frontend -->|"\1"| Parse["Parse Tool Call"] Parse -->|"Extract Tool Name & Args"| ToolExecutor["Frontend Tool Executor"] ToolExecutor -->|"Execute Local Function/API Call"| ExternalService["Local Function / External API"] ExternalService -->|"Tool Result"| ToolExecutor ToolExecutor -->|"Update UI with Result"| Frontend Frontend -->|\1| DisplayText["Display Text"] DisplayText --> User

Tool Definition and Schema

For an AI to understand how to call a tool, it needs a description. This is typically provided as a JSON schema. For example, a getWeather tool might be described like this:

{
  "name": "getWeather",
  "description": "Get the current weather for a given location",
  "parameters": {
    "type": "object",
    "properties": {
      "location": {
        "type": "string",
        "description": "The city and state, e.g. San Francisco, CA"
      },
      "unit": {
        "type": "string",
        "enum": ["celsius", "fahrenheit"],
        "description": "The unit of temperature to return"
      }
    },
    "required": ["location"]
  }
}

This schema tells the LLM:

  • The tool’s name (getWeather).
  • What it does (description).
  • What arguments it expects (parameters), their types, and if they are required.

While you define these schemas (usually on the backend where the LLM is configured), your frontend needs to be aware of the names of the tools and what their arguments mean so it can correctly execute them.

Agent Orchestration from the Client (High-Level)

In a fully “agentic” system, the LLM often orchestrates multiple tools and reasoning steps. From a frontend perspective, we’re primarily concerned with:

  1. Sending the initial user prompt.
  2. Receiving the streaming response, which might contain text or tool calls.
  3. Acting on those tool calls (executing the tool, displaying results).
  4. Potentially sending the tool’s output back to the LLM for further reasoning, especially in complex multi-step agents. For this chapter, we’ll focus on the frontend executing the tool and displaying its output directly, which covers many common use cases.

Step-by-Step Implementation: Handling Tool Calls

Let’s modify our chat interface from Chapter 4 to handle simulated tool calls. We’ll simulate a getWeather tool that the AI might “request.”

First, ensure you have a basic React or React Native project set up. We’ll use a simple functional component for this example.

Prerequisites:

  • A React/React Native project (e.g., created with npx create-react-app my-ai-app or npx react-native@latest init my-ai-app --version 0.73.5).
  • Familiarity with useState and useEffect hooks.
  • Understanding of asynchronous operations.

We’ll assume you have a ChatInterface component that manages messages and an input field. We’ll focus on the handleSendMessage logic and how to process the AI’s response.

1. Define Our Available Tools

Let’s create a simple object to hold our “tools.” In a real application, these would be functions that make API calls or perform complex logic. For now, they’ll just return mock data.

Create a new file, say src/tools.js:

// src/tools.js

export const availableTools = {
  getWeather: async ({ location, unit = 'celsius' }) => {
    console.log(`TOOL CALL: Getting weather for ${location} in ${unit}...`);
    // Simulate an API call delay
    await new Promise(resolve => setTimeout(resolve, 1500));

    // In a real app, you'd call a weather API here.
    // For now, let's return some mock data.
    if (location.toLowerCase().includes('new york')) {
      return {
        temperature: unit === 'celsius' ? 10 : 50,
        conditions: 'Cloudy',
        location: 'New York, USA',
        unit: unit
      };
    } else if (location.toLowerCase().includes('london')) {
      return {
        temperature: unit === 'celsius' ? 8 : 46,
        conditions: 'Rainy',
        location: 'London, UK',
        unit: unit
      };
    }
    return { error: `Weather data not available for ${location}` };
  },
  // We'll add more tools in the mini-challenge!
};

Explanation:

  • availableTools: An object mapping tool names (strings) to their corresponding asynchronous functions.
  • getWeather: Our first tool. It takes location and unit as arguments.
  • console.log: Helps us see when the tool is “called.”
  • setTimeout: Simulates the delay of a real API call.
  • Mock Data: Returns different weather based on the location.

2. Simulate an AI Response with a Tool Call

Instead of a real AI API for now, we’ll simulate a response that includes a tool call. Our mockStreamAIResponse function will return a sequence of messages, some of which are text and one of which is a tool call.

Modify your App.js (or ChatInterface.js) where you handle AI responses. Let’s create a helper to simulate streaming:

// src/App.js (or wherever your chat logic resides)
import React, { useState, useRef } from 'react';
import { availableTools } from './tools'; // Import our tools

// ... (other imports for styling, etc.)

// Helper to simulate a streaming response, now including tool calls
const mockStreamAIResponse = async (userMessage, onNewToken, onToolCall, onComplete) => {
  const responses = {
    "what's the weather in new york?": [
      { type: 'text', content: "Okay, I'm checking the weather for New York..." },
      { type: 'tool_call', name: 'getWeather', args: { location: 'New York', unit: 'fahrenheit' } },
      { type: 'text', content: "The weather data is in!" }
    ],
    "what's the weather like in london?": [
      { type: 'text', content: "One moment, looking up London's forecast..." },
      { type: 'tool_call', name: 'getWeather', args: { location: 'London', unit: 'celsius' } }
    ],
    "hello": [
      { type: 'text', content: "Hello there! How can I assist you today?" }
    ],
    "default": [
      { type: 'text', content: "I'm not sure how to respond to that yet. Try asking about the weather!" }
    ]
  };

  const lowerCaseMessage = userMessage.toLowerCase();
  const chosenResponse = responses[lowerCaseMessage] || responses["default"];

  for (const item of chosenResponse) {
    await new Promise(resolve => setTimeout(resolve, 50)); // Simulate network delay for each item

    if (item.type === 'text') {
      for (const char of item.content) {
        onNewToken(char);
        await new Promise(resolve => setTimeout(resolve, 20)); // Simulate character-by-character streaming
      }
    } else if (item.type === 'tool_call') {
      // When a tool call is encountered, we immediately pass it to the handler
      onToolCall(item.name, item.args);
      // Wait for the tool to execute before continuing the AI's "text" stream
      // In a real scenario, the LLM might wait for tool output before generating more text.
      // Here, we simulate that pause.
      await new Promise(resolve => setTimeout(resolve, 2000));
    }
  }
  onComplete();
};

function App() {
  const [messages, setMessages] = useState([]);
  const [inputMessage, setInputMessage] = useState('');
  const [isTyping, setIsTyping] = useState(false);
  const currentBotMessageRef = useRef(''); // To build up the streamed response

  const handleSendMessage = async () => {
    if (!inputMessage.trim()) return;

    const newUserMessage = { id: Date.now(), text: inputMessage, sender: 'user' };
    setMessages(prev => [...prev, newUserMessage]);
    setInputMessage('');
    setIsTyping(true); // AI is now "typing"

    // Add a placeholder for the bot's response immediately
    const botMessageId = Date.now() + 1;
    setMessages(prev => [...prev, { id: botMessageId, text: '', sender: 'bot', isToolExecuting: false }]);

    currentBotMessageRef.current = ''; // Reset for new bot message

    await mockStreamAIResponse(
      inputMessage,
      (token) => {
        // This runs for each character of text
        currentBotMessageRef.current += token;
        setMessages(prevMessages =>
          prevMessages.map(msg =>
            msg.id === botMessageId ? { ...msg, text: currentBotMessageRef.current } : msg
          )
        );
      },
      async (toolName, toolArgs) => {
        // This runs when a tool_call instruction is received
        setMessages(prevMessages =>
          prevMessages.map(msg =>
            msg.id === botMessageId ? { ...msg, text: `${currentBotMessageRef.current}\n\n*Agent is executing tool: ${toolName} with args: ${JSON.stringify(toolArgs)}*`, isToolExecuting: true } : msg
          )
        );

        // Execute the tool
        if (availableTools[toolName]) {
          try {
            const toolResult = await availableTools[toolName](toolArgs);
            console.log(`Tool ${toolName} executed. Result:`, toolResult);

            // Append tool result to the bot's message
            setMessages(prevMessages =>
              prevMessages.map(msg => {
                if (msg.id === botMessageId) {
                  const resultText = toolResult.error
                    ? `Error: ${toolResult.error}`
                    : `**${toolName} Result:**\nTemperature: ${toolResult.temperature}°${toolResult.unit.toUpperCase()}\nConditions: ${toolResult.conditions}\nLocation: ${toolResult.location}`;
                  return {
                    ...msg,
                    text: `${msg.text}\n\n${resultText}`,
                    isToolExecuting: false // Tool execution complete
                  };
                }
                return msg;
              })
            );
          } catch (error) {
            console.error(`Error executing tool ${toolName}:`, error);
            setMessages(prevMessages =>
              prevMessages.map(msg =>
                msg.id === botMessageId ? { ...msg, text: `${msg.text}\n\n*Error executing tool: ${error.message}*`, isToolExecuting: false } : msg
              )
            );
          }
        } else {
          setMessages(prevMessages =>
            prevMessages.map(msg =>
              msg.id === botMessageId ? { ...msg, text: `${msg.text}\n\n*Unknown tool: ${toolName}*`, isToolExecuting: false } : msg
            )
          );
        }
      },
      () => {
        // This runs when the AI response stream is complete
        setIsTyping(false);
      }
    );
  };

  return (
    <div style={{ maxWidth: '600px', margin: '20px auto', border: '1px solid #ccc', borderRadius: '8px', padding: '15px', fontFamily: 'sans-serif' }}>
      <h2 style={{ textAlign: 'center' }}>AI Assistant</h2>
      <div style={{ height: '400px', overflowY: 'scroll', border: '1px solid #eee', padding: '10px', marginBottom: '10px', borderRadius: '4px', backgroundColor: '#f9f9f9' }}>
        {messages.map(msg => (
          <div key={msg.id} style={{
            marginBottom: '10px',
            textAlign: msg.sender === 'user' ? 'right' : 'left',
          }}>
            <span style={{
              display: 'inline-block',
              padding: '8px 12px',
              borderRadius: '15px',
              backgroundColor: msg.sender === 'user' ? '#007bff' : '#e9ecef',
              color: msg.sender === 'user' ? 'white' : 'black',
              whiteSpace: 'pre-wrap', // Preserve newlines
              fontWeight: msg.isToolExecuting ? 'bold' : 'normal', // Highlight when tool is executing
              fontStyle: msg.isToolExecuting ? 'italic' : 'normal',
            }}>
              {msg.text}
            </span>
            {msg.isToolExecuting && (
              <span style={{ marginLeft: '10px', fontSize: '0.8em', color: '#666' }}>
                (Processing...)
              </span>
            )}
          </div>
        ))}
        {isTyping && (
          <div style={{ textAlign: 'left', marginBottom: '10px' }}>
            <span style={{ display: 'inline-block', padding: '8px 12px', borderRadius: '15px', backgroundColor: '#e9ecef', color: 'black' }}>
              ...
            </span>
          </div>
        )}
      </div>
      <div style={{ display: 'flex' }}>
        <input
          type="text"
          value={inputMessage}
          onChange={(e) => setInputMessage(e.target.value)}
          onKeyPress={(e) => e.key === 'Enter' && handleSendMessage()}
          placeholder="Type your message..."
          style={{ flexGrow: 1, padding: '10px', borderRadius: '4px', border: '1px solid #ccc', marginRight: '10px' }}
          disabled={isTyping}
        />
        <button onClick={handleSendMessage} disabled={isTyping || !inputMessage.trim()} style={{ padding: '10px 15px', borderRadius: '4px', border: 'none', backgroundColor: '#007bff', color: 'white', cursor: 'pointer' }}>
          Send
        </button>
      </div>
    </div>
  );
}

export default App;

Explanation of Changes:

  • mockStreamAIResponse Update:
    • Now returns an array of objects, each with a type (text or tool_call).
    • If type is tool_call, it includes name and args.
    • It takes an onToolCall callback to handle tool execution.
  • handleSendMessage Logic:
    • We added an isToolExecuting flag to the bot’s message state. This allows us to visually indicate when a tool is being run.
    • The onToolCall callback is where the magic happens:
      • It immediately updates the UI to show the user that a tool is being executed.
      • It then looks up the toolName in our availableTools object.
      • If found, it awaits the execution of the tool function with the provided toolArgs.
      • Once the tool returns a toolResult, the UI is updated again to display this result, and isToolExecuting is set back to false.
      • Error handling is included in case a tool fails or is not found.
  • UI Updates:
    • The bot’s message now dynamically updates to show both the AI’s preamble, the tool execution status, and the tool’s result.
    • isToolExecuting state is used to conditionally render a “(Processing…)” indicator next to the message and apply bold/italic styling.
    • whiteSpace: 'pre-wrap' is important for displaying multi-line tool results nicely.

Now, run your React app (npm start or yarn start) and try typing:

  • “What’s the weather in New York?”
  • “What’s the weather like in London?”
  • “Hello”
  • Any other message

You should see the AI’s response stream, then a message indicating the tool call, a pause (simulating tool execution), and finally the tool’s result integrated into the chat!

Mini-Challenge: Add a New Tool

You’ve successfully integrated a getWeather tool. Now, let’s expand our agent’s capabilities!

Challenge:

  1. Create a new tool named getCurrentTime in your src/tools.js file.
    • This tool should accept an optional timezone argument (defaulting to ‘UTC’).
    • It should return the current time as a formatted string (e.g., “HH:MM:SS (Timezone)”).
  2. Update mockStreamAIResponse in src/App.js to trigger this new tool.
    • Add a new user input that would cause the AI to call getCurrentTime (e.g., “What time is it in Tokyo?”).
    • Ensure the tool_call object specifies the name as getCurrentTime and passes the appropriate args.
  3. Test your implementation by asking the AI about the current time.

Hint:

  • You can use new Date().toLocaleTimeString('en-US', { timeZone: timezone || 'UTC' }) to get the current time in a specific timezone.
  • Remember to add a default response to mockStreamAIResponse for the new prompt as well, if the tool call is conditional.

What to observe/learn:

  • How easy it is to extend the agent’s functionality by just defining new tools.
  • The seamless integration of a new tool into the existing UI-driven execution flow.
  • The importance of clear tool definitions and arguments.

Common Pitfalls & Troubleshooting

  1. Misinterpreting tool_calls:
    • Problem: The frontend expects a specific structure for tool_call objects (e.g., name, args), but the AI’s response (or the mock response) sends something different.
    • Troubleshooting: Double-check the exact format of the tool_call object you’re receiving from your AI service (or your mock). Ensure your parsing logic (onToolCall in our example) matches this structure precisely. Use console.log to inspect the toolName and toolArgs as they are received.
  2. Security Risks with Client-Side Tool Execution:
    • Problem: Directly executing arbitrary code or making unvalidated API calls based on AI output on the client-side can be a huge security vulnerability. An attacker could craft a prompt to make the AI generate a tool call that performs malicious actions.
    • Troubleshooting:
      • Never expose sensitive API keys or credentials directly in frontend tools. All tools that require sensitive access should proxy through your secure backend.
      • Validate ALL arguments: Before executing a tool, always validate its arguments on the client-side. For example, if a tool expects a URL, ensure it’s a valid and safe URL. If it expects a number, ensure it’s within expected bounds.
      • Whitelist tools: Only allow the execution of explicitly defined and safe tools. Do not dynamically load or execute tools based on arbitrary AI output. Our if (availableTools[toolName]) check is a basic form of whitelisting.
      • User Consent: For sensitive actions, consider adding a user confirmation step before executing the tool (e.g., “The AI wants to delete this item. Confirm?”).
  3. Handling Complex Tool Arguments:
    • Problem: Tools might require complex JSON objects or arrays as arguments, and parsing these correctly from a potentially stringified AI response can be tricky.
    • Troubleshooting: Ensure the AI’s tool call output is consistently valid JSON for its arguments. Use JSON.parse() carefully, wrapped in try-catch blocks, as invalid JSON will crash your application. Modern LLM APIs often return tool_calls as structured objects directly, which simplifies this.

Summary

Congratulations! You’ve successfully integrated UI-driven tool calling into your React application. You now understand:

  • What tool calling is and why it’s essential for building truly agentic AI experiences.
  • The critical role of the frontend in receiving, interpreting, executing, and displaying the results of AI-requested tools.
  • How to define simple tools and integrate them into your application’s logic.
  • How to update your UI dynamically to reflect tool execution status and results.
  • Key security considerations when implementing client-side tool execution.

This capability is a game-changer, moving your AI applications beyond just conversation to active participation. In the next chapter, we’ll delve deeper into managing the AI’s state, memory, and context within your React application, ensuring your agents maintain coherence and understanding across multiple interactions. Get ready to build even smarter and more context-aware AI experiences!

References

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.