Welcome back, future AI architect! In this exciting chapter, we’re going to put all the pieces together and build something truly practical and engaging: a multi-LLM chatbot. This isn’t just any chatbot; it’s one that can intelligently switch between different Large Language Model (LLM) providers using any-llm, leveraging their unique strengths and capabilities.

By the end of this chapter, you’ll have a functional Python chatbot that demonstrates dynamic LLM provider selection, manages conversation history, and incorporates robust error handling. This hands-on project will solidify your understanding of any-llm’s core features and prepare you for real-world AI application development. Ready to bring your multi-LLM vision to life? Let’s dive in!

Before we begin, ensure you’re comfortable with the concepts covered in previous chapters, especially:

  • any-llm installation and basic usage.
  • Configuring and authenticating with multiple LLM providers (e.g., OpenAI, Mistral, Ollama).
  • Understanding the completion API and its parameters.
  • Basic error handling with try-except blocks.

The Vision: A Smart, Adaptable Chatbot

Imagine a chatbot that knows when to use a super-fast, cost-effective model for simple questions and when to switch to a more powerful, nuanced model for complex reasoning tasks. That’s the power we’re unlocking with any-llm. Our chatbot will simulate this intelligence by:

  1. Taking user input.
  2. Maintaining conversation history.
  3. Deciding which LLM provider to use based on a simple strategy (for now, we’ll make it explicit).
  4. Calling the chosen LLM via any-llm.
  5. Displaying the response.

Let’s visualize this flow:

flowchart TD A["Start Chatbot"] --> B{"User Input?"}; B -->|Yes| C["Capture Input"]; C --> D["Update Chat History"]; D --> E{"Determine LLM Strategy"}; E -->|"e.g., 'fast' or 'smart'"| F["Configure any-llm Provider"]; F --> G["Make any-llm Completion Call"]; G --> H{"Call Successful?"}; H -->|Yes| I["Display LLM Response"]; H -->|No| J["Handle Error & Report"]; I --> D; J --> D; B -->|No| K["End Chatbot"];

This diagram illustrates the continuous loop of our chatbot: taking input, processing it, interacting with an LLM, and responding, all while intelligently picking the right tool for the job.

Step-by-Step Implementation: Building Our Chatbot

We’ll build our chatbot incrementally, adding features one by one.

1. Project Setup and Initial File

First, let’s create a new Python file for our chatbot. Let’s call it multi_llm_chatbot.py.

touch multi_llm_chatbot.py

Now, open multi_llm_chatbot.py in your favorite code editor.

2. Importing any-llm and Setting Up Providers

Before we write any chat logic, we need to import any_llm and ensure our environment is ready. Remember, any-llm-sdk (current stable version 1.x.x as of 2025-12-30) needs to be installed with your desired providers.

If you haven’t already, install it:

pip install any-llm-sdk[openai,mistral,ollama]

(You can choose other providers like anthropic if you prefer, just ensure you have the necessary API keys or local Ollama server running.)

Next, ensure your API keys are set as environment variables. For example:

export OPENAI_API_KEY="YOUR_OPENAI_KEY"
export MISTRAL_API_KEY="YOUR_MISTRAL_KEY"
# For Ollama, you usually don't need a key if running locally, but ensure it's active.

Now, in multi_llm_chatbot.py, let’s import the necessary components:

# multi_llm_chatbot.py
import os
from any_llm import completion, LLMCompletionError

# Define our available LLM providers and their default models
# In a real application, you might load these from a config file.
LLM_PROVIDERS = {
    "openai": {
        "model": "gpt-4o-2025-12-30", # Assuming latest model for Dec 2025
        "api_key_env": "OPENAI_API_KEY"
    },
    "mistral": {
        "model": "mistral-large-2025-12-30", # Assuming latest model for Dec 2025
        "api_key_env": "MISTRAL_API_KEY"
    },
    "ollama": {
        "model": "llama3:8b", # Example Ollama model, ensure it's pulled locally
        "api_key_env": None # Ollama typically doesn't use an API key env var
    }
}

# This will store our conversation history
chat_history = []

print("Welcome to the Multi-LLM Chatbot! Type 'quit' to exit.")
print("You can switch providers by typing 'switch to openai', 'switch to mistral', or 'switch to ollama'.")
print(f"Current default provider: {list(LLM_PROVIDERS.keys())[0]}") # Just pick the first one as default

Explanation:

  • import os: Needed to check environment variables for API keys.
  • from any_llm import completion, LLMCompletionError: We import the completion function to interact with LLMs and LLMCompletionError for specific error handling.
  • LLM_PROVIDERS: A dictionary to map simple provider names (like “openai”) to their specific configuration (model name, and the environment variable where its API key is stored, if applicable). This makes it easy to manage and switch.
  • chat_history: An empty list to store our conversation. This is crucial for maintaining context in a chatbot.
  • Initial print statements: Friendly welcome messages and instructions for the user.

3. The Main Chat Loop

Now, let’s create the interactive loop where the user can type messages.

# ... (previous code) ...

def get_current_provider_config(provider_name):
    """Retrieves the configuration for a given provider, checking API keys."""
    config = LLM_PROVIDERS.get(provider_name)
    if not config:
        print(f"Error: Provider '{provider_name}' not recognized.")
        return None
    
    # Check for API key if required
    if config["api_key_env"] and not os.getenv(config["api_key_env"]):
        print(f"Warning: {provider_name.upper()} API key not found. Please set the '{config['api_key_env']}' environment variable.")
        return None
    return config

# Initialize current provider
current_provider_name = list(LLM_PROVIDERS.keys())[0] # Default to the first one
current_provider_config = get_current_provider_config(current_provider_name)

while True:
    user_input = input("\nYou: ")

    if user_input.lower() == 'quit':
        print("Goodbye!")
        break
    
    # Add user message to history
    chat_history.append({"role": "user", "content": user_input})

    # For now, let's just echo the input
    print(f"Bot (using {current_provider_name}): Thinking...")

    # We'll replace this "thinking" part with actual LLM calls next!

Explanation:

  • get_current_provider_config: A helper function to safely retrieve a provider’s configuration and perform a quick check for its API key. This helps prevent any-llm errors later.
  • current_provider_name and current_provider_config: Variables to keep track of which LLM we’re currently using.
  • while True:: This creates an infinite loop, typical for chatbots, until the user types ‘quit’.
  • input("\nYou: "): Prompts the user for input.
  • chat_history.append(...): Every user message is added to our chat_history. This is vital for conversational context.

4. Integrating any-llm for Responses

Now, let’s replace our placeholder “Thinking…” with an actual call to any_llm.completion.

# ... (previous code) ...

# Initialize current provider
current_provider_name = list(LLM_PROVIDERS.keys())[0] # Default to the first one
current_provider_config = get_current_provider_config(current_provider_name)

while True:
    user_input = input("\nYou: ")

    if user_input.lower() == 'quit':
        print("Goodbye!")
        break
    
    # Check for provider switching commands
    if user_input.lower().startswith("switch to "):
        requested_provider = user_input.lower().replace("switch to ", "").strip()
        if requested_provider in LLM_PROVIDERS:
            new_config = get_current_provider_config(requested_provider)
            if new_config:
                current_provider_name = requested_provider
                current_provider_config = new_config
                print(f"Bot: Switched to {current_provider_name} provider.")
                continue # Skip LLM call for this command
            else:
                print(f"Bot: Could not switch to {requested_provider}. Check configuration or API key.")
                continue
        else:
            print(f"Bot: Unknown provider '{requested_provider}'. Available: {', '.join(LLM_PROVIDERS.keys())}.")
            continue # Skip LLM call for this command
    
    # Add user message to history
    chat_history.append({"role": "user", "content": user_input})

    # Ensure we have a valid provider config before attempting a call
    if not current_provider_config:
        print("Bot: No valid LLM provider configured. Please set up API keys or switch to a working provider.")
        # Remove the last user message as it wasn't processed by an LLM
        chat_history.pop() 
        continue

    print(f"Bot (using {current_provider_name}): Thinking...")
    
    try:
        # Make the completion call
        # We pass the full chat_history to maintain context
        response = completion(
            messages=chat_history,
            provider=current_provider_name,
            model=current_provider_config["model"],
            temperature=0.7, # A bit creative, but not too wild
            max_tokens=150   # Limit response length
        )

        bot_response_content = response.content
        print(f"Bot: {bot_response_content}")
        
        # Add bot's response to history
        chat_history.append({"role": "assistant", "content": bot_response_content})

    except LLMCompletionError as e:
        print(f"Bot Error (from {current_provider_name}): {e}")
        print("Please try again or switch to a different provider.")
        # Remove the last user message from history if the LLM call failed
        chat_history.pop()
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        print("Please try again or switch to a different provider.")
        chat_history.pop()

Explanation:

  • Provider Switching Logic: We added a if user_input.lower().startswith("switch to "): block. This allows the user to explicitly tell the chatbot to use a different LLM. We update current_provider_name and current_provider_config accordingly.
  • Pre-call Check: if not current_provider_config: ensures we don’t try to call completion if the selected provider isn’t properly configured (e.g., missing API key).
  • completion(...) call:
    • messages=chat_history: This is crucial! any-llm expects a list of message objects ({"role": "user", "content": "..."} or {"role": "assistant", "content": "..."}) to maintain conversation context. By passing the full chat_history, the LLM sees the entire conversation.
    • provider=current_provider_name: Tells any-llm which LLM service to use.
    • model=current_provider_config["model"]: Specifies the exact model within that service.
    • temperature=0.7, max_tokens=150: Standard LLM parameters to control creativity and response length.
  • Adding Bot Response to History: After receiving a response, we add it to chat_history with {"role": "assistant", "content": bot_response_content}. This ensures the next turn of the conversation includes the bot’s previous statement.
  • Error Handling:
    • try...except LLMCompletionError as e:: Catches specific errors from any-llm (e.g., invalid model, rate limits, API key issues). We print the error and inform the user.
    • except Exception as e:: A general catch-all for any other unexpected Python errors.
    • chat_history.pop(): If an LLM call fails, we remove the user’s last message from chat_history. Why? Because the LLM didn’t successfully process it, so including it in the next turn would be misleading or could lead to repeated errors.

5. Running Your Multi-LLM Chatbot!

Save your multi_llm_chatbot.py file. Now, open your terminal, ensure your API keys are set as environment variables, and run:

python multi_llm_chatbot.py

Try it out!

  • Ask a simple question.
  • Then try switch to mistral (or switch to openai, etc.).
  • Ask another question and observe if the response style changes.
  • Ask a follow-up question to see if the conversation history is maintained.
  • Finally, type quit to exit.

This is your first multi-LLM chatbot! Pretty neat, right?

Mini-Challenge: Enhance Provider Switching

Our current provider switching is explicit (the user types “switch to”). Let’s make it a bit smarter.

Challenge: Modify the chatbot so that if the user asks a question containing the word “code” or “programming”, it automatically switches to a provider that you designate as “good for code” (e.g., OpenAI’s latest model or a specific Mistral model), if it’s not already using it. If the user then asks a general question, it should revert to a “general purpose” provider (e.g., a cheaper one).

Hint:

  • You’ll need a variable to store your “current strategy” or “preferred provider” based on the last user input.
  • Use if "code" in user_input.lower() or "programming" in user_input.lower(): to detect keywords.
  • You might want to add a default_provider_name that it reverts to.

What to Observe/Learn: This challenge helps you understand how to implement basic “routing” logic based on user intent, a common pattern in advanced AI agents. You’ll see how any-llm makes it trivial to swap out the underlying model without changing your core completion call.

Common Pitfalls & Troubleshooting

  1. Missing API Keys:

    • Symptom: LLMCompletionError: Missing API key for provider 'openai', or similar.
    • Fix: Ensure you’ve correctly set the OPENAI_API_KEY, MISTRAL_API_KEY, etc., environment variables in the same terminal session before running the script. Remember export commands are session-specific.
    • Best Practice: Never hardcode API keys directly in your script!

  2. Incorrect Model Name or Provider:

    • Symptom: LLMCompletionError: Invalid model 'gpt-4.0-turbo' or LLMCompletionError: Provider 'unknown_llm' not found.
    • Fix: Double-check the model names in your LLM_PROVIDERS dictionary against the official documentation for each provider. Ensure the provider name matches what any-llm expects (e.g., openai, mistral, ollama).

  3. Ollama Not Running/Model Not Pulled:

    • Symptom: If using Ollama, LLMCompletionError: Could not connect to Ollama server or LLMCompletionError: Model 'llama3:8b' not found locally.
    • Fix: Ensure the Ollama desktop application or server is running. If the model isn’t found, run ollama pull llama3:8b (or your chosen model) in your terminal.

  4. Conversation History Issues:

    • Symptom: The chatbot seems to forget previous turns, or responses are repetitive.
    • Fix: Verify that chat_history is correctly updated with both user and assistant messages ({"role": "user", ...} and {"role": "assistant", ...}) and that the entire chat_history list is passed to the messages parameter of the completion function on every call.

Summary

Congratulations! You’ve successfully built a multi-LLM chatbot using any-llm. Here are the key takeaways from this hands-on project:

  • Unified Interface: any-llm provides a seamless way to interact with diverse LLM providers using a single completion function.
  • Dynamic Provider Switching: You learned to implement logic that allows your application to switch between LLMs based on user commands or predefined strategies.
  • Conversation Management: Maintaining chat_history is crucial for building stateful, conversational AI applications.
  • Robustness with Error Handling: Incorporating try-except blocks, especially for LLMCompletionError, makes your application more resilient to external API issues.
  • Practical Application: This project demonstrates how any-llm simplifies the integration of powerful LLMs into real-world applications.

What’s Next?

In the next chapter, we’ll delve deeper into advanced any-llm features, including how to leverage different output types (like structured JSON), explore embedding generation, and discuss more sophisticated production patterns for building scalable and reliable AI systems. You’re well on your way to becoming an any-llm master!

References

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.