Microsoft Introduces AI Routing and Failover for .NET

Microsoft has introduced new experimental routing and failover capabilities in Microsoft.Extensions.AI, giving .NET developers more flexibility when working with AI models and providers. The new primitives are designed to help applications manage key challenges such as cost, uptime, and latency by routing requests between AI services and providing failover options when issues occur.

Here’s what you need to know:

• Route AI Requests Where They Make the Most Sense

  RoutingChatClient lets your application choose an AI client for each request based on your own rules. For example, you could send simple requests to a lower-cost model while directing more complex tasks to a more capable one.

• Route Based on What the User Is Asking

  SemanticRoutingChatClient takes routing a step further by looking at the meaning of a message, rather than relying only on predefined rules. It compares the request with example messages and sends it to the client with the strongest semantic match.

• Keep AI Applications Running with Failover

  FailoverChatClient can retry a request with another AI client when the selected provider fails before any response is sent to the user. This can help applications remain available when a model or provider experiences an issue.

• Use a Simple Backup Order

  OrderedFailoverChatClient provides a ready-to-use approach for failover by trying AI clients in a defined order. If the primary client fails, the request moves to the next available option.

• Build Smarter Routing Strategies

  These capabilities can also support cost-aware, latency-aware, health-aware, capability-aware, and region-based routing. Developers can use performance data such as response time and time to first update to make better routing decisions over time.

• Get Started

  The new routing and failover types are available in Microsoft.Extensions.AI version 10.9.0, although they are currently marked experimental. Developers can add the package with dotnet add package Microsoft.Extensions.AI.

Looking for AI Agents tailored to your organisation? Discover how customised AI solutions can streamline workflows and boost productivity. Contact us today to get started!