This post explores our implementation of Anthropic's Claude 3.5 Sonnet and the development of our LLM-agnostic architecture that enables flexible integration with multiple AI models. Secoda users can now choose between Claude 3.5 Sonnet and GPT-4o through AI settings, while maintaining consistent security and performance standards.

Why Claude 3.5 Sonnet

Our evaluation of Claude 3.5 Sonnet revealed four key technical advantages:

• Superior query handling performance: Reduces SQL query errors and hallucinations, particularly with column names and table relationships
• Enhanced context management: Maintains RAG system effectiveness by respecting zero-based pagination, preventing skipped documents
• Built-in security controls: Implements complex filter structure generation and validation for catalog search functionality, enabling comprehensive parameter validation
• Function calling reliability: Correctly implements search function parameters without hallucination, supporting advanced operations within increased function description capacity limits

Common function calling challenges

Our journey with function calling implementations revealed several challenges across different LLMs. A typical search function structure in our system looks like this:

{

       “name”: “search”,

       “parameters”: {

    “page”: “the current page of results”,

    “page_size”: “number of results per page”,

    “term”: “the search term”

    }

}

In our experience with OpenAI models, we encountered several consistent challenges with function calling:

• Persistent disregard for instructions: A fundamental issue we encountered was OpenAI models consistently ignoring provided instructions. This manifested in multiple ways:
  
  • In our search tool, despite explicitly specifying zero-based pagination, the model would ignore this and start from page 1. This directly impacted our RAG system's effectiveness, as it would skip the most relevant documents by fetching the second page of results instead of the first.
  • With our run SQL tool, the model would use invalid column names, and when instructed how to fix the query, it would simply abandon the task instead of applying the correction.‍
• ‍Function parameter hallucinations: Models would frequently use incorrect parameter names despite clear specifications‍
• Function description length constraints: OpenAI's 1024-character limit on function descriptions impacted our ability to implement complex tools. This was particularly limiting for our catalog search functionality, which needs to go beyond simple keyword or vector search. The tool requires detailed instructions for:
  • Complex filter structure generation
  • Sort specification construction
  • Multi-parameter validation

While we attempted to work around these limitations using a planner agent architecture, the additional latency made this solution impractical. These challenges led us to explore alternative models, ultimately leading to our integration of Claude 3.5 Sonnet, which has demonstrated more reliable function calling capabilities and better adherence to instructions.

With Claude's increased function description capacity, we've been able to provide more comprehensive instructions for these complex operations, resulting in more reliable handling of advanced search queries.

What to expect from the new Secoda AI 

Our model-agnostic runtime combined with Claude's capabilities delivers several key improvements that Secoda users will immediately notice:

More complex query handling

The system can now differentiate between simple lookup queries and complex analytical questions. For example:

• Basic question: "Show me all tables" (Traditional AI approach: Vectorizes question into [0.1, 0.2, 0.11, 0.12, ...] and performs similarity matching against indexed data)
• Intermediate question: "Show me tables used in the last 30 days" (Hybrid approach: Uses vector search with basic temporal parameters)
• Sophisticated question: "What are our most queried tables?" (Enhanced approach: System recognizes this requires catalog search instead of vector matching. It creates specific filters on entity_type="table" and sorts by external_usage to accurately measure query frequency)

The key difference lies in how the AI processes these queries. While most AI solutions rely solely on vector search (converting questions into number sequences for similarity matching), our system intelligently switches between:

1. Vector search when appropriate for natural language queries
2. Catalog search with precise filters and sort parameters when the user's intent requires structured data analysis

This intelligence in choosing the right search strategy leads to:

• Better handling of complex data relationships and multi-step analyses
• More reliable catalog search with advanced filtering and sorting
• More accurate results that match the user's actual search intent rather than just finding similar text

Improved accuracy

The new system delivers significant improvements in several key areas:

• Significantly reduced SQL query errors and hallucinations
• More reliable function calling with fewer parameter mistakes
• Better adherence to pagination and other technical requirements

Faster response times

Users will experience enhanced performance through:

• Immediate response streaming back to users
• Fewer steps needed to complete queries
• More efficient function calling reducing overall latency

Developer experience benefits

Our LLM-agnostic architecture provides several key technical advantages:

• Easy integration of new LLMs without system redesign
• Simplified security management through standardized controls
• Faster development cycles without framework-specific dependencies
• Support for A/B testing different models on specific functions

For teams considering a similar approach, key implementation considerations include:

• Design for model-agnostic function calling from the start
• Implement comprehensive validation and error handling
• Focus on cross-model security vulnerability management
• Consider latency implications of model switching mechanisms

This architecture creates a foundation that's both powerful and adaptable, allowing teams to integrate and test new LLMs as they become available while maintaining consistent performance standards.

Looking forward

The combination of Claude's robust function calling capabilities, GPT4o’s strengths, and Secoda AI’s flexible architecture provides teams with options that can be tailored to their specific needs. This multi-model approach, backed by our custom runtime, creates a foundation that's both powerful and adaptable.

By focusing on a model-agnostic architecture from the start, we've built Secoda AI to not only solve immediate data intelligence challenges but also provide the flexibility to integrate and test new LLMs as they become available - ensuring our customers always have access to the most effective tools for their data stack.

Try Secoda today and see how our multi-model approach can transform your data operations.