Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: Extend function calling model to support strict mode #9786

Open
SergeyMenshykh opened this issue Nov 21, 2024 · 2 comments · May be fixed by #9924
Open

.Net: Extend function calling model to support strict mode #9786

SergeyMenshykh opened this issue Nov 21, 2024 · 2 comments · May be fixed by #9924
Labels
ai connector Anything related to AI connectors function_calling .NET Issue or Pull requests regarding .NET code sk team issue A tag to denote issues that where created by the Semantic Kernel team (i.e., not the community)

Comments

@SergeyMenshykh
Copy link
Member

Context: {Azure}OpenAI support the 'strict' mode for function calling that tells the model to adhere to JSON schema of the function definition:
Image
Source: https://platform.openai.com/docs/guides/function-calling

Enabling this mode should ensure that the arguments generated by the model for a function call exactly match the JSON Schema provided in the function definition.

ToDo: Add the new AllowStrictSchemaAdherence property to the FunctionChoiceBehaviorOptions class allowing activation of the strict mode for function calling.

@SergeyMenshykh SergeyMenshykh added ai connector Anything related to AI connectors function_calling labels Nov 21, 2024
@markwallace-microsoft markwallace-microsoft added .NET Issue or Pull requests regarding .NET code triage labels Nov 21, 2024
@SergeyMenshykh SergeyMenshykh moved this to Backlog in Semantic Kernel Nov 21, 2024
@markwallace-microsoft markwallace-microsoft added sk team issue A tag to denote issues that where created by the Semantic Kernel team (i.e., not the community) and removed triage labels Nov 22, 2024
@baywet
Copy link
Member

baywet commented Nov 22, 2024

Here are some early testing results I'd like to share with anybody who wants to look at this.

Switching the strict mode to true pushes additional requirements on the schem:

  • can't use minimum with integer type (makes sense from a JSON schema spec perspective)
  • can't use uniqueItems on array types ???
  • all query parameters need to be required ???

This last one is really problematic in a scenario where you have more parameters than there's context for.
E.g. with GET https://graph.microsoft.com/v1.0/me/messages with supports:

  • includeHiddenMessages
  • $search
  • $top
  • $skip
  • $filter
  • $select
  • $expand
  • $orderby

And the prompt using only the subject, recap my last 5 emails ordered by created date time, the model only has enough information for top/select/orderby, and might be able to figure out to set skip to 0. Gpt4 then gives up saying it couldn't call the function, and doesn't do a better job with the orderby parameter formatting, gpt4o defaults all others to the default for the type, which ends up failing on the service since the request doesn't make sense.

@baywet
Copy link
Member

baywet commented Dec 5, 2024

@SergeyMenshykh it looks like the work to normalize the schema is already started at #9807

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ai connector Anything related to AI connectors function_calling .NET Issue or Pull requests regarding .NET code sk team issue A tag to denote issues that where created by the Semantic Kernel team (i.e., not the community)
Projects
Status: Backlog
Development

Successfully merging a pull request may close this issue.

3 participants