Generate adaptive cards when creating API Plugins

[maisarissi]

Current Challenge

Today, the Teams Toolkit (TTK) VS Code extension generates adaptive cards when scaffolding the project. However, with the integration of Kiota into TTK to generate API plugins, there is a potential issue. When regenerating plugins, Kiota may override the plugin manifest content, deleting any previously added adaptive cards. This is because Kiota currently does not have adaptive card generation as part of its logic.

Proposal

As part of the Kiota plugin generator history, we should generate adaptive cards when generating API plugins. This way, whether a user is generating a plugin for the first time or regenerating an existing plugin, Kiota can provide initial adaptive cards based on the selected endpoints. These cards can later be improved by the user.
We need to do reverse engineering in TTK's code and add the same logic they are using to generate adaptive cards into Kiota generation.

Description

Adaptive cards information are generated for each function that returns a complex data structure under the response_semantics in the API Plugin. The response_semantics property instructs the agent how it should display data it receives from the API. It consists of 3 properties: data_path, properties, and static_template.

So for example, given the /search/issues#GET operation, based on the GitHub's OpenAPI description, this is the reponse:

"responses": {
  "200": {
    "description": "Response",
    "content": {
      "application/json": {
        "schema": {
          "type": "object",
          "required": [
            "total_count",
            "incomplete_results",
            "items"
          ],
          "properties": {
            "total_count": {
              "type": "integer"
            },
            "incomplete_results": {
              "type": "boolean"
            },
            "items": {
              "type": "array",
              "items": {
                "$ref": "#/components/schemas/issue-search-result-item"
              }
            }
          }
        },
        "examples": {
          "default": {
            "$ref": "#/components/examples/issue-search-result-item-paginated"
          }
        }
      }
    }
  }

The data_path specifies a JSON path expression that points to the relevant part of the API response. In the above scenario, the data_path should point to items.
The next part of response semantics are properties that are used to tell the agent which of the data properties from the API response represent the item’s properties such as title, description or URL.
The final property of response semantics is static_template which is used to define an Adaptive Card template to show the data from the API.

Here is an example of the desired structure:

"functions": [
    {
        "name": "search_issues_and_pull_requests",
        "description": "Find issues by state and keyword. This method returns up to 100 results [per page](https://docs.gith",
        "capabilities": {
            "response_semantics": {
                "data_path": "$.items",
                "properties": {
                    "title": "$.title",
                    "subtitle": "$.body",
                    "url": "$.url"
                },
                "static_template": {
                    "type": "AdaptiveCard",
                    "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
                    "version": "1.5",
                      "body": [
                          {
                              "type": "Container",
                              "items": [
                                  {
                                      "type": "TextBlock",
                                      "text": "user.login: ${if(user.login, user.login, 'N/A')}"
                                  },
                                  {
                                      "type": "Image",
                                      "url": "${if(user.avatar_url != null && user.avatar_url != '', user.avatar_url, '')}"
                                  },
                                  {
                                      "type": "TextBlock",
                                      "text": "created_at: ${if(created_at, created_at, 'N/A')}",
                                      "wrap": true
                                  },
                                  {
                                      "type": "TextBlock",
                                      "text": "closed_at: ${if(closed_at, closed_at, 'Still Open')}",
                                      "wrap": true
                                  },
                                  {
                                      "type": "TextBlock",
                                      "text": "assignee: ${if(assignee, assignee, 'No assignees to work on this issue')}",
                                      "wrap": true
                                 }
                             ]
                         }
                     ]
                }
            }
        }
    }
]

Acceptance Criteria:

1. When generating or regenerating API plugins, Kiota should automatically generate adaptive cards based on the selected endpoints.
2. For API endpoints that return a JSON schema, Kiota should add an adaptive card.
3. For API endpoints that don't return a JSON schema, Kiota should still generate the function but with no response_semantics.
4. Every function with response_semantics must include data_path, properties with at least title and the static_template.
5. Kiota should always generate a valid adaptive card based on the following schema: http://adaptivecards.io/schemas/adaptive-card.json
6. When regenerating Kiota should display a window asking whether the user wants to regenerate the API plugin including the adaptive cards.
7. When regenerating, if the user says "YES" to regenerating including adaptive cards, Kiota should regenerate everything in the API plugin including the adaptive card as well as regen the sliced OpenAPI description.
8. When regenerating, if the user says "NO" to regenerating the adaptive cards , Kiota should regenerate the API plugin to include/exclude new endpoints (functions) but must not override the existing adaptive cards. Kiota should also regen the sliced OpenAPI description.
9. Tests and validations should be performed to ensure that the adaptive cards are generated correctly and can be displayed in the Copilot.

[sebastienlevert]

On AC 8, let's say that an endpoint changes, but the user says no to regen the adaptive card, are we failing? Are we generating the adaptive card because the endpoint changed? Or do we do a "mix" of we generate the updated function, but we inject the existing adaptive card?

Last option seems pretty dangerous...

[maisarissi]

In the API plugin we reference the OpenAPI description through the function[n].name, so if there was a change in the endpoint the new sliced OpenAPI description should be updated to have the new changes. The adaptive cards in the API plugin though, shouldn't be updated in the user selects "No" (AC 8).
We should always created the adaptive card checking the parameter:
${if(api_parameter, api_paramenter, '')}"
that way, even if the endpoint changes and the api_parameter don't exist anymore, the adaptive won't break.

If the user says "No" to regen adaptive cards, we should not fail if there was a change in the endpoint. We should not regen the adaptive card even if there was a change in the endpoint. We should just add the new selected endpoints/remove unselected endpoints in the API plugin and regen the sliced OpenAPI description.

Does that make sense?

[baywet]

This brings so many questions.

First off, what if I'm generating a plugin manifest with the goal of feeding it to semantic kernel/something other than TTK? Do I always get an adaptive card? That I need to manually delete every time?

Also, the experience you're describing is very VS Code centric, what about the CLI? Are we going to start interrupting the shell and ask "modal" prompts? or are we adding additional switches?

The adaptive card is deeply dependent on TTK (I think you can also use adaptive cards in windows apps, outlook, and SPFx, but let's ignore those for now as they are not copilot scenarios). I DO NOT think it makes sense architecturally to start mixing concerns, and take ownership of something additional we don't know much about.

It'd probably make much more sense to "handover back" to TTK saying "here is the schema, do what you must". TTK has a deep expertise of adaptive cards, validation libraries, etc... that we don't have and would have to build.

Lastly, mixing generated code and manual edits will lead to a poor experience, especially because of the overwrite problem. Adaptive cards are meant to be designed, sometimes by other audiences than developers (this is why they are a bunch of simplified controls, personalization points etc...). Getting "in the way of the developer" and starting to ask to overwrite files is going to lead to a lot of friction in the experience.

Being able to work on two separate files, once that describes the data structure, auto-generated, and another that describes the card "layout" handcrafted, would probably deliver a MUCH better end user experience. But I don't know if this is something supported by adaptive cards today.

[maisarissi]

Hey Vincent, thanks for asking question to get as much information as possible.

About the CLI experience, we had an internal conversation a few days ago and yes, we decided to add a new and optional switch --adaptive-card (alias --ac) as part of plugin command, defaulting to false when not provided. So we will always assume one doesn't want to generate adaptive cards if not said otherwise. For the VS Code extension, when generating the plugin the first time, we will always generate adaptive cards, and for every regeneration, we should ask the user whether or not they want to regenerate adaptive cards as well and let them know that any manual change will be override. I'm going to update the CLI spec to cover the new switch 😄

We also believe that Kiota + TypeSpec for authoring APIs can bring a massive benefit with new decorators, some of them related to tagging which properties should be used in the adaptive card and Kiota would be able to actually create something based on the developer preference. So I don't agree with the adaptive card is deeply dependent on TTK as adaptive cards will also be used by Copilot to render information, they are actually part of the plugin definition, and because Kiota wants to be the tool to create plugins, adaptive cards will be an important part of it.
I do believe that if the OpenAPI description has the new custom OpenAPI extensions (still working on those), Kiota should use it to create the adaptive cards, otherwise, we should create something based in our best effort to guess the properties (we should use TTK logic as a start point for it).

Now, about mixing generated code and manual edits, I do agree with you. This is very trick. Because of that, we have talked to TTK and asked them to support adaptive cards in a different file and use an import mechanism when provisioning/publishing to merge both files when generating the package OfficeDev/teams-toolkit#12665
So, because TTK team is targeting adaptive cards in a separate file for next milestone, it might make sense for us just go with the "final approach" and start generating adaptive cards in a separate file.