Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Application Insights warning on llm model deprecation #2524

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Application Insights warning on llm model deprecation #2524

Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
8068a6e
Slice 548931: [AI][Public Preview]Copilot Toolkit: Targeted ISV commu…
Oct 31, 2024
07d1124
added current deprecated models to deprecation list
Nov 1, 2024
4caeb89
modified dates on model deprecation list to be deprecated from novemb…
Nov 1, 2024
22f3f66
corrected deprecated models from gpt4o to gpt4
Nov 15, 2024
0f7c2f8
Added EventId to log message
Nov 22, 2024
d3e8d63
Apply suggestions from code review
christian-andersen-msft Nov 27, 2024
46fd9f0
Merge commit 'f9210ad33ddccfdcc713e52e8d4bcf7d58839e27' of https://gi…
Dec 2, 2024
86d84dd
Merge branch 'main' of https://github.com/microsoft/BCApps into isv-t…
Dec 2, 2024
10ba1ee
Added debug telemetry 0000AD0
christian-andersen-msft Dec 6, 2024
a6be283
Added additional telemetry tests
christian-andersen-msft Dec 6, 2024
9388ab9
Merge branch 'main' into isv-telemetry-on-llm-deprecation-rebranch
christian-andersen-msft Dec 9, 2024
bbc44ef
temporarely removed duplicate CustomDimension entry
christian-andersen-msft Dec 10, 2024
6b5d7fb
Cleaned up code for release
christian-andersen-msft Dec 10, 2024
e7cc871
Merge branch 'main' into isv-telemetry-on-llm-deprecation-rebranch
christian-andersen-msft Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#if not CLEAN25
using System.Environment;
#endif
using System.Telemetry;

codeunit 7769 "AOAI Deployments Impl"
{
Expand All @@ -15,11 +16,15 @@
InherentPermissions = X;

var
Telemetry: Codeunit Telemetry;
UnableToGetDeploymentNameErr: Label 'Unable to get deployment name, if this is a third party capability you must specify your own deployment name. You may need to contact your partner.';
GPT4oLatestLbl: Label 'gpt-4o-latest', Locked = true;
GPT4oPreviewLbl: Label 'gpt-4o-preview', Locked = true;
GPT4oMiniLatestLbl: Label 'gpt-4o-mini-latest', Locked = true;
GPT4oMiniPreviewLbl: Label 'gpt-4o-mini-preview', Locked = true;
DeprecatedDeployments: Dictionary of [Text, Date];
DeprecationDatesInitialized: Boolean;
DeprecationMessageLbl: Label 'Deployment %1 deprecated from %2. Check out codeunit 7768 AOAI Deployments', Comment = 'Telemetry message where %1 is the name of the deployment and %2 is the date of deprecation';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we translate these strings or do we emit partner telemetry always in english?
If we translate them, you are good, but if we emit in english, you need to lock this string for translations, by adding , Locked = true after you close the quote of the comment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make the text a bit more dev friendly.
We detected usage of the Azure OpenAI deployment "%1". This model is obsoleted starting %2 and the quality of your results might vary after that date. Check out codeunit 7768 AOAI Deployments to find the supported deployments.

#if not CLEAN25
GPT4LatestLbl: Label 'gpt-4-latest', Locked = true;
GPT4PreviewLbl: Label 'gpt-4-preview', Locked = true;
Expand Down Expand Up @@ -103,12 +108,55 @@
exit(GetDeploymentName(GPT4oMiniLatestLbl, CallerModuleInfo));
end;

// Initializes dictionary of deprecated models
local procedure InitializeDeploymentDeprecationDates()
begin
if DeprecationDatesInitialized then
exit;

// Add deprecated deployments with their deprecation dates here
DeprecatedDeployments.Add(Turbo0301SaasLbl, DMY2Date(1, 11, 2024));

Check failure on line 118 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'Turbo0301SaasLbl' does not exist in the current context.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This label is defined in the file, but it's between the #if not CLEAN25 and the #endif preprocessor directive. That means it will be automatically removed in a couple of major updates. At that point, your code will no longer compile, so you need to add the lines for this model and the other obsoleted models within an #if not CLEAN25 and an #endif as well.

We can talk about this in person if you like :)

DeprecatedDeployments.Add(GPT40613SaasLbl, DMY2Date(1, 11, 2024));

Check failure on line 119 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT40613SaasLbl' does not exist in the current context.
DeprecatedDeployments.Add(Turbo0613SaasLbl, DMY2Date(1, 11, 2024));

Check failure on line 120 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'Turbo0613SaasLbl' does not exist in the current context.
DeprecatedDeployments.Add(GPT35TurboLatestLbl, DMY2Date(1, 11, 2024));

Check failure on line 121 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT35TurboLatestLbl' does not exist in the current context.
DeprecatedDeployments.Add(GPT35TurboPreviewLbl, DMY2Date(1, 11, 2024));

Check failure on line 122 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT35TurboPreviewLbl' does not exist in the current context.
DeprecatedDeployments.Add(GPT4PreviewLbl, DMY2Date(1, 11, 2024));

Check failure on line 123 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT4PreviewLbl' does not exist in the current context.
DeprecatedDeployments.Add(GPT4LatestLbl, DMY2Date(1, 11, 2024));

Check failure on line 124 in src/System Application/App/AI/src/Azure OpenAI/AOAIDeploymentsImpl.Codeunit.al

View workflow job for this annotation

GitHub Actions / Build System Application and Tools (Clean) / System Application and Tools (Clean) 

AL0118 The name 'GPT4LatestLbl' does not exist in the current context.

DeprecationDatesInitialized := true;
end;

// Application Insights telemetry on deprecated models
local procedure LogDeprecationTelemetry(DeploymentName: Text)
var
CustomDimensions: Dictionary of [Text, Text];
IsDeprecated: Boolean;
DeprecatedDate: Date;
begin
InitializeDeploymentDeprecationDates();
IsDeprecated := DeprecatedDeployments.ContainsKey(DeploymentName);
if IsDeprecated then begin
DeprecatedDate := DeprecatedDeployments.Get(DeploymentName);
CustomDimensions.Add('DeploymentName', DeploymentName);
CustomDimensions.Add('DeprecationDate', Format(DeprecatedDate));
Telemetry.LogMessage('0000AD1',
StrSubstNo(DeprecationMessageLbl, DeploymentName, DeprecatedDate),
Verbosity::Warning,
DataClassification::SystemMetadata,
Enum::"AL Telemetry Scope"::All,
CustomDimensions);
end;
end;

local procedure GetDeploymentName(DeploymentName: Text; CallerModuleInfo: ModuleInfo): Text
var
AzureOpenAiImpl: Codeunit "Azure OpenAI Impl";
CurrentModuleInfo: ModuleInfo;
begin
LogDeprecationTelemetry(DeploymentName);

NavApp.GetCurrentModuleInfo(CurrentModuleInfo);

if (CallerModuleInfo.Publisher <> CurrentModuleInfo.Publisher) and not AzureOpenAiImpl.IsTenantAllowlistedForFirstPartyCopilotCalls() then
Error(UnableToGetDeploymentNameErr);

Expand Down
Loading