fix(amazonq): Error handling and telemetry for Unit test generation. #5192

laileni-aws · 2024-12-11T19:21:31Z

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Description

Adding 4XX vs 5XX httpStatusCode field to amazonq_utgGenerateTests event.
Improving error handling in unit test generation.

Checklist

My code follows the code style of this project
I have added tests to cover my changes
A short description of the change has been added to the CHANGELOG if the change is customer-facing in the IDE.
I have added metrics for my changes (if required)

License

I confirm that my contribution is made under the terms of the Apache 2.0 license.

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

...software/aws/toolkits/jetbrains/services/codewhisperer/util/CodeWhispererZipUploadManager.kt

+                    "UploadTestArtifactToS3Error: $errorMessage",
+                    "403",
+                    "UploadTestArtifactToS3Error",
+                    message("testgen.error.generic_technical_error_message")


...software/aws/toolkits/jetbrains/services/codewhisperer/util/CodeWhispererZipUploadManager.kt

+                "CreateUploadUrlError: $errorMessage",
+                "500",
+                "CreateUploadUrlError",
+                message("testgen.error.generic_technical_error_message")


...software/aws/toolkits/jetbrains/services/codewhisperer/util/CodeWhispererZipUploadManager.kt

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+
+package software.aws.toolkits.jetbrains.services.codewhisperer.codetest
+
+import software.aws.toolkits.resources.message


...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+    throw CodeTestException(message("codewhisperer.codescan.file_too_large_telemetry"), "400", "ProjectZipError")
+
+internal fun cannotFindFile(errorMessage: String, filepath: String): Nothing =
+    error(message("codewhisperer.codescan.file_not_found", filepath, errorMessage))


...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

ctlai95 · 2024-12-13T22:47:35Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+            throw codeTestServerException(
+                "CreateTestJobError: $errorMessage",
+                "400",


Should this be renamed? It mentions server exception but uses 4xx

Yeah this needs to be e.statusCode() but for now this can be 5XX.

rli · 2024-12-13T22:49:25Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+        } catch (e: Exception) {
+            LOG.error(e) { "Failed to create test generation job" }
+            // TODO: Not able to emit e.statusCode directly as statusCode is private property
+            // Cannot access 'statusCode': it is invisible (private in a supertype) in 'ThrottlingException'


try statusCode()

Yeah, I tried statusCode(), no luck.

ctlai95 · 2024-12-13T22:50:28Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -173,7 +190,13 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
                    }
                    // update test summary card
                } else {
-                    throw Exception(message("testgen.message.failed"))
+                    // If job status is Completed and has no ShortAnswer then there might be some issue in the backend.
+                    throw codeTestServerException(


Have you verified that this works? It looks like you are throwing a function which throws an exception

Verified in Kibana and its working fine. Attached related links in Slack.

rli · 2024-12-13T22:50:56Z

...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+
+fun codeTestServerException(
+    errorMessage: String,
+    statusCode: String?,


status code should always be a number

statusCode is string in MetricBase in VSC so to align with VSC and JB I initialized this to String.

Thought of using number but in our Telemetry statusCode is declared as String.
aws-toolkit-common

If required I can use number here and can convert to string while emitting?

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+                        "TestGenFailedError: " + message("testgen.message.failed"),
+                        "500",
+                        "TestGenFailedError",
+                        message("testgen.error.generic_technical_error_message")


.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+                        message("testgen.error.maximum_generations_reach")
+
+                    e is CodeTestException -> e.uiMessage
+                    e is JsonParseException -> message("testgen.error.generic_technical_error_message")


...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+    override val message: String?,
+    val statusCode: String? = "400",
+    val code: String? = "DefaultError",
+    val uiMessage: String? = message(


...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

rli · 2024-12-13T23:23:15Z

...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+internal fun cannotFindValidFile(errorMessage: String): Nothing =
+    throw CodeTestException(errorMessage, "400", "ProjectZipError")
+
+internal fun cannotFindBuildArtifacts(errorMessage: String): Nothing =
+    throw CodeTestException(errorMessage, "400", "ProjectZipError")
+
+internal fun invalidSourceZipError(): Nothing =
+    throw CodeTestException(message("codewhisperer.codescan.invalid_source_zip_telemetry"), "400", "InvalidSourceZipError")


i am pretty sure none of these are actually error code 400, please use the actual error code from the service

These are client side errors so there wont be any error codes from service

if it is client side error then why would we have an http status code

Its a request from manager to see the statusCodes accordingly in the Kibana to differentiate the errors.

rli · 2024-12-13T23:24:14Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+                    e is CodeTestException && e.statusCode == "400" &&
+                        e.message?.startsWith("CreateTestJobError: Maximum") == true ->


is there a better way of doing this

Here Quota exceeded is subset of Throttling exception so opted for checking the error message.

rli · 2024-12-13T23:24:40Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

@@ -517,8 +554,9 @@ class CodeWhispererUTGChatManager(val project: Project, private val cs: Coroutin
                    jobGroup = session.testGenerationJobGroupName,
                    jobId = session.testGenerationJob,
                    result = if (e.message == message("testgen.message.cancelled")) MetricResult.Cancelled else MetricResult.Failed,
-                    reason = e.javaClass.name,
-                    reasonDesc = e.message,
+                    reason = (e as CodeTestException).code ?: "DefaultError",


why don't we just let it be null?

It can be but placing this under the bucket of DefaultError to be consistent in Kibana.

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+
+                    e is CodeTestException -> e.uiMessage
+                    e is JsonParseException -> message("testgen.error.generic_technical_error_message")
+                    else -> message("testgen.error.generic_error_message")


rli · 2024-12-14T01:28:37Z

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+                e is ThrottlingException -> e.statusCode()
+                else -> 500


Suggested change

e is ThrottlingException -> e.statusCode()

else -> 500

e is SdkServiceException -> e.statusCode()

else -> ??? // not a service exception, so it would be ???

.../src/software/aws/toolkits/jetbrains/services/amazonqCodeTest/CodeWhispererUTGChatManager.kt

+            throw CodeTestException(
+                "CreateTestJobError: $errorMessage",
+                "CreateTestJobError",
+                message("testgen.error.generic_technical_error_message")


...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+) : RuntimeException()
+
+internal fun noFileOpenError(): Nothing =
+    throw CodeTestException(message("codewhisperer.codescan.no_file_open"), "ProjectZipError")


...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+    throw CodeTestException(message("codewhisperer.codescan.no_file_open"), "ProjectZipError")
+
+internal fun fileTooLarge(): Nothing =
+    throw CodeTestException(message("codewhisperer.codescan.file_too_large_telemetry"), "ProjectZipError")


...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+    throw CodeTestException(errorMessage, "ProjectZipError")
+
+internal fun invalidSourceZipError(): Nothing =
+    throw CodeTestException(message("codewhisperer.codescan.invalid_source_zip_telemetry"), "InvalidSourceZipError")


...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+    throw CodeTestException(message("codewhisperer.codescan.invalid_source_zip_telemetry"), "InvalidSourceZipError")
+
+fun testGenStoppedError(): Nothing =
+    throw CodeTestException(message("testgen.message.cancelled"), "TestGenCancelled", message("testgen.message.cancelled"))


...ity/src/software/aws/toolkits/jetbrains/services/codewhisperer/codetest/CodeTestException.kt

+    throw CodeTestException(message("codewhisperer.codescan.invalid_source_zip_telemetry"), "InvalidSourceZipError")
+
+fun testGenStoppedError(): Nothing =
+    throw CodeTestException(message("testgen.message.cancelled"), "TestGenCancelled", message("testgen.message.cancelled"))


laileni-aws · 2024-12-14T15:06:21Z

/retryBuilds

laileni-aws added 3 commits December 11, 2024 10:29

Error handling for UTG

b733179

Error handling for UTG

bc08f99

Merge branch 'aws:main' into main

9ee9b13

laileni-aws requested review from a team as code owners December 11, 2024 19:21

github-advanced-security bot found potential problems Dec 11, 2024

View reviewed changes

laileni-aws marked this pull request as draft December 11, 2024 19:43

Minor edits addresing comments

c04de5e

laileni-aws force-pushed the main branch from 9f7151d to c04de5e Compare December 12, 2024 02:44

laileni-aws added 2 commits December 12, 2024 06:54

Lint errors

b22eca6

Merge branch 'main' into main

10b39bb

github-advanced-security bot found potential problems Dec 12, 2024

View reviewed changes

laileni-aws added 6 commits December 12, 2024 21:12

Merge branch 'main' into main

7fc0959

Merge branch 'main' into main

ee4de19

Merge branch 'main' into main

95b7f1b

Fixing build failures

75a6254

Fixing build failures

3a87313

Merge branch 'main' into main

4461636

github-advanced-security bot found potential problems Dec 13, 2024

View reviewed changes

Removing CodeTestServerException from CodeTestException file

143c8c7

laileni-aws marked this pull request as ready for review December 13, 2024 22:34

Adding Generic error message for CX facing UX

d893719

ctlai95 reviewed Dec 13, 2024

View reviewed changes

rli reviewed Dec 13, 2024

View reviewed changes

ctlai95 reviewed Dec 13, 2024

View reviewed changes

rli reviewed Dec 13, 2024

View reviewed changes

github-advanced-security bot found potential problems Dec 13, 2024

View reviewed changes

Changing status code from 4XX to 5XX for CreateTestJobError

4698590

rli reviewed Dec 13, 2024

View reviewed changes

laileni-aws added 2 commits December 13, 2024 15:56

Moving statusCode datatype from string to int

a0c6fc1

Adding perfClientLatency to telemetry event (#4)

0bab6b6

laileni-aws requested review from ctlai95 and rli December 14, 2024 00:00

github-advanced-security bot found potential problems Dec 14, 2024

View reviewed changes

Addressing comments

1c89a6e

ctlai95 approved these changes Dec 14, 2024

View reviewed changes

github-advanced-security bot found potential problems Dec 14, 2024

View reviewed changes

rli approved these changes Dec 14, 2024

View reviewed changes

laileni-aws added 4 commits December 13, 2024 17:39

Addressing Comments

f200c7c

Merge branch 'main' into main

f86d1cb

Removing the statusCode from UTG telemetry

d62ae9c

Minor edits

584547b

github-advanced-security bot found potential problems Dec 14, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(amazonq): Error handling and telemetry for Unit test generation. #5192

fix(amazonq): Error handling and telemetry for Unit test generation. #5192

laileni-aws commented Dec 11, 2024

ctlai95 Dec 13, 2024

laileni-aws Dec 13, 2024

rli Dec 13, 2024

laileni-aws Dec 13, 2024

rli Dec 13, 2024

ctlai95 Dec 13, 2024

laileni-aws Dec 13, 2024

rli Dec 13, 2024

ctlai95 Dec 13, 2024

laileni-aws Dec 13, 2024 •

edited

Loading

laileni-aws Dec 13, 2024

rli Dec 13, 2024

laileni-aws Dec 13, 2024

rli Dec 13, 2024

laileni-aws Dec 13, 2024

rli Dec 13, 2024

laileni-aws Dec 14, 2024 •

edited

Loading

rli Dec 13, 2024

laileni-aws Dec 13, 2024

rli Dec 13, 2024

laileni-aws Dec 13, 2024

rli Dec 14, 2024

laileni-aws commented Dec 14, 2024


		package software.aws.toolkits.jetbrains.services.codewhisperer.codetest

		import software.aws.toolkits.resources.message

		e is CodeTestException && e.statusCode == "400" &&
		e.message?.startsWith("CreateTestJobError: Maximum") == true ->

fix(amazonq): Error handling and telemetry for Unit test generation. #5192

Are you sure you want to change the base?

fix(amazonq): Error handling and telemetry for Unit test generation. #5192

Conversation

laileni-aws commented Dec 11, 2024

Types of changes

Description

Checklist

License

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laileni-aws Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laileni-aws Dec 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laileni-aws commented Dec 14, 2024

laileni-aws Dec 13, 2024 •

edited

Loading

laileni-aws Dec 14, 2024 •

edited

Loading