Check logs for errors at smoke tests cleanup #8111

smola · 2024-12-18T16:20:53Z

What Does This Do

Add boolean isErrorLog(String line) defining what is considered an error log for a given smoke test. Most smoke tests can just rely on the default. This PR includes several examples on how to extend it, both to add additional errors, or to exclude some errors.
Add void assertNoErrorLogs(final Closure<Boolean> extraIsErrorLog), which can be used to assert there are no error logs. Most tests should not use directly, and when they do, they should do it only after the test process has finished (otherwise, the test may be flaky).
Call assertNoErrorLogs on cleanupSpec for every smoke test. Most suites were not checking error logs.
Use processTestLogLines when we need to wait for a log line that is supposed to be printed. This should be less flaky than assertNoErrorLogs in the case where the test process has not ended.
Remove checkLogPostExit, which was superseded by the above methods.

Motivation

We have various flaky tests on smoke tests, and sometimes non-flaky failures that are hard to diagnose, that show up in errors all just as:

Condition not satisfied:

!logHasErrors
||
|true
false

This means there was an error printed to logs, but to find which one we need to download reports.tar and dig into the logs.

After this PR, they look like this:

Condition not satisfied:

errorLogs.isEmpty()
|         |
|         false
[16:39:08,836 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@5:21 - no applicable action for [immediateFlush], current pattern is [[configuration][appender][immediateFlush]], 16:39:08,984 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@15:21 - no applicable action for [immediateFlush], current pattern is [[configuration][appender][immediateFlush]]]

Test application log contains 2 errors:
1: 16:39:08,836 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@5:21 - no applicable action for [immediateFlush], current pattern is [[configuration][appender][immediateFlush]]
2: 16:39:08,984 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@15:21 - no applicable action for [immediateFlush], current pattern is [[configuration][appender][immediateFlush]]


	at datadog.smoketest.ProcessManager.assertNoErrorLogs(ProcessManager.groovy:294)
	at datadog.smoketest.ProcessManager.assertNoErrorLogs(ProcessManager.groovy:259)
	at datadog.smoketest.AbstractSmokeTest.cleanupSpec(AbstractSmokeTest.groovy:246)

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2024-12-18T17:00:24Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/smoke-test-log-errors
git_commit_date	1734517793	1734557643
git_commit_sha	`a19f73a`	`acfbbb9`
release_version	1.45.0-SNAPSHOT~a19f73a5ea	1.45.0-SNAPSHOT~acfbbb9657

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1734560185	1734560185
ci_job_id	744804160	744804160
ci_pipeline_id	51504253	51504253
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 52 metrics, 11 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~acfbbb9657, baseline=1.45.0-SNAPSHOT~a19f73a5ea

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.092 s) : 0, 1091748
Total [baseline] (10.47 s) : 0, 10469628
Agent [candidate] (1.102 s) : 0, 1101504
Total [candidate] (10.461 s) : 0, 10460996
section appsec
Agent [baseline] (1.226 s) : 0, 1225662
Total [baseline] (10.69 s) : 0, 10690144
Agent [candidate] (1.227 s) : 0, 1227089
Total [candidate] (10.704 s) : 0, 10704325
section iast
Agent [baseline] (1.219 s) : 0, 1219471
Total [baseline] (10.984 s) : 0, 10984301
Agent [candidate] (1.225 s) : 0, 1224894
Total [candidate] (11.051 s) : 0, 11050643
section profiling
Agent [baseline] (1.333 s) : 0, 1332783
Total [baseline] (10.93 s) : 0, 10929966
Agent [candidate] (1.325 s) : 0, 1324895
Total [candidate] (10.898 s) : 0, 10898440

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.092 s	-
Agent	appsec	1.226 s	133.914 ms (12.3%)
Agent	iast	1.219 s	127.723 ms (11.7%)
Agent	profiling	1.333 s	241.035 ms (22.1%)
Total	tracing	10.47 s	-
Total	appsec	10.69 s	220.517 ms (2.1%)
Total	iast	10.984 s	514.673 ms (4.9%)
Total	profiling	10.93 s	460.338 ms (4.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.102 s	-
Agent	appsec	1.227 s	125.586 ms (11.4%)
Agent	iast	1.225 s	123.39 ms (11.2%)
Agent	profiling	1.325 s	223.392 ms (20.3%)
Total	tracing	10.461 s	-
Total	appsec	10.704 s	243.329 ms (2.3%)
Total	iast	11.051 s	589.647 ms (5.6%)
Total	profiling	10.898 s	437.444 ms (4.2%)

gantt
    title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~acfbbb9657, baseline=1.45.0-SNAPSHOT~a19f73a5ea

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (695.1 ms) : 0, 695100
BytebuddyAgent [candidate] (701.279 ms) : 0, 701279
GlobalTracer [baseline] (317.379 ms) : 0, 317379
GlobalTracer [candidate] (319.773 ms) : 0, 319773
AppSec [baseline] (54.125 ms) : 0, 54125
AppSec [candidate] (55.116 ms) : 0, 55116
Remote Config [baseline] (659.637 µs) : 0, 660
Remote Config [candidate] (666.534 µs) : 0, 667
Telemetry [baseline] (10.789 ms) : 0, 10789
Telemetry [candidate] (10.833 ms) : 0, 10833
section appsec
BytebuddyAgent [baseline] (712.146 ms) : 0, 712146
BytebuddyAgent [candidate] (713.44 ms) : 0, 713440
GlobalTracer [baseline] (314.734 ms) : 0, 314734
GlobalTracer [candidate] (314.779 ms) : 0, 314779
AppSec [baseline] (165.425 ms) : 0, 165425
AppSec [candidate] (165.368 ms) : 0, 165368
Remote Config [baseline] (666.748 µs) : 0, 667
Remote Config [candidate] (668.677 µs) : 0, 669
Telemetry [baseline] (8.44 ms) : 0, 8440
Telemetry [candidate] (8.46 ms) : 0, 8460
IAST [baseline] (21.141 ms) : 0, 21141
IAST [candidate] (21.265 ms) : 0, 21265
section iast
BytebuddyAgent [baseline] (812.616 ms) : 0, 812616
BytebuddyAgent [candidate] (817.932 ms) : 0, 817932
GlobalTracer [baseline] (306.052 ms) : 0, 306052
GlobalTracer [candidate] (305.947 ms) : 0, 305947
AppSec [baseline] (57.293 ms) : 0, 57293
AppSec [candidate] (57.947 ms) : 0, 57947
Remote Config [baseline] (599.358 µs) : 0, 599
Remote Config [candidate] (618.068 µs) : 0, 618
Telemetry [baseline] (7.47 ms) : 0, 7470
Telemetry [candidate] (7.43 ms) : 0, 7430
IAST [baseline] (21.707 ms) : 0, 21707
IAST [candidate] (21.195 ms) : 0, 21195
section profiling
BytebuddyAgent [baseline] (697.865 ms) : 0, 697865
BytebuddyAgent [candidate] (691.638 ms) : 0, 691638
GlobalTracer [baseline] (437.303 ms) : 0, 437303
GlobalTracer [candidate] (437.005 ms) : 0, 437005
AppSec [baseline] (54.156 ms) : 0, 54156
AppSec [candidate] (54.061 ms) : 0, 54061
Remote Config [baseline] (662.271 µs) : 0, 662
Remote Config [candidate] (679.167 µs) : 0, 679
Telemetry [baseline] (7.761 ms) : 0, 7761
Telemetry [candidate] (7.734 ms) : 0, 7734
ProfilingAgent [baseline] (95.369 ms) : 0, 95369
ProfilingAgent [candidate] (94.57 ms) : 0, 94570
Profiling [baseline] (95.392 ms) : 0, 95392
Profiling [candidate] (94.595 ms) : 0, 94595

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~acfbbb9657, baseline=1.45.0-SNAPSHOT~a19f73a5ea

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.103 s) : 0, 1102751
Total [baseline] (8.681 s) : 0, 8680713
Agent [candidate] (1.095 s) : 0, 1095404
Total [candidate] (8.665 s) : 0, 8664584
section iast
Agent [baseline] (1.231 s) : 0, 1231164
Total [baseline] (9.278 s) : 0, 9277673
Agent [candidate] (1.222 s) : 0, 1221573
Total [candidate] (9.213 s) : 0, 9212887
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.221 s) : 0, 1221441
Total [baseline] (9.177 s) : 0, 9176704
Agent [candidate] (1.221 s) : 0, 1221184
Total [candidate] (9.19 s) : 0, 9190090
section iast_TELEMETRY_OFF
Agent [baseline] (1.223 s) : 0, 1223022
Total [baseline] (9.182 s) : 0, 9181697
Agent [candidate] (1.228 s) : 0, 1227528
Total [candidate] (9.193 s) : 0, 9192864

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.103 s	-
Agent	iast	1.231 s	128.413 ms (11.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.221 s	118.69 ms (10.8%)
Agent	iast_TELEMETRY_OFF	1.223 s	120.271 ms (10.9%)
Total	tracing	8.681 s	-
Total	iast	9.278 s	596.96 ms (6.9%)
Total	iast_HARDCODED_SECRET_DISABLED	9.177 s	495.992 ms (5.7%)
Total	iast_TELEMETRY_OFF	9.182 s	500.985 ms (5.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.095 s	-
Agent	iast	1.222 s	126.169 ms (11.5%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.221 s	125.779 ms (11.5%)
Agent	iast_TELEMETRY_OFF	1.228 s	132.124 ms (12.1%)
Total	tracing	8.665 s	-
Total	iast	9.213 s	548.303 ms (6.3%)
Total	iast_HARDCODED_SECRET_DISABLED	9.19 s	525.506 ms (6.1%)
Total	iast_TELEMETRY_OFF	9.193 s	528.28 ms (6.1%)

gantt
    title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~acfbbb9657, baseline=1.45.0-SNAPSHOT~a19f73a5ea

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (701.72 ms) : 0, 701720
BytebuddyAgent [candidate] (697.072 ms) : 0, 697072
GlobalTracer [baseline] (319.253 ms) : 0, 319253
GlobalTracer [candidate] (318.316 ms) : 0, 318316
AppSec [baseline] (55.002 ms) : 0, 55002
AppSec [candidate] (54.305 ms) : 0, 54305
Remote Config [baseline] (671.291 µs) : 0, 671
Remote Config [candidate] (670.972 µs) : 0, 671
Telemetry [baseline] (12.262 ms) : 0, 12262
Telemetry [candidate] (11.304 ms) : 0, 11304
section iast
BytebuddyAgent [baseline] (819.624 ms) : 0, 819624
BytebuddyAgent [candidate] (814.507 ms) : 0, 814507
GlobalTracer [baseline] (310.02 ms) : 0, 310020
GlobalTracer [candidate] (306.619 ms) : 0, 306619
AppSec [baseline] (57.597 ms) : 0, 57597
AppSec [candidate] (57.032 ms) : 0, 57032
Remote Config [baseline] (624.696 µs) : 0, 625
Remote Config [candidate] (610.875 µs) : 0, 611
Telemetry [baseline] (7.509 ms) : 0, 7509
Telemetry [candidate] (7.424 ms) : 0, 7424
IAST [baseline] (21.941 ms) : 0, 21941
IAST [candidate] (21.634 ms) : 0, 21634
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (814.059 ms) : 0, 814059
BytebuddyAgent [candidate] (813.804 ms) : 0, 813804
GlobalTracer [baseline] (306.463 ms) : 0, 306463
GlobalTracer [candidate] (306.083 ms) : 0, 306083
AppSec [baseline] (58.052 ms) : 0, 58052
AppSec [candidate] (56.564 ms) : 0, 56564
Remote Config [baseline] (620.042 µs) : 0, 620
Remote Config [candidate] (618.745 µs) : 0, 619
Telemetry [baseline] (7.46 ms) : 0, 7460
Telemetry [candidate] (7.471 ms) : 0, 7471
IAST [baseline] (21.046 ms) : 0, 21046
IAST [candidate] (22.897 ms) : 0, 22897
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (817.195 ms) : 0, 817195
BytebuddyAgent [candidate] (818.682 ms) : 0, 818682
GlobalTracer [baseline] (305.509 ms) : 0, 305509
GlobalTracer [candidate] (308.328 ms) : 0, 308328
AppSec [baseline] (58.083 ms) : 0, 58083
AppSec [candidate] (57.064 ms) : 0, 57064
Remote Config [baseline] (598.623 µs) : 0, 599
Remote Config [candidate] (608.866 µs) : 0, 609
Telemetry [baseline] (7.309 ms) : 0, 7309
Telemetry [candidate] (7.405 ms) : 0, 7405
IAST [baseline] (20.499 ms) : 0, 20499
IAST [candidate] (21.574 ms) : 0, 21574

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-12-18T21:46:01	2024-12-18T21:52:57
git_branch	master	smola/smoke-test-log-errors
git_commit_date	1734517793	1734557643
git_commit_sha	`a19f73a`	`acfbbb9`
release_version	1.45.0-SNAPSHOT~a19f73a5ea	1.45.0-SNAPSHOT~acfbbb9657
start_time	2024-12-18T21:45:47	2024-12-18T21:52:44

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1734559129	1734559129
ci_job_id	744804162	744804162
ci_pipeline_id	51504253	51504253
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~acfbbb9657, baseline=1.45.0-SNAPSHOT~a19f73a5ea
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.336 ms) : 1316, 1355
.   : milestone, 1336,
appsec (1.748 ms) : 1724, 1771
.   : milestone, 1748,
appsec_no_iast (1.748 ms) : 1723, 1773
.   : milestone, 1748,
iast (1.501 ms) : 1479, 1523
.   : milestone, 1501,
profiling (1.497 ms) : 1473, 1521
.   : milestone, 1497,
tracing (1.49 ms) : 1466, 1514
.   : milestone, 1490,
section candidate
no_agent (1.342 ms) : 1323, 1362
.   : milestone, 1342,
appsec (1.735 ms) : 1712, 1758
.   : milestone, 1735,
appsec_no_iast (1.747 ms) : 1722, 1772
.   : milestone, 1747,
iast (1.49 ms) : 1468, 1512
.   : milestone, 1490,
profiling (1.515 ms) : 1492, 1538
.   : milestone, 1515,
tracing (1.486 ms) : 1461, 1511
.   : milestone, 1486,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.336 ms [1.316 ms, 1.355 ms]	-
appsec	1.748 ms [1.724 ms, 1.771 ms]	411.97 µs (30.8%)
appsec_no_iast	1.748 ms [1.723 ms, 1.773 ms]	412.083 µs (30.8%)
iast	1.501 ms [1.479 ms, 1.523 ms]	165.125 µs (12.4%)
profiling	1.497 ms [1.473 ms, 1.521 ms]	161.512 µs (12.1%)
tracing	1.49 ms [1.466 ms, 1.514 ms]	154.225 µs (11.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.342 ms [1.323 ms, 1.362 ms]	-
appsec	1.735 ms [1.712 ms, 1.758 ms]	392.798 µs (29.3%)
appsec_no_iast	1.747 ms [1.722 ms, 1.772 ms]	404.589 µs (30.1%)
iast	1.49 ms [1.468 ms, 1.512 ms]	147.551 µs (11.0%)
profiling	1.515 ms [1.492 ms, 1.538 ms]	172.186 µs (12.8%)
tracing	1.486 ms [1.461 ms, 1.511 ms]	143.419 µs (10.7%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~acfbbb9657, baseline=1.45.0-SNAPSHOT~a19f73a5ea
    dateFormat X
    axisFormat %s
section baseline
no_agent (374.483 µs) : 353, 396
.   : milestone, 374,
iast (488.21 µs) : 467, 510
.   : milestone, 488,
iast_FULL (646.883 µs) : 625, 668
.   : milestone, 647,
iast_GLOBAL (516.146 µs) : 494, 538
.   : milestone, 516,
iast_HARDCODED_SECRET_DISABLED (487.353 µs) : 466, 509
.   : milestone, 487,
iast_INACTIVE (451.867 µs) : 430, 473
.   : milestone, 452,
iast_TELEMETRY_OFF (476.608 µs) : 455, 498
.   : milestone, 477,
tracing (447.562 µs) : 427, 469
.   : milestone, 448,
section candidate
no_agent (371.084 µs) : 351, 391
.   : milestone, 371,
iast (491.001 µs) : 469, 513
.   : milestone, 491,
iast_FULL (643.783 µs) : 622, 665
.   : milestone, 644,
iast_GLOBAL (525.045 µs) : 502, 548
.   : milestone, 525,
iast_HARDCODED_SECRET_DISABLED (484.06 µs) : 463, 505
.   : milestone, 484,
iast_INACTIVE (447.485 µs) : 426, 469
.   : milestone, 447,
iast_TELEMETRY_OFF (474.91 µs) : 453, 497
.   : milestone, 475,
tracing (449.544 µs) : 428, 471
.   : milestone, 450,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	374.483 µs [352.906 µs, 396.06 µs]	-
iast	488.21 µs [466.607 µs, 509.813 µs]	113.727 µs (30.4%)
iast_FULL	646.883 µs [625.438 µs, 668.328 µs]	272.4 µs (72.7%)
iast_GLOBAL	516.146 µs [494.321 µs, 537.972 µs]	141.663 µs (37.8%)
iast_HARDCODED_SECRET_DISABLED	487.353 µs [466.024 µs, 508.683 µs]	112.87 µs (30.1%)
iast_INACTIVE	451.867 µs [430.25 µs, 473.485 µs]	77.384 µs (20.7%)
iast_TELEMETRY_OFF	476.608 µs [455.311 µs, 497.905 µs]	102.125 µs (27.3%)
tracing	447.562 µs [426.584 µs, 468.54 µs]	73.078 µs (19.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	371.084 µs [351.24 µs, 390.928 µs]	-
iast	491.001 µs [469.269 µs, 512.734 µs]	119.917 µs (32.3%)
iast_FULL	643.783 µs [622.148 µs, 665.419 µs]	272.699 µs (73.5%)
iast_GLOBAL	525.045 µs [502.195 µs, 547.896 µs]	153.961 µs (41.5%)
iast_HARDCODED_SECRET_DISABLED	484.06 µs [462.902 µs, 505.218 µs]	112.976 µs (30.4%)
iast_INACTIVE	447.485 µs [426.346 µs, 468.625 µs]	76.401 µs (20.6%)
iast_TELEMETRY_OFF	474.91 µs [453.115 µs, 496.706 µs]	103.826 µs (28.0%)
tracing	449.544 µs [428.262 µs, 470.827 µs]	78.46 µs (21.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	smola/smoke-test-log-errors
git_commit_date	1734517793	1734557643
git_commit_sha	`a19f73a`	`acfbbb9`
release_version	1.45.0-SNAPSHOT~a19f73a5ea	1.45.0-SNAPSHOT~acfbbb9657

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1734559733	1734559733
ci_job_id	744804164	744804164
ci_pipeline_id	51504253	51504253
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~acfbbb9657, baseline=1.45.0-SNAPSHOT~a19f73a5ea
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.043 s) : 15043000, 15043000
.   : milestone, 15043000,
appsec (14.735 s) : 14735000, 14735000
.   : milestone, 14735000,
iast (19.102 s) : 19102000, 19102000
.   : milestone, 19102000,
iast_GLOBAL (17.822 s) : 17822000, 17822000
.   : milestone, 17822000,
profiling (15.43 s) : 15430000, 15430000
.   : milestone, 15430000,
tracing (14.922 s) : 14922000, 14922000
.   : milestone, 14922000,
section candidate
no_agent (15.191 s) : 15191000, 15191000
.   : milestone, 15191000,
appsec (14.929 s) : 14929000, 14929000
.   : milestone, 14929000,
iast (19.102 s) : 19102000, 19102000
.   : milestone, 19102000,
iast_GLOBAL (18.066 s) : 18066000, 18066000
.   : milestone, 18066000,
profiling (14.913 s) : 14913000, 14913000
.   : milestone, 14913000,
tracing (15.092 s) : 15092000, 15092000
.   : milestone, 15092000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.043 s [15.043 s, 15.043 s]	-
appsec	14.735 s [14.735 s, 14.735 s]	-308.0 ms (-2.0%)
iast	19.102 s [19.102 s, 19.102 s]	4.059 s (27.0%)
iast_GLOBAL	17.822 s [17.822 s, 17.822 s]	2.779 s (18.5%)
profiling	15.43 s [15.43 s, 15.43 s]	387.0 ms (2.6%)
tracing	14.922 s [14.922 s, 14.922 s]	-121.0 ms (-0.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.191 s [15.191 s, 15.191 s]	-
appsec	14.929 s [14.929 s, 14.929 s]	-262.0 ms (-1.7%)
iast	19.102 s [19.102 s, 19.102 s]	3.911 s (25.7%)
iast_GLOBAL	18.066 s [18.066 s, 18.066 s]	2.875 s (18.9%)
profiling	14.913 s [14.913 s, 14.913 s]	-278.0 ms (-1.8%)
tracing	15.092 s [15.092 s, 15.092 s]	-99.0 ms (-0.7%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~acfbbb9657, baseline=1.45.0-SNAPSHOT~a19f73a5ea
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.465 ms) : 1453, 1476
.   : milestone, 1465,
appsec (2.337 ms) : 2295, 2379
.   : milestone, 2337,
iast (2.087 ms) : 2034, 2140
.   : milestone, 2087,
iast_GLOBAL (2.128 ms) : 2075, 2182
.   : milestone, 2128,
profiling (1.936 ms) : 1894, 1978
.   : milestone, 1936,
tracing (1.922 ms) : 1881, 1962
.   : milestone, 1922,
section candidate
no_agent (1.464 ms) : 1452, 1475
.   : milestone, 1464,
appsec (2.328 ms) : 2286, 2370
.   : milestone, 2328,
iast (2.074 ms) : 2021, 2127
.   : milestone, 2074,
iast_GLOBAL (2.13 ms) : 2076, 2184
.   : milestone, 2130,
profiling (2.422 ms) : 2244, 2599
.   : milestone, 2422,
tracing (1.926 ms) : 1885, 1967
.   : milestone, 1926,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.465 ms [1.453 ms, 1.476 ms]	-
appsec	2.337 ms [2.295 ms, 2.379 ms]	872.333 µs (59.6%)
iast	2.087 ms [2.034 ms, 2.14 ms]	622.189 µs (42.5%)
iast_GLOBAL	2.128 ms [2.075 ms, 2.182 ms]	663.551 µs (45.3%)
profiling	1.936 ms [1.894 ms, 1.978 ms]	471.523 µs (32.2%)
tracing	1.922 ms [1.881 ms, 1.962 ms]	456.941 µs (31.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.464 ms [1.452 ms, 1.475 ms]	-
appsec	2.328 ms [2.286 ms, 2.37 ms]	864.281 µs (59.0%)
iast	2.074 ms [2.021 ms, 2.127 ms]	610.557 µs (41.7%)
iast_GLOBAL	2.13 ms [2.076 ms, 2.184 ms]	666.241 µs (45.5%)
profiling	2.422 ms [2.244 ms, 2.599 ms]	957.991 µs (65.4%)
tracing	1.926 ms [1.885 ms, 1.967 ms]	462.08 µs (31.6%)

smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Dec 18, 2024

smola force-pushed the smola/smoke-test-log-errors branch 6 times, most recently from 2e82afe to 112f4f1 Compare December 18, 2024 20:53

Check logs for errors at smoke tests cleanup

acfbbb9

smola force-pushed the smola/smoke-test-log-errors branch from 112f4f1 to acfbbb9 Compare December 18, 2024 21:34

smola marked this pull request as ready for review December 18, 2024 22:24

smola requested review from a team as code owners December 18, 2024 22:24

smola requested review from manuel-alvarez-alvarez and Mariovido December 18, 2024 22:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check logs for errors at smoke tests cleanup #8111

Check logs for errors at smoke tests cleanup #8111

smola commented Dec 18, 2024 •

edited

Loading

pr-commenter bot commented Dec 18, 2024 •

edited

Loading

Check logs for errors at smoke tests cleanup #8111

Are you sure you want to change the base?

Check logs for errors at smoke tests cleanup #8111

Conversation

smola commented Dec 18, 2024 • edited Loading

What Does This Do

Motivation

Additional Notes

Contributor Checklist

pr-commenter bot commented Dec 18, 2024 • edited Loading

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

smola commented Dec 18, 2024 •

edited

Loading

pr-commenter bot commented Dec 18, 2024 •

edited

Loading