Temporal 1.24.1 <- 1.23.1 upgrade error

[harish4459]

Expected Behavior

Application should work normally after upgrading

Actual Behavior

Application is crashing after upgrading to version 1.24.1

Steps to Reproduce the Problem

postgreSQL Schema upgrade to PostgreSQL schema v1.12 from v1.11 is successfully done.
Application upgrade to version 1.24.1 from older version was successful .

Our setup is we are using single db schema (core) and not using visibility schema but there are few tables from visibility schema in our core db.
As per logs error seems to be it is hitting visibility schema tables but not sure what is causing this ?

Specifications

• Version: 1.24.1 with postgresSQL
• Platform: AWS EKS version 1.29

temporal-frontend logs-

{"level":"error","ts":"2024-11-20T06:57:21.362Z","msg":"service failures","operation":"CountWorkflowExecutions","wf-namespace":"default","error":"CountWorkflowExecutions operation failed. Query failed: pq: column \"temporalnamespacedivision\" does not exist","logging-call-at":"telemetry.go:411","stacktrace │ │ ":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).handleError\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/te │ │ lemetry.go:411\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:202\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/ │ │ server.go:1186\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation.func2\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:245\ngo.temporal.io/server/service/frontend.(*NoopRedirectionPolicy).WithNamespaceRedirect │ │ \n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/dc_redirection_policy.go:125\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleRedirectAPIInvocation\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:242\ng │ │ o.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/frontend/redirection_interceptor.go:202\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo │ │ .temporal.io/server/common/authorization.(*Interceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/authorization/interceptor.go:181\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/ser │ │ ver/service/frontend.GrpcServerOptionsProvider.NewServerMetricsContextInjectorInterceptor.func2\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo │ │ .opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:315\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/ │ │ mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_logger.go:85\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/ │ │ pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/namespace_validator.go:135\ngoogle.golang.org/grpc.getChainUnaryHa │ │ ndler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/common/utf8validator.(*Validator).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/utf8validator/validate.go:182\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/ │ │ runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1186\ngo.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServiceErrorInterceptor.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/grpc.go:178\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors. │ │ chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1177\ngo.temporal.io/api/workflowservice/v1._WorkflowService_CountWorkflowExecutions_Handler\n\t/home/runner/go/pkg/mod/go.temporal.io/[email protected]/workflowservice/v1/service_grpc.pb.go:2217\ngoogle.golang.org/g │ │ rpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1369\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1780\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/go │ │ ogle.golang.org/[email protected]/server.go:1019"}