Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problems encountered by codeql in building chromium QL library 2 #13489

Open
cooolie opened this issue Jun 17, 2023 · 13 comments
Open

Problems encountered by codeql in building chromium QL library 2 #13489

cooolie opened this issue Jun 17, 2023 · 13 comments
Labels
question Further information is requested

Comments

@cooolie
Copy link

cooolie commented Jun 17, 2023

According to the method mentioned here,
i removed \out\debug\obj\components\autofill

codeql:2.13.3.0
error:
"../../third_party/protobuf/src/google/protobuf/arena.h", line 486: error: no instance of overloaded "operator new" matches the argument list
            argument types are: (unsigned int, void *)
        return new (ptr) T(static_cast<Args&&>(args)...);

[T 20:53:09 24244] invocation: D:\codeql\bin\codeql-win64\codeql\cpp\tools\win64\extractor.exe, args: --mimic "d:\chromium\src\third_party\llvm-build\release+asserts\bin\clang-cl.exe" /c ../../components/autofill/content/browser/content_autofill_router.cc /Foobj/components/autofill/content/browser/browser/content_autofill_router.obj /nologo /showIncludes:user "-imsvcC:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.35.32215\include" "-imsvcC:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.35.32215\ATLMFC\include" "-imsvcC:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Auxiliary\VS\include" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-imsvcC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-imsvcC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-imsvcC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-imsvcC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" -DDCHECK_ALWAYS_ON=1 -DUSE_AURA=1 -D_HAS_NODISCARD -D_CRT_NONSTDC_NO_WARNINGS -D_WINSOCK_DEPRECATED_NO_WARNINGS -D_LIBCPP_ENABLE_ASSERTIONS=1 "-DCR_CLANG_REVISION=\"llvmorg-17-init-12166-g7586aeab-3\"" -DCOMPONENT_BUILD -DCR_LIBCXX_REVISION=055b2e17ae4f0e2c025ad0c7508b01787df17758 -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=2 -DWIN32 -D_SECURE_ATL -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=NTDDI_WIN10_NI -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DLIBYUV_DISABLE_NEON -DWEBP_EXTERN=extern -DUSE_EGL -DVK_USE_PLATFORM_WIN32_KHR -D_WTL_NO_AUTOMATIC_NAMESPACE -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DGOOGLE_PROTOBUF_INTERNAL_DONATE_STEAL_INLINE=0 -DPROTOBUF_USE_DLLS -DABSL_CONSUME_DLL -DBORINGSSL_SHARED_LIBRARY -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DLEVELDB_PLATFORM_CHROMIUM=1 -DLEVELDB_SHARED_LIBRARY -DSK_ENABLE_SKSL -DSK_UNTIL_CRBUG_1187654_IS_FIXED "-DSK_USER_CONFIG_HEADER=\"../../skia/config/SkUserConfig.h\"" -DSK_WIN_FONTMGR_NO_SIMULATIONS -DSK_DISABLE_LEGACY_SKSURFACE_METHODS -DSK_DISABLE_LEGACY_GRAPHITE_IMAGE_FACTORIES -DSK_DISABLE_LEGACY_GRAPHITE_IMAGE_METHODS -DSK_DISABLE_LEGACY_SKSURFACE_FACTORIES -DSK_DISABLE_LEGACY_SKSURFACE_FLUSH -DSK_DISABLE_LEGACY_SKSURFACE_AS_IMAGE -DSK_DISABLE_LEGACY_SKSURFACE_DISPLAYLIST -DSK_CODEC_DECODES_JPEG -DSK_CODEC_DECODES_PNG -DSK_CODEC_DECODES_WEBP -DSK_ENCODE_JPEG -DSK_ENCODE_PNG -DSK_ENCODE_WEBP -DSK_HAS_WUFFS_LIBRARY -DSKIA_DLL -DSKCMS_API=__declspec(dllexport) -DGR_GL_FUNCTION_TYPE=__stdcall -DSK_GANESH "-DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\"" -DSK_GL -DSK_VULKAN=1 -DSK_GRAPHITE -DSK_DAWN -DWGPU_SHARED_LIBRARY "-DI18N_ADDRESS_VALIDATION_DATA_URL=\"https://chromium-i18n.appspot.com/ssl-aggregate-address/\"" -DCRASHPAD_ZLIB_SOURCE_EXTERNAL -DWEBRTC_ENABLE_SYMBOL_EXPORT -DWEBRTC_ENABLE_AVX2 -DRTC_ENABLE_WIN_WGC -DWEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0 -DWEBRTC_CHROMIUM_BUILD -DWEBRTC_WIN -DABSL_ALLOCATOR_NOTHROW=1 -DLOGGING_INSIDE_WEBRTC -DV8_USE_EXTERNAL_STARTUP_DATA -DUSING_V8_SHARED -DV8_ENABLE_CHECKS -DV8_COMPRESS_POINTERS -DV8_COMPRESS_POINTERS_IN_SHARED_CAGE -DV8_31BIT_SMIS_ON_64BIT_ARCH -DV8_ENABLE_SANDBOX -DV8_DEPRECATION_WARNINGS -DCPPGC_CAGED_HEAP -DCPPGC_YOUNG_GENERATION -DCPPGC_POINTER_COMPRESSION -DCPPGC_SLIM_WRITE_BARRIER -I../.. -Igen -I../../buildtools/third_party/libc++ -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -I../../third_party/libyuv/include -I../../third_party/jsoncpp/source/include -Igen/third_party/dawn/include -I../../third_party/dawn/include -I../../third_party/libwebp/src/src -Igen/third_party/private_membership/src -Igen/third_party/shell-encryption/src -Igen/components/pol[E 20:53:09 20676] Warning[extractor-c++]: In index_template: [2001c5d5868] Unexpected template kind 9.
icy/proto -I../../third_party/khronos -I../../gpu -I../../third_party/vulkan-deps/vulkan-headers/src/include -I../../third_party/wtl/include -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/abseil-cpp -I../../third_party/boringssl/src/include -I../../third_party/ipcz/include -I../../third_party/ced/src -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/leveldatabase -I../../third_party/leveldatabase/src -I../../third_party/leveldatabase/src/include -I../../net/third_party/quiche/overrides -I../../net/third_party/quiche/src/quiche/common/platform/default -I../../net/third_party/quiche/src -Igen/net/third_party/quiche/src -I../../third_party/skia -Igen/third_party/skia -I../../third_party/wuffs/src/release/c -I../../third_party/vulkan/include -I../../third_party/libaddressinput/src/cpp/include -I../../third_party/crashpad/crashpad -I../../third_party/crashpad/crashpad/compat/win -I../../third_party/zlib -I../../third_party/webrtc_overrides -I../../third_party/webrtc -Igen/third_party/webrtc -I../../third_party/libwebm/source -I../../third_party/mesa_headers -I../../third_party/libaom/source/libaom -I../../v8/include -Igen/v8/include /W4 -Wimplicit-fallthrough -Wextra-semi -Wunreachable-code-aggressive -Wthread-safety /WX -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -Wloop-analysis -Wno-unneeded-internal-declaration -Wno-nonportable-include-path -Wenum-compare-conditional -Wno-ignored-pragma-optimize -Wno-deprecated-builtins -Wno-bitfield-constant-conversion -Wno-deprecated-this-capture -Wshadow -fno-delete-null-pointer-checks -fno-ident -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 /clang:-ffp-contract=off -fcomplete-member-pointers /Gy /FS /bigobj /utf-8 /Zc:twoPhase -ffile-reproducible /Zc:sizedDealloc- /D__WRL_ENABLE_FUNCTION_STATICS__ -fmsc-version=1934 /Zc:dllexportInlines- -m64 -msse3 /Brepro -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern /Od /Ob0 /GF /Z7 -gno-codeview-command-line -gcodeview-ghash -Xclang -fuse-ctor-homing /guard:cf,nochecks /MDd -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ref-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang check-stack-allocated -Xclang -plugin-arg-find-bad-constructs -Xclang check-raw-ptr-fields -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=testing/platform_test.h -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=sandbox/mac/ -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=chrome/browser/ui/cocoa/touchbar/browser_window_default_touch_bar.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=device/bluetooth/test/mock_bluetooth_central_manager_mac.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=device/bluetooth/test/mock_bluetooth_cbperipheral_mac.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=ui/views/controls/native/native_view_host_mac_unittest.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=ui/views/widget/native_widget_mac_unittest.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=ui/views/controls/native/native_view_host_mac_unittest.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=um/winnt.h -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wno-redundant-parens -Wno-null-pointer-subtraction -DPROTOBUF_ALLOW_DEPRECATED=1 -Wno-redundant-parens -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare /std:c++20 -Wno-trigraphs /TP /GR- -I../../buildtools/third_party/libc++/trunk/include /Fd"obj/components/autofill/content/browser/browser_cc.pdb"
[T 20:53:09 24244] trace_languages(1): [cpp]
[E 20:53:09 20676] Warning[extractor-c++]: In index_template: [2001c45a540] Unexpected template kind 9.
[E 20:53:09 20676] Warning[extractor-c++]: In index_template: [2001c459970] Unexpected template kind 9.
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_undef.inc
[E 20:53:09 26764] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/stubs/stl_util.h
[E 20:53:09 26764] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_def.inc
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/generated_message_util.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/C_/Program Files (x86)/Windows Kits/10/Include/10.0.22621.0/ucrt/assert.h
[E 20:53:09 12776] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/buildtools/third_party/libc++/trunk/include/__iterator/counted_iterator.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/stubs/once.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_def.inc
[E 20:53:09 12776] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/buildtools/third_party/libc++/trunk/include/__concepts/common_with.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_undef.inc
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/any.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/message_lite.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/metadata_lite.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_def.inc
[E 20:53:09 12776] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/buildtools/third_party/libc++/trunk/include/__iterator/front_insert_iterator.h
"../../third_party/protobuf/src/google/protobuf/arena.h", line 486: error: no instance of overloaded "operator new" matches the argument list
            argument types are: (unsigned int, void *)
        return new (ptr) T(static_cast<Args&&>(args)...);
               ^
          detected during:
            instantiation of "T *google::protobuf::Arena::InternalHelper<T>::Construct(void *, Args &&...) [with T=autofill::StrikeData, Args=<google::protobuf::Arena *>]" at line 659
            instantiation of "T *google::protobuf::Arena::DoCreateMessage<T,Args...>(Args &&...) [with T=autofill::StrikeData, Args=<>]" at line 584
            instantiation of "T *google::protobuf::Arena::CreateMessageInternal<T>(google::protobuf::Arena *) [with T=autofill::StrikeData]" at line 269 of "gen/components/autofill/core/browser/proto/strike_data.pb.cc"

[E 20:53:09 15972] Warning[extractor-c++]: In construct_message: "../../third_party/protobuf/src/google/protobuf/arena.h", line 486: error: no instance of overloaded "operator new" matches the argument list
            argument types are: (unsigned int, void *)
        return new (ptr) T(static_cast<Args&&>(args)...);
               ^
          detected during:
            instantiation of "T *google::protobuf::Arena::InternalHelper<T>::Construct(void *, Args &&...) [with T=autofill::StrikeData, Args=<google::protobuf::Arena *>]" at line 659
            instantiation of "T *google::protobuf::Arena::DoCreateMessage<T,Args...>(Args &&...) [with T=autofill::StrikeData, Args=<>]" at line 584
            instantiation of "T *google::protobuf::Arena::CreateMessageInternal<T>(google::protobuf::Arena *) [with T=autofill::StrikeData]" at line 269 of "gen/components/autofill/core/browser/proto/strike_data.pb.cc"
@cooolie cooolie added the question Further information is requested label Jun 17, 2023
@cooolie
Copy link
Author

cooolie commented Jun 17, 2023

"Just to clarify, the compilation was successful, it's just that many cc files will not appear in the CodeQL database."

D:\chromium\src>codeql database create "D:/codeql/autofill_06172" --language=cpp --command="ninja -j 18 -C out/debug chrome"
Initializing database at D:\codeql\autofill_06172.
Running build command: [ninja, -j, 18, -C, out/debug, chrome]
[2023-06-17 21:09:10] [build-stdout] ninja: Entering directory `out/debug'
[2023-06-17 21:09:10] [build-stdout] [1/14] CXX obj/components/autofill/content/browser/browser/fingerprint.obj
[2023-06-17 21:09:10] [build-stdout] [2/14] CXX obj/components/autofill/content/browser/browser/bad_message.obj
[2023-06-17 21:09:10] [build-stdout] [3/14] CXX obj/components/autofill/content/browser/browser/form_forest.obj
[2023-06-17 21:09:10] [build-stdout] [4/14] CXX obj/components/autofill/content/browser/browser/content_autofill_driver.obj
[2023-06-17 21:09:10] [build-stdout] [5/14] CXX obj/components/autofill/content/browser/browser/content_autofill_router.obj
[2023-06-17 21:09:25] [build-stdout] [6/14] CXX obj/components/autofill/content/browser/browser/content_autofill_client.obj
[2023-06-17 21:09:25] [build-stdout] [7/14] CXX obj/components/autofill/content/browser/browser/autofill_log_router_factory.obj
[2023-06-17 21:09:25] [build-stdout] [8/14] CXX obj/components/autofill/content/browser/browser/content_autofill_driver_factory.obj
[2023-06-17 21:09:25] [build-stdout] [9/14] LIB obj/components/autofill/content/browser/browser.lib
[2023-06-17 21:09:25] [build-stdout] [10/14] LINK(DLL) chrome.dll chrome.dll.lib chrome.dll.pdb
Finalizing database at D:\codeql\autofill_06172.
Successfully created database at D:\codeql\autofill_06172.

@aibaars
Copy link
Contributor

aibaars commented Jun 19, 2023

The error about "../../third_party/protobuf/src/google/protobuf/arena.h" look strange and may be related. When the CodeQL extractor.exe encounters an error it usually carries on and adds the source file to the database anyway, unless there are too many errors.

You said there were many CC files missing. Could you search for a couple of them in the build-tracer.log to see if they are mentioned at all. If they are not mentioned then they are likely not needed by the build process (they may have already been compiled earlier or are part of a different build target). If the files are mentioned in invocation: D:\codeql\bin\...\extractor.exe ... lines then there should be an explanation usually mentioning a "Catastrophic error".

Attaching the entire build-tracer log to this issue is probably not possible, but if you can attach a snippet with of the log related to the trace of a single CC file that would be great for debugging.

@cooolie
Copy link
Author

cooolie commented Jun 19, 2023

build-tracer.log

Deleted Obj file list

06/17/2023  09:09 PM           130,359 autofill_log_router_factory.obj
06/17/2023  09:09 PM           687,810 bad_message.obj
06/17/2023  09:09 PM         1,074,288 content_autofill_client.obj
06/17/2023  09:09 PM         2,865,175 content_autofill_driver.obj
06/17/2023  09:09 PM         2,278,732 content_autofill_driver_factory.obj
06/17/2023  09:09 PM         3,592,710 content_autofill_router.obj
06/17/2023  09:09 PM         2,407,846 fingerprint.obj
06/17/2023  09:09 PM         3,270,934 form_forest.obj

@aibaars
Copy link
Contributor

aibaars commented Jun 19, 2023

Thanks for providing a build-tracer log. I see a couple of things like :

66 errors and 1 catastrophic error detected in the compilation of "../../components/autofill/content/browser/risk/fingerprint.cc".
Compilation aborted.

I think these could explain why some files are missing. I'll pass it on to the team, hopefully they know what is causing the problem.

@aibaars
Copy link
Contributor

aibaars commented Jun 19, 2023

There may be a fix for your problem in the pre-release version of CodeQL 2.13.4. Could you try if that helps? See: https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.13.4

@cooolie
Copy link
Author

cooolie commented Jun 20, 2023

build-tracer.log
Better than before, but there are still file processing failures, see log

Deleted Obj file list

obj\chrome\browser\ui\ui\manage_passwords_ui_controller.obj

@jketema
Copy link
Contributor

jketema commented Jun 20, 2023

Thanks for the updated log. The log seems to indicate that only one file was compiled, not the 8 from #13489 (comment). Is that correct?

@cooolie
Copy link
Author

cooolie commented Jun 20, 2023

yes, i only delete one obj file (obj\chrome\browser\ui\ui\manage_passwords_ui_controller.obj)

@jketema
Copy link
Contributor

jketema commented Jun 20, 2023

Can you delete all the relevant object files and re-try and share the log. Currently I cannot judge how severe the problem is.

@cooolie
Copy link
Author

cooolie commented Jun 20, 2023

build-tracer.zip

Deleted Obj file list
obj\chrome\browser\ui\ui\*.obj


CodeQL C++ extractor: Current location: ../../chrome/browser/ui/passwords/manage_passwords_ui_controller.cc:152317,5
CodeQL C++ extractor: Current physical location: ../../base/compiler_specific.h:104,5
"../../base/compiler_specific.h", line 104: internal error: assertion failed: macro_invocation: unknown special predefined macro (macro.c, line 7359 in macro_invocation)

  #if HAS_CPP_ATTRIBUTE(no_unique_address)
      ^

[E 16:25:08 17016] Warning[extractor-c++]: In construct_message: "../../base/compiler_specific.h", line 104: internal error: assertion failed: macro_invocation: unknown special predefined macro (macro.c, line 7359 in macro_invocation)

  #if HAS_CPP_ATTRIBUTE(no_unique_address)
      ^


3 errors and 1 catastrophic error detected in the compilation of "../../chrome/browser/ui/passwords/manage_passwords_ui_controller.cc".
Compilation aborted.

@jketema
Copy link
Contributor

jketema commented Jun 20, 2023

Thanks for the complete log. I cannot easily reproduce this. I'll add this problem to our internal tracking issue for these kinds of problems, but I cannot promise this will be solved any time soon.

@cooolie
Copy link
Author

cooolie commented Jun 20, 2023

Reproduce step:
Guided by this article
https://chromium.googlesource.com/chromium/src/+/HEAD/docs/windows_build_instructions.md

build args.gn

Set build arguments here. See gn help buildargs.

is_component_build = true
enable_nacl = false
target_cpu = "x64"
blink_symbol_level = 2
v8_symbol_level = 2
symbol_level = 2

delete obj in obj\chrome\browser\ui\ui*.obj

codeql database create "D:/codeql/test" --language=cpp --command="ninja -j 18 -C out/debug chrome"

@jketema
Copy link
Contributor

jketema commented Jun 20, 2023

Thanks. I'm fully aware how to build Chromium. This doesn't mean it's easy to create a test case that reproduces the problem.

@jketema jketema mentioned this issue Jun 25, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cooolie @aibaars @jketema