C++ extractor fails to process code based on Unreal Engine

[artem-smotrakov]

Hey team,

I am trying to build a CodeQL database for a sample Unreal project. Building the code base on Unreal Engine includes two main steps:

1. Running UnrealHeaderTool that parses the C++ headers for Unreal-related class metadata and generates custom C++ code.
2. Running the normal C++ compiler to compile the generated code.

Unreal Engine also has UnrealBuildTool that runs the above steps.

I use the following command to build the CodeQL database:

codeql database create --language=cpp --overwrite --command=run_build_tool.bat -- codeql-db

where run_build_tool.bat contains this:

"c:\Program Files\Epic Games\UE_4.27\Engine\Binaries\DotNET\UnrealBuildTool.exe" FirstPersonTemplate Win64 Development -Project=d:\projects\codeql\unreal\FirstPersonTemplate\FirstPersonTemplate.uproject -log=build.log

The codeql database create ... command finishes successfully. The codeql database print-baseline codeql-db shows about 3K lines of C++ code. The following query returns a list of files included into the database:

import cpp

from File f
select f, "file"

However, the database doesn't not include any function call, if-blocks, etc although the code have them. For example, the following query doesn't return anything:

import cpp

from FunctionCall call
select call, "function call"

I see multiple errors in build-tracer.log like this:

CodeQL C++ extractor: Current location: E:\projects\unreal\FirstPersonTemplate\Source\FirstPersonTemplate\FirstPersonTemplateGameMode.cpp:507675,0
CodeQL C++ extractor: Current physical location: E:\projects\unreal\FirstPersonTemplate\Source\FirstPersonTemplate\FirstPersonTemplateGameMode.cpp:17,0 (end of source)
Internal error: assertion failed at: "class_decl.c", line 33485 in diagnose_duplicate_capture


[E 16:25:56 12932] Warning[extractor-c++]: In construct_message: Internal error: assertion failed at: "class_decl.c", line 33485 in diagnose_duplicate_capture



1 catastrophic error detected in the compilation of "E:\projects\unreal\FirstPersonTemplate\Source\FirstPersonTemplate\FirstPersonTemplateGameMode.cpp".
Compilation aborted.

Does it look like a bug in the C++ extractor, or am I doing anything wrong?

Steps to reproduce:

1. Install Unreal Engine 4
2. Create a sample first-person project, select C++ instead of Blueprint
3. Build a CodeQL database as described above
4. Check errors in codeql-db/log/build-tracer.log
5. Run the query above that looks for function calls

My testing environment:

• CodeQL CLI v2.14.2
• Windows 10 x64
• Visual Studio 2019 toolchain (tested with 2022 -- same results)
• Unreal Engine 4.27

[aibaars]

Thanks for reporting! I'll pass it on to the team. I think this issue will be solved by improvements to the frontend the team is working on. However, I don't know when those changes will be released.

[artem-smotrakov]

Thanks Arthur! I'd appreciate it if you could post a short update here if/when the team confirms it's a bug in the extractor, and when it can be possibly fixed. Thank you!

[sashabu]

@artem-smotrakov We've confirmed that this is a bug and are testing a fix internally. If all goes well, the fix will be included in CodeQL CLI v2.14.4.

Note that while I was able to reproduce the bug with Unreal Engine 4.27.2 following your instructions, the corresponding sample project in Unreal Engine 5.2.1 was fine. I'm not familiar with UE, but it might be worth trying UE 5.2.1 as a workaround to use in the meantime.

[artem-smotrakov]

Hi @sashabu ! Thank you for the update!

If all goes well, the fix will be included in CodeQL CLI v2.14.4.

Sounds awesome! 🤞

the corresponding sample project in Unreal Engine 5.2.1 was fine. I'm not familiar with UE, but it might be worth trying UE 5.2.1 as a workaround to use in the meantime.

Thanks for the info -- I have not tested it with UE5. In my case, it's only UE4 unfortunately. I wish it were UE5 though :)

[artem-smotrakov]

Hi @sashabu Please let me know if I can help with testing the fix or anything else. Thanks!

[sashabu]

Hi @sashabu Please let me know if I can help with testing the fix or anything else. Thanks!

Hi @artem-smotrakov Thanks for the ping. The fix is now expected the next release, CodeQL CLI v2.15.2, later this month.

[artem-smotrakov]

Sounds great, thank you for the update!

[artem-smotrakov]

Hi @sashabu v2.15.2 seems to have the fix, doesn't it? At least I now see function calls and if-blocks in the database built with v2.15.2. I have not yet done better testing though.

I still see about 300 errors in build-tracer.log like those I mentioned in the description above.

[sashabu]

Hi @sashabu v2.15.2 seems to have the fix, doesn't it?

Hi @artem-smotrakov That's correct.

I still see about 300 errors in build-tracer.log like those I mentioned in the description above.

Could you confirm whether you're seeing the "catastrophic error" message specifically, or only the "Warning[extractor-c++]"?

[artem-smotrakov]

Hi @sashabu Thank you for the confirmation!

I still see "catastrophic error" in the logs

[E 12:15:55 22760] Warning[extractor-c++]: In construct_text_message: "Runtime\RHI\Public\RHIContext.h", line 53: catastrophic error: out of memory
        {
        ^


1 catastrophic error detected in the compilation of "E:\projects\unreal\FirstPersonTemplate\Source\FirstPersonTemplate\FirstPersonTemplateGameMode.cpp".
Compilation terminated.

I gave it about 16G via --ram but still see these errors. Does it look like a problem with the extractor? The build runs well without CodeQL.

[jketema]

Hi @artem-smotrakov ,

The --ram option will have no effect on the extractor, so it's correct that you do not see any improvement with that option.

Are all the catastrophic errors now out of memory ones, or do you see others too?

[artem-smotrakov]

Hi @jketema

The --ram option will have no effect on the extractor, so it's correct that you do not see any improvement with that option.

Is there any way how I can give the extractor more memory to get rid of the catastrophic error: out of memory errors?

Are all the catastrophic errors now out of memory ones, or do you see others too?

Mainly out of memory but I see several other errors.

build-tracer.log

[jketema]

Hi @jketema

The --ram option will have no effect on the extractor, so it's correct that you do not see any improvement with that option.

Is there any way how I can give the extractor more memory to get rid of the catastrophic error: out of memory errors?

Not really, as it's a 64-bit C++ program running out of heap space. There might be a memory leak somewhere of course, but we've not observed such errors on other code bases recently. So, we'll need to investigate.

Are all the catastrophic errors now out of memory ones, or do you see others too?

Mainly out of memory but I see several other errors.

build-tracer.log

Thanks for logs, as far as I can tell all the errors are in fact out-of-memory errors.