[cpp] database create problems #16239
Comments
|
Thanks for opening a new issue for this. Could you tell me which version of CodeQL you are using? If this is not the latest version, could you try with the latest version, which should have better error in the area where the crash seems to occur. |
2.16.5 |
|
We've looked a bit closer at this and the following: combined with your remark that "(I stopped running because it always print this error messages in different trap files)" makes us suspect that the machine you're using to create the database might have bad RAM. Would it be possible for you to try using a different machine to create the database to see if the problem persists there? |
I got builds today in the same machine, It shows that "exausted all available IDs" here : Maybe It is not RAM problem? |
|
The "We have exausted all available IDs in the disk pool" message is what we expected. The other errors you were seeing should not have been possible because it should have reached this particular message. So we still suspect there is something wrong with your machine, but on top of that the tool cannot handle the size of the database you're trying to create. The latest version (if you were not using that already) might perform slightly better, but that might still not sufficient in your case. If this is a big "mono repo" with many different applications in it, you might be better of creating a separate database per application. If this is not the case, then what we've discussed on #16237 is possibly your only option at the moment. Note that we have been actively looking at the underlying causes of the "We have exausted all available IDs in the disk pool" message, and it is something we have been trying to improve. However, I cannot give you any time lines of when we might have something that would work in your case. |
Yes, I have updated to 2.17.0. And the latest log above is runned in 2.17.0. Maybe I have to reduce build scale, or maybe this is another way to reduce trace scale (for example : ignore trace some folders)? just to confirm |
For compiled languages like C++, the tool does not support ignoring folders, so that's not an option. As I wrote above your options are basically:
|
I get it , thank you |
|
@jketema I divided into several parts. But It still have logs with "We have exausted all available IDs in the disk pool"。 And in the build-tracer.log I found many errors (6000+ times in build-tracer.log) like |
The only thing I can recommend is cut things up in even smaller parts, if possible. All the warnings are pretty inconsequential, and the one that occurs most "In index_stmk_decl: No source sequence entry for declaration statement." is fixed in the latest CodeQL version. To help you with "3 errors detected in the compilation of "xxx.cpp"." I would need to see the actually error messages (which should occur higher up in the log). |
In fact, today I downgraded codeql to 2.15.4. Now the divided project can be compiled normally. The compiled database size is 13G and the time is within 6 hours (the intermediate files during the compilation process may be cleaned up at the end). In the before when upgrading to a newer version For versions such as (2.16.5 and 2.17.1 (released a few days ago)), compilation will take more than 12 hours and eventually fail. |
Note that this might be because (i) 2.15.4 is able to extract significantly less code, so you will have less coverage, (ii) 2.1.5.4 doesn't check for exhausting of IDs in the disk pool, so you might silently end up with a corrupted database. |
The log occured in higher up is similar to what I just posted. I roughly categorized it (whole log is GB sizes): Maybe the logs are messed up due to concurrency. Hopefully It is useful. |
No, this is not useful. If there are lines like "3 errors detected in the compilation of "xxx.cpp".", that means there should also be lines that contain the text "error: ". Those are suspiciously absent. |
Thanks, I search some errors in build-tracer.log : |
Those seem to be mostly related to the boost library, so they likely have little effect on any analysis results related to your code. |
|
We've made some changes that should mitigate the "We have exausted all available IDs in the disk pool" problem. These will be part of CodeQL 2.18.2, which should be released in about two weeks. |
exciting, I will try |
I use command creating databases , like :
codeql database create C:\test\codeql-database --source-root "E:\test-project-code\src" --language=cpp --command="call build_win_codeql.bat" --threads=0 --verbose --overwrite --mode=clear --min-disk-free=100000it built project successful,
but it takes too much time and costs too much disks . and after build it costs much more time to scanning trap file.
is it something wrong
all build-trace like :
...
...
[2024-04-16 11:42:00] [build-stdout] MSBuild version 17.9.8+b34f75857 for .NET Framework
[2024-04-16 11:42:00] [build-stdout] Build started 4/16/2024 11:42:00 AM.
...
[2024-04-16 21:04:28] [build-stdout] 4134 Warning(s)
[2024-04-16 21:04:28] [build-stdout] 0 Error(s)
[2024-04-16 21:04:28] [build-stdout] Time Elapsed 09:22:26.82
[2024-04-16 21:04:28] Plumbing command codeql database trace-command completed.
[2024-04-16 21:04:28] [PROGRESS] database create> Finalizing database at C:\test\devops-codeql-database.
[2024-04-16 21:04:28] Running plumbing command: codeql database finalize --threads=0 --mode=clear --min-disk-free=10000 --no-db-cluster -- C:\test\devops-codeql-database
[2024-04-16 21:04:28] Using pre-finalize script C:\codeql-home\codeql\cpp\tools\pre-finalize.cmd.
[2024-04-16 21:04:28] [PROGRESS] database finalize> Running pre-finalize script C:\codeql-home\codeql\cpp\tools\pre-finalize.cmd in C:\devops\p-6ac70a2931f74eb2a2452fb8f52372e1\src.
[2024-04-16 21:04:28] Running plumbing command: codeql database trace-command --working-dir=C:\devops\p-6ac70a2931f74eb2a2452fb8f52372e1\src --no-tracing --threads=0 -- C:\test\devops-codeql-database C:\codeql-home\codeql\cpp\tools\pre-finalize.cmd
[2024-04-16 21:04:28] [PROGRESS] database trace-command> Running command in C:\devops\p-6ac70a2931f74eb2a2452fb8f52372e1\src: [C:\codeql-home\codeql\cpp\tools\pre-finalize.cmd]
[2024-04-16 21:04:28] Plumbing command codeql database trace-command completed.
[2024-04-16 21:04:28] [PROGRESS] database finalize> Running TRAP import for CodeQL database at C:\test\devops-codeql-database...
[2024-04-16 21:04:28] Running plumbing command: codeql dataset import --dbscheme=C:\codeql-home\codeql\cpp\semmlecode.cpp.dbscheme --threads=0 -- C:\test\devops-codeql-database\db-cpp C:\test\devops-codeql-database\trap\cpp
[2024-04-16 21:04:28] Clearing disk cache since the version file C:\test\devops-codeql-database\db-cpp\default\cache\version does not exist
[2024-04-16 21:04:29] Tuple pool not found. Clearing relations with cached strings
[2024-04-16 21:04:29] Trimming disk cache at C:\test\devops-codeql-database\db-cpp\default\cache in mode clear.
[2024-04-16 21:04:29] Sequence stamp origin is -6195435911585769745
[2024-04-16 21:04:29] Pausing evaluation to hard-clear memory at sequence stamp o+0
[2024-04-16 21:04:29] Unpausing evaluation
[2024-04-16 21:04:29] Pausing evaluation to quickly trim disk at sequence stamp o+1
[2024-04-16 21:04:29] Unpausing evaluation
[2024-04-16 21:04:29] Pausing evaluation to zealously trim disk at sequence stamp o+2
[2024-04-16 21:04:29] Unpausing evaluation
[2024-04-16 21:04:29] Trimming completed (12ms): Purged everything.
[2024-04-16 21:04:29] Scanning for files in C:\test\devops-codeql-database\trap\cpp
[2024-04-16 21:05:01] Found 18594 files on disk containing 7859950 TRAP files (239.02 GiB)
[2024-04-16 21:05:01] [PROGRESS] dataset import> Grouping TRAP files by link target
[2024-04-16 21:11:37] [PROGRESS] dataset import> Grouping unlinked TRAP files together
[2024-04-16 21:12:11] [PROGRESS] dataset import> Scanning TRAP files
...
[2024-04-16 21:55:32] Scanning trace_log.cc.11991bd7.trap (trace_log.cc.11991bd7_0.trap.tar.br) (6891731 of 7859950)
...
[2024-04-17 00:06:20] Importing 96f07e7c3376ac5f473f4bab.trap (trace_log.cc.11991bd7_0.trap.tar.br) for no link target (6974499 of 7859950) [2024-04-17 00:06:20] [ERROR] dataset import> 9fafff7e4bf067c0c66b0ca7.trap (connection.cc.a0da4be2_0.trap.tar.br) for no link target, 38: com.semmle.util.exception.CatastrophicError: ID 94380083 is already mapped to 19293495 com.semmle.inmemory.util.DiskIdStore.append(DiskIdStore.java:63) com.semmle.inmemory.util.NonSequentialDiskPool.insert(NonSequentialDiskPool.java:105) com.semmle.inmemory.util.NonSequentialDiskPool.insertBucket(NonSequentialDiskPool.java:47) com.semmle.inmemory.util.DiskPool.getIdWithFreshness(DiskPool.java:171) com.semmle.inmemory.util.NonSequentialDiskPool.getIdWithFreshness(NonSequentialDiskPool.java:14) com.semmle.inmemory.populate.IdPool.lookupWithFreshness(IdPool.java:66) com.semmle.inmemory.trap.SynchronizedIdAllocator.getElementIdAndFreshness(SynchronizedIdAllocator.java:23) com.semmle.inmemory.trap.FreshIdAllocator.getElementId(FreshIdAllocator.java:22) com.semmle.inmemory.trap.TRAPReader$MetaStringBuilder.getElementId(TRAPReader.java:1016) com.semmle.inmemory.trap.TRAPReader.computeID(TRAPReader.java:1090) com.semmle.inmemory.trap.TRAPReader.computeID(TRAPReader.java:1085) com.semmle.inmemory.trap.TRAPReader.scanLabelKey(TRAPReader.java:829) com.semmle.inmemory.trap.TRAPReader.scanLabelValue(TRAPReader.java:801) com.semmle.inmemory.trap.TRAPReader.scanTuplesAndLabels(TRAPReader.java:505) com.semmle.inmemory.trap.TRAPReader.importTuples(TRAPReader.java:414) com.semmle.inmemory.trap.ImportTasksProcessor.process(ImportTasksProcessor.java:234) com.semmle.inmemory.trap.ImportTasksProcessor.lambda$importTrap$1(ImportTasksProcessor.java:154) com.semmle.util.concurrent.FutureUtils.lambda$mapAsync_$8(FutureUtils.java:161) java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) java.base/java.lang.Thread.run(Unknown Source) at (failed to read line: asked for -1 bytes which seems wrong!)
...
...
(I stopped running because it always print this error messages in different trap files)
The text was updated successfully, but these errors were encountered: