-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUILD] Graal compiler build process fails #1243
Comments
I'll close this issue, as the build does not fail anymore. I suggest take a look at this patch, see neomatrix369/awesome-graal@628e5e2#diff-12d564a25796a5c2eb64e51509bbc11f (Line 35 onwards) - the fix might be needed when Graal is built over JDKs which HotSpot/JVM string does not contain a space. /cc @dougxc |
@dougxc i hope this is fix help, in my experience the bug triggers for builds that do not have spaces around the brackets in the HotSpot string |
I have been able to reproduce this build failure again, see https://circleci.com/gh/neomatrix369/awesome-graal/390, each run happens in a clean environment. I ran the build with the Can you please take a look at this and comment? |
Sorry, there's nothing in that log that points to the cause of the error. Is there a hs_err log file lying around that provide more info on the alleged javac compiler daemon crash? |
Just re-running them to fetch the When a build process crashes which other logs are helpful? Instead of the |
So far I haven't been able to get hold of the crash logs, although I have these artefacts (see under the Artifacts tab) - https://circleci.com/gh/neomatrix369/awesome-graal/407#artifacts/containers/0 |
I have now modified the scripts to pass in Java-args that enable recording crash logs and heap dump, but from these logs https://circleci.com/gh/neomatrix369/awesome-graal/412, it appears that the commands that are failing (look for CalledProcessError) are not using these passed in args. Do I need to pass in something else, in addition to the changes I made to build_JDK_JVMCI.sh and buildGraalCompiler.sh - look for |
Looking into the build logs I found these:
Are the compiler daemons JVM instances? Looking at the code at https://github.com/graalvm/mx/blob/master/mx.py#L4276 but maybe the logs are async But looking at the [While debugging this issue, I raised an issue related to |
One more question, will this line work? Line 49 in 8a3c7d6
I think the Unless the compiler does some other inferencing to resolve it. Here's my stack trace wrt to the above:
|
The |
Yes, they are separate JVM processes.
Correct. The args are set up here.
Yes. Since the daemon is a separate JVM started before any compilation, it's too late to pass JVM args for a compilation. |
Thanks @dougxc ! I'll look into the failing builds further to see why it is happening, could be very specific to that environment. It could be because it's running inside the container environment. I'm thinking Travis does the same. Any hints on how you would go about tracing failing compiler daemon failures. Could it be memory/resources constraints? |
I'm also trying to understand this command:
Looking at the commands in /home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product/bin/, there is no |
That output line comes from here: https://github.com/graalvm/mx/blob/master/mx.py#L4285 |
Thanks for that I should have noticed it earlier, I think this line: Should have been, which will fix the space issue and also add the right command - i.e.
Semantically similar to: I can do a quick PR to fix this, misleading debug log messages can sink up debug time. |
This issue is down to running in an instance with low memory, closing issue! |
How did you figure out the memory issue? This kind of thing should be made clearer by mx logging. |
No loggings present during any of the failures - how can we improve this? I did the following:
The failures were mainly around the javac compiler crashing most of the times:
So its sort of a guess that its memory related, but maybe only reproducible in lighter environments. Although I think it might be the python As rule of thumb I try to use only 75% of the CPU/Threading resources when allocating multiple tasks. I haven't looked at the Also, I think specific to mx.py, the compiler daemons could take the |
When I run a
If I do this in mx, I also see output indicating the low memory conditions:
So it's still a mystery as to why you don't see something. |
The way to pass |
I concur. Could it be that there is something happening at low-level and the container might just be terminating processes silently? |
Would the jvm flags to capture OOME and heap dumps help? I had those in mind, so we could get some info just when it crashed. I'll use the mx flag you mentioned to pass the JVM args to the javac compiler and see what the outcome is. |
I think I might have found out the reason for the failure, builds are back to green (https://circleci.com/gh/neomatrix369/awesome-graal/534), here's the changed code neomatrix369/awesome-graal@3d40ee1#diff-e09099e42fa9f20401d67bf58248adcd I was setting |
kudos: loving the fact you guys built incremental compiling into the build system - makes rebuilds quick and it does it quite efficiently 👍 |
On the back of issue #1233, the reported issues were fixed and we have been able to progress, but now the building of the Graal compiler fails with:
See full logs at https://circleci.com/gh/neomatrix369/awesome-graal/343
The text was updated successfully, but these errors were encountered: