Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUILD] Graal compiler build process fails #1243

Closed
neomatrix369 opened this issue May 6, 2019 · 24 comments
Closed

[BUILD] Graal compiler build process fails #1243

neomatrix369 opened this issue May 6, 2019 · 24 comments

Comments

@neomatrix369
Copy link
Contributor

On the back of issue #1233, the reported issues were fixed and we have been able to progress, but now the building of the Graal compiler fails with:

<---snipped--->
Compiling com.oracle.truffle.api.debug.test with javac-daemon(JDK 1.8)... [dependency TRUFFLE_DSL_PROCESSOR_INTEROP_INTERNAL updated]
Archiving TRUFFLE_INSTRUMENT_TEST... [dependency com.oracle.truffle.api.instrumentation.test updated]
Compiling org.graalvm.compiler.truffle.test with javac-daemon(JDK 1.8)... [dependency GRAAL_NODEINFO_PROCESSOR updated]
Compiling com.oracle.truffle.api.benchmark with javac-daemon(JDK 1.8)... [dependency TRUFFLE_DSL_PROCESSOR updated]


Compiling org.graalvm.compiler.truffle.test with javac-daemon(JDK 1.8) failed
Compiling com.oracle.truffle.api.benchmark with javac-daemon(JDK 1.8) failed
2 build tasks failed
Exited with code 1

See full logs at https://circleci.com/gh/neomatrix369/awesome-graal/343

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented May 9, 2019

I'll close this issue, as the build does not fail anymore.

I suggest take a look at this patch, see neomatrix369/awesome-graal@628e5e2#diff-12d564a25796a5c2eb64e51509bbc11f (Line 35 onwards) - the fix might be needed when Graal is built over JDKs which HotSpot/JVM string does not contain a space.

/cc @dougxc

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented May 15, 2019

I'll close this issue, as the build does not fail anymore.

I suggest take a look at this patch, see neomatrix369/awesome-graal@628e5e2#diff-12d564a25796a5c2eb64e51509bbc11f (Line 35 onwards) - the fix might be needed when Graal is built over JDKs which HotSpot/JVM string does not contain a space.

/cc @dougxc

@dougxc i hope this is fix help, in my experience the bug triggers for builds that do not have spaces around the brackets in the HotSpot string

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented Jun 7, 2019

I have been able to reproduce this build failure again, see https://circleci.com/gh/neomatrix369/awesome-graal/390, each run happens in a clean environment. I ran the build with the -V flag to highlight the cause of the failure.

Can you please take a look at this and comment?

@neomatrix369 neomatrix369 reopened this Jun 7, 2019
@dougxc
Copy link
Member

dougxc commented Jun 8, 2019

Sorry, there's nothing in that log that points to the cause of the error. Is there a hs_err log file lying around that provide more info on the alleged javac compiler daemon crash?
BTW, the -V level of logging doesn't help in this case.

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented Jun 8, 2019

Sorry, there's nothing in that log that points to the cause of the error. Is there a hs_err log file lying around that provide more info on the alleged javac compiler daemon crash?
BTW, the -V level of logging doesn't help in this case.

Just re-running them to fetch the hs_err logs.

When a build process crashes which other logs are helpful?

Instead of the -V or -v flags which ones are best to use to get more details on failed builds?

@neomatrix369
Copy link
Contributor Author

So far I haven't been able to get hold of the crash logs, although I have these artefacts (see under the Artifacts tab) - https://circleci.com/gh/neomatrix369/awesome-graal/407#artifacts/containers/0

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented Jun 9, 2019

I have now modified the scripts to pass in Java-args that enable recording crash logs and heap dump, but from these logs https://circleci.com/gh/neomatrix369/awesome-graal/412, it appears that the commands that are failing (look for CalledProcessError) are not using these passed in args.

Do I need to pass in something else, in addition to the changes I made to build_JDK_JVMCI.sh and buildGraalCompiler.sh - look for JAVA_OPTS and -J.

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented Jun 9, 2019

Looking into the build logs I found these:

...[snipped]...
/home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product/bin/javac -g -d /home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.hotspot.amd64/bin -classpath /home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product/jre/lib/jvmci/jvmci-api.jar:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.api.replacements/bin:/home/graal/project/graal/sdk/mxbuild/dists/jdk1.8/graal-sdk.jar:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.util/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.serviceprovider/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.graphio/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.options/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.debug/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.core.common/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.asm/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.nodeinfo/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.bytecode/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.graph/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.code/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.lir/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.nodes/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.phases/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.phases.common/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.virtual/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.loop/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.loop.phases/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.core/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.asm.amd64/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.lir.amd64/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.java/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.core.amd64/bin:/home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product/jre/lib/jvmci/jvmci-hotspot.jar:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.api.runtime/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.api.directives/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.word/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.replacements/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.printer/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.runtime/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.hotspot/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.replacements.amd64/bin -parameters -processorpath /home/graal/project/graal/compiler/mxbuild/dists/jdk1.8/graal-processor-common.jar:/home/graal/project/graal/compiler/mxbuild/dists/jdk1.8/graal-serviceprovider-processor.jar:/home/graal/project/graal/compiler/mxbuild/dists/jdk1.8/graal-nodeinfo-processor.jar:/home/graal/project/graal/compiler/mxbuild/dists/jdk1.8/graal-replacements-processor.jar -s /home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.hotspot.amd64/src_gen -target 1.8 -source 1.8 -verbose @/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.hotspot.amd64/javafilelist.txt -Xlint:all,-auxiliaryclass,-processing,-path,-options -XDignore.symbol.file -encoding UTF-8 -Xmaxerrs 10000
[Compiler daemon process appears to have crashed][Compiler daemon process appears to have crashed]

[Compiler daemon process appears to have crashed]
[Compiler daemon process appears to have crashed]
[Compiler daemon process appears to have crashed]
[Compiler daemon process appears to have crashed]
Process Process-212:
Process Process-213:
Process Process-194:
Process Process-205:
Process Process-210:
Process Process-211:
Process Process-202:
Process Process-208:
Process Process-209:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):

...[snipped]...

  File "/home/graal/project/mx/mx.py", line 4185, in compile
    return self.daemon.compile(nonJvmArgs)
CalledProcessError: Command '/home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product/bin/java-g -d /home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.hotspot.management/bin -classpath /home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product/jre/lib/jvmci/jvmci-api.jar:/home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product/jre/lib/jvmci/jvmci-hotspot.jar:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.api.runtime/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.api.directives/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.api.replacements/bin:/home/graal/project/graal/sdk/mxbuild/dists/jdk1.8/graal-sdk.jar:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.util/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.serviceprovider/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.graphio/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.options/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.debug/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.core.common/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.asm/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.nodeinfo/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.bytecode/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.graph/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.code/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.lir/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.nodes/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.phases/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.java/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.loop/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.phases.common/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.loop.phases/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.word/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.replacements/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.virtual/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.core/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.printer/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.runtime/bin:/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.hotspot/bin -parameters -processorpath /home/graal/project/graal/compiler/mxbuild/dists/jdk1.8/graal-processor-common.jar:/home/graal/project/graal/compiler/mxbuild/dists/jdk1.8/graal-serviceprovider-processor.jar -s /home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.hotspot.management/src_gen -target 1.8 -source 1.8 -verbose @/home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.hotspot.management/javafilelist.txt -Xlint:all,-auxiliaryclass,-processing,-path,-options -XDignore.symbol.file -encoding UTF-8 -Xmaxerrs 10000' returned non-zero exit status -1

Are the compiler daemons JVM instances? Looking at the code at https://github.com/graalvm/mx/blob/master/mx.py#L4276 but maybe the logs are async

But looking at the CalledProcessError I don't see any of the flags in opts passed into that call. What does nonJVMArgs signify? Does it mean the -J args are not passed in for such calls?

[While debugging this issue, I raised an issue related to mx, see https://github.com/graalvm/mx/issues/194]

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented Jun 9, 2019

One more question, will this line work?

I think the ValidTruffleObject5MRForeign file has not been checked in.

Unless the compiler does some other inferencing to resolve it.

Here's my stack trace wrt to the above:

[snipped]
[skipping GRAAL_TRUFFLE_COMPILER_LIBGRAAL]
/home/graal/scripts/graal/truffle/src/com.oracle.truffle.api.dsl.test/src/com/oracle/truffle/api/dsl/test/interop/ValidTruffleObject5.java:49: error: cannot find symbol
        return ValidTruffleObject5MRForeign.ACCESS;
               ^
  symbol:   variable ValidTruffleObject5MRForeign
  location: class ValidTruffleObject5
1 error
Result = 1
[exit code: 1]
  File "/home/graal/scripts/mx/mx.py", line 19177, in <module>
    main()
  File "/home/graal/scripts/mx/mx.py", line 19158, in main
    retcode = c(command_args)
  File "/home/graal/scripts/mx/mx_commands.py", line 147, in __call__
    return self.command_function(*args, **kwargs)
  File "/home/graal/scripts/mx/mx.py", line 12878, in build
    task.proc.start()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/usr/lib/python2.7/multiprocessing/forking.py", line 126, in __init__
    code = process_obj._bootstrap()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/graal/scripts/mx/mx.py", line 12861, in executeTask
    task.execute()
  File "/home/graal/scripts/mx/mx.py", line 964, in execute
    self.build()
  File "/home/graal/scripts/mx/mx.py", line 3772, in build
    self.compiler.compile(self.compileArgs)
  File "/home/graal/scripts/mx/mx.py", line 4099, in compile
    return self.daemon.compile(nonJvmArgs)
  File "/home/graal/scripts/mx/mx.py", line 4202, in compile
    abort(retcode)
  File "/home/graal/scripts/mx/mx.py", line 12406, in abort
    traceback.print_stack()
[snipped]

@dougxc
Copy link
Member

dougxc commented Jun 10, 2019

The ValidTruffleObject5MRForeign class is generated by the com.oracle.truffle.api.interop.Resolve annotations in ValidTruffleObject5MR.java. The compilation error you see means that relevant annotation processor is not being configured properly. I'm not sure why this is as we're not seeing this problem in either our Travis or internal gates.

@dougxc
Copy link
Member

dougxc commented Jun 10, 2019

Are the compiler daemons JVM instances? Looking at the code at https://github.com/graalvm/mx/blob/master/mx.py#L4276 but maybe the logs are async

Yes, they are separate JVM processes.

But looking at the CalledProcessError I don't see any of the flags in opts passed into that call.

Correct. The args are set up here.

What does nonJVMArgs signify? Does it mean the -J args are not passed in for such calls?

Yes. Since the daemon is a separate JVM started before any compilation, it's too late to pass JVM args for a compilation.

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented Jun 10, 2019

Thanks @dougxc ! I'll look into the failing builds further to see why it is happening, could be very specific to that environment. It could be because it's running inside the container environment. I'm thinking Travis does the same.

Any hints on how you would go about tracing failing compiler daemon failures. Could it be memory/resources constraints?

@neomatrix369
Copy link
Contributor Author

I'm also trying to understand this command:

/home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product/bin/java-g -d /home/graal/project/graal/compiler/mxbuild/src/org.graalvm.compiler.api.directives.test/bin
...

Looking at the commands in /home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product/bin/, there is no java-g and the rest of the parameters won't go with the java launcher, how do we read this log line, is it correctly printed?

@dougxc
Copy link
Member

dougxc commented Jun 10, 2019

That output line comes from here: https://github.com/graalvm/mx/blob/master/mx.py#L4285
There's a missing space but the more misleading part is that this is really a fake CalledProcessError since it's actually reporting the failure of a compilation request sent to a compiler daemon.

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented Jun 10, 2019

That output line comes from here: https://github.com/graalvm/mx/blob/master/mx.py#L4285
There's a missing space but the more misleading part is that this is really a fake CalledProcessError since it's actually reporting the failure of a compilation request sent to a compiler daemon.

Thanks for that I should have noticed it earlier, I think this line:
https://github.com/graalvm/mx/blob/fe69cd8b27384c85a5ab9506110e7a0d30a7f023/mx.py#L4285

Should have been, which will fix the space issue and also add the right command - i.e. javac not java:

raise subprocess.CalledProcessError(retcode, self.jdk.javac +  ' ' + ' '.join(compilerArgs))

Semantically similar to:
https://github.com/graalvm/mx/blob/fe69cd8b27384c85a5ab9506110e7a0d30a7f023/mx.py#L4270

I can do a quick PR to fix this, misleading debug log messages can sink up debug time.

@neomatrix369
Copy link
Contributor Author

This issue is down to running in an instance with low memory, closing issue!

@dougxc
Copy link
Member

dougxc commented Jun 17, 2019

How did you figure out the memory issue? This kind of thing should be made clearer by mx logging.

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented Jun 17, 2019

How did you figure out the memory issue? This kind of thing should be made clearer by mx logging.

No loggings present during any of the failures - how can we improve this?

I did the following:

  • ran the build on my local machine (2CPU, 16GB RAM) - build passes
  • ran the build on CircleCI VM (2CPU, 8GB RAM) - build passes (see https://circleci.com/gh/neomatrix369/awesome-graal/503)
  • ran the build on CircleCI docker container (2CPU, 4GB RAM) - build fails all the time, except a few times out of every 40-50 builds

The failures were mainly around the javac compiler crashing most of the times:

[snipped]
Compiling com.oracle.truffle.api.test with javac-daemon(JDK 1.8) failed
Compiling org.graalvm.compiler.jtt with javac-daemon(JDK 1.8) failed
Compiling com.oracle.truffle.nfi.test with javac-daemon(JDK 1.8) failed
Compiling org.graalvm.compiler.hotspot.aarch64 with javac-daemon(JDK 1.8) failed
Compiling org.graalvm.compiler.hotspot.sparc with javac-daemon(JDK 1.8) failed
Compiling org.graalvm.compiler.truffle.compiler.hotspot with javac-daemon(JDK 1.8) failed
Compiling org.graalvm.compiler.hotspot.amd64 with javac-daemon(JDK 1.8) failed
Compiling org.graalvm.compiler.hotspot.test with javac-daemon(JDK 1.8) failed
[Stopped javac-daemon on port 40275 for Java 1.8.0_212 (1.8) from /home/graal/project/graal-jvmci-8/openjdk1.8.0_212/linux-amd64/product]
  File "/home/graal/project/mx/mx.py", line 19498, in <module>
    main()
  File "/home/graal/project/mx/mx.py", line 19479, in main
    retcode = c(command_args)
  File "/home/graal/project/mx/mx_commands.py", line 147, in __call__
    return self.command_function(*args, **kwargs)
  File "/home/graal/project/mx/mx.py", line 13120, in build
    abort('{0} build tasks failed'.format(len(failed)))
  File "/home/graal/project/mx/mx.py", line 12629, in abort
    traceback.print_stack()
8 build tasks failed
Exited with code 1
[snipped]

So its sort of a guess that its memory related, but maybe only reproducible in lighter environments. Although I think it might be the python multiprocessing package that might be jam-packing processes/threads and running multiple daemons at a time leading to these failures in low resource environments, see https://circleci.com/gh/neomatrix369/awesome-graal/475 for most recent failures.

As rule of thumb I try to use only 75% of the CPU/Threading resources when allocating multiple tasks. I haven't looked at the multiprocessing impementation in mx.py closely to find out how this is handled - is this something that might be causing it?

Also, I think specific to mx.py, the compiler daemons could take the --J / -J flags passed by the user, so we could trap the reasons for the crash. It's not using any of them atm - is this worth looking into?

@dougxc
Copy link
Member

dougxc commented Jun 17, 2019

When I run a javac compilation with too little memory, I see:

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

If I do this in mx, I also see output indicating the low memory conditions:

mx build -A-J-Xmx30M
...
Compiling com.oracle.truffle.api.object with javac-daemon(JDK 1.8)... [/Users/dnsimon/graal/graal/truffle/mxbuild/src/com.oracle.truffle.api.object/bin/com/oracle/truffle/api/object/DynamicObject.class does not exist]




The system is out of resources.
Consult the following stack trace for details.



The system is out of resources.
The system is out of resources.
Consult the following stack trace for details.

The system is out of resources.
Consult the following stack trace for details.
Consult the following stack trace for details.
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded


The system is out of resources.
java.lang.OutOfMemoryError: GC overhead limit exceeded
...

So it's still a mystery as to why you don't see something.

@dougxc
Copy link
Member

dougxc commented Jun 17, 2019

Also, I think specific to mx.py, the compiler daemons could take the --J / -J flags passed by the user, so we could trap the reasons for the crash. It's not using any of them atm - is this worth looking into?

The way to pass -J to javac is with the -A flag to mx build as shown above.
What flags would you pass to trap the crash reason(s)?

@neomatrix369
Copy link
Contributor Author

When I run a javac compilation with too little memory, I see:

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

If I do this in mx, I also see output indicating the low memory conditions:

mx build -A-J-Xmx30M
...
Compiling com.oracle.truffle.api.object with javac-daemon(JDK 1.8)... [/Users/dnsimon/graal/graal/truffle/mxbuild/src/com.oracle.truffle.api.object/bin/com/oracle/truffle/api/object/DynamicObject.class does not exist]




The system is out of resources.
Consult the following stack trace for details.



The system is out of resources.
The system is out of resources.
Consult the following stack trace for details.

The system is out of resources.
Consult the following stack trace for details.
Consult the following stack trace for details.
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded


The system is out of resources.
java.lang.OutOfMemoryError: GC overhead limit exceeded
...

So it's still a mystery as to why you don't see something.

I concur. Could it be that there is something happening at low-level and the container might just be terminating processes silently?

@neomatrix369
Copy link
Contributor Author

Also, I think specific to mx.py, the compiler daemons could take the --J / -J flags passed by the user, so we could trap the reasons for the crash. It's not using any of them atm - is this worth looking into?

The way to pass -J to javac is with the -A flag to mx build as shown above.
What flags would you pass to trap the crash reason(s)?

Would the jvm flags to capture OOME and heap dumps help? I had those in mind, so we could get some info just when it crashed.

I'll use the mx flag you mentioned to pass the JVM args to the javac compiler and see what the outcome is.

@neomatrix369
Copy link
Contributor Author

neomatrix369 commented Jun 19, 2019

I think I might have found out the reason for the failure, builds are back to green (https://circleci.com/gh/neomatrix369/awesome-graal/534), here's the changed code neomatrix369/awesome-graal@3d40ee1#diff-e09099e42fa9f20401d67bf58248adcd

I was settingHOTSPOT_BUILD_JOBS to max cpus and maybe this was causing issues with certain builds. @dougxc

@neomatrix369
Copy link
Contributor Author

kudos: loving the fact you guys built incremental compiling into the build system - makes rebuilds quick and it does it quite efficiently 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants