Updated hyperexecute file to handle timeout error and run on different platforms

[gauravchawhan]

No description provided.

[gauravchawhan]

Hi @josdejong,
I have updated the changes to handle timeouts and added support for running unit tests on Multiple OS.
Please try with the latest changes in the PR, let us know if you face any issues.

Thanks

[josdejong]

Thanks @gauravchawhan , I've updated the configuration in hyperexecute.yaml (in the develop branch) and now the tests run fine 🎉.

I'll work out the configuration further this week and let you know (replace samle project urls, and test on all different browsers like Firefox, Chrome, Edge, Safari, trigger the testing on push instead of workflow_dispatch etc).

[josdejong]

@gauravchawhan I can use some more help:

1. Is there any documentation on what browsers and platforms are available? I try to configure testing on Chrome, Firefox, Safari, Edge, and Android Browser but miss information about, see: https://github.com/josdejong/mathjs/blob/develop/test/browser-test-config/lambdatest-karma.js#L27-L61
2. Is there a way to get more informative logs in Github Actions, showing something like "Succesfully executed X tests"? Currently it shows how long running the tests take, but from the logs I cannot see whether it did actually run actual tests or something. Here an example of the current output: https://github.com/josdejong/mathjs/actions/runs/11663745399/job/32472823537

[gauravchawhan]

Hi @josdejong,

Here is the documentation for the supported OS and browser combinations, which can be updated in the capabilities section of your configuration here.

Additionally, I have updated the code to allow printing the number of tests by executing a custom script. Please give it a try, and let me know if you encounter any issues.

[josdejong]

Thanks @gauravchawhan

1. I now found the following endpoint which returns all available platforms and browsers: https://api.lambdatest.com/automation/api/v1/platforms. However I can't manage to get the safari_mac and android tests working, and the error messages do not give any pointer on what's wrong. How do I make this work?

   See code: https://github.com/josdejong/mathjs/blob/develop/test/browser-test-config/lambdatest-karma.js#L27-L58

2. Thanks for the script to output the total count. It loooks quite fragile though, for example it will break when I rename the test folder or karma config file name. I'm not sure if that is a good way ahead

   Isn't there a way to automatically output the logs coming from karma/mocha itself (like with BrowserStack, see this example)? I tried by configuring logLevel: config.LOG_INFO, but that only introduces many extra blank lines in the logs that I can view in the HyperExecute dashboard (see for example run 27), and none of the logs become available in the Github Actions logs.

3. Right now, BrowserStack tests run in less than 2 minutes (for 5 browsers), and HyperExecute runs in about 3 minutes (currently 3 browsers). That is slower. However, in the logs of HyperExecute I see the actual tests themselves take 1:10 minutes. So apparently there is a lot of overhead between the actual tests and the GitHub Actions. In the config I see that:

   a. First we start a GitHub action running Windows, that downloads hyperexecute.exe and starts it
   b. Then HyperExecute starts a new windows machine and goes through it's config
   c. THen HyperExecute starts a linux machine that runs karma
   d. Then karma starts the different browsers on specific machines

   It feels like we're starting three machines in series where we would only need one? Or am I misunderstanding what's going on? (I.e. the BrowserStack config directly starts karma in 1 step running Ubuntu and that's it, and all the logs are available directly in GitHub Actions).

[josdejong]

@gauravchawhan @SparshKesari any help?

[gauravchawhan]

Hi @josdejong,

1. Here is the API that will help you fetch all the OS-browser combinations specifically for HyperExecute:
   https://api.hyperexecute.cloud/v2.0/browsers. Additionally, here is the list of all available devices: https://mobile-api.lambdatest.com/mobile-automation/api/v1/list.

2. Currently, each configuration and device maintains its own logs, which can be downloaded or can be viewed on the hyperexecute dashboard. At the moment, we do not natively support aggregating logs from all machines in one place to keep the logging process simple.

3. You're correct that HyperExecute introduces some overhead due to its multi-machine orchestration, which optimizes scalability and performance for larger, complex test suites by isolating processes. Since HyperExecute is tailored for End-to-End (E2E) testing, rather than speed-focused unit testing, this explains the speed lags you're observing during unit tests.

[josdejong]

@gauravchawhan thanks for your reply

1. I tried updating the config for Mac+Safari via 84f571b but without luck, what am I overlooking? Furthermore, the url https://mobile-api.lambdatest.com/mobile-automation/api/v1/list requires a username and password. I tried my LambdaTest username and password but that doesn't work. How can I access this endpoint?
2. Clear, thanks
3. Clear, thanks.

[josdejong]

I'll close this PR now. It looks like HyperExecute isn't a good fit for the "speed-focused unit testing" of mathjs, at least not right now. It runs much slower than Browserstack, I guess due to its "multi-machine orchestration". And the lack of easy accessible test logs is actually a step back.

Thanks for all the help during this experiment @gauravchawhan and @SparshKesari, I'm sorry to see that it didn't work out.