[Disussion] Benchmarks!

[jcbhmr]

This is a discussion thread to discuss WHY the benchmarks are the way they are and how to improve on them

#4 (comment)

[jcbhmr]

@jimmywarting re your thread from #4 (comment)

Yeah, I'm getting similarish results.

tinylet/test/async-to-sync-comparison.bench.js

Lines 33 to 60 in 72fe63f

	const synckit = createSyncFn(process.cwd() + "/synckit-runAsWorker.mjs");
	bench.add("synckit", () => {
	synckit(testFile);
	});
	const tinylet = redlet(async (f) => {
	const { readFile } = await import("node:fs/promises");
	return await readFile(f);
	});
	bench.add("tinylet", () => {
	tinylet(testFile);
	});
	const await_sync = createWorker()(async (f) => {
	const { readFile } = await import("node:fs/promises");
	return await readFile(f);
	});
	bench.add("await-sync", () => {
	await_sync(testFile);
	});
	const make_synchronous = makeSynchronous(async (f) => {
	const { readFile } = await import("node:fs/promises");
	return await readFile(f);
	});
	bench.add("make-synchronous", () => {
	make_synchronous(testFile);
	});

I think the reason that the synckit is so much faster is because it's not transferring the data: URL each time. It's ONLY transferring the arguments array. That's it.

const msg: MainToWorkerMessage<Parameters<T>> = { sharedBuffer, id, args }
//                                              👆 obj is 1      👆 id is 3
//                                                    👆 sab ptr is 2 👆 args are N

👆 That's only N+3 "things" that need to get serialized/transfered each call. Compare that to:

port.postMessage([lockBuffer, executorURL, this, [...arguments]]);
//               👆 array is 1  👆 str is M length, needs to be copied
//                                           👆 this is usually 1 (undefined)
//                                                     👆 arguments are N

👆 This is N+M+4. I think that might be why it's slower than synckit?

[jimmywarting]

I think the reason that the synckit is so much faster is because it's not transferring the data: URL each time

O_o

In my own test i just mostly only bench-testing the functions execution time. not the time it takes to load up a new worker.
So my bench test only calls this function once:

  const url = "data:text/javascript," + encodeURIComponent(code)
  const { default: fn } = await import(url)

therefore the data url is only transfered once.

[jimmywarting]

my assumption to why synckit is faster is b/c it cheats and uses receiveMessageOnPort
it dose not use any (de)serialize methods to transfer the data from the worker to the main thread via SharedArrayBuffer

it uses postMessages instead - which is a no go for other env solutions.

[jcbhmr]

When I remove the 200 bytes of data: URL that was getting transferred each time, it reduced the time enough that now tinylet/redlet() is the fastest!

I'm currently using a very crude caching system. I need to make it a bit more robust to failure so that having something throw doesn't mean game over 😅

child worker doing the recieving

tinylet/src/redlet-node.js

Lines 35 to 41 in e3d21f4

	if (typeof executorURLOrId === "number") {
	f = rememberedFunctions[executorURLOrId];
	} else {
	const module = await import(executorURLOrId);
	f = await module.default;
	rememberedFunctions.push(f);
	}

parent caller outside worker

tinylet/src/redlet-node.js

Lines 103 to 108 in e3d21f4

	if (rememberedURLs.includes(executorURL)) {
	port.postMessage([lockBuffer, rememberedURLs.indexOf(executorURL), args]);
	} else {
	port.postMessage([lockBuffer, executorURL, args]);
	rememberedURLs.push(executorURL);
	}

[jcbhmr]

my assumption to why synckit is faster is b/c it cheats and uses receiveMessageOnPort it dose not use any (de)serialize methods to transfer the data from the worker to the main thread via SharedArrayBuffer

it uses postMessages instead - which is a no go for other env solutions.

You may be right. I think that having a specialized export for Node.js that uses recieveMessageOnPort() to get 🏎🏎 speed and then a normal browser-compatible Deno-compatible version is the ideal end-game. Both exposed as the same entry point so that you don't need to care about the implementation, it just auto-routes it to the best option for your platform using export conditions

[jimmywarting]

using postMessage and recieveMessageOnPort have some advantages... it can transfer all structural clonable objects. including things such as ReadableStreams, ArrayBufferViews, Dates, Error Objects, Blob / Files, Regex and everything.
Blob & files can easily just be references points instead of fully cloned content

and while you are at it you could also use the transferable option to instead of cloning a typed array you would then instead transfer it. so using postMessage(data, [ transferList ]) have some advantages... but the con is that deno, bun, browser dose not have recieveMessageOnPort

[jcbhmr]

Is this just me or is Deno just really really slow?

Deno:

Node.js:

[jcbhmr]

Tried converting Deno benchmarks to the native https://deno.land/[email protected]/tools/benchmarker Deno.bench() and still terrible results... 😭😭😭

[jcbhmr]

@jimmywarting This is very interesting. Deno has a not-so-great postMessage() serialization and transfer procedure. This means that your trick of doing everything in a SharedArrayBuffer polling loop is orders of magnitude faster! Awesome trick! 👍

denoland/deno#11561