Approaches to track memory leaks in async php

[maciejfikus]

As I am having really tough time with debugging my already production-deployed app I wonder how do you track memory issues in async php?

I mean, it took me a few hours by trial and error to figure out what part of my app is responsible for memory leak. I wasn't sure if it is X itself, my own code or some package i.e. PgAsync. By now I am 99% sure it is fault of PgAsync (voryx/PgAsync#48) but after all I am not able to locate exact place and reason there.

I tried to debug it with php-memprof, but it doesn't say much to me, not sure if it is suitable for async envs tho. What approaches and tools do you use to find out stuff like that?

[clue]

@maciejfikus Thank for reporting, this is an interesting one!

To the best of our knowledge, Framework X does not exhibit any memory leaks on its own and can safely run for months straight. PHP has a deterministic garbage collector which should make it "virtually impossible" to leak memory in unpredictable places.

That said, it's definitely possible to "waste" memory depending on how your application logic works. For instance, if you keep adding an element to an array for each incoming request, you will see that memory usage keep growing for each additional HTTP request. X is carefully designed to avoid any unneeded memory allocations, but depending on your application logic, this memoization strategy is definitely something we've seen in the wild and should be avoided for long-running application servers.

If you think you've found a memory leak in any of the underlying components, reporting this upstream is definitely the way to go. I see you've reported this upstream in PgAsync. We don't maintain this component, but I'm also not aware of any known memory leaks here. I did take a quick look at your ticket and don't see anything wrong after eyeballing this. If you can pinpoint this to a single component, creating a simple reproducer definitely helps. Perhaps you can work out a set of commands to reproduce this locally (such as starting a blank database in a container and a simple gist to reproduce this)?

I hope this helps 👍

[maciejfikus]

Hey @clue, thanks for an answer.
As we discussed this issue before, I will provide some simple code to reproduce the issue out of the box. Had hope to have this done already, but unfortunately I've got some health issues recently. I will do my best to do it this week.

[maciejfikus]

@clue, I've included sample project in mentioned issue: voryx/PgAsync#48 (comment)
It's just simple PoC, probably far from perfect. You can achieve the same thing by passing a client instance to container in FrameworkX, running this query in controller and spamming that route with some http testing tool. The result will be pretty much the same.

[clue]

@maciejfikus Good, let's continue the discussion in voryx/PgAsync#48 👍

If you can reproduce the same memory issues in Framework X without PgAsync, make sure to report back here.

[maciejfikus]

@clue, In fact, I've gotten to the point where I believe FrameworkX also contributed to the memory problem. Otherwise, this is normal GC behavior in PHP when using fibers.

In the last example I posted here even though the memory is leaking (as I mentioned in the thread), manually running the garbage collector in a loop resets the reserved memory to its normal state.

\ReactEventLoop::addPeriodicTimer(60, function () {.
    gc_collect_cycles();
    gc_mem_caches();
    var_dump('gc_collect_cycles');
});

The same workaround will not work when using FrameworkX.
Here we have two scenarios, the first, presented below, and the second - where Connection goes to the Container, while the /test route is moved to the Controller. In the first scenario, GC reclaim about 5% of the memory, while with Container and Controller – running GC has no effect at all.

<?php

declare(strict_types=1);

require __DIR__ . '/../vendor/autoload.php';

$connection = new \PgAsync\Connection([
    'host'     => '172.29.0.1',
    'port'     => 54320,
    'user'     => 'devuser',
    'password' => 'devsecret',
    'database' => 'devdb'
], \React\EventLoop\Loop::get());

$app = new FrameworkX\App();

$app->get('/test', function (\Psr\Http\Message\ServerRequestInterface $request) use ($connection) {
    for ($i = 1; $i <= 30000; $i++) {
        $connection->query('SELECT 1')->subscribe(
            fn($result) => var_dump($result),
            fn($ex) => 5,
        );
    }

    return \React\Http\Message\Response::json([]);
});

\React\EventLoop\Loop::addPeriodicTimer(60, function () {
    gc_collect_cycles();
    gc_mem_caches();
    var_dump('gc_collect_cycles');
});

$app->run();

I believe that PgAsync is not working properly, but at this point I am wondering why GC is not able to recover memory as soon as FrameworkX and its routing joins the party?

[clue]

From what you're saying, it sounds like PgAsync might be at fault and to the best of our knowledge, Framework X does not exhibit any memory leaks. If you can provide a simple reproducer with Framework X only, I'm happy to take a look here, but as it stands, it sounds like this should be handled upstream instead. If you need this for a commercial project and you're interested in professional support, please reach out via email and we might be able to take a look at this for you.

[maciejfikus]

@clue To be honest I don't have an idea how to paraphrase my message. We already agreed that PgAsync in at fault of leak. Above message was focused on garbage collector workings in framework x. I would like to understand why manually running GC clears leaked memory in global scope and can't do same from framework x controllers. Please take a look on provided example for a reference. PgAsync doesn't matter in this context anymore.

[SimonFrings]

If you can provide a simple reproducer with Framework X only, I'm happy to take a look here

@maciejfikus I think what @clue meant is that your example above still contains PgAsync and doesn't show an example with only Framework X. If the same behavior still consists without PgAsync in it, we can take a closer look. That way we can make better guesses as to whether Framework X is causing this or something else.

[maciejfikus]

@SimonFrings I see. It would probably require finding another way, probably in pure PHP, to leak some memory – after all it probably doesn't matter what is leaking while behavior of GC change in relation to using X. Anyway, I got your point :-)