[META] Chromium performance issues

[Sparticuz]

Chromium seems to be running slower after upgrading from puppeteer 10 -> 14

See attached comment:

I am genuinely surprised that the chromium upgrade has had such a significant increase to our costs. I am sharing our AWS lambda costs below as reference, these are grouped by regions. We made the upgrade around July 1st.

P.S I am only sharing this to show you what I mean and not for you to help us with our costs problems 😉 .

Originally posted by @hasan-wajahat in #15 (comment)

[codykaup]

We're seeing similar behavior in our testing. It can take long enough to load where it times out the Lambda (even if we set the timeout to 5 minutes).

We've also seeing more flaky rendering where text blocks or images are randomly missing from the screenshot. The network logs look good so I'm wondering if it's during the render somewhere. 🤔

Do we have any leads on what causes the performance difference? Maybe any workarounds?

[hasan-wajahat]

Hey @codykaup we have reduced some of the problems by passing more optimization flags to the chrome browser.
This is what we used:

   '--disable-background-timer-throttling',
        '--disable-gpu',
        '--disable-dev-shm-usage',
        '--disable-extensions',
        '--disable-features=TranslateUI,BlinkGenPropertyTrees',
        '--disable-accelerated-2d-canvas',
        '--disable-ipc-flooding-protection',
        '--disable-renderer-backgrounding',
        '--enable-features=NetworkService,NetworkServiceInProcess',
        '--no-first-run',

This significantly improved performance because when you disable gpu stuff then natually chrome's performance improves.
We also increased the memory available to aws lambda.

However the core problem hasn't been resolved which is puppeteer consuming much more resources now.

[codykaup]

Thanks @hasan-wajahat, I tested things again with those flags and it worked much better than before! We also had to increase the amount of memory provided to the Lambda since the memory footprint is larger within Puppeteer.

One issue we're still having is flakiness within our snapshots. For whatever reason, local icons/images/videos/etc. are randomly missing on the page even if we disable WebGL or add a sleep after the render. Did you happen to run into cases like that on your end?

[Sparticuz]

I can probably bring a few of those into CAL.

[Sparticuz]

Chromium had major graphic stack changes, which is why I had to fork the package. I'm suspecting that that is the majority of the performance issues. Disabling the graphics stack, which is what you did, most likely will bypass many of the issues.

[hasan-wajahat]

Hey @codykaup I haven't noticed issues with screenshots yet. But I still occasionally get missing elements in the dom or sometimes the puppeteer just times out. I assume your screenshot issue has the same cause as my issues.

Like Sparticuz is saying there must have been major changes in Chrome because I don't get any of the problems if I downgrade to version 10 of Chrome aws lambda.

In any case I feel this issue is worth exploring because I am sure this problem will effect a lot of people but it only shows up if you run the lambda functions with a significant frequency.

[codykaup]

But I still occasionally get missing elements in the dom

I'm glad I'm not the only one seeing this! 😅

Do we have any leads on why this would be happening? I've been doing some testing but nothing sticks out at the moment. 🤔

[Sparticuz]

Puppeteer did some updates on the wait function a few versions ago, perhaps that is the culprit.

[Sparticuz]

@codykaup @hasan-wajahat Try v17 and see if you get similar results. It's the same chromium version as v15 and v16, but perhaps there were some puppeteer improvements that helped?

It's really amazing how much faster v17 is so a huge THANKS for adding that @Sparticuz ! The Lambda runs about 3 times as fast!

With v14:

With v17:

Originally posted by @QAnders in #13 (comment)

[QAnders]

I did some additional runs with and without the flags @hasan-wajahat posted and with v17 I can't say they are doing any difference in time now (average of 10 invokes), but a slight decrease in memory used (about 4-5% less).