"Task timed out" error on some Laravel endpoints

[nickrupert7]

Description:

We have recently migrated one of our Laravel 8 projects to use serverless deployment with Bref on AWS Lambda.

SOME of our endpoints are consistently failing with the error {"message":"Internal Server Error"} (Task timed out after 28.03 seconds in CloudWatch logs) while other endpoints work perfectly.

I have seen #475 but I am not convinced that the error has to do with the database being inaccessible due to our VPC, because again, other endpoints which use the database seem to have no such connection issues. It is also worth noting that our serverless.yml file is configured to host the lambdas in the VPC.

I'm also not convinced that the error is a failure to connect to some other external resource, because logging indicates that the code is crashing before boot in AppServiceProvider is called, which means that the error is not caused by any code specifically within the failing endpoints.

The nature of this error suggests that it should be happening on every request, or not at all, which is why it's weird to me that it's only happening on SOME endpoints. And again, it's consistent about which endpoints work and don't work. I also haven't noticed any sort of pattern that links the consistently failing endpoints together.

How to reproduce:

I'm not sure what is causing the issue, so I can't provide generic steps to reproduce.

[mnapoli]

Hey! Just to be sure, are you aware of this limitation: https://bref.sh/docs/environment/database.html#accessing-the-internet

[t-richard]

Yes, there is no real way to get around it all. Sometimes, setting default_socket_timeout in INI file can help, also setting cURL timeout option globally can get around libraries not allowing to set the timeout directly.

[nickrupert7]

Adding the NAT gateway fixed my issues! I didn't even have to mess with the security groups, or timeout settings.

Would it be possible to publish this conversation publicly so somebody else in the future might find it and benefit from it?

The logging situation is still super weird to me, and the fact that log messages on failing endpoints weren't going to CloudWatch at all led me to incorrect conclusions about what was happening, so again, if somebody else comes along in the future with a similar issue I hope that this conversation might help them in some small way.

In conclusion, I think that the log messages must go to some kind of "buffer" in the Lambda function memory, and that buffer doesn't get dumped to CloudWatch until the request exists - even if the response is an error. But because it's timing out, the function itself never actually issues a response, and so the log buffer never gets dumped. At least I THINK that's what's happening.

[nickrupert7]

Also, I tried to mark your answer as the accepted answer, but GitHub isn't letting me. The button doesn't even seem to do anything. 🤷‍♂️

[mnapoli]

Adding the NAT gateway fixed my issues!

Would it be possible to publish this conversation publicly so somebody else in the future might find it and benefit from it?

😅 This is why I added a whole section in the documentation about that, with a big "WARNING" at the top:

https://bref.sh/docs/environment/database.html#accessing-the-internet

[nickrupert7]

Yeah that's my bad for not reading it! Like i said, the thing that really threw me off was that my logging wasn't working, so I really was convinced that the endpoints weren't even running at all. With trouble "accessing the internet" I wouldn't expect the code to hang up until the line that actually calls the internet.

But alas, you were right, and I thank you for your patience!