Intermittent prisma crashes

@danny When I first posted this issue I was on RW 0.13.0. What version are you upgrading from?

howdy all! quick note that this is on my (our) radar. reading through the comments and history… oh my :open_mouth:

Feels like Prisma <–> Postgres connection issue at first glance. Will figure out some next steps.

@danny to confirm, you’re seeing this on production using Netlify and DigitalOcean (for DB)?

@Tobbe to confirm, you’re seeing this on production using Netlify and Heroku (free tier DB)?

Correct

. . . . . . . . Something something, min characters in a message…

1 Like

We’re upgrading from 0.12, probably not a coincidence :slight_smile:

We’re holding off prod deployment for a little while, we’re seeing this on staging using Netlify and AWS RDS (no proxy). I’m going to switch staging to a DigitalOcean DB and see if it happens as well. Report back in a few!

1 Like

That’s great to know. So something could have changed from 0.12.0 to 0.13.0

Finally got around to deliberately test this. Unfortunately still crashes.

OK, updates - unfortunately connection pooling doesn’t make a difference either. I forked my prod DB to connect to staging to try, all the settings are the same.

Some thoughts:

  1. It’s super interesting that the error happens always after that 90s wait in Tobbe’s script. Exactly the same situation in mine, although I’m using a graphql curl to test (amazing find @Tobbe!)
  2. I definitely don’t see this issue happening on RW 0.12 (my prod version), and the prisma client wasn’t changed between 0.12 and 0.13 https://github.com/redwoodjs/redwood/compare/v0.12.0...v0.13.0#diff-e5f0c02e2caab3031448f49aeeece0edR12 (press Files Changed, then scroll down to api/package.json) - which would mean its something else in RW?
  3. Tobbe wasn’t able to reproduce on vanilla prisma (please confirm!)

Managed to turn on some more logging (thanks @dthyresson) and this is what I see when it errors out

1:09:12 AM: 2020-08-04T23:09:12.498Z engine {
1:09:12 AM: error: SocketError: other side closed
1:09:12 AM: at Socket.<anonymous> (/var/task/src/node_modules/@prisma/client/runtime/index.js:1:81651)
1:09:12 AM: at Socket.emit (events.js:327:22)
1:09:12 AM: at endReadableNT (_stream_readable.js:1221:12)
1:09:12 AM: at processTicksAndRejections (internal/process/task_queues.js:84:21) {
1:09:12 AM: code: 'UND_ERR_SOCKET'
1:09:12 AM: }
1:09:12 AM: }
1 Like

Correct.

I have identical lambda functions in RW and in a project based on https://github.com/prisma/deployment-example-netlify But I can only trigger the error in the RW project

RW project

$ yarn list --pattern "@prisma/client"
yarn list v1.22.4
└─ @prisma/client@2.2.2
Done in 1.43s.

prisma example project

$ npm ls @prisma/client
deployment-example-netlify@1.0.0 C:\Users\tobbe\dev\redwood\prisma\deployment-example-netlify
`-- @prisma/client@2.0.0-beta.1

What prisma client version did we use in RW 0.12.0?

On prod (RW 0.12)

yarn list --pattern "@prisma/client"
yarn list v1.22.4
└─ @prisma/client@2.1.1

Same as it says in the diff. Want to try downgrading and seeing if it makes a difference? I’m too far ahead with 0.15.3

I’m doing it the other way around first, because that was easier - I’m upgrading my prisma project to 2.2.2 to see if that makes it reproducible for me.

Updating wasn’t as easy as I had hoped

1:53:28 AM: 2020-08-04T23:53:28.109Z 93ef6208-c34c-4875-974a-108c4bf67cb6
ERROR PrismaClientInitializationError: Invalid prisma.product.findOne() invocation in/var/task/src/functions/product.js:9:42
Query engine exited with code 1error: Found argument ‘–enable-experimental’ which wasn’t expected, or isn’t valid in this context
Did you mean --enable-playground?

USAGE:
query-engine-rhel-openssl-1.0.x --enable-playground
For more information try --help at PrismaClientFetcher.request (/var/task/src/node_modules/@prisma/client/runtime/index.js:1:211294) at processTicksAndRejections (internal/process/task_queues.js:97:5)
1:53:28 AM: Duration: 152.85 ms Memory Usage: 92 MB Init Duration: 210.15 ms
1:53:33 AM: 2020-08-04T23:53:33.219Z
3b3525c3-025e-41e5-89e0-95012af08152 ERROR PrismaClientInitializationError: Invalid prisma.product.findOne() invocation in/var/task/src/functions/product.js:9:42 Query engine exited with code 1error: Found argument ‘–enable-experimental’ which wasn’t expected, or isn’t valid in this context
Did you mean --enable-playground?

USAGE:
query-engine-rhel-openssl-1.0.x --enable-playground
For more information try --help at PrismaClientFetcher.request (/var/task/src/node_modules/@prisma/client/runtime/index.js:1:211294) at processTicksAndRejections (internal/process/task_queues.js:97:5)
1:53:33 AM: Duration: 2.16 ms Memory Usage: 92 MB


Will have to continue tomorrow, too tired now

Omg, our latest lead… https://github.com/prisma/prisma-client-js/issues/789

Going to try patch packaging and upgrading the prisma client to 2.4.0

1 Like

Great find! I’ll try to upgrade as well tomorrow when I’m out of bed

Sadly, my hacky attempts to upgrade prisma weren’t succesful. I’ll wait for @peterp or @thedavid to suggest a way to upgrade just the prisma dependencies. Maybe install redwood from a branch on my github with changes?

Anyway, one for the morning :slight_smile:

Thank you Tobbe, this was fun! To be continued…

1 Like

Ah, the dreaded socket/connection reset by peer type errors. Contentful would give me those from time to time when making multiple repeated queries – which is why I almost wrapped in a retry.

Glad to see Prisma may have fixed in in 2.4.

o/

I’m upgrading over here: https://github.com/redwoodjs/redwood/pull/937

I’ll merge and give you the canary version in a few minutes. I’ve changed the way the the client was specified so that you can upgrade it yourself in the future. (With the caveat that we haven’t tested it.)

This canary version will also include:

I have not tested the scalar changes, but they should be OK. If you’re unable to log in then the scalar changes are NOT ok :confused:

2 Likes

I’ll try that right now!

1 Like

Success! Thanks for the build @peterp Feels good to finally have this resolved

2 Likes

Thank you Peter! Just upgraded, seems to have fixed the issue for me too! :tada: No auth issues either

Well, this was an awesome experience. Open source debugging FTW - thank you all for your help :)!


Summary ⍞

To summarise for anyone who stumbles across this thread - there was an issue introduced in Prisma clients 2.2.x to 2.3.x, and as a result seen in Redwood version >=0.13 and <=0.15.3.

This has been fixed as of 2.4.0 of prisma client, and expect to see the RW fix in 0.15.4.

2 Likes

If someone comes here at the future:
I had the same issue when I tried to upgrade to @prisma/cli: “2.7.1”, not using redwood at all and in a very very simple project.
Downgrading to 2.4.0 made the issue go away for me

1 Like