@danny ok, lots to parse here. My goal is two-fold: give you some straight-up opinions and help towards a manageable next step. But the caveat is that I’m most definitely not our resident expert at Auth.
But let me start with this → woah, this is awesome. Lots of moving pieces against a young framework and you pulled it off. Way. To. Go. 
Opinions
I’ve built a lot of things. Can’t say I’m necessarily great at it, but something I seem to do increasingly over time is utilize existing solutions at the beginning of a project – especially when my team is small and cognitive overhead is a precious resource. Not knowing your specific circumstances/needs, I have to +1 the suggestion about Auth0 → it’s what I would use. You get 7,000 users on the free tier. 7,000! 
@chris-hailstorm (who’s one of our resident AuthMasters™) is a big advocate of not rolling your own Auth because of all the downstream capabilities you end up needing to address that can add a lot of overhead: 2 factor, Multiple identity providers, logging and auditing, security, etc. I agree with him.
But, to be clear, you don’t have to agree. So consider this a Fwiw…
Env Var Workarounds
You mention one of the primary blocking issues you’re facing is maxing out the size of env var (that definitely sucks – do you know if that’s Lamdba restriction or something on Netlify’s sides? Also, what’s the diff between size limit and your jwt secret?)
As a master of hacking things together, what I’d attempt to do is store the JWT secret in something like AWS Secrets Manager (or similar) and use the Lambda Env Var to access the Secrets Manager. That might be terrible and/or flawed due to serverless structure. But, well, you asked 
RW Support for JWT + Roll-your-own Auth
Regarding " Step 3: Patch authHeaders in @redwood/api/auth", this could be of interest to support on the framework side.
@peterp (our resident, resident AuthMaster™) is currently working on some additions and restructuring of the Auth package. See #637. So the timing is right to discuss if something like this is possible, specifically this comment from above:
In node_modules/@redwoodjs/api/dist/auth/authHeaders.js, add a new case to the switch statement so auth-provider jwt is allowed
Also, I need Peter’s help clarifying these two questions:
So whats the best way to pass in the public key here without modifying redwood’s source (assuming the case statement exists). Is there some way we could pass it in when calling
createGraphQLHandler?
^^ I don’t believe this is possible
In custom auth client, I had to add
window.__REDWOOD__USE_AUTH = useCustomAuth
This gets called in componentDidMount of the authProvider, but I don’t see why I had to do this considering this is set in the useCustomAuth, and is done exactly the same way as auth0 provider is setup.
^^ unclear to me