RedwoodJS v7, BullMQ & Bullboard

jtomchak · February 21, 2024, 8:39pm

I thought it would be helpful to see, now with the ability to easily configure the server with fasify plugins, I was able to add BullMQ & the BullBoard for monitoring in just a couple quick easy steps.

Use the fastify plugins GitHub - felixmosh/bull-board: 🎯 Queue background jobs inspector
Configure your server.ts file like this RedwoodJS v7 Server Config with BullMQ & BullBoard · GitHub

and you’re in business

dom · February 22, 2024, 12:04am

Psyched to hear, that’s exactly what it’s for!

jamesj · July 6, 2024, 11:41pm

This was a godsent as I recently discovered I’d have to compute some heavy data and wanted to find a way to externalize that to a worker. I just learned about message queues and Kafka or RabbitMQ seem like overkill but your solution really helped me out. Thank you!

dthyresson · July 8, 2024, 12:20am

We’re working on it!

github.com/redwoodjs/redwood

[WIP] Adds background job scheduling and execution

redwoodjs:main ← redwoodjs:rc-background-jobs

opened 09:42PM - 02 Jul 24 UTC

cannikin

+2659 -0

This new package provides scheduling and processing of background jobs. We want …everything needed to run a modern web application to be included in Redwood itself—you shouldn't need any third party integrations if you don't want. Background jobs have been sorely missed, but the time has come! (If you do want to use a third party service we have had an [integration with Inngest](https://community.redwoodjs.com/t/ship-background-jobs-crons-webhooks-and-reliable-workflows-in-record-time-with-inngest-and-redwoodjs/4866) since May of 2023!) ## What's Included * A base `RedwoodJob` class from which your own custom jobs will extend. You only need to fill out the details of a single `perform()` action, accepting whatever arguments you want, and the underlying RedwoodJob code will take care of scheduling, delaying, running, and, in the case your job fails, recording the error and rescheduling in the future for a retry. * Backend adapters for storing your jobs. Today we're shipping with a `PrismaAdapter` but we also provide a `BaseAdapter` from which you can extend and build your own. * A persistent process to watch for new jobs and execute them. It can be run in dev mode, which stays attached to your console so you can monitor and execute jobs in development, or in daemon mode which detaches from the console and runs in the background forever (you'll use this mode in production). Decoupling the jobs from their backends means you can swap out backends as your app grows, or even use different backends for different jobs! The actual Worker and Executor classes that know how to find a job and work on it are self-contained, so you can write your own runner if you want. ## Features * Named queues: you can schedule jobs in separate named queues and have a different number of workers monitoring each one—makes it much easier to scale your background processing * Priority: give your jobs a priority from 1 (highest) to 100 (lowest). Workers will sort available jobs by priority, working the most important ones first. * Configurable delay: run your job as soon as possible (default), wait a number of seconds before running, or run at a specific time in the future * Run inline: instead of scheduling to run in the background, run immediately * Auto-retries with backoff: if your job fails it will back off at the rate of `attempts ** 4` for a default of 24 tries, the time between the last two attempts is a little over three days. The number of max retries is configurable per job. * Integrates with Redwood's [logger](https://docs.redwoodjs.com/docs/logger): use your existing one in `api/src/lib/logger` or create a new one just for job logging ## How it Works Using the `PrismaAdapter` means your jobs are stored in your database. The `yarn rw setup jobs` script will add a `BackgroundJob` model in your `schema.prisma` file. Any job that is invoked with `.performLater()` will add a row to this table: ``` WelcomeEmailJob.performLater({ user.email }) ``` If using the `PrismaAdapter`, any arguments you want to give to your job must be serializable as JSON since the values will be stored in the database as text. The persistent job workers (started in dev with `yarn rw jobs work` or detached to run in the background with `yarn rw jobs start`) will periodically check the database for any jobs that are qualified to run: not already locked by another worker and with a `runAt` time before or equal to right now. They'll lock the record, instantiate your job class and call `perform()` on it, passing in the arguments you gave when scheduling it. * If the job succeeds it is removed from the database * If the job fails the error is recorded, the job is rescheduled to try again, and the lock is removed Repeat until the queue is empty! ## Usage ### Setup To simplify the setup, run the included setup script: ``` yarn rw setup jobs ``` This creates `api/src/lib/jobs` with the basic config included to get up and running, as well as the model added to your `schema.prisma` file. You can generate a job with the shell ready to go: ``` yarn rw g job WelcomeEmail ``` This creates a file at `api/src/jobs/WelcomeEmailJob.js` along with the shell of your job. All you need to is fill out the `perform()` function: ```javascript // api/src/jobs/WelcomeEmailJob.js export class WelcomeEmailJob extends RedwoodJob { perform(email) { // send email... } } ``` ### Scheduling A typical place you'd use this job would be in a service. In this case, let's add it to the `users` service after creating a user: ```javascript // api/src/services/users/users.js export const createUser = async ({ input }) { const user = await db.user.create({ data: input }) await WelcomeEmailJob.performLater(user.email) return user }) ``` With the above syntax your job will run as soon as possible, in the queue named "default" and with a priority of 50. You can also delay your job for, say, 5 minutes: ```javascript OnboardingJob.set({ wait: 300 }).performLater(user.email) ``` Or run it at a specific time in the future: ```javascript MilleniumReminderJob.set({ waitUntil: new Date(2999, 11, 31, 12, 0, 0) }).performLater(user.email) ``` There are lots of ways to customize the scheduling and worker processes. Check out the docs for the full list! ### Execution To run your jobs, start up the runner: ```bash yarn rw jobs work ``` This process will stay attached the console and continually look for new jobs and execute them as they are found. To work on whatever outstanding jobs there are and then exit, use the `workoff` mode instead. To run the worker(s) in the background, use the `start` mode: ```bash yarn rw jobs start ``` To stop them: ```bash yarn rw jobs stop ``` You can start more than one worker by passing the `-n` flag: ```bash yarn rw jobs start -n 4 ``` If you want to specify that some workers only work on certain named queues: ```bash yarn rw jobs start -n default:2,email:1 ``` Make sure you pass the same flags to the `stop` process as the `start` so it knows which ones to stop. You can `restart` your workers as well. In production you'll want to hook the workers up to a process monitor as, just like with any other process, they could die unexpectedly. More on this in the docs. ## The Future * More adapters (maybe the community wants to get involved?): Redis, SQS, RabbitMQ * Studio integration: monitor the state of your outstanding jobs * Baremetal integration: if jobs are enabled, monitor the workers * Recurring jobs (performEvery?) ## TODO * [ ] Typescript * [ ] `yarn rw setup jobs` * [ ] `yarn rw g job` * [ ] Documentation

jamesj · July 8, 2024, 1:33am

Oh my god this made my night. That is so amazing. Especially this line:

Decoupling the jobs from their backends means you can swap out backends as your app grows, or even use different backends for different jobs!

If I have a very large database update I’d love to run that on another container that processes jobs so my api backend isn’t overloaded in production as multiple requests will be hitting it during heavy workload.

The future is amazing. The future is RedwoodJS