How do I run background or cron jobs?

Get Help and Help Others Solutions
1

Maybe you want to send emails, process data, run some lengthly calculations, fan out tasks to/from a queue. How would I do that with RedwoodJS? Truth be told, the long running and/or scheduled job has been a missing piece in the Jamstack puzzle for some time.

Bear in mind that you can deploy your RedwoodJS app to a serverless (Netlify, Vercel) or a serverful (Render, bare metal) provider. In the serverless world, lambdas only run for a short time – typically 10 seconds. So, your job you have to run rather quick.

Thankfully, there are now some options.

  1. If you deploy to a serverful context, like Render then you do have access to cron, redis, and a job runner that can perform these tasks. You may want to consider Creating a Background Worker with Exec and Faktory and Redwood’s exec script CLI command.

  2. Netlify’s background functions allow you to set up serverless function processes that run longer than 10 seconds and up to 15 minutes.

  3. Supabase offers cron inside Postgres where you can schedule requests that RedwoodJS can consume via a signed, secure webhook. It’s a great fit for that asynchronous background Netlify function.

  4. Want integrated cron? Netlify recently added scheduled functions that enable you to run them on a regular and consistent schedule, much like a cron job. But, you can define that schedule in code like schedule("@hourly", handler).

  5. Repeater is a Job Queue as a Service … service. It provides an API for you to schedule tasks that will call an http endpoint to run (similar to the Supabase cron approach).

  6. bare metal job running option TODO: @rob can you explain this? As done in Algostake

2 Likes
2

I’m going to be updating the baremetal deploy docs with this info soon, but here’s the Stack Overflow post that I used to unlock the magic! How to make a task job with PM2? - Stack Overflow

Note that you don’t have to use the baremetal deploy option to do this, but you do need a real server that you can connect to and run pm2.

1 Like
3

Also Ryan Chenkie wrote up how to uses queues with RedwoodJS!

1 Like
4

Hey Rob,
this is great and all, but it just focuses on the deployment. I am still wondering how I can “mock” the production server on my local.

Thing is: I’d like to implement a scheduler/cron job that executes one of my scripts/ on a regular basis - and I would like to test this on my local before moving to deploy on production. Is that possible?

I have researched a bunch and my blocker comes down to the non-existence of a Service.ts or similar, basically where the App starts and where I could initialize the cron service.

Or is the solution to create a wholly new service myself, write the pm2 launch config and let my redwood service interact with my custom written scheduler service?

Any advice would be much appreciated. I’ve also seen a request for a first-level support for cron jobs and just want to give that suggestion a +1!

Love redwood, keep rocking

5

I think I figured it out, here is how I handle the cron jobs on my local now (the same thing should work for any baremetal linux server):

  1. Install and setup a cronjob manager, I opted for cronie as described in the Arch Wiki
  2. Then added a new crontab with crontab -e and added the following: */5 * * * * cd /path/to/my/redwood/project && /usr/bin/yarn rw exec nameOfScript
  3. Now, when I check the systemctl status cronie I can see the output of my script, indicating that it has run.

Pretty basic, somewhat a workaround, since we don’t have the “classical” cron implementation for NodeJS. But seems to be more in line with the Redwood design.

Something like this could be used for syncing from an external service once a day. Just put the functionality in the script and call it with the crontab at the desired time.

1 Like
6

Nice work!

I was thinking something similar but if the job was a Webhook, to use a tunneler like ngrok and then have the cron job make a http request to the hook.

But, having the cron task be a script exec make perfect sense.

7

Nice! I’ve got an open issue for adding recurring jobs to pm2 and/or cron, but I didn’t have any plan for getting them running locally with a scheduler: whenever I had them in previous apps I wouldn’t have them actually running on recurring basis locally, I’d just start/stop them off on an as-needed basis to make sure they’re working. Having them in cron locally means they’d be running forever which wasn’t ideal.

So I’d just manually run yarn rw exec scriptName when I need to have the job run.

But maybe another process, like the api and web servers, that will start with yarn rw dev and only run while that command is running…that’d be great for simulating a persistent runner, but without eating up resources even when the rest of the app isn’t running. And it just runs yarn rw exec scriptName for you on whatever schedule.

1 Like
8

Task scheduling is the obvious call out here, but if you ever consider doing some baked into Redwood implementation for task scheduling, I hope it wont be overkill to consider doing it based on a work queue with a scheduled fallback so that you have the option of firing off a worker prior to the scheduled check if the work queue gets too full. The example I can think of here is Flume where it sends logs in batches when the batch is full, but if the batch never gets full due to a lull in traffic, you still have a minimum SLA met by a scheduler that will take care of whatever work is available.

1 Like
9

I would guess something like GitHub - graphile/worker: High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue) might work too?

But I’m unsure, how to connect redwood and it, so that I can have tasks run by the worker, but use prisma to insert into the database etc

10

This is our favorite, new answer to this question:

11

That’s what send me down searching again, as it’s been in the backlog for us for some time and came back to mind due to the new release.

That’s not going to work for us, as that’s in the cloud.

Also probably overkill for what I’m trying to do.

12

So graphile-worker seems to work great.

I had to create a new script in redwood and now I can start an additional docker container with rw exec worker to run that worker.

image

Also needed to copy the scripts folder, which I didn’t do before
image

My worker.ts looks like this:

image
image686×787 66.6 KB

Implementing the tasks is pretty straight forward and I seem to be able to use services and database stuff without a problem.

Things I’m not happy with, but seem non breaking:

  1. Importing stuff to a task in the script folder seems weird. As vscode doesn’t seem to figure out automatically, where to import stuff from.
  2. I’m not fully sure, where packages get pulled from. It seems to work fine to import them in the package.json in the api folder. But it seems a bit out of control, it would be nice, if the scripts folder had it’s own package.json
2 Likes
13

Hi.
For somebody looking for async job worker working code examples, I created a sample implementation with graphile-worker and redwood.js.
It’s working beautifully at my local environment.

1 Like