How do I start background Jobs on a baremetal deploy?

Get Help and Help Others
1

I’ve been messing with the baremetal deploys for a few months now and I really like it. I however have an issue.

If I deploy yarn rw deploy baremetal production I cannot seem to get the background jobs to start.

I’ve been following the Advanced Job Workers and trying to match that up with the ecosystem.js

I’ve tried a number of configurations and tried this with the first-run option to get these running…I mentioned it in Esteban’s BareMetal Cheat sheet

module.exports = {
  apps: [
/*    {      name: 'api',    },*/// not relevant here
    { //adv. job test
      name: 'job',
      cwd: 'current',
      script: 'node_modules/.bin/rw-jobs-worker',
      args: '--index=0 --id=0',
      instances: 'max',
      exec_mode: 'cluster',
      wait_ready: true,
      listen_timeout: 10000,
    },
{//esteban suggested this, but it didnt start it
        name: 'bg-jobs',
        cwd: 'current',
        script: 'node_modules/.bin/rw',
        args: 'jobs start',
        instances: 'max',
        exec_mode: 'cluster',
        wait_ready: true,
        listen_timeout: 10000,
      },
  ],
}

My workaround is to do the deploy, then ssh to the machine, cd into the /var/www/project/current and run yarn rw jobs start

This works until the current symlinked direcotry gets dropped, but really i need to reboot the server and restart the job. I had it working with pm2, but the runner was tied to the … dated directory and not current.

Anyone have any ideas on how to get these background jobs to start and restart with baremetal deploys?

2 Likes
2

Hi @jacebenson just to clarify- you are looking for a way to start jobs on boot?

Or it fails when starting up on boot/reboot?

3

I would like it to start on boot, and restart on deploy, if not running on deploy to start.

4

My go to ChatGPT…

To solve this issue with pm2 and ensure that your background jobs always run in the current directory after deployment, you can use a combination of a pm2 ecosystem file and a dynamic cwd (current working directory) to always point to the symlinked current directory. Here’s an approach that should help:

  1. Create an Ecosystem File
    Use pm2’s ecosystem file to define your process, ensuring that the cwd points to the current directory and not a dated one.

    {
  "apps": [
    {
      "name": "rw-jobs",
      "cwd": "/var/www/project/current",
      "script": "yarn",
      "args": "rw jobs start",
      "watch": false,
      "env": {
        "NODE_ENV": "production"
      }
    }
  ]
}

    This makes sure that whenever pm2 restarts, it picks up the current symlink directory.

  2. Deploy with pm2 reload
    During your deployment process, after you update the symlink to point to the new release, you can issue:

    pm2 reload ecosystem.config.js

    This will reload the rw-jobs process and restart it, ensuring that it uses the newly symlinked current directory.

  3. Use post-deploy Hook
    If you are using a deployment tool like pm2 deploy, you can add a post-deploy hook to automate the reloading of pm2 after each deploy.

    For example, in your deploy configuration:

    {
  "deploy": {
    "production": {
      "post-deploy": "yarn install && pm2 reload ecosystem.config.js --env production"
    }
  }
}

This should ensure that every time the server or pm2 process is restarted, it runs the background jobs from the symlinked current directory.

1 Like
5

Thanks! I’ll try that and report back!

6

I haven’t actually tried setting up Jobs with Baremetal yet, but in my mind I was going to do the same thing you did in your first config option: start a single worker with rw-jobs-worker and then pm2 can take it from there, watching for fails and restarting.

You might need to tweak the config a bit:

  • You’ll only want instances: 1 so there’s a single instance of that worker with that index/id. If you need a second one, add another pm2 block and give it a unique --id
  • You don’t want wait_ready —that’s something we added for the api server so that pm2 won’t start sending requests to it until the api server has reported that it’s up and running…that “I’m ready” signal isn’t sent by the jobs stuff at all, so just remove that option altogether.
  • You can probably remove exec_mode as well, since there’s only 1 instance running it won’t matter (default is to fork which should be fine)

What’s your full ecosystem config file look like? I’m in the processes of converting Spoke over to use jobs, and my proposed config was going to look like this (nginx handles the web side so I only need Redwood to serve the api side):

module.exports = {
  apps: [
    {
      name: 'api',
      cwd: 'current',
      script: 'node_modules/.bin/rw',
      args: 'serve api',
      instances: 'max',
      exec_mode: 'cluster',
      wait_ready: true,
      listen_timeout: 10000,
    },
    {
      name: 'jobs',
      cwd: 'current',
      script: 'node_modules/.bin/rw-jobs-worker',
      args: '--index=0 --id=0',
      instances: 1,
    },
  ],
}

I’m very curious if that wait_ready is the key…I think with that set to true (and the fact that we never send the “I’m ready” message means that that job worker would never restart. Is that what you’re seeing, it’s always looking at an old version of the deployed code?

1 Like
7

I’m going to play around with this, but I’ll try this configuration this weekend and see if I can manually get the job running and just restart on deploy.

Chaning the ecosysystem and doing a yarn rw deploy baremetal production --first-run on deploy didn’t attach the job to my pm2 managed processes.

image
image1042×991 50 KB

I am also running nginx so I only need the api side to run.

8

Can you try starting the services manually? I think you need do it from the root of the deployed app (the directory containing all the datestamp folders and the current symlink). Stop the current running stuff:

pm2 stop all

And then start it manually, telling it to get the settings from your latest ecosystem file:

pm2 start current/ecosystem.config.js

If THAT doesn’t start the job process then something really weird is going on…

9

Looks promising. Let me… do a few deploys.
image

Okay so after a few deploys it’s still ticking along. I had to run a pm2 save to keep it after reboot. I think this does the trick! Thank you!