Repeater.dev now available in beta! (background jobs for the Jamstack)

Hello! If you’ve been to the last couple community get-togethers on Hopin (and if you haven’t been to one yet, what are you waiting for??) you saw a demo of a secret project I’ve been working on that I’m calling Repeater. It was made to fill a hole in the Jamstack architecture, namely “how do I do something in the future?”

Use cases include:

  • Sending an email
  • Sending a reminder to someone at a specific date/time in the future
  • Periodically doing some processing on a dataset

Right now Repeater does one thing: call to a URL at a specific datetime and record the response. That’s it. But in most cases that’s all you need for the Jamstack—you’ve got function endpoints that are available on the internet, right? If you’ve deployed to Netlify then these are called Functions. You can create your own with Redwood pretty easily.

The idea is to have your function perform whatever task you need to run. That task is triggered by the URL being hit by Repeater. You can include a body and/or headers along with your request and they’ll be available to your function.

Usage

The Repeater API is GraphQL. You’ll create/edit/delete jobs using this interface. Eventually you’ll be able to create/edit/delete jobs through the UI on the website as well, but for now you can view jobs and job results within the UI.

You’ll need to create an account at https://repeater.dev and then create an Application (you can only create an Application through the UI, no GraphQL interface for that yet). When you do, you’ll get a unique token. That token will need to be included in an Authorization header to all GraphQL calls:

Authorization: Bearer c0d234fb00f4387c69865684afda3392

Now you can create a job with a GraphQL call to https://api.repeater.dev/graphql:

mutation {
  createJob(
    name: "first-job"
    endpoint: "https://mysite.com/.netlify/functions/success"
    verb: "get"
    runAt: "2020-07-24T22:00:00Z"
    body: "{\"foo\":\"bar\"}"
  ) {
    name
  }
}

From within Redwood the most secure way to create jobs would be from the server side, specifically from a Redwood service. Here’s a generic backgroundJobs service I’m using in another project (note this requires the graphql-request npm package which is a super-slim GraphQL client):

// api/src/services/contacts/backgroundJobs.js

import { GraphQLClient } from 'graphql-request'

const CREATE_OR_UPDATE_JOB = `
  mutation CreateOrUpdateJobMutation($name: String!, $body: String!, $endpoint: String!, $runAt: String!) {
    createOrUpdateJob(
      name: $name,
      body: $body,
      endpoint: $endpoint,
      runAt: $runAt,
      verb: "post"
    ) {
      name
    }
  }
`

const DELETE_JOB = `
  mutation($name: String!) {
    deleteJob(name: $name) {
      name
    }
  }
`

const graphQLClient = new GraphQLClient('https://api.repeater.dev/graphql', {
  headers: {
    authorization: `Bearer ${process.env['REPEATER_APP_TOKEN']}`,
  },
})

const jobName = (contact) => {
  return `contact-${contact.id}-reminder`
}

export const createBackgroundJob = async (contact) => {
  const variables = {
    name: jobName(contact),
    body: JSON.stringify({ contact: { id: contact.id } }),
    endpoint: `https://mysite.com/.netlify/functions/sendReminder`,
    runAt: contact.reminder.toISOString(),
  }
  await graphQLClient.request(CREATE_OR_UPDATE_JOB, variables)
}

export const deleteBackgroundJob = async ({ contact }) => {
  const variables = { name: jobName(contact) }
  await graphQLClient.request(DELETE_JOB, variables)
}

I invoke the service in my contacts service, after a contact is created or updated in the database:

// api/src/services/contacts/contacts.js

import { createBackgroundJob, deleteBackgroundJob } from 'src/services/backgroundJobs/backgroundJobs'
import { db } from 'src/lib/db'

export const createContact = async ({ input }) => {
  const contact = await db.contact.create({
    data: input,
  })

  if (contact.reminder) {
    await createBackgroundJob(contact)
  }

  return contact
}

export const deleteContact = async ({ id }) => {
  const contact = await db.contact.findOne({where: { id }})
  await deleteBackgroundJob(contact)
  await db.contact.delete({where: { id }})

  return contact
}

In this case I’m sending a reminder to contact someone, so I schedule the job to be sent at that reminder time via runAt and in the body of the job I include an object containing the ID of the contact that the owner is going to be reminded about. When the function’s URL is hit it will deserialize the body and look up the contact’s ID and then send them the email. The function itself doesn’t care about time, it assumes it will be called when the time is right—it just sends the email.

Note that I’m using the createOrUpdate endpoint, which will create the named job if it doesn’t exist, otherwise it’ll update an existing job with the same name. When a job is updated any outstanding background tasks are canceled and rescheduled based on the runAt and runEvery arguments.

You can introspect the GraphQL endpoint at https://api.repeater.dev/graphql for documentation on each endpoint. I’m working on regular HTML docs that will be available on https://repeater.dev soon! If you’re on a Mac I’ve found the Insomnia to be an excellent GUI for playing with APIs.

Documentation is available at https://docs.repeater.dev

Limitations

This probably goes without saying but this is extremely beta so please don’t use it to move money between bank accounts or manage your nuclear reactor.

  • You can create a max of 100 jobs in a 24 period
  • For recurring jobs the fastest they can recur is once per minute

Be aware that on the Free plan on Netlify, your functions will timeout after 10 seconds. Pro and Enterprise plans have longer timeouts. So whatever you have to do, do it fast! :slight_smile:

TODO

I’ve got a list of features I’m working on next, including:

  • Create/edit jobs through the repeater.dev UI
  • Configurable timeouts per job
  • Set a single job to run at multiple specific times
  • Ability to set job priority
  • Webhooks/notifications when a job completes (success or failure)
  • Integrate into Redwood, @redwoodjs/jobs perhaps?
  • Generic npm package that anyone can import and use

I’m sure one of the first requests will be for the ability to have Repeater itself run arbitrary code for you. I hear you, and we’re thinking about it, but now we just call URLs.

Found a bug? Got a feature request?

Repeater is closed source for now but you can report issues over at the repeater-issues repo: https://github.com/redwoodjs/repeater-issues/issues

7 Likes

Already signed up, and have an easy feature request!

Instead of writing our own graphql wouldn’t it make sense to just have a node module that I can import and use since all the repeater calls always have the same payload format?

I’m not going to have a chance to try it out this week, but will give it a go in a couple after my break!

1 Like

Instead of writing our own graphql wouldn’t it make sense to just have a node module that I can import and use since all the repeater calls always have the same payload format?

For sure, I’ll add that to the TODO list! The next step for me was going to make a nice interface for Redwood, something like a simple scheduleJob or scheduleRecurringJob call from your services. But having an NPM package that anyone can use would be great as well.

1 Like

I mean… just amazing :rofl:

Well done, RC!

2 Likes

holy smokes this is awesome. thanks guys for sharing

1 Like

Hey @rob, thanks for opening up the beta. Just signed up and excited to use it.
However, I noticed that GET http://api.repeater.dev/graphql returns a 404 and I’m unable to introspect the schema.

1 Like

Hey @rob, This looks really cool :grinning:

1 Like

I noticed that GET http://api.repeater.dev/graphql returns a 404 and I’m unable to introspect the schema.

Yeah apparently you need to send a regular GraphQL request to introspect, which is a POST. The one that the Insomnia GUI sends looks like:

query IntrospectionQuery {
  __schema {
    queryType { name }
    mutationType { name }
    subscriptionType { name }
    types {
      ...FullType
    }
    directives {
      name
      description
      locations
      args {
        ...InputValue
      }
    }
  }
}

fragment FullType on __Type {
  kind
  name
  description
  fields(includeDeprecated: true) {
    name
    description
    args {
      ...InputValue
    }
    type {
      ...TypeRef
    }
    isDeprecated
    deprecationReason
  }
  inputFields {
    ...InputValue
  }
  interfaces {
    ...TypeRef
  }
  enumValues(includeDeprecated: true) {
    name
    description
    isDeprecated
    deprecationReason
  }
  possibleTypes {
    ...TypeRef
  }
}

fragment InputValue on __InputValue {
  name
  description
  type { ...TypeRef }
  defaultValue
}

fragment TypeRef on __Type {
  kind
  name
  ofType {
    kind
    name
    ofType {
      kind
      name
      ofType {
        kind
        name
        ofType {
          kind
          name
          ofType {
            kind
            name
            ofType {
              kind
              name
              ofType {
                kind
                name
              }
            }
          }
        }
      }
    }
  }
}
1 Like

That worked. Thanks for the info.

1 Like

I’m working on docs now, you can see them in progress at https://docs.repeater.dev

Update First pass at docs complete! All the query/mutation types are documented as well as field lists and argument lists for each.

1 Like

Used repeater.dev for first time last night to periodically rebuild/deploy a Netlify site via a Build hook.

Worked great!

FYI - any build script you run has access to the INCOMING_HOOK_URL env on Netlify, so can check that and conditionally build differently via a webhook vs normal.

In my case, delete data, restart sequence and re-seed.

2 Likes

Hey! I have a project that creates many (more than 500, and growing) repeater jobs per day and I’m getting rate limited. Is Repeater still the best scheduler for a heavy use case like that or is there another solution that would be better suited for creating that many jobs?

FWIW it’s been simple to work with and super helpful – I’d be willing to pay a reasonable amount to increase the limit :slight_smile:

1 Like

^^ @rob you have a PowerUser™ question :rocket:

@tctrautman what are you building? Do TELL!! :laughing:

1 Like

Thanks @thedavid!

I’m building https://slp.careers/, a site to help Speech-Language Pathologists (aka SLPs or speech therapists) anonymously share how much they make and figure out if they’re being paid fairly for their combination of city, experience level, and type of employer. So far over 4,300 anonymous compensations have been shared.

I’m using Repeater, though, to seed a yet-to-be launched job board with scraped jobs, which I then categorize in ways that are meaningful to SLPs and overlooked by generalized job boards for many different types of jobs.

Hello! It sounds like you’re creating one-off background jobs vs. repeating jobs? What kind of load do you expect in the next month, 6 months, 1 year? Is there a reasonable cap to the number of jobs you’d expect to add in a day, or could it be millions at some point?

In the meantime I raised your rate limit to 1000 jobs per day, let me know if that helps!

Thank you for the increase and quick response @rob!

It sounds like you’re creating one-off background jobs vs. repeating jobs

That’s right – I have a few repeating jobs, but the vast majority of them are one-off.

What kind of load do you expect in the next month, 6 months, 1 year? Is there a reasonable cap to the number of jobs you’d expect to add in a day?

I’ll have a better estimate in the next few weeks, but right now my guess is that I’ll max-out at ~20-30k per day, and it’s likely that in 6 to 9 months I’ll start to roll-back the number of jobs I do each day, probably to ~70% of what I max out at.

I totally understand if this is a bit beyond what you had imagined supporting as a free service, and I can absolutely look into other solutions if that’s what you think makes the most sense!