Best/canonical ways to interact with backend services?

Hi all - @thedavid suggested I move this here from Discord.

Thus far there is excellent support for going directly to a datastore via prisma, with the business logic written in a rw service. there is also the nice netlify functions integration - though this must be lightweight as it doesn’t allow for a lot of config, monitoring/logging, or etl/cron as far as i can tell.

this doesn’t cover a lot of the backend services i want to interact with, though. i’ve checked out the 3rd party api tutorial - and clearly i can just call any api from the client like a blackbox (totally fine), but i’m mostly curious about how (if at all) you plan to extend support for server-side api integrations?

i could imagine a generator based on an OpenAPI spec (though that’s not universally adopted)? or perhaps just to reduce some of the boilerplate… even just that would be huge for a (hypothetical, of course) pythonista who is struggling to write business logic in javascript

if there’s anything on a roadmap or general thoughts here, do let me know. if you want to point me to other discussions/threads, that’d kick ass to!

@rob - David says that you’re up to “some fancy API fanciness” - would love to see what you’re thinking?

2 Likes

related post - this appears to have be codified into the “cookbook” i mentioned:

Hi @tessier0ashpool thanks for taking me up on kicking this off! There are a lot of related topics to your question, which means it might take some time and searching to touch on them all. Either way, keep ping’ing us as the questions come — you’re definitely not the only one interested in this. :grinning:

One recent conversation about the Redwood API in general is here “Grokking Redwood”. There’s a lot of momentum to add help + docs + tutorials + guides for making it easier to access and learn about the API. Services is an important concept and component of the API in general.

Speaking of Services, @dom did a great write-up about them here, “Understanding Services”.

Also, did you know there’s a generator for Services?

i’ve checked out the 3rd party api tutorial - and clearly i can just call any api from the client like a blackbox (totally fine), but i’m mostly curious about how (if at all) you plan to extend support for server-side api integrations?

This is a super intriguing question. Could be answered via code snippets (e.g. generators), Cookbooks/guides, npm packages…

Could you start with a specific example of an API integration/service? Or a list? Maybe the next step would be to share some code about what you’re trying to do (no worries, we’re all beginners with it comes to Redwood!) and see if that leads to both progress and ideas about ways to generalize and help everyone?

Hi @tessier0ashpool! Good question. That 3rd party API integration cookbook recipe does talk about making the API calls from the server side, and then exposing the results to the web side via GraphQL. Ideally anything you’re interacting with in the outside world is abstracted away in a service. And services don’t have to be exposed to the web side via GraphQL, they can be used only by other services.

I’m working on a secret project (to be announced soon!). It’s not a Redwood app, but is intended to be used by Redwood apps. It has a GraphQL API. Let’s call it Project B.

I’ve got a Redwood app which is going to consume this API. The way I’ve made that work is to create a service in the Redwood app, which itself talks GraphQL to Project B. The service (in the Redwood app) does not need to be available to the web side, it’s only used by other Redwood services, so it doesn’t have an associated SDL (which provides a GraphQL interface to the web site).

Redwood Service <---> GraphQL <---> Project B

I used graphql-request (coincidentally also from Prisma) to talk GraphQL from the service. (Redwood uses Apollo for talking GraphQL, but Apollo Client is only used on the web site and Apollo Server on the api side. By default you don’t have a GraphQL client on the API side, and Apollo Client provides all kinds of extra functionality [like caching] that I don’t need, so I went with something lighter weight.)


If you want to go one level deeper (but is veering off the original topic) keep reading!

Where it gets really interesting is that Project B, at some point in the future, calls back to a custom function in the Redwood app!

Project B <---> HTTP <---> Redwood App Netlify Function

In theory this could be a GraphQL call as well, but in this case all the important stuff is happening in the function itself and it doesn’t need to return anything to Project B once it’s done doing what it needs to do, so I thought GraphQL was overkill. Just calling the URL of the function (and providing a single parameter either in the query string or the body) is all that’s needed to give the function all the context it needs to do its job.


So the short answer is I don’t think we’ve thought too much about codifying talking to 3rd party services from the api side (other than the idea that they should be wrapped in a Redwood service)—the APIs that are available in the world are just so varied it’s hard to generalize them into a single pattern. We could include a GraphQL client library so that if the 3rd party talks GraphQL then that’s one less package you need to manually add to your project, but other than that I’m not sure what else we can do to make people’s lives easier. But if you have any ideas, please share!

1 Like

Best/canonical ways to interact with backend services?

I read a tweet today that said not to trust any advice unless it starts off with “it depends”.

So, it depends :wink:

Going to share how I think about interacting with other APIs – not best, not canonical, just how I think … and have started to shape this in my head from a RW perspective.

  1. Do I need to present the data back to the web in a cell?

For example, am I calling a CMS like Contentful or Sanity and want to render a list of the results?

Case in point, I made a Contentful space for my niece over a year ago where she can upload pictures of various :cupcake:"cupcake characters" she’s made out of some :princess: Disney Princess figurines/games – she’s 8. And she uses Siri and an iPad to take photos of them and dictate little descriptions.

I can use Contentful’s SDK to fetch the cupcakes in a RW service and the sdl so that I can use cells to render a list of cupcakes or show a single cupcake:

import { createClient } from 'contentful'

const client = createClient({
  space: process.env.CONTENTFUL_SPACE,
  accessToken: process.env.CONTENTFUL_DELIVERY_API_KEY,
})

const renderAsset = (fields) => {
  return { title: fields.title, file: fields.file }
}

const renderAssets = (assets) => {
  return assets.map((asset) => renderAsset(asset.fields))
}

const renderEntry = (entry) => {
  return {
    id: entry.sys.id,
    name: entry.fields.name,
    description: entry.fields.description,
    price: entry.fields.price,
    rating: entry.fields.rating,
    slug: entry.fields.slug,
    photos: renderAssets(entry.fields.photos),
  }
}

export const cupcakes = async () => {
  const response = await client.getEntries({
    content_type: 'cupcake',
    limit: 1000,
    order: 'fields.name',
  })

  return response.items.map((entry) => renderEntry(entry))
}

export const cupcake = async ({ id }) => {
  const entry = await client.getEntry(id)

  return renderEntry(entry)
}

and SDL

import gql from 'graphql-tag'

export const schema = gql`
  type Cupcake {
    id: String!
    name: String!
    description: String!
    price: Float
    rating: Int
    slug: String!
    photos: [ContentfulAsset]
  }
  type Query {
    cupcakes: [Cupcake!]!
    cupcake(id: String!): Cupcake!
  }
`

RW is none the wiser that my Cupcakes didn’t come from Prisma/SQL database, but came from the Contentful API call.

  1. I don’t need to render anything, the API I’m interacting with maybe posts some request or or returns some other response

Here, I’m thinking about an API like Mailgun.

In this case, I’d probably use a function vs a service – or probable a function that calls a service like MailManager.

The Mailgun API endpoints I might call are:

I wouldn’t make a SDL for this or want to interact with the API via gql, I don’t think.

I do have some ideas for some best practices around functions and this is where I really like your thinking:

i could imagine a generator based on an OpenAPI spec (though that’s not universally adopted)? or perhaps just to reduce some of the boilerplate

For me, there are things I do in functions over and over when calling a 3rd party API:

  • security – check that whoever is calling the function is allowed to call it
  • http status codes. handle 200 success. handle 500 error. 401 unauthorized 400 bad request, 403 forbidden. 204 no content, etc.
  • parse/process request
  • return result

If you think about it, it’s no so unlike cells with “loading” “error” “empty” and “success” cases.

So, to that end, could there be something imported into a function that establishes those states and handles in the response is empty -> 204 or unauthorized -> 401 or not permitted -> 403 “for you”?

Also, 7 times out of 10 if you are making a a call to another API, you might use got or as @rob mentioned graphql-request which I’ve used in services

import { GraphQLClient } from 'graphql-request'

export const request = async (
  query = {},
  domain = process.env.HASURA_DOMAIN
) => {
  const endpoint = `https://${domain}/v1/graphql`

  const graphQLClient = new GraphQLClient(endpoint, {
    headers: {
      'x-hasura-admin-secret': process.env.HASURA_KEY,
    },
  })

  try {
    return await graphQLClient.request(query)
  } catch (error) {
    console.log(error)
    return error
  }
}

to make some requests to a Hasura API I already had around.

So, could RW have pre-built some small clients to just set the endpoint and something to process the response? In the Hasura case, I just had to reference the key

import { requireAuth } from 'src/lib/auth.js'
import { request } from 'src/lib/hasuraClient'

export const stories = async () => {
  requireAuth()

  const query = `
  {
    stories {
      author
      categories
      categoryGroup
      channelEmojiIcon
      channelId
      channelName
      companyNames
      conceptEmojiIcons
      .. lots more
      url
    }
  }
 `


  const data = await request(query, process.env.HASURA_DOMAIN)

  return data['stories']
}

and now can present the data in a cell just as normal.

But, back to functions.

  • security
  • http status codes
  • parse/process request
  • return result

I’d definitely like to see either via a generator or imports something to help with the above.

  • some JWT and auth header helpers
  • some basic small clients (http / gql) that can set the endpoint, auth, response handler
  • handle response codes
  • hook into processing result/response

the APIs that are available in the world are just so varied it’s hard to generalize them into a single pattern

Have to agree there.

I’ll end with one last idea that you brought up:

i could imagine a generator based on an OpenAPI spec (though that’s not universally adopted)? or perhaps just to reduce some of the boilerplate

I Googled/discovered this just now: Translate APIs described by OpenAPI Specifications (OAS) or Swagger into GraphQL.

Repo.

No idea how/if this works, but can see it accelerating the SDL generation and once you have a SDL can generate a service.

That would be interesting. I wonder if there is something similar for JSON Schema

Thanks for reading!

2 Likes

thanks both of you. so I think the biggest things I’ve gleaned here - and totally make sense to me - is that any api that we’re pulling in whether it involves GQL or not is going to be interacted with via a service.

the case that @rob talks about (and would love to see your secret project – before it’s ready or when you’re ready to release it :slight_smile: maybe its something like a long-running job or has an async element where you’d do a callback via a netlify function… presumably you’d handle that callback with a service as well?

it sounds like there’s a bit more flexibility here than the admonition in the Custom Function cookbook that everything be done through the gql interface - a single interface. this seemed implied in the 3rd party api cookbook so i shall embrace it until rebuked.

most of my needs will be like the Mailgun example that @dthyresson presents. the examples here are SUPER helpful… particularly with regards to when to use GQL and more importantly, a Cell. this was an area of confusing for me, since the boilerplate I’m referring to reducing would be using something very akin to a Cell!

what i was thinking of here is very much what @dthyresson lists out:
generator/imports for:

  • security
  • http status codes
  • helpers to parse/process request
  • header helpers
  • some basic small clients (http / gql) that can set the endpoint, auth, response handler

really when i think about it, the draw for RedwoodJS for this pythonista is that you select and glue together some nice bits that reduce my search costs in a very chaotic js ecosystem. secondary draw - you generate a bunch of not-garbage code that i can modify. so the set of other recommended clients (like Got) go a really long way for people like me who don’t know any better. I don’t think they need to be included but maybe there is a recommended set of imports? or some educational material for things like GQL, parsing/processing requests in JS, etc… if this is a more general challenge and people can help me figure out what to write about, i’d be up for writing some docs to cover a path to learning these!

as a final thought - the SDL/Service generator using Swagger is something I’d love to see here - maybe like @rob’s secret project this is something that could be ancillary to Redwood… you can generate all kinds of SDKs with swagger / OAS in general. here’s i think the classic example: https://petstore.swagger.io/#/
and you can imagine almost a point-n-click ability to generate services boilerplate for different routes.

3 Likes

I realized after I wrote this that there is also a good case for using the MailManager service from a graphql query/mutation (and not a function).

Say you have a contact form post and you save the form details in Contact with a message, email, etc. After an insert, you might send the mail – so a service to saveContact and then that calls mailManger.sendEmail()

More often than not 8-out-of-10 I like to use functions for “external” api calls like incoming webhooks. Like an outward facig api endpoint (which is why proxying them w/ Netlify to shield the /.netlify/functions with an api redirect is also so common).

Well, maybe not gql, but definitely services (if dealing with the app’s data).

Because sometimes, the external call like an incoming webhook (via an Auth0 rule or a Zapier webhook or an incoming Slack webhook or even a Netlify Identity webhook) won’t speak gql just straight up https request/post.

But what’s important and so nice about RW is that functions could and should call services. Then they have super easy access to your data layer, auth layer, etc.

Yup. Agreed. As I was trying to argue: if a graphql call for a cell has loading, empty, success, error … and graphql is just a function, then why might not functions have some equivalent event handling?

Because sometimes, the external call like an incoming webhook (via an Auth0 rule or a Zapier webhook or a an incoming Slack webhook or even a Netlify Identity webhook) wont speak gql just straight up https request/post.

yes to this. the workflow of wrapping with gql doesn’t feel good. driving everything through a service, by contrast, feels very ergonomic.

also - going back through the 3rd party api cookbook, i noticed that i naturally didn’t put services into the correct folder structure - i think because of how the graphql is setup. it’s kind of silly but if i can outsource this to a generator that does it right every time, it is a great way to avoid n00b mistakes.

note the nesting difference:

// api/src/graphql/weather.sdl.js

vs.

// api/src/services/weather/weather.js

i’m tempted to try to build a little tutorial to demonstrate this - perhaps with a Slack webhook. any bright ideas about the use case? i’ll see if my noggin will shake itself out of slumber this afternoon…

@tessier0ashpool FYI there are generators for both sdl and services – and I believe generating sdl will create both the sdl from the Prisma schema as well as the service.

The generator can overwrite, too, (via the -f flag) so I’ve been keeping my generated Model-based services alone (like services/users.js) so I can re-generate them from any schema changes I make to the User model and placing custom user services (some findUserByEmail()) in something like services/userManager.js.

Nice!

Something fairly simple and that stays w/in the Netlify ecosystem – so no need to have Slack or Zapier as many people will use Netlify w/ RW – would be to use their outgoing webhooks to post the deploy status:

You could then save the deploy status to a Deploy model.

More about Netlify webhooks here.

I’m doing something similar in an app I haven’t had a chance to work on in a bit that intends to make a dashboard of deploy build/fails/succeeds over time, but will get the deploy info success/fail from a build plugin.

@tessier0ashpool Just checking in to say Hi and see how things are going?