Testing in Redwood

:rotating_light: Long post alert! :rotating_light:

Hi all! Yesterday during the community call there was a brief discussion on testing, and from what I understood; the config and setup for jest has been ping-ponged around a bit, but there were some troubles setting up. I’d love to help out setting up. It would be great if someone (I believe @peterp and @thedavid worked on this?) could clarify what the difficulties were and how we could jump in!

Other than getting stuff to run I have some more thoughts on making testing easier/more convenient for Redwood users!

Jest

Web

Low Hanging Fruit

  • Stub SVG/Image imports.

  • Expose a custom render function that includes RedwoodProvider and Router stuff.

Mocking

I believe it was @rob who mentioned that in rails it’s common to not mock anything except the database. I like this thought! I like to mock as little as possible to keep tests as close to reality as possible.

At the moment I noticed scaffolds generate tests in the following shape:

import { render, cleanup, screen } from '@testing-library/react'

import { Loading, Empty, Failure, Success } from './PostsCell'

describe('PostsCell', () => {
  afterEach(() => {
    cleanup()
  })

  it('Loading renders successfully', () => {
    render(<Loading />)
    // Use screen.debug() to see output.
    expect(screen.queryByText('Loading...')).toBeInTheDocument()
  })

  it('Empty renders successfully', () => {
    render(<Empty />)
    expect(screen.queryByText('Empty')).toBeInTheDocument()
  })

  it('Failure renders successfully', () => {
    render(<Failure error={{ message: 'Oh no!' }} />)
    expect(screen.queryByText('Error: Oh no!')).toBeInTheDocument()
  })

  it('Success renders successfully', () => {
    render(
      <Success posts={[{ title: 'Post Title' }]} />
    )
    expect(
      screen.queryByText('Post 1 ')
    ).toBeInTheDocument()
  })
})

This tests all the units of the Cell, but it doesn’t hit the QUERY part! You could turn it into something like this:

// ========== Mocking the API ==========
import { render } from '@redwood/testing'

import Posts from './PostsCell'

describe('PostsCell', () => {
  it('Loading renders successfully', () => {
    const { getByText } = render(<Posts />)

    // It should immediately start to load
    expect(getByText('Loading...')).toBeInTheDocument()
  })

  it('Empty renders successfully', async () => {
    const mocks = [
      {
        request: {
          query: QUERY,
          variables: {},
        },
        result: {
          data: { posts: [] },
        },
      },
    ]

    const { findByText } = render(<Posts />, { mocks })

    // After resolving without posts, "Empty" will be displayed
    expect(await findByText('Empty')).toBeInTheDocument()
  })

  it('Failure renders successfully', async () => {
    const mocks = [
      {
        request: {
          query: QUERY,
          variables: {},
        },
        error: new Error('Whoops'),
      },
    ]

    const { findByText } = render(<Posts />, { mocks })

    // After resolving with an error, the error state will be displayed
    expect(await findByText('Error: Oh no!')).toBeInTheDocument()
  })

  it('Success renders successfully', async () => {
    const mocks = [
      {
        request: {
          query: QUERY,
          variables: {},
        },
        result: {
          data: {
            posts: [
              {
                title: 'Post 1',
              },
            ],
          },
        },
      },
    ]

    const { findByText } = render(<Posts />, { mocks })

    // After retrieving the posts, display them!
    expect(await findByText('Post 1')).toBeInTheDocument()
  })
})

This is a mocking method for ApolloClient using MockProvider (abstracted away): https://www.apollographql.com/docs/react/development-testing/testing/

It mocks any QUERY's with the exact variables passed and returns the defined data.

This is pretty neat, but what if the mocked result doesn’t match the API by accident? Or my API changes. I’d need to update all the mocks of the domain that changed. MirageJS has a neat way of mocking API’s that works in all test frameworks. It’s kind of like what @mojombo mentioned in yesterday’s call about using factories to create stuff in your (fake) database.

Mirage JS is an API mocking library that lets you build, test and share a complete working JavaScript application without having to rely on any backend services.

I love using this tool in a corporate setting where frontend and backend development is completely split. By using mirage we can mimick the backend without relying on another team.

Buuuuut… in this setting we don’t rely on another team. We have our own backend! Instead of rewriting our entire backend in something like Mirage, why can’t we just use our actual backend? You could do something like this:

// ========== No mocking but the database. ==========
import { render, makeServer } from '@redwood/testing'

import Posts from './PostsCell'

describe('PostsCell', () => {
  let server

  beforeEach(() => {
    server = makeServer()
  })

  it('Loading renders successfully', () => {
    const { getByText } = render(<Posts />)

    // It should immediately start to load
    expect(getByText('Loading...')).toBeInTheDocument()
  })

  it('Empty renders successfully', async () => {
    const { findByText } = render(<Posts />)
    expect(await findByText('Empty')).toBeInTheDocument()
  })

  it('Failure renders successfully', () => {
    const mocks = [
      {
        request: {
          query: QUERY,
          variables: {},
        },
        error: new Error('Whoops'),
      },
    ]

    const { getByText } = render(<Posts />, { mocks })

    expect(getByText('Error: Oh no!')).toBeInTheDocument()
  })

  it('Success renders successfully', async () => {
    server.create('post')

    const { getByText, findByText } = render(<Posts />)

    // The loading state test could be combined into this one!
    expect(getByText('Loading...')).toBeInTheDocument()

    // After a while, posts will be loaded.
    expect(await findByText('Post 1')).toBeInTheDocument()
  })
})

That would be cool! I’m not sure if it’s possible, maybe this is even too much, but I think it maybe, possibly could be done by intercepting requests using Pretender (This is also what Mirage uses), somehow forwarding those requests to an Apollo Test Client, no HTTP server required, should be able to run in Jest since it’s NodeJS, then add stuff to the database through Redwood services (or a “factory” layer on top of services?) or maybe even through the Prisma Client (or a “factory” layer on top of the Prisma Client?).

A big tradeoff is of course speed. But I’d rather have good tests than fast tests (there is a limit to my patience of course : - )). One thing that could make this easier is In-Memory SQLite databases in Prisma

Another thing that makes both example 2 and 3 difficult at the moment is the rendering of Cells. @peterp @mojombo do you think it’s possible to move the processing of Cells from a Webpack Loader to Babel? This would make it usable in Jest as well!

Api

I’m not very experienced with backend testing, but I guess in a similar fashion it would be great to

  1. Insert into DB.
  2. Check response of queries are correct.

And

  1. Call mutations.
  2. Check database contains what you expect it to contain.

Cypress

In Cypress you could do very similar things, I’ve already created a little experiment for that! The difference is that there’s no weird setup required; you just run your full app front to back to database.

One thing to consider here (and in Jest tests as well) would be mocking Auth. If you’re using Auth0 there’s some tutorials out there telling you to create a separate M2M tenant to programmatically obtain tokens for using in your e2e tests. I don’t really like that approach because you’re still hitting the 3rd party endpoint from your automated test regularly, which could generate troubles with rate limits and all that. Another thing I like to do is create a new fake user for every test (as opposed to having a file with precreated users), which would fill up your tenant DB pretty quickly, tasking you with wiping it somehow every so often. I’m not very experienced with this however, so please correct me if I’m wrong!

Another thing I was struggling with is how to run these tests in your CI/CD pipeline. With the new Netlify beta it’s now possible to run Cypress tests post-build (awesome). How can I tell my already deployed app to use a different database (wouldn’t want to run automated tests on my production database I guess). Perhaps intercepting requests with Pretender and forwarding them to a node process running in the post-build job? But wouldn’t that be mocking? :thinking:

Conclusion

Most importantly I’d like to help creating an initial setup so people can decently run some tests!

Anyway look at the time, I g2g! If you read my entire post; thanks! Please let me know what you think. (Also if you didn’t read the entire post!)

Cheers!

3 Likes

Thanks for starting the subject @Robert :slight_smile:

I’m not an expert in testing so I’m afraid I won’t be able to offer solutions, but here are remarks/ideas that came up reading your post:

Rails

First, Rails’ idea is actually not to mock the DB ^^
Tests are run against a test DB that’s created and seeded for the test run, and filled with fixtures data. That’s what Rob and Tom described: having to fill in so many different fixtures files (one YAML file for posts, one for users, one for…) that the thing quickly becomes unmanageable.
But in the end that does not really change the reasoning of your post I think.

Fixtures/mocks maintenance

This is pretty neat, but what if the mocked result doesn’t match the API by accident? Or my API changes. I’d need to update all the mocks of the domain that changed.

We definitely don’t want that!
Whatever the solution we go with though, there will be some maintenance somewhere if a tested part of the API changes. But if it’s exclusively the tests themselves, that’d be best.

Boilerplate

Seeing this:

  beforeEach(() => {
    server = makeServer()
  })

And this:

  afterEach(() => {
    cleanup()
  })

… makes me wanna stress how much I’d love not having to think about this kind of boilerplate code anytime I write tests :slight_smile:
I think we should take the same approach in the testing framework than what’s done on the dev side: provide the most common, sensible way to do things out of the box, and let people override/change it for their specific needs.

In-memory SQLite & staying close to reality

One thing that could make this easier is In-Memory SQLite databases in Prisma

Slow tests in large projects are quite a pain, so I’m all for fast tests.
However I don’t think we can override the dev’s chosen database by default: SQLite for instance, doesn’t support enums. Postgres ans MySQL do. So tests might break just for that simple reason, and for hundreds of even more subtle differences.
So if we want to stay close to reality and not mock the DB, we need to keep whatever DB the developer has chosen.

@David / Rob: related but off-topic

btw that’s one thing I’m not a big fan of in the tutorial @thedavid @rob, “encouraging” the use of SQLite locally and Postgres in prod only. It makes things much simpler for a tutorial, but might be worth a warning?

Rails’ approach to this is to provide a DB config for each environment (dev, test, prod). They’re all configured with the same DB kind by default, but you’re free to change anything if you want.

API testing

That’s exactly how I’d go about it :slight_smile:

Test scenario factories

I absolutely loved that idea @mojombo, it was very inspiring.
Since we control the entire stack with Redwood, I think we can afford to be pretty creative.

Thinking a bit too big maybe, but we could imagine something à la GraphQL.
Say I want to test that an admin can edit all posts, and that a regular user can only edit theirs.

  • I write a “test scenario query” for my test suite:

    User(count: 2) {           // I want 2 users
      isAdmin: [true, false]   // one will have `isAdmin === true`, the other `isAdmin === false`
      posts(count: 1)          // each will have their `posts` relationship populated with a single `Post`
    }
    
  • At the setup phase of the test suite, the DB is cleaned up and the test scenario is executed.
    Because we have access to the Prisma schema, we can:

    • Find the definition of the User model
    • Generate a realistic value for all scalar fields using something like Faker.js
    • Find that we have to include a Post payload as well thanks to the posts relation in the schema
    • Call our service’s creation function with the generated data to save our data to the test DB
      Edit: I think that might be the trickiest part? Can we actually solely rely on the existing services? If not, maybe we could push the GraphQL analogy and have some sort of “resolvers”, which would be our test fixtures factories that generate parts of the data from the scenario?
  • My tests are provided with a fixtures object that I can query based on the attributes that were specifically set in the scenario:

    let adminUser = fixtures.users.having({ isAdmin: true })
    let regularUser = fixtures.users.having({ isAdmin: false })
    // now check that `adminUser` can edit `regularUser.posts[0]`
    

:point_up: please tell me what’s wrong with this idea, 'cause I want it now! :grimacing:

@olance Thanks for your input! I’m certainly no expert either, but I suppose if we bundle all our thoughts we can come up with something great!

But in the end that does not really change the reasoning of your post I think.

Test DB, Mock DB, Whatever you might call it I think we want the same thing! But test DB sounds more like what it is indeed.

Whatever the solution we go with though, there will be some maintenance somewhere if a tested part of the API changes. But if it’s exclusively the tests themselves, that’d be best.

You are right! What I meant to express is if I would like to only change my test if the behavior of the test subject changed, if that makes sense.

Boilerplate

I completely agree. Stuff like cleanup is also recommended by @testing-library to put in your global jest config.

I’m not entirely sure about generalizing makeServer, but I guess that comes down to the exact implementation if it becomes a thing!

So if we want to stay close to reality and not mock the DB, we need to keep whatever DB the developer has chosen.

I’m not so hot on databases, but what you’re saying about differences in DB’s definitely makes sense. I could see using an actual DB for integration tests complicating the setup or slowing tests down severely, but I’m not sure. Only one way to find out I guess!

Test Scenario Factories

Did you take a look at MirageJS’s Factories? I think it shares some similarities with what you describe here and I think it works pretty well!

1 Like

I just did, but I don’t think it really matches what I have in mind: based on the few minutes reading/skimming through the docs, their API feels very imperative and seems to expect that you’ll define fields values that you need for your test.
Their factories kinda look like Rails’ YAML fixtures on steroids, and I’m afraid they’d suffer from the same maintenance issues on the long run?

The approach I’ve (tried to) described is more on the declarative side: instead of saying what movie you need, you just say that you need a movie. The data factories are part of the framework and generate the proper objects based on the schema files and a faker lib.

Ideally the “save to the db” part would also be fully automated through the API but we’d have to check to what extent that would work. That’s where I think we might need “factories”, which would be more like “data persisters” actually.

1 Like

Sure! So we got blocked when we wanted to start test in the web side. Our goal is to make yarn rw test work when you create a new app, and when you add generated components.

Web

The major blocker was when we wanted to test pages that used the router.link() in a page. The way that the “Router” determines which pages are available is done at runtime based on the children that’s passed into it:

So whenever you’re testing a page that has a route.nameOfRoute it would blow up because that route doesn’t exist. Maybe there’s a way to map the named routes outside of the instantiation of the router, or some magical way to mock route.<nameOfRoute> automatically. I think this would be priority # 1.

Maybe once we have a way to share typescript definitions from the api side with the web side this might be easier? Love where your mind is here!

That would be amazing! I think this would be priority #2.

Api

I think the majority of the focus should be on testing the services, we can write a utility that throws an exception during development and testing if you don’t correctly match a resolver to a service.

Thanks again for this Robert!

1 Like

Ah, that’s a great point that I hadn’t really kept in the front of my mind.

Prisma is a query engine with the model definition as a graph of the database - I wonder if there’s a way that they could store a representation of data-in and data-out that isn’t a real database. This is wishful thinking considering that they mocked an in memory sqlite database, but there may be some relief in the future from Prisma and Nexus.

1 Like

Thanks for clearing that up @peterp

To reach the goal of making yarn rw test work out of the box I think we’ve come to 3 steps:

  1. Get Jest up and running:
    a. Setup Jest Configs.
    b. Make router.<nameOfRoute> work in tests.

  2. Convert Cell Webpack Loader to Babel Plugin.

  3. Continue discussion on potential testing utilities like factories.

Shall I create GitHub issues for the first 2 “steps”? I’m happy to get started on the first one. The webpack to babel conversion can be picked up in parallel I think if someone is interested!

1 Like

Yes please! We already have a bunch of jest related config files over here: https://github.com/redwoodjs/redwood/tree/master/packages/core/config

I don’t know what’s missing, or how it can be improved for the end user. I guess we don’t have a way for the user to define their own “setup” steps, but maybe that’s when you walk off the golden path, since I don’t think jest supports merging or extending configs.

1 Like

On it!

I don’t think Jest has an extend method, but since a Jest config is .js, I guess you could require('@redwoodjs/<path-to-jest-config>) and then merge it? I’ll check online what other people did.

I’ll have a look at your configs and go from there!

3 Likes

All kinds of :rocket: about this. I’ll dig in deeper tomorrow.

@Robert For context and reference, here’s the original tracking issue including the aspirational vision for testing in a Redwood App: https://github.com/redwoodjs/redwood/issues/181

And this is where we last left off about a month and a half ago: https://github.com/redwoodjs/redwood/issues/265

All is open to revision + improvement, of course, based on your recommendations here. Mainly want to help connect dots with current needs and work-done-to-date.

Thanks again!

2 Likes

Thanks for the links @thedavid, I had found the first but not the second link. Helpful to know what you already went through! :metal:

Oh my, this is my new favorite forum conversation!! I’ll do my best to keep up. But in general, yes please!! :star_struck:

Here’s the list of priorities I’d like to suggest:

  1. Get Jest up and running: [Issue#502 :tada:]
    a. Setup Jest Configs.
    b. Make router.<nameOfRoute> work in tests.
  2. Convert Cell Webpack Loader to Babel Plugin. [Issue#503 :tada:]
  3. Generator Test Templates: improve existing and make sure all pass tests out of the box.
    Bonus: Check to make sure all generated templates pass linter tests
  4. Start shipping Create Redwood Apps with built-in GitHub Action CI. See PR#25
  5. Continue discussion on potential testing utilities like factories.

RWJS Testing

It might be helpful to separate this conversation into two categories:

  1. Built-In Testing: Related to testing, what should RWJS do out of the box? E.g. at time of installation, when running generators, etc.
  2. Extended Capabilities: what additional config+capabilities should make it easy for developers to extend and enhance testing their Apps?

Built-In

Aspirationally and directionally, we want developers to be thinking about testing from the moment they are up and running with a Redwood App. (I attempted to address this in the “Aspirational” section of this Issue, which is very much open to feedback.) We want to model some best practices and provider directional guidance, create momentum through boilerplate and CI feedback, and most of all make it easy for them to take next steps.

So some things we’ve been doing so far:

  • if you generate code, you should also generate test code
    question: should we also include the CLI opinion to only generate test files if needed for a specific type? I haven’t thought about this before
  • this includes mostly components (web) and services (API). Rob and I never could figure out how to test SDL generated file

Extended Capabilities

It seems to me a lot of the DB conversation so far would fall into this category. It’s not something RW will do for you, but it is pre-configured for you to set up if you follow an outlined path.

I think other types of testing capability might fall in this category, too: Cypress, full EtoE, etc.


Helpful… thoughts/comments/suggestions?

1 Like

Yeah, this is a good point that actually been coming up recently. We’re not going to change it in the tutorial, but we could add a callout as well as a “Where to go from here:” section at the end, which might recommend setting up Postgres locally as well as trying a Cookbook doc.

[Edit: Rob does mention setting up postgres for local deployment with a link to the doc. And I just created an Issue about improving the “Wrapping Up” final section of the Tutorial with some specific next steps.]

2 Likes

LGTM!

I have something in the works for 1 a & b as well as 2. I think I’ll open a draft PR for them today or tomorrow to get some feedback!

While working on the Cell plugin I realized that that part also immediately requires a solution for handling backend calls, since all Cells involve backend calls. :smile: I’m thinking the easiest solution for now would be to use the Apollo MockProvider and let users mock their own backend calls. While we discuss what could be a cooler solution!

Shall we open a GH issue for #3 as well? We’ll definitely need that! Maybe an Umbrella issue for all things testing? Could be good for overview? :man_shrugging:

  • How would you decide which types don’t need tests?
  • I was thinking you can do this using createTestClient from Apollo, but @peterp mentioned writing a utility that checks if you correctly matched your services to your SDL file, which also seems good to me, and with less overhead!

Thanks for the input! Very excited over this topic as well! :metal:

For Router, I came up with this approach. It’s a quick fix to get at least some render test to work. Would love to hear your thoughts:

2 Likes

Awesome stuff in this thread! I thought I’d add a quick overview about how we handle testing in our apps for comparison purposes (many of them are an identical stack to Redwood, just pieced together manually).

Here’s how we divide things out:

Frontend

  • Unit Tests via Jest
  • Integration Tests via React Testing Library (single component and multi-component interaction)
  • E2E Tests via Cypress

When we generate a new util function, we automatically generate a Jest unit test for it. Same thing for component level tests, just a RTL test instead of Jest.

For Integration Tests that have GraphQL queries/mutations (Cells), we leverage some helpers that wrap the Apollo mocking. Here’s an example of how that works:

const mocks: MockedResponse[] = [
  {
    request: {
      query: QUERY,
        variables: { userId: user.id },
       },
       result: {
         data: {
           oauthAccounts: [oauthAccount],
          },
        },
      },
  },
];

render(
  <WithMockedTestState currentUser={mockedUser} apolloMocks={mocks}>
    <MyComponent />
  </WithMockedTestState>
);

Admittedly, WithMockedTestState isn’t perfect yet, but it feels like a good general direction. Especially the mocking of currentUser in the helper, since that’s something that needs to happen frequently.

E2E tests are typically only for critical “happy path” features. We start the server in dev mode, spin up an in-memory Postgres db (on CI, it’s a local test db when running locally), and run tests against that.

API

  • Unit Tests via Jest
  • Integration (request) Tests via Jest and Supertest

Unit tests on the API are pretty straightforward (utils, etc). We do not test GraphQL resolvers with unit tests. Instead favoring request tests.

Request tests spin up a test db that is truncated between each test run. By doing this, we can test the GraphQL API from request -> database -> response and properly check for errors, permissions, and edge cases.

Factories
The factory pattern has worked well. We use Chance to generate fake data in Factories. Factories are used to setup state correctly in request tests, but also on the Frontend (for example to build an object that populates a component test)


Hope that perspective is helpful! I’m happy to expand on anything here or over Zoom sometime and see how I can help out with implementation. I’ll also wrap my head around the linked issues you posted when I get some cycles in the next few days.

Because Redwood controls the full stack, I think it is uniquely positioned to add this type of full stack testing that includes a test database without making devs jump through setup hoops to make it happen, or make too many incorrect assumptions.

5 Likes

@cball Thanks for that! It’s good to get a real-world perspective on all of this!

I’ve created a PR that implements Apollo’s MockProvider as an inbetween step for something more advanced like factories: https://github.com/redwoodjs/redwood/pull/521

If you could take a look and maybe review if this matches what you have in mind that would be great!

@dnprock I like your idea for rendering the Routes before running the test itself. I had a similar idea which is also in the PR I linked! In my case I do a bit of mocking to prevent loading all the pages if you have many.

2 Likes

@Robert sounds great. I’ll take a look!

@Robert I added some feedback to the linked PR. Let me know what you think!

1 Like

This pattern is great. Frontend approach is similar to what we’re doing at Airbnb (our API side is not nearly as straightforward, plus Node is in very limited use).