Baremetal Deployment on NixOS - issues with environment variables

I am trying to do a baremetal deployment to a NixOS server. But when running the yarn rw deploy baremetal production --first-run my CI pipeline fails at the database migration step with the following error:

Deploy failed!
Error while running command `yarn rw prisma migrate deploy`:
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ prisma:warn Prisma failed to detect the libssl/openssl version to use, and may not work as expected. Defaulting to "openssl-1.1.x". │
│ Please manually install OpenSSL and try installing Prisma again.                                                                    │
│ Warning Precompiled engine files are not available for nixos.                                                                       │
│ Error: Env var PRISMA_QUERY_ENGINE_LIBRARY is provided but provided path engines/lib/libquery_engine.node can't be resolved.        │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

By adding these steps in the deploy.toml I can see that the user is correct, but the $PRISMA_QUERY_ENGINE_LIBRARY is indeed wrong.

before.install = "echo $PRISMA_QUERY_ENGINE_LIBRARY > prisma_engine.txt"
after.install = "whoami > user.txt"

:thinking: Now this is a headscretcher, since I have indeed set these variables in my configuration.nix along with the prisma-engines. When I do a normal ssh connection to my server and echo the variable it also shows a correct location to the nix store.

  environment.systemPackages = with pkgs; [
    # required for rw
    git
    yarn
    nodejs-18_x
    nodePackages.prisma
    prisma-engines
  ];


  environment.variables = with pkgs; {
    PRISMA_MIGRATION_ENGINE_BINARY = "${prisma-engines}/bin/migration-engine";
    PRISMA_QUERY_ENGINE_BINARY = "${prisma-engines}/bin/query-engine";
    PRISMA_QUERY_ENGINE_LIBRARY = "${prisma-engines}/lib/libquery_engine.node";
    PRISMA_INTROSPECTION_ENGINE_BINARY =
      "${prisma-engines}/bin/introspection-engine";
    PRISMA_FMT_BINARY = "${prisma-engines}/bin/prisma-fmt";
  };


When developing locally I use the shell.nix as follows:

{ pkgs ? import <nixpkgs> { } }:

with pkgs;

mkShell {

  nativeBuildInputs = with pkgs; [ bashInteractive ];
  buildInputs = with pkgs; [ nodejs-18_x nodePackages.prisma prisma-engines ];
  shellHook = with pkgs; ''
    export BROWSER=none
    export PRISMA_MIGRATION_ENGINE_BINARY="${prisma-engines}/bin/migration-engine"
    export PRISMA_QUERY_ENGINE_BINARY="${prisma-engines}/bin/query-engine"
    export PRISMA_QUERY_ENGINE_LIBRARY="${prisma-engines}/lib/libquery_engine.node"
    export PRISMA_INTROSPECTION_ENGINE_BINARY="${prisma-engines}/bin/introspection-engine"
    export PRISMA_FMT_BINARY="${prisma-engines}/bin/prisma-fmt"
  '';
}

But I don’t know how to deploy redwood with this…

Anyway, my feeling is that the env variable for prisma is not picked up by some weird reason. However I don’t know how to “look into” the deployment, what is being done, or how I could use the variable. The difficulty is to pass the dynamic location to the nix store - and I think this can only be done in a .nix file.

What I tried

  • I tried to “hardcode” the PRISMA_QUERY_ENGINE_LIBRARY in my .env, but that didn’t work.
  • I tried to use the shell.nix before migration with:
before.install = "nix-shell shell.nix"

… but that didn’t work either.

I’ve never heard of NixOS before, so I’m going to have trouble helping to diagnose. I know that on a standard Linux install, there’s almost always an issue with SSH using a non-login shell, which doesn’t initialize any of the ENV vars you’d expect, proper versions of node, etc. Is there something similar in NixOS?

If you look at the source for baremetal you can see the commands it’s running: https://github.com/redwoodjs/redwood/blob/main/packages/cli/src/commands/deploy/baremetal.js#L381

1 Like

Thanks so much for the link to the repo, this is already very helpful!

NixOS is a “normal” linux distribution, using it’s own package manager nix. The cool thing is, that everything is declarative in configuration files and the system is then built deterministically. Anyhow, I have a working non-login ssh connection, and when I use a normal terminal to enter, the environment variable is set correctly.

echo $PRISMA_QUERY_ENGINE_LIBRARY
/nix/store/xvjlm9vnyvpzqp39i2xgz4hhm18nzpi0-prisma-engines-4.6.0/lib/libquery_engine.node

Everything works as expected, I can cd into the just created, but failed to migrate project directory and manually execute yarn rw prisma migrate deploy without issue. It just seems to be broken in the context of the baremetal deploy pipeline.


Looking through the code you linked, I don’t see anything obvious that could cause the issue… What puzzles me is that even when I do commands like these in my deploy.toml

before.install = "PRISMA_QUERY_ENGINE_LIBRARY=/nix/store/xvjlm9vnyvpzqp39i2xgz4hhm18nzpi0-prisma-engines-4.6.0/lib/libquery_engine.node"
after.install = "echo $PRISMA_QUERY_ENGINE_LIBRARY > prisma_engine.txt"

the contents of this prisma_engine.txt are still just

cat prisma_engine.txt
engines/lib/libquery_engine.node

I have to conclude that the steps are somehow isolated from each other.

This may very well be a quirk of nixOS, but I just find it weird that i cannot manually set the env var in the deploy pipeline.

So after connecting, each of the commands should run in sequence in the same session, so any ENV vars should persist. Can you try:

export PRISMA_QUERY_ENGINE_LIBRARY=/path/to/libquery_engine.node

In a before.install (with the real path, of course)? That should make it global, available everywhere and so in theory should be set for all of the following commands…

Good idea, but still the issue remains.

I’m not sure what else to try…we’ve got plenty of people using baremetal without issue so I don’t think the core functionality is faulty. It sure sounds like something about session persistence/login shell shenanigans.

It would be a bit of work, but have you tried a different *nix distro? I’m an Ubuntu man myself. You could run one in a local VM and test the deploy…