Can't overwrite the env values from docker run

I have a dockerfile where I have web build and api build and I need a web build arg which needs to be overwritten later at the runtime. When I do as below then it does override as I see new value passed at the runtime in the printenv but when the url is called it still shows the old build value. Here is the Dockerfile:

# base
# ------------------------------------------------
FROM node:18-bookworm-slim as base
USER node
WORKDIR /home/node/app
RUN  yarn install


# api build
# ------------------------------------------------
FROM base as api_build
RUN yarn prisma generate

# web build
# ------------------------------------------------
FROM base as web_build

# Add web environment variables here as args
ARG ENV1=val1
RUN yarn  build web --no-prerender

# serve api
# ------------------------------------------------
FROM node:18-bookworm-slim as serve

#Add runtime envs here
ENV ENV1=${ENV1}
ENV NODE_ENV=production
USER node
WORKDIR /home/node/app
RUN yarn install
RUN yarn add api-server web-server
EXPOSE 8910
CMD ["sh", "-c", "node_modules/.bin/rw-server api & node_modules/.bin/rw-server web"]

I run following command to run the image after bulding
docker run -e ENV1=“val2” -dt -p 8910:8910 imagename

WHen I do docker exec and check the printenv then I can see the updated value but when I run the webapp and the ENV1 value is the redirect url the it uses the val1 from the build argument rather then the val2 from the environment variable
The redwood.toml has following code as well

[web]
  title = "EFileEasy"
  port = 8910
  includeEnvironmentVariables = ['ENV1']

Also I was nota able to run the docker when I had this in the CMD
CMD [ “node_modules/.bin/rw-server”, “–load-env-files”, “&&”, “node_modules/.bin/rw-web-server” ]
as I was getting error that --load-env-file is not valid arugument so not sure if that is causing some error to load the env variables as well.

Hey @dakhadka :wave:

In my head, I would generally expect all the web environment variables you need to be available at build time and then their values inlined into the built web dist. Our docs on includeEnvironmentVariables here mention that these variables are available at both dev and build - notably not serve though.

Are you trying to do something that this behaviour wouldn’t support?

I am trying to overwrite the values of environment variables set in the redwood.toml.
This is a SECURE_AUTH_DOMAIN (IAM solution from 3rd party) and I need it to be changed dynamically when I run the docker. In my dockerfile I have added same variable as below:

# web build
# ------------------------------------------------
FROM base as web_build

# Add redwood.toml environment variables here as args
ARG SECURE_AUTH_DOMAIN=testttt

and usually if I want to overwrite it I usually declare them in the serve part of Dockerfile where there is an comment which says add runtime envs here. So I added ENV $SECURE_AUTH_DOMAIN=${SECURE_AUTH_DOMAIN}.
and when I run the docker run I pass the new value for this variable as I want the secure auth domain to be different for dev and test. I am trying to build one dockerfile with default valule for SECURE_AUTH_DOMAIN and then overwrite it at a runtime so that when I run the docker container in the dev it would have one value and when I run it in the test it will have a different value.
The problem is that when I check the docker printenv it shows that the new value is updated but at the functionaly where it is always using the same value that was baked while creating the docker image i.e the value that was passed as ARG in the web build stage

Here is my Dockerfile and I cannot overwrite the value of SECURE_AUTH_REDIRECT_URI with following docker run command :
docker run -e SECURE_AUTH_REDIRECT_URI =“val2” -dt -p 8910:8910 imagename

# base

# ------------------------------------------------

FROM proget.ascensus.com/dockerio/node:18 as base

WORKDIR /home/node/app

COPY --chown=node:node .yarn/releases .yarn/releases

COPY --chown=node:node .yarnrc.yml .

COPY --chown=node:node package.json .

RUN yarn install

# api build

# ------------------------------------------------

FROM base as api_build

# If your api side build relies on build-time environment variables,

# specify them here as ARGs. (But don't put secrets in your Dockerfile!)

# ARG MY_BUILD_TIME_ENV_VAR

ARG DATABASE_URL=""

RUN yarn redwood build api

RUN yarn redwood prisma migrate deploy

RUN yarn redwood prisma generate

# web build

# ------------------------------------------------

FROM base as web_build

# Add redwood.toml environment variables here as args

ARG SECURE_AUTH_DOMAIN=testt

# ARG MY_BUILD_TIME_ENV_VAR

#ARG NODE_ENV=dev


RUN npx update-browserslist-db@latest

RUN yarn redwood build web --no-prerender

# serve api

# ------------------------------------------------

FROM proget.ascensus.com/dockerio/node:18 as serve

#FROM node:18-bookworm-slim as serve

ARG DATABASE_URL=""

#Add runtime envs here

#ENV DATABASE_URL=${DATABASE_URL}
ENV SECURE_AUTH_DOMAIN=${SECURE_AUTH_DOMAIN}
ENV NODE_ENV=production

USER node

WORKDIR /home/node/app

RUN --mount=type=cache,target=/home/node/app/.yarn/berry/cache,uid=1000 \

--mount=type=cache,target=/home/node/app/.cache,uid=1000 \

CI=1 yarn workspaces focus api --production

#RUN yarn install

RUN yarn add @redwoodjs/api-server @redwoodjs/web-server

EXPOSE 8910

#CMD [ "node_modules/.bin/rw-server", "--load-env-files", "&&", "node_modules/.bin/rw-web-server" ]

CMD ["sh", "-c", "node_modules/.bin/rw-server api & node_modules/.bin/rw-server web"]

All I want is to have a different value that I can pass at runtime for SECURE_AUTH_DOMAIN but it is always giving the value testt as passed as an ARG in the Dockerfile. Eventhough the .env file or printenv shows different value for SECURE_AUTH_DOMAIN in the docker container cli the value does not seems to be changed.

I’ll ask the team for their input here too but my understanding is that once you’ve built the web side the environment variable at build time is in that code and it isn’t changed at runtime. You’d have to build with the value you want or the value that can change comes from the API side. Again I might just be misunderstanding so I’ll let another one of the team come in here too.

1 Like

thank you very much , I will be looking forward for it as we are halted at this point. If we have to build various images for various environment variable then we won’t be able to use the same image to deploy in dev and test environment as both have different callback url’s and other environment variables that would be different at build time and different at runtime.

any updates on this one?

So in Redwood, when you build the web side, any references to ENV vars are literally replaced with strings, so the resulting code just contains the value of the ENV var, with no indication/memory of the original ENV var name:

// source code
const secret = "${process.env.SECRET}"

// compiled code
const secret = "g0anetasdf9hagoasd8fhaoer"

So there’s no way to know where SECRET was originally used at runtime so that it could be swapped out.

One way to work around this would be to just run yarn rw build web again right before you start serving it: that way the string is swapped into its original position, but with the new value.

If that doesn’t work for some reason, you could try something a little more complicated:

Set yourself a comment, so you have something to target, and then have a custom script that swaps it out on the fly:

const secret = "${process.env.SECRET}" // SECRET

After the build runs, that becomes:

const secret = "g0anetasdf9hagoasd8fhaoer" // SECRET

Create a simple script that contains the code to look for that comment (maybe with a nice regex) and re-write the file before the server starts:

yarn rw g script swapSecret

That creates scripts/swapSecret.js. Open that file, find/replace the text before // SECRET. And then in your last CMD before the server starts, execute that script first:

CMD ["sh", "-c", "yarn rw exec swapSecret && node_modules/.bin/rw-server api & node_modules/.bin/rw-server web"]

Of course that means your NEW var value is now just sitting in plain text in the file, but it’s no different than at build time when the previous value was sitting there in plain text.

I tried adding yarn rw build web in various spots as in code below in the serve api but all the time build failed so was not even able to check. Please correct if I am adding that in the wrong place. Also I am using yarn rw build web --no-prerender so not sure if it will make any difference:

# web build
# ------------------------------------------------
FROM base as web_build

# Add redwood.toml environment variables here as args
#
ARG SECURE_AUTH_DOMAIN="testttttt"
ARG SECURE_AUTH_CLIENT_ID="somevalllllll"
ARG SECURE_AUTH_REDIRECT_URI="http://0.0.0.0:8910/auth/callback"
ARG APP_URL="http://0.0.0.0:8910"
# ARG MY_BUILD_TIME_ENV_VAR
#ARG NODE_ENV=dev
COPY --chown=node:node web web
COPY --chown=node:node scripts scripts
COPY --chown=node:node efast efast
RUN npx update-browserslist-db@latest
RUN yarn redwood build web --no-prerender

# serve api
# ------------------------------------------------
FROM proget.ascensus.com/dockerio/node:18-slim as serve
#FROM node:18-bookworm-slim as serve
ARG DATABASE_URL=""

#Add runtime envs here

ENV SECURE_AUTH_DOMAIN="someval"
ENV SECURE_AUTH_CLIENT_ID="someval"
ENV SECURE_AUTH_REDIRECT_URI="http://0.0.0.0:8910/auth/callback"
ENV APP_URL="http://0.0.0.0:8910"

ENV NODE_ENV=production

USER node
WORKDIR /home/node/app

COPY --chown=node:node .yarn/releases .yarn/releases
COPY --chown=node:node .yarnrc.yml .
COPY --chown=node:node api/package.json .
COPY --chown=node:node yarn.lock .

RUN --mount=type=cache,target=/home/node/app/.yarn/berry/cache,uid=1000 \
    --mount=type=cache,target=/home/node/app/.cache,uid=1000 \
    CI=1 yarn workspaces focus api --production
RUN yarn install
#RUN yarn redwood build web --no-prerender
COPY --chown=node:node redwood.toml .
COPY --chown=node:node graphql.config.js .
COPY --chown=node:node .env.defaults .env.defaults

COPY --chown=node:node --from=api_build /home/node/app/api/dist /home/node/app/api/dist
COPY --chown=node:node --from=api_build /home/node/app/api/db /home/node/app/api/db
COPY --chown=node:node --from=api_build /home/node/app/node_modules/.prisma /home/node/app/node_modules/.prisma
COPY --chown=node:node --from=api_build /home/node/app/efast /home/node/app/efast
COPY --chown=node:node --from=web_build /home/node/app/web /home/node/app/web

COPY --chown=node:node --from=web_build /home/node/app/web/dist /home/node/app/web/dist


#RUN yarn add @redwoodjs/api-server @redwoodjs/web-server
RUN yarn add @redwoodjs/api-server@6.6.2 @redwoodjs/web-server@6.6.2

EXPOSE 8910

CMD [ "node_modules/.bin/rw-server", "--load-env-files", ".env", "&&", "node_modules/.bin/rw-web-server",  "--load-env-files", ".env" ]


Hmm, what were the errors during build? If you built once, you should be able to build again as many times as you want as long as nothing changed in src

so if I add RUN Yarn redwood build web --no-prerender in serve api part either before EXPOSE or after RUN --mount and before copy --chown=node:node redwood.toml following is the error:

[serve 11/21] RUN yarn redwood build web --no-prerender:
1.180 node:internal/modules/cjs/loader:1137
1.180   throw err;
1.180   ^
1.180
1.180 Error: Cannot find module '@redwoodjs/cli/package.json'
1.180 Require stack:
1.180 - /home/node/app/node_modules/@redwoodjs/api/dist/bins/redwood.js
1.180     at Module._resolveFilename (node:internal/modules/cjs/loader:1134:15)
1.180     at Function.resolve (node:internal/modules/helpers:188:19)
1.180     at Object.<anonymous> (/home/node/app/node_modules/@redwoodjs/api/dist/bins/redwood.js:5:59)
1.180     at Module._compile (node:internal/modules/cjs/loader:1356:14)
1.180     at Module._extensions..js (node:internal/modules/cjs/loader:1414:10)
1.180     at Module.load (node:internal/modules/cjs/loader:1197:32)
1.180     at Module._load (node:internal/modules/cjs/loader:1013:12)
1.180     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:128:12)
1.180     at node:internal/main/run_main_module:28:49 {
1.180   code: 'MODULE_NOT_FOUND',
1.180   requireStack: [ '/home/node/app/node_modules/@redwoodjs/api/dist/bins/redwood.js' ]
1.180 }
1.180
1.180 Node.js v18.19.1
------
Dockerfile:118
--------------------
 116 |         CI=1 yarn workspaces focus api --production
 117 |     #RUN yarn install
 118 | >>> RUN yarn redwood build web --no-prerender
 119 |     COPY --chown=node:node redwood.toml .
 120 |     COPY --chown=node:node graphql.config.js .
--------------------
ERROR: failed to solve: process "/bin/sh -c yarn redwood build web --no-prerender" did not complete successfully: exit code: 1

but If I don’t repeate it anywhere in the serve api stage then the image builds without and issue

we resolved this by running the build again in the entrypoint so that the envs are refreshed again.

Hi
I am getting a similar issue

> [api api_build 2/2] RUN yarn rw build api: 20.95 ✖ Generating Prisma Client... [FAILED: Command failed with exit code 1: node "/home/node/app/node_modules/prisma/build/index.js" generate --schema="/home/node/app/api/db/schema.prisma" 20.95 Error: Attempted to load binaryTargets value using env(BINARY_TARGET)but it was not present. Please ensure that BINARY_TARGET is present in your Environment Variables 20.95 Prisma schema loaded from db/schema.prisma]

I have set the BINARY_TARGET environment variable in the docker compose file “docker-compose.prod.yml” but it still throws this error.

Any hints would be greatly appreciated

we resolved this with a regex script as suggested by rob.

# web serve
# ------------------------------------------------
FROM node:20-bookworm-slim as web_serve
RUN corepack enable

USER node
WORKDIR /home/node/app

COPY --chown=node:node .yarnrc.yml .
COPY --chown=node:node package.json .
COPY --chown=node:node web/package.json web/
COPY --chown=node:node yarn.lock .

RUN mkdir -p /home/node/.yarn/berry/index
RUN mkdir -p /home/node/.cache

RUN --mount=type=cache,target=/home/node/.yarn/berry/cache,uid=1000 \
    --mount=type=cache,target=/home/node/.cache,uid=1000 \
    CI=1 yarn workspaces focus web --production

COPY --chown=node:node redwood.toml .
COPY --chown=node:node graphql.config.js .
COPY --chown=node:node .env.defaults .env.defaults

ENV NODE_ENV=production \
    API_PROXY_TARGET=http://api:8911 \
    CLERK_PUBLISHABLE_KEY=clerk-publishable-key

COPY --chown=node:node --from=web_build /home/node/app/web/dist /home/node/app/web/fresh

ENTRYPOINT ["/bin/sh", "-c", "\
    rm -rf /home/node/app/web/dist && \
    cp -r /home/node/app/web/fresh /home/node/app/web/dist && \
    grep -rl \"{}.CLERK_PUBLISHABLE_KEY\" ./web/dist | xargs sed -i 's/{}.CLERK_PUBLISHABLE_KEY/\"'\"$CLERK_PUBLISHABLE_KEY\"'\"/g' && \
    exec node_modules/.bin/rw-web-server --api-proxy-target $API_PROXY_TARGET"]
1 Like

for those trying this or something similar. keep in mind that the env.defaults mustn’t include the variables you want to replace. so either remove them from it, remove the file copy in the Dockerfile or create a second default file like env.prod.defaults to be copied in the container instead.