Skip to content

Authentication Flow

Updated 8 min read

Detailed explanation of how OIDC authentication works for Nebari Software Packs.

Overview

Section titled “Overview”

When a NebariApp has auth.enabled: true, the nebari-operator sets up a complete OIDC authentication flow using Keycloak and Envoy Gateway. Users are required to log in before accessing the application.

Components

Section titled “Components”
ComponentRole
Envoy GatewayReverse proxy that enforces the SecurityPolicy (OIDC filter)
KeycloakOIDC identity provider that handles login and issues tokens
nebari-operatorCreates and manages all the glue resources (HTTPRoute, SecurityPolicy, Certificate, Keycloak client)
cert-managerProvisions TLS certificates for the application hostname

The Flow

Section titled “The Flow”
                                        Nebari Cluster
    User        Envoy Gateway         Keycloak          Your App
     |               |                    |                 |
     |--1. GET /---->|                    |                 |
     |               |--2. No session---->|                 |
     |<--3. 302 -----|   cookie?          |                 |
     |   redirect    |                    |                 |
     |               |                    |                 |
     |--4. Login ----|---------------+--->|                 |
     |   page        |               |   |                 |
     |               |               |   |                 |
     |--5. Submit----|---------------+--->|                 |
     |   credentials |               |   |                 |
     |               |               |   |                 |
     |<--6. 302 -----|<--auth code--------|                 |
     |   redirect    |                    |                 |
     |               |                    |                 |
     |--7. GET / --->|                    |                 |
     |  (with code)  |--8. Exchange------>|                 |
     |               |    code for tokens |                 |
     |               |<--9. ID + Access---|                 |
     |               |    tokens          |                 |
     |               |                    |                 |
     |<--10. Set ----|                    |                 |
     |  cookies +    |                    |                 |
     |  redirect     |                    |                 |
     |               |                    |                 |
     |--11. GET / -->|                    |                 |
     |  (with        |--12. Forward-------|---------------->|
     |   cookies)    |    request         |                 |
     |               |                    |                 |
     |<--13. Response from your app------|<-----------------|

Step by step

Section titled “Step by step”

  1. User visits the app at https://my-pack.nebari.example.com

  2. Envoy Gateway checks for session cookies. The OIDC filter (configured by the SecurityPolicy) looks for valid IdToken-* and AccessToken-* cookies.

  3. No valid session - redirect to Keycloak. Envoy Gateway sends a 302 redirect to the Keycloak authorization endpoint with the client ID, redirect URI, and requested scopes.

  4. Keycloak presents the login page. The user sees the Keycloak login form (or SSO if already authenticated with Keycloak).

  5. User submits credentials. Keycloak validates the username/password (or delegates to an external IdP if configured).

  6. Keycloak redirects back with an authorization code. The redirect goes to the redirectURI configured in the NebariApp (default: /oauth2/callback), which is handled by Envoy Gateway’s OIDC filter.

  7. Browser follows the redirect back to Envoy Gateway with the authorization code.

  8. Envoy Gateway exchanges the code for tokens. A server-to-server call from Envoy Gateway to Keycloak’s token endpoint.

  9. Keycloak returns ID token, access token, and refresh token.

  10. Envoy Gateway sets session cookies. The tokens are stored in cookies:

    • IdToken-<suffix> (JWT containing user claims)
    • AccessToken-<suffix>
    • OauthHMAC-<suffix>, OauthExpires-<suffix>, RefreshToken-<suffix>

    The <suffix> is an 8-character hex string derived from the SecurityPolicy’s Kubernetes UID (e.g., IdToken-a1b2c3d4).

  11. Browser retries the original request with the session cookies attached.

  12. Envoy Gateway validates the cookies and forwards the request to your service via the HTTPRoute.

  13. Your app receives the request. The IdToken cookies are available for your app to read if it needs user identity information.

Section titled “Cookie Format”

Envoy Gateway’s OIDC filter sets cookies with the following naming convention:

IdToken-<suffix>
AccessToken-<suffix>
OauthHMAC-<suffix>
OauthExpires-<suffix>
RefreshToken-<suffix>
OauthNonce-<suffix>

The <suffix> is an 8-character hexadecimal string generated by FNV-32a hashing the SecurityPolicy resource’s Kubernetes UID. This ensures unique cookie names when multiple SecurityPolicies exist on the same domain.

For example: IdToken-a1b2c3d4, AccessToken-a1b2c3d4.

Cookie names can be customized via the cookieNames field in the SecurityPolicy’s OIDC configuration.

Reading user identity in your app

Section titled “Reading user identity in your app”

The IdToken is a JWT signed by Keycloak. Verify its signature before you trust any claim in it. Envoy Gateway does not do this for you:

  • Envoy’s OAuth2 filter never checks the JWT signature. It reads the token only for its expiry, and protects its own cookies with an HMAC.
  • Requests can reach your app without passing through that filter. The Service is reachable from any pod in the cluster unless you add a NetworkPolicy, and paths listed in routing.publicRoutes are served by an HTTPRoute with no SecurityPolicy attached. A request on any of those paths can carry any IdToken-* cookie it likes.

Whether the platform should state or enforce this itself is tracked in nebari-operator#194.

Verification needs three values, all available to your pod:

ValueWhere it comes from
Client ID (the token’s aud)client-id key of the <nebariapp-name>-oidc-client Secret
Issuer (the token’s iss)issuer-url key of the same Secret. It is empty unless the operator runs with KEYCLOAK_EXTERNAL_URL; in that case use the issuer your Keycloak puts in tokens.
JWKS URLKeycloak serves it at <issuer>/protocol/openid-connect/certs. Use the in-cluster Keycloak URL if your pods cannot reach the public one.

With PyJWT (PyJWT[crypto]>=2.10.1; 2.10.0 has a broken issuer check, CVE-2024-53861):

import logging


import jwt


log = logging.getLogger(__name__)


# The JWK set is cached for 5 minutes. Avoid cache_keys=True: its per-key cache never
# expires, so a key Keycloak has removed would stay trusted until the process restarts.
jwks = jwt.PyJWKClient(JWKS_URL, timeout=5)




def verified_claims(request) -> dict | None:
    tokens = [v for k, v in request.cookies.items() if k.startswith("IdToken-")]
    if len(tokens) != 1:  # none, or an extra cookie someone added
        return None
    try:
        key = jwks.get_signing_key_from_jwt(tokens[0])
        return jwt.decode(
            tokens[0],
            key.key,
            algorithms=["RS256"],
            audience=CLIENT_ID,
            issuer=ISSUER,
            options={"require": ["exp", "iss", "aud"]},
        )
    except jwt.PyJWKClientConnectionError:
        log.warning("JWKS unreachable; treating request as unauthenticated")
        return None
    except jwt.PyJWTError:
        return None

The auth-fastapi example does this end to end, including wiring the three values from the Secret in its Helm chart.

To keep traffic from bypassing the gateway entirely, also restrict ingress to your pods to the Envoy proxies. The auth-fastapi chart ships an optional NetworkPolicy for this (networkPolicy.enabled: true).

Common JWT claims

Section titled “Common JWT claims”
ClaimDescription
preferred_usernameKeycloak username
emailUser’s email address
nameDisplay name
given_nameFirst name
family_nameLast name
groupsKeycloak group memberships (if groups scope requested)
realm_access.rolesKeycloak realm roles
subUnique subject identifier
issToken issuer URL (Keycloak realm)
expToken expiration timestamp

What the Operator Creates

Section titled “What the Operator Creates”

When auth.enabled: true, the nebari-operator creates these resources:

1. Keycloak Client (when provisionClient: true)

Section titled “1. Keycloak Client (when provisionClient: true)”

The operator calls the Keycloak Admin API to create an OIDC client:

  • Client ID: <namespace>-<nebariapp-name> (namespace-scoped to prevent collisions)
  • Client protocol: openid-connect
  • Access type: confidential (not public)
  • Standard Flow: enabled (OAuth2 Authorization Code flow)
  • Redirect URIs: Both HTTP and HTTPS variants of the hostname
  • Web Origins: * (allows CORS)
  • Scopes: As configured in spec.auth.scopes

2. Kubernetes Secret

Section titled “2. Kubernetes Secret”

Client credentials are stored in a Secret named <nebariapp-name>-oidc-client:

apiVersion: v1
kind: Secret
metadata:
  name: <nebariapp-name>-oidc-client
  labels:
    app.kubernetes.io/name: nebariapp
    app.kubernetes.io/instance: <nebariapp-name>
    app.kubernetes.io/managed-by: nebari-operator
data:
  client-id: <base64-encoded>       # Always present. Value: <namespace>-<nebariapp-name>
  client-secret: <base64-encoded>   # Always present. Cryptographically generated.
  issuer-url: <base64-encoded>      # Always present. Empty unless the operator has
                                    # KEYCLOAK_EXTERNAL_URL set; then the public issuer
                                    # (e.g., https://keycloak.example.com/realms/nebari)
  spa-client-id: <base64-encoded>   # Present when spaClient is enabled.
  device-client-id: <base64-encoded> # Present when deviceFlowClient is enabled.

The operator also creates a Role and RoleBinding that let spec.serviceAccountName (default: the NebariApp name) get this Secret through the Kubernetes API:

  • Role: <nebariapp-name>-oidc-secret-reader
  • RoleBinding: <nebariapp-name>-oidc-secret-reader

You only need this if your app reads the Secret through the API. Referencing it with env.valueFrom.secretKeyRef or a volume works without any Role, because the kubelet fetches it. The Role does not stop anyone else from reading the Secret either; see “Who can read the OIDC Secret” in the NebariApp CRD reference.

3. Envoy Gateway SecurityPolicy (when enforceAtGateway: true)

Section titled “3. Envoy Gateway SecurityPolicy (when enforceAtGateway: true)”
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: <nebariapp-name>-security
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: <nebariapp-name>-route
  oidc:
    provider:
      # In-cluster Keycloak URL, used only by Envoy Gateway's control plane
      issuer: http://<keycloak-service>.<keycloak-namespace>.svc.cluster.local/.../realms/<realm>
      tokenEndpoint: <in-cluster Keycloak>/protocol/openid-connect/token
      # Browser-facing endpoints, set when the operator has KEYCLOAK_EXTERNAL_URL
      authorizationEndpoint: <public Keycloak>/protocol/openid-connect/auth
      endSessionEndpoint: <public Keycloak>/protocol/openid-connect/logout
    clientID: <namespace>-<nebariapp-name>
    clientSecret:
      name: <nebariapp-name>-oidc-client
    redirectURL: https://<hostname><redirectURI>
    logoutPath: /logout
    scopes: [openid, profile, email]

The policy targets only <nebariapp-name>-route. Paths in routing.publicRoutes are served by a second HTTPRoute, <nebariapp-name>-public-route, which has no SecurityPolicy. The policy contains no authorization rules: auth.groups is not enforced here.

4. HTTPRoute

Section titled “4. HTTPRoute”
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: <nebariapp-name>-route   # plus <nebariapp-name>-public-route for publicRoutes
spec:
  parentRefs:
    - name: <gateway-name>
      namespace: <gateway-namespace>
  hostnames:
    - <hostname>
  rules:
    - backendRefs:
        - name: <service-name>
          port: <service-port>

5. cert-manager Certificate (when routing.tls.enabled: true)

Section titled “5. cert-manager Certificate (when routing.tls.enabled: true)”

Created only when the operator has a cert-manager ClusterIssuer configured and routing.tls.secretName is not set. Certificates live in the Gateway’s namespace, so their names include the app’s namespace:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: <nebariapp-name>-<namespace>-cert
  namespace: envoy-gateway-system
spec:
  secretName: <nebariapp-name>-<namespace>-tls
  dnsNames:
    - <hostname>
  issuerRef:
    name: <cluster-issuer>
    kind: ClusterIssuer

App-Native OAuth

Section titled “App-Native OAuth”

Some applications handle OAuth natively (e.g., Grafana, Superset, Gitea). For these apps, the operator provisions the Keycloak client and stores credentials, but the app handles the OAuth flow itself. This is useful when the app needs deeper integration with the OAuth flow, such as mapping Keycloak groups/roles to app-internal roles.

Gateway-only auth (app reads JWT from cookies)

Section titled “Gateway-only auth (app reads JWT from cookies)”

If your app just needs user identity (not role mapping), use enforceAtGateway: true (the default) and read and verify the IdToken cookie as described above.

App-native auth only (no gateway enforcement)

Section titled “App-native auth only (no gateway enforcement)”

Set enforceAtGateway: false to skip gateway auth. The operator will:

  • Provision a Keycloak client
  • Store credentials in a Secret
  • NOT create a SecurityPolicy
auth:
  enabled: true
  provider: keycloak
  provisionClient: true
  enforceAtGateway: false
Section titled “Dual-layer auth (recommended for RBAC apps)”

Use both gateway enforcement AND app-native OAuth. The gateway ensures users are authenticated, while the app reads roles/groups for authorization:

auth:
  enabled: true
  provider: keycloak
  provisionClient: true
  # enforceAtGateway defaults to true

The app also authenticates against the same Keycloak client to get roles/groups.

Wiring credentials to your app

Section titled “Wiring credentials to your app”

Reference the operator-created OIDC secret to inject credentials as environment variables. Use valueFrom.secretKeyRef to map specific keys:

# In your Helm values (e.g., for an upstream chart's extraEnv/extraEnvRaw)
extraEnvRaw:
  - name: OAUTH_CLIENT_ID
    valueFrom:
      secretKeyRef:
        name: <nebariapp-name>-oidc-client
        key: client-id
  - name: OAUTH_CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: <nebariapp-name>-oidc-client
        key: client-secret
  - name: OAUTH_ISSUER_URL
    valueFrom:
      secretKeyRef:
        name: <nebariapp-name>-oidc-client
        key: issuer-url
        optional: true  # Written by the operator when it provisions the client (empty unless
                        # KEYCLOAK_EXTERNAL_URL is set); with provisionClient: false, you write it

The OIDC discovery URL can be constructed as: <issuer-url>/.well-known/openid-configuration

Your app then configures its OAuth provider using these environment variables.

Note on extraEnv vs extraEnvRaw: Many upstream Helm charts support multiple env var formats. Use extraEnvRaw (or the equivalent) when you need valueFrom.secretKeyRef syntax. Check your upstream chart’s documentation for the correct field name.

NebariApp CRD vs Envoy Gateway SecurityPolicy

Section titled “NebariApp CRD vs Envoy Gateway SecurityPolicy”

The fields documented in the NebariApp API reference are the fields the operator understands - they go on spec.auth of the NebariApp resource. At runtime, the operator generates an Envoy Gateway SecurityPolicy from the NebariApp, and that SecurityPolicy has its own (much larger) set of OIDC tuning knobs.

For the OIDC filter fields specifically, mentally place each one in one of these buckets:

  1. Surfaced on NebariApp. The operator exposes the field as a NebariApp spec.auth.* field and copies it into the SecurityPolicy at reconcile time. Currently this includes forwardAccessToken (auth.forwardAccessToken) and redirect-deny rules (auth.denyRedirect). Set these on the NebariApp.

  2. Not surfaced on NebariApp. Most fine-grained OIDC filter fields - including cookieNames, disableIdToken, disableAccessToken, passThroughAuthHeader, custom logout URLs, and similar - are not exposed on NebariApp today. To use them, either:

    • File an issue / PR on nebari-operator asking for the field to be plumbed through.
    • Set auth.enforceAtGateway: false and manage your own SecurityPolicy resource alongside the NebariApp. The operator will still provision the Keycloak client and Secret, but won’t generate a SecurityPolicy to conflict with yours.

Refer to the Envoy Gateway SecurityPolicy reference for the full set of OIDC fields.

Limitations

Section titled “Limitations”

  • Local development: The dev/ Makefile builds a kind cluster with Keycloak, Envoy Gateway, cert-manager and the operator, so you can test the full login flow locally: cd dev && make up-fastapi, then make update-hosts so the browser can resolve keycloak.nebari.local, then log in with the Keycloak credentials it prints. The FastAPI example shows “Not Authenticated” when no valid IdToken cookie is present.

  • Token expiration: Envoy Gateway handles token refresh automatically via refresh tokens stored in cookies. Your app does not need to handle token refresh.

  • Cookie size: Very large JWTs (many groups/roles) may exceed browser cookie size limits (typically 4KB). If this is an issue, reduce token size by limiting scopes/claims at the Keycloak level. Token-cookie suppression (disableIdToken/disableAccessToken) is a SecurityPolicy field the operator does not currently expose - see the boundary section above for how to use it if you need to.