Installation

Warning

Both provided installation options serve as templates for productive use only. Even though they can run out of the box, they will need proper configuration for the requirements of the environment they will be installed in. This includes additional hardening and security measures.

Docker Compose

SecObserve provides 2 Docker Compose files as templates for productive use in the folder docker-compose: docker-compose-prod-postgres.yml and docker-compose-prod-mysql.yml. Both start Traefik as an edge router as well as the SecObserve frontend and backend plus a database (either PostgreSQL or MySQL).

Without any changes to the Docker Compose file, 3 URL's are available:

• Frontend: http://secobserve.localhost
• Backend: http://secobserve-backend.localhost (base URL)
• Traefik: http://traefik.localhost (dashboard)

docker-compose-prod-postgres.yml

name: "secobserve_prod"

volumes:
  prod_postgres_data:

networks:
  traefik:
  database:

services:

  traefik:
    image: "traefik:v3.7.13"
    container_name: "prod_traefik"
    command:
      - "--log.level=INFO"
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=secobserve_prod_traefik"
      - "--entrypoints.web.address=:80"
    labels:
      - "traefik.enable=true"
      # - "traefik.http.middlewares.traefik-ipallowlist.ipallowlist.sourcerange=172.18.0.1/24"
      # - "traefik.http.routers.api.middlewares=traefik-ipallowlist@docker"
      - "traefik.http.routers.api.entrypoints=web"
      - "traefik.http.routers.api.rule=Host(`traefik.localhost`)"
      - "traefik.http.routers.api.service=api@internal"
    ports:
      - "80:80"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - default
      - traefik

  frontend:
    image: ghcr.io/secobserve/secobserve-frontend:1.61.0
    container_name: "prod_secobserve_frontend"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.frontend.rule=Host(`secobserve.localhost`)"
      - "traefik.http.routers.frontend.entrypoints=web"
      - "traefik.http.services.frontend.loadbalancer.server.port=3000"
    environment:
      API_BASE_URL: ${SO_API_BASE_URL:-http://secobserve-backend.localhost/api}
      OIDC_ENABLE: ${SO_OIDC_ENABLE:-false}
      OIDC_AUTHORITY: ${SO_OIDC_AUTHORITY:-dummy}
      OIDC_CLIENT_ID: ${SO_OIDC_CLIENT_ID:-dummy}
      OIDC_REDIRECT_URI: ${SO_OIDC_REDIRECT_URI:-http://secobserve.localhost}
      OIDC_POST_LOGOUT_REDIRECT_URI: ${SO_OIDC_POST_LOGOUT_REDIRECT_URI:-http://secobserve.localhost}
      OIDC_SCOPE: ${SO_OIDC_SCOPE:-openid profile email}
      OIDC_PROMPT: ${SO_OIDC_PROMPT:-}
    networks:
      - traefik

  backend:
    image: ghcr.io/secobserve/secobserve-backend:1.61.0
    container_name: "prod_secobserve_backend"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.backend.rule=Host(`secobserve-backend.localhost`)"
      - "traefik.http.routers.backend.entrypoints=web"
    depends_on:
      - postgres
    environment:
      # --- Admin user ---
      ADMIN_USER: ${SO_ADMIN_USER:-admin}
      ADMIN_PASSWORD: ${SO_ADMIN_PASSWORD:-admin}
      ADMIN_EMAIL: ${SO_ADMIN_EMAIL:-admin@example.com}
      # --- Gunicorn ---
      GUNICORN_WORKERS: ${SO_GUNICORN_WORKERS:-3}
      GUNICORN_THREADS: ${SO_GUNICORN_THREADS:-10}
      GUNICORN_LIMIT_REQUEST_FIELD_SIZE: ${SO_GUNICORN_LIMIT_REQUEST_FIELD_SIZE:-16380}
      # --- Database ---
      DATABASE_ENGINE: ${SO_DATABASE_ENGINE:-django.db.backends.postgresql}
      DATABASE_HOST: ${SO_DATABASE_HOST:-postgres}
      DATABASE_PORT: ${SO_DATABASE_PORT:-5432}
      DATABASE_DB: ${SO_DATABASE_DB:-secobserve}
      DATABASE_USER: ${SO_DATABASE_USER:-secobserve}
      DATABASE_PASSWORD: ${SO_DATABASE_PASSWORD:-secobserve}
      # --- Security ---
      ALLOWED_HOSTS: ${SO_ALLOWED_HOSTS:-secobserve-backend.localhost}
      CORS_ALLOWED_ORIGINS: ${SO_CORS_ALLOWED_ORIGINS:-http://secobserve.localhost}
      DJANGO_SECRET_KEY: ${SO_DJANGO_SECRET_KEY:-NxYPEF5lNGgk3yonndjSbwP77uNJxOvfKTjF5aVBqsHktNlf1wfJHHvJ8iifk32r}
      FIELD_ENCRYPTION_KEY: ${SO_FIELD_ENCRYPTION_KEY:-DtlkqVb3wlaVdJK_BU-3mB4wwuuf8xx8YNInajiJ7GU=}
      # --- OpenID Connect ---
      OIDC_AUTHORITY: ${SO_OIDC_AUTHORITY:-}
      OIDC_CLIENT_ID: ${SO_OIDC_CLIENT_ID:-}
      OIDC_USERNAME: ${SO_OIDC_USERNAME:-}
      OIDC_FIRST_NAME: ${SO_OIDC_FIRST_NAME:-}
      OIDC_LAST_NAME: ${SO_OIDC_LAST_NAME:-}
      OIDC_FULL_NAME: ${SO_OIDC_FULL_NAME:-}
      OIDC_EMAIL: ${SO_OIDC_EMAIL:-}
      OIDC_GROUPS: ${SO_OIDC_GROUPS:-}
    networks:
      - traefik
      - database

  postgres:
    image: postgres:15.19-alpine
    container_name: "prod_postgres"
    environment:
      POSTGRES_DB: ${SO_POSTGRES_DB:-secobserve}
      POSTGRES_USER: ${SO_POSTGRES_USER:-secobserve}
      POSTGRES_PASSWORD: ${SO_POSTGRES_PASSWORD:-secobserve}
    volumes:
      - prod_postgres_data:/var/lib/postgresql/data
    networks:
      - database

Configuration for Traefik

• The Traefik dashboard should either be configured with authentication or disabled, see The Dashboard.
• Encrypted communiction should be configured for frontend and backend. Traefik supports given certificates and automatic configuration with Let's Encrypt, see HTTPS & TLS.

Configuration for SecObserve

The Docker Compose file sets default values for the SecObserve configuration, so that the containers can run out of the box. All default values can be overriden, by setting respective environment variables in the shell before starting Docker Compose. To avoid name collisions, the environment variables in the shell need to have a SO_ prefix in front of the name as it is stated in Configuration.

Some values should be changed for productive use, to avoid using the default values for secrets:

• SO_ADMIN_PASSWORD
• SO_DATABASE_PASSWORD
• SO_DJANGO_SECRET_KEY
• SO_FIELD_ENCRYPTION_KEY

Startup

• The database structure is initialized with the first start of the backend container.
• The URLs for frontend and backend are available after approximately 30 seconds, after the healthcheck of the containers has been running for the first time.

Kubernetes

SecObserve provides a Helm chart as a template for productive use. Resource and bundled PostgreSQL names are derived from the Helm release name. With the default values, the frontend is accessible at https://secobserve.dev/.

Database

The PostgreSQL database is provided by Bitnami's Helm chart. Bitnami doesn't provide updates for their free tier anymore, see Upcoming changes to the Bitnami Catalog and the Docker image is pulled from the bitnamilegacy repository.

This is ok to test the Kubernetes installation, but not suitable for production use. A productive environment has to use an update-to-date database, e.g. installed as an operator like CloudNativePG or a managed service of a cloud provider.

The chart automatically selects the PostgreSQL primary Service for standalone and replication architectures. When using postgresql.auth.existingSecret, the backend also uses the configured Secret and postgresql.auth.secretKeys.userPasswordKey.

For an external database, disable the subchart and provide the connection and password Secret explicitly:

postgresql:
  enabled: false

database:
  host: postgres.example.internal
  port: 5432
  name: secobserve
  username: secobserve
  passwordSecret:
    name: secobserve-database
    key: password

Secrets

Three values are read from a Secret, which has to be created manually before installing the chart:

• ADMIN_PASSWORD
• DJANGO_SECRET_KEY
• FIELD_ENCRYPTION_KEY

The command to setup the secret can look like this:

kubectl create secret generic my-release-secobserve-secrets \
    --namespace ... \
    --from-literal=password='...' \
    --from-literal=django_secret_key='...' \
    --from-literal=field_encryption_key='...'

See Configuration for more information how to set these values.

The example uses the release name my-release. For a release named secobserve, the generated Secret name remains secobserve-secrets. Set backend.existingSecret to use a different name.

Background tasks

With the default architecture: single, all roles run in one Pod and the chart supports exactly one application replica.

Set architecture: ha to run the backend roles as separate workloads instead: an init Job for migrations, admin user, parsers and licenses, a scalable api Deployment, a background Deployment for the Huey consumer, and a separate frontend Deployment. Scale the API with backend.api.replicaCount and the frontend with frontend.replicaCount.

backend.background.replicaCount is limited to 1. The Huey scheduler is enabled on every consumer and the flushes on consumer startup clear the locks and in-flight entries of the whole queue, so a second consumer would enqueue periodic tasks twice and reset the state of the first one. For the same reason the background Deployment uses the Recreate strategy.

The background container has a liveness probe that restarts it when a task has been running for longer than HUEY_TASK_MAX_RUNTIME_HOURS, because a worker thread that is blocked forever keeps the lock of its task and all later runs of the task are skipped. The restart runs the startup flushes, which release the locks and mark the interrupted periodic tasks as failed. Tasks that other worker threads are running at that moment are lost and run again at their next schedule. When the probe can't reach the database, it passes, because a restart doesn't help then. Configure the probe with backend.background.livenessProbe.* or disable it with backend.background.livenessProbe.enabled=false. The single architecture has no such probe, because the consumer shares its container with the API.

For SQLite installations the queue is persisted in a dedicated PersistentVolumeClaim. Configure huey.persistence.existingClaim to reuse a claim or huey.persistence.storageClass to select a storage class. The PersistentVolumeClaim is not created for architecture: ha, which requires PostgreSQL or MySQL.

Scaling the API

Two limits are shared by all API replicas and should be checked before raising backend.api.replicaCount.

Database connections. CONN_MAX_AGE is not configured, so connections are opened per request rather than held open. At peak an API replica can still use up to GUNICORN_WORKERS * GUNICORN_THREADS connections, which is 30 with the defaults of 3 and 10, see Configuration. The bundled PostgreSQL allows 100 connections by default, so raise max_connections through postgresql.primary.extendedConfiguration, or lower GUNICORN_THREADS, before running more than three API replicas.

Rate limiting. SecObserve uses the Django in-memory cache, which is local to each Gunicorn worker process. The rate limits of the REST API are therefore counted per process, not per installation: the effective limit is replicas * GUNICORN_WORKERS times the configured rate. This already applies to a single Pod and is not specific to architecture: ha.