dd-mani-viti

Kubernetes manifests for Dokumentdeling REST API on the Viti platform.

Table of Contents

• Architecture
• Repo Structure
  • Manifests Reference
• Environments
  • Cluster details
• PostgreSQL (pgEdge Active-Active)
  • Architecture
  • Cluster Details
  • Connection Details
  • Cross-Cluster Networking
  • TLS Certificates
  • Wiring Spock Replication
  • Managed Roles
• Bootstrapping a Cluster
  • Cluster-specific parameters
• Rendering Templates Locally
• Secrets Management
  • OpenBao (CSI Secrets Store)
  • Secrets synced to Kubernetes Secrets
  • CNPG-managed secrets (auto-generated per cluster)
• ArgoCD Sync Waves
• Monitoring & Alerting
• Network Access

Architecture

flowchart TD
    subgraph External
        Client[Client / Helsenett]
        OpenBao[OpenBao Vault]
        S3[S3 Backup Storage]
    end

    subgraph Cluster["Kubernetes Cluster"]
        subgraph NS["Namespace: dokumentdeling"]
            GW[Gateway<br/>hnet-private VIP]
            HR[HTTPRoute]
            SVC[Service :8443]
            Deploy[Deployment<br/>dokumentdeling-api]
            CM[ConfigMap]
            SA[ServiceAccount]
            NP[NetworkPolicy]
            Cert[internal-cert<br/>cert-manager]
            SecretApp[dokumentdeling-secret]
            SecretIPAM[ipam-secret]

            subgraph DB["PostgreSQL (CNPG + pgEdge)"]
                PG[Cluster<br/>pgEdge Spock]
                PGSvc[Replication LB<br/>MetalLB]
                PGMon[PodMonitor]
                PGBackup[ScheduledBackup]
            end

            subgraph Secrets["OpenBao CSI Sync"]
                SPC_App[SecretProviderClass<br/>app secrets]
                SPC_S3[SecretProviderClass<br/>S3 creds]
                SPC_Certs[SecretProviderClass<br/>pgEdge certs]
            end

            MigJob[DB Migration Job]
        end
    end

    Client -->|HTTPS| GW
    GW --> HR --> SVC --> Deploy
    Deploy --- CM
    Deploy --- SA
    Deploy --- Cert
    Deploy --- SecretApp
    Deploy -.->|connects| PG
    MigJob -.->|migrates| PG

    OpenBao -->|CSI sync| SPC_App --> SecretApp
    OpenBao -->|CSI sync| SPC_S3 --> PGBackup
    OpenBao -->|CSI sync| SPC_Certs --> PG
    OpenBao -->|IPAM secret| SecretIPAM --> GW

    PG --> PGSvc
    PGSvc <-->|Spock replication| External
    PGBackup -->|WAL + base| S3
    PGMon -.->|metrics| PG

    NP -.->|controls| Deploy
    NP -.->|controls| PG

Repo Structure

dd-mani-viti/
├── apps/           ArgoCD App of Apps (parent application)
├── manifests/      Main API Helm chart
├── monitoring/     Prometheus probes, alerts and Alertmanager configs
└── scripts/        Operational scripts (cert push, README generation)

Manifests Reference

manifests/templates/ — all Kubernetes resources deployed by the API chart

File	What it does	Key tunables
namespace.yaml	Creates the dokumentdeling namespace	-
serviceaccount.yaml	dokumentdeling-sa SA for pods	-
configmap.yaml	Static env vars (ASPNETCORE_ENVIRONMENT, ASPNETCORE_URLS)	-
deployment.yaml	Main API deployment	replicaCount, api.image, api.envVariables
service.yaml	ClusterIP service on port 8443	api.service.port
httproute.yaml	Gateway API route for external traffic	api.hosts, gateway.ipZone, gateway.clusterIssuer
networkpolicy.yaml	Egress/ingress firewall rules	Allows DNS, all external, and pgEdge on 5432
internal-cert.yaml	Self-signed TLS cert for pod-to-pod HTTPS	internalSsl.commonName, internalSsl.names
openbao-kubernetes-auth.yaml	SA + Secret + ClusterRoleBinding for OpenBao auth	-
openbao-secrets.yaml	SecretProviderClass: syncs app secrets from OpenBao	openbao.kubernetesMountPath
database-migration-job.yaml	Runs DB migrations before deploy (wave -1)	databaseMigration.enabled
job-reader-role.yaml	RBAC Role to read Job status	-
job-reader-rolebinding.yaml	Binds job-reader to the SA	-
PostgreSQL
postgres.yaml	CNPG Cluster (only when role != pgedge)	postgres.instances, postgres.storage.size
pg-replication-svc.yaml	MetalLB LoadBalancer for Spock cross-cluster traffic	postgres.pgedge.loadBalancerIP
pg-podmonitor.yaml	Prometheus PodMonitor for CNPG metrics	postgres.monitoring.enabled
pg-scheduled-backup.yaml	Daily S3 backup via Barman (when backup enabled)	postgres.backup.bucket, postgres.backup.endpointURL
openbao-pg-s3.yaml	SecretProviderClass for S3 backup credentials	postgres.backup.secretPath
openbao-pgedge-certs.yaml	SecretProviderClass: syncs pgEdge TLS certs from OpenBao (non-TRD only)	-
pgedge-cert-sync-job.yaml	Job that triggers CSI cert sync (non-TRD only)	-

manifests/values — key values you might want to tune

Value	Default	Dev override	What it controls
replicaCount	2	3	Number of API pods
postgres.role	primary	pgedge	DB mode: standalone or active-active
postgres.pgedge.localNode	n1	set per cluster	Which pgEdge node this cluster runs
postgres.storage.size	10Gi	5Gi	PVC size for PostgreSQL
postgres.monitoring.enabled	true	true	Enable PodMonitor for Prometheus
databaseMigration.enabled	true	false	Run migration Job on sync
openbao.kubernetesMountPath	kubernetes	per cluster	OpenBao auth mount for this cluster
gateway.ipamSecret	""	dokumentdeling-ipam-secret	Multi-cluster VIP failover
api.image	-	sha256 digest	Container image for the API
api.hosts	[]	[viti-api.dev.pjd.nhn.no]	External hostnames

Environments

Env	Hosts	Clusters	Failover	Auto-sync
dev	viti-api.dev.pjd.nhn.no	d-osl-dd-viti, d-trd-dd-viti	IPAM shared VIP	yes
test	viti-api.pjd.test.nhn.no	t-trd-dd-viti	-	yes
qa	viti-api.pjd.qa.nhn.no	q-trd-dd-viti	-	yes
prod	api.pjd.nhn.no	p-dd-000 (Tanzu)	-	no

Cluster details

Cluster	Location	Vitistack ID	OpenBao auth mount	pgEdge node	MetalLB IP
d-osl-dd-viti	OSL	vitistack-dokdel-xb9h	dokumentdeling-auth-mount-osl	n2	10.204.153.179
d-trd-dd-viti	TRD	vitistack-dokdeling-qr46	dokumentdeling-auth-mount	n1	10.204.153.180
prod	PROD	``	``	-	-
qa	QA	``	``	-	-
test	TEST	``	``	-	-

PostgreSQL (pgEdge Active-Active)

The database layer uses pgEdge with Spock logical replication for active-active multi-master across clusters. Each cluster runs an independent CNPG-managed PostgreSQL instance that replicates bidirectionally.

Architecture

graph LR
n2["d-osl-dd-viti<br/>dokumentdeling-n2<br/>LB: 10.204.153.179"]
n1["d-trd-dd-viti<br/>dokumentdeling-n1<br/>LB: 10.204.153.180"]
n1 <-->|"Spock replication"| n2

Cluster Details

Property	OSL (n2)	TRD (n1)
CNPG Cluster	dokumentdeling-n2	dokumentdeling-n1
Spock node	n2	n1
MetalLB IP	10.204.153.179	10.204.153.180
Image	ncr.sky.nhn.no/ghcr/pgedge/pgedge-postgres:16-spock5-standard	same
Storage	5Gi (cephfs)	same
Cert source	cert-manager	OpenBao
Extensions	Spock 5.x + Snowflake	same

Connection Details

Applications connect using the CNPG-generated secret:

Cluster	Secret Name	Service (rw)	Database	User
OSL	dokumentdeling-n2-app	dokumentdeling-n2-rw.dokumentdeling:5432	app	app
TRD	dokumentdeling-n1-app	dokumentdeling-n1-rw.dokumentdeling:5432	app	app

The secret contains keys: host, port, dbname, user, password, uri, jdbc-uri.

How the app gets its connection string

In pgEdge mode (dev), the deployment builds ConnectionStrings__DefaultConnection directly from the CNPG secret -- no manual connection string in OpenBao needed:

ConnectionStrings__DefaultConnection = "Host=$(DB_HOST);Port=5432;Database=app;Username=$(DB_USER);Password=$(DB_PASSWORD)"

Env var	Source
DB_HOST (OSL)	dokumentdeling-n2-app → host
DB_USER (OSL)	dokumentdeling-n2-app → username
DB_PASSWORD (OSL)	dokumentdeling-n2-app → password
DB_HOST (TRD)	dokumentdeling-n1-app → host
DB_USER (TRD)	dokumentdeling-n1-app → username
DB_PASSWORD (TRD)	dokumentdeling-n1-app → password

In non-pgEdge envs (test/qa/prod), ConnectionStrings__DefaultConnection is fetched from OpenBao (kv-v2-dokumentdeling/data/shared) as before.

Cross-Cluster Networking

Clusters have overlapping pod CIDRs so Cilium ClusterMesh cannot be used. Instead, MetalLB LoadBalancer services with routable IPs in the 10.204.153.0/24 range provide connectivity for Spock replication on port 5432.

TLS Certificates

pgEdge uses mutual TLS for replication. The first cluster (OSL) provisions certificates via cert-manager, then they are pushed to OpenBao and synced to other clusters via the CSI Secrets Store driver.

Secret	Purpose
client-ca-key-pair	Client CA (shared across all clusters)
admin-client-cert	Admin client cert
app-client-cert	App client cert
pgedge-client-cert	pgEdge replication client cert
streaming-replica-client-cert	Streaming replica cert

OpenBao path: kv-v2-dokumentdeling/data/pgedge-certs/dokumentdeling

Pushing certs to OpenBao (one-time after first cluster deploy)

export BAO_ADDR=https://openbao.test.nhn.no
export BAO_NAMESPACE=PJD
bao login -method=oidc
./scripts/push-pgedge-certs-to-openbao.sh

Wiring Spock Replication

After all clusters are deployed, create Spock nodes and subscriptions.

Step 1: Create Spock nodes

# OSL (n2)
kubectl --context d-osl-dd-viti -n dokumentdeling exec dokumentdeling-n2-1 -c postgres -- \
  psql -U postgres -d app -c "SELECT spock.node_create('n2', 'host=10.204.153.179 port=5432 dbname=app user=pgedge sslmode=require sslcert=/projected/pgedge/certificates/tls.crt sslkey=/projected/pgedge/certificates/tls.key');"

# TRD (n1)
kubectl --context d-trd-dd-viti -n dokumentdeling exec dokumentdeling-n1-1 -c postgres -- \
  psql -U postgres -d app -c "SELECT spock.node_create('n1', 'host=10.204.153.180 port=5432 dbname=app user=pgedge sslmode=require sslcert=/projected/pgedge/certificates/tls.crt sslkey=/projected/pgedge/certificates/tls.key');"

Step 2: Create subscriptions (each node subscribes to every other)

# OSL subscribes to TRD
kubectl --context d-osl-dd-viti -n dokumentdeling exec dokumentdeling-n2-1 -c postgres -- \
  psql -U postgres -d app -c "SELECT spock.sub_create('sub_n1_n2', 'host=10.204.153.180 port=5432 dbname=app user=pgedge sslmode=require sslcert=/projected/pgedge/certificates/tls.crt sslkey=/projected/pgedge/certificates/tls.key', '{default,default_insert_only,ddl_sql}', synchronize_data := true);"

# TRD subscribes to OSL
kubectl --context d-trd-dd-viti -n dokumentdeling exec dokumentdeling-n1-1 -c postgres -- \
  psql -U postgres -d app -c "SELECT spock.sub_create('sub_n2_n1', 'host=10.204.153.179 port=5432 dbname=app user=pgedge sslmode=require sslcert=/projected/pgedge/certificates/tls.crt sslkey=/projected/pgedge/certificates/tls.key', '{default,default_insert_only,ddl_sql}', synchronize_data := true);"

Verify replication status

# OSL
kubectl --context d-osl-dd-viti -n dokumentdeling exec dokumentdeling-n2-1 -c postgres -- \
  psql -U postgres -d app -c "SELECT * FROM spock.sub_show_status();"

# TRD
kubectl --context d-trd-dd-viti -n dokumentdeling exec dokumentdeling-n1-1 -c postgres -- \
  psql -U postgres -d app -c "SELECT * FROM spock.sub_show_status();"

All subscriptions should show status = replicating.

Managed Roles

Role	Capabilities
app	Default application user (owner of app database)
admin	Superuser for administration
pgedge	Superuser + replication (used by Spock)

Bootstrapping a Cluster

Each cluster has a bootstrap manifest in bootstrap/:

bootstrap/
├── dev-osl.yaml
├── dev-trd.yaml
├── prod.yaml
├── qa.yaml
├── test.yaml

Apply bootstrap per cluster

# OSL (d-osl-dd-viti)
kubectl apply -f bootstrap/dev-osl.yaml --context d-osl-dd-viti

# TRD (d-trd-dd-viti)
kubectl apply -f bootstrap/dev-trd.yaml --context d-trd-dd-viti

# PROD (prod)
kubectl apply -f bootstrap/prod.yaml --context prod

# QA (qa)
kubectl apply -f bootstrap/qa.yaml --context qa

# TEST (test)
kubectl apply -f bootstrap/test.yaml --context test

Cluster-specific parameters

Each bootstrap file sets these Helm parameters to tell the apps which cluster they're on:

d-osl-dd-viti (OSL)

Parameter	Value
vitistackName	vitistack-dokdel-xb9h
clusterName	d-osl-dd-viti
openbaoMountPath	dokumentdeling-auth-mount-osl
postgresSite	cluster-osl
autoSync	true

d-trd-dd-viti (TRD)

Parameter	Value
vitistackName	vitistack-dokdeling-qr46
clusterName	d-trd-dd-viti
openbaoMountPath	dokumentdeling-auth-mount
postgresSite	cluster-trd
autoSync	true

prod (PROD)

Parameter	Value
vitistackName	``
clusterName	prod
openbaoMountPath	``
postgresSite	``
autoSync	true

qa (QA)

Parameter	Value
vitistackName	``
clusterName	qa
openbaoMountPath	``
postgresSite	``
autoSync	true

test (TEST)

Parameter	Value
vitistackName	``
clusterName	test
openbaoMountPath	``
postgresSite	``
autoSync	true

ArgoCD then manages child applications:

• dokumentdeling-api (from manifests/)
• dokumentdeling-monitoring (from monitoring/)
• dokumentdeling-pgedge (pgEdge Helm chart from pgedge.github.io)
• nhn-appmarket (CSI driver, OpenBao provider, k8sapibridge, CNPG operator)

Rendering Templates Locally

Helm template commands

# API manifests (dev)
helm template dokumentdeling manifests/ \
  -f manifests/values.yaml \
  -f manifests/values.dev.yaml

# Apps chart (per cluster)

# OSL
helm template dokumentdeling-apps apps/ \
  -f apps/values.yaml \
  -f apps/values.dev.yaml \
  --set postgresSite=cluster-osl \
  --set openbaoMountPath=dokumentdeling-auth-mount-osl

# TRD
helm template dokumentdeling-apps apps/ \
  -f apps/values.yaml \
  -f apps/values.dev.yaml \
  --set postgresSite=cluster-trd \
  --set openbaoMountPath=dokumentdeling-auth-mount

# PROD
helm template dokumentdeling-apps apps/ \
  -f apps/values.yaml \
  -f apps/values.prod.yaml \
  --set postgresSite= \
  --set openbaoMountPath=

# QA
helm template dokumentdeling-apps apps/ \
  -f apps/values.yaml \
  -f apps/values.qa.yaml \
  --set postgresSite= \
  --set openbaoMountPath=

# TEST
helm template dokumentdeling-apps apps/ \
  -f apps/values.yaml \
  -f apps/values.test.yaml \
  --set postgresSite= \
  --set openbaoMountPath=

Secrets Management

OpenBao (CSI Secrets Store)

Env	OpenBao Server	Namespace	Auth Mounts	KV Path
dev	openbao.test.nhn.no	PJD	dokumentdeling-auth-mount-osl, dokumentdeling-auth-mount	kv-v2-dokumentdeling/data/shared
test/qa	openbao.test.nhn.no	PJD	kubernetes	kv-v2-dokumentdeling/data/shared
prod	openbao.nhn.no	PJD	kubernetes	kv-v2-dokumentdeling/data/shared

Secrets synced to Kubernetes Secrets

K8s Secret Name	Keys	Purpose
dokumentdeling-secret	All app secrets	envFrom in deployment
dokumentdeling-api-slack-alertmanagerconfig-secret	apiSecret, apiUrl	Alertmanager Slack bot
dokumentdeling-ipam-secret	secret	IPAM shared secret for failover
admin-client-cert	tls.crt, tls.key, ca.crt	pgEdge admin cert (non-primary clusters)
client-ca-key-pair	tls.crt, tls.key, ca.crt	pgEdge shared CA (non-primary clusters)
pgedge-client-cert	tls.crt, tls.key, ca.crt	pgEdge replication cert (non-primary clusters)

CNPG-managed secrets (auto-generated per cluster)

Secret	Keys	Purpose
dokumentdeling-n2-app	host, port, dbname, user, password, uri, jdbc-uri	OSL DB credentials
dokumentdeling-n1-app	host, port, dbname, user, password, uri, jdbc-uri	TRD DB credentials

ArgoCD Sync Waves

Wave	Resources
-4	OpenBao pgEdge certs SecretProviderClass (non-primary clusters)
-3	pgEdge cert-sync Job (non-primary clusters)
-2	Namespace, ServiceAccount, Secrets, Service, Cert, RBAC, ConfigMap, NetworkPolicy, LoadBalancer
-1	Database migration Job
0	Deployment, HTTPRoutes, PodMonitor

Monitoring & Alerting

The monitoring/ chart installs:

• Blackbox Probe -- HTTP health check against /health
• PrometheusRule with alerts: SlowHttp, HttpFailure, ProbeFailed, SSLCertExpiringSoon
• AlertmanagerConfig (OS Dashboard) -- webhook to NHN OS Dashboard
• AlertmanagerConfig (Slack) -- posts to Slack via bot token
• PodMonitor for CNPG PostgreSQL metrics

Network Access

The API is published on Helsenett private (ipam.vitistack.io/zone: hnet-private) with security policy expose-nhn-office-client. The NetworkPolicy allows:

• Ingress on port 8443
• Egress to DNS (kube-system), all external IPs, and PostgreSQL (port 5432)