Skip to main content
Version: Next

Upgrade instructions

Run the following instructions to upgrade an existing RADAR-Kubernetes cluster.

Upgrade to RADAR-Kubernetes version 1.3.0

This version introduces postgresql and Timescaledb clusters managed by the CloudNativePG operator.

Update mods/migration/1.3.0.yaml mods file

This file provides configuration for database migration. In the cloudnative_postgresql: section, remove any database from the databases: list that is not needed. For instance:

...
databases:
- managementportal
# - appconfig
# - appserver
# - kratos
- restsourceauthorizer
# - uploadconnector
source:
...

Update production.yaml file

  1. Remove any line beginning with _chart_version:.

  2. If desired, add a section that changes the default storage size of the Postgresql cluster to be created:

cloudnative_postgresql:
cluster:
cluster:
storage:
size: 10Gi
  1. If using a TimescaleDB database, set legacy versions for Timescaledb in jdbc-connector sections. If desired, change the storage size of the respective database(s) that is in use:

Note: major version upgrades performed by the CloudNativePG operator are currently under development. When v1.26 is released, the Timescaledb databases can be upgraded to the a newer version.

radar_jdbc_connector_grafana:
timescaledb:
cluster:
# Do not remove: needed for legacy version. Can be removed after upgrade to new postgresql version (handled in future RADAR-base update).
version:
postgresql: "15"
Timescaledb: "2.11"
cluster:
storage:
# Change this to the desired size
size: 50Gi
radar_jdbc_connector_data_dashboard_backend:
timescaledb:
cluster:
# Do not remove: needed for legacy version. Can be removed after upgrade to new postgresql version (handled in future RADAR-base update).
version:
postgresql: "15"
Timescaledb: "2.11"
cluster:
storage:
size: 508Gi
radar_jdbc_connector_realtime_dashboard:
timescaledb:
cluster:
# Do not remove: needed for legacy version. Can be removed after upgrade to new postgresql version (handled in future RADAR-base update).
version:
postgresql: "15"
Timescaledb: "2.11"
cluster:
storage:
size: 50Gi
  1. Duplicate entry grafana_metrics_db_user ane rename to grafana_metrics_endpoint_user (keep the same value).

  2. Rename kratos: to radar_kratos: and add a kratos: indent to the radar_kratos: section like so:

radar_kratos:
...
kratos: <- new indent
kratos:
courier:
...
  1. Rename kratos_ui: to radar_self_enrolment_ui:.

Update production.yaml.gotmpl file

For the radar_grafana: section add a grafana: indent like so:

radar_grafana:
grafana: <- new indent
dashboards:
...

Update secrets.yaml file

  1. Duplicate secret grafana_metrics_db_password and rename to grafana_metrics_endpoint_password (keep the same value).
  2. Add the following secrets and replace secret with strong passwords:
radar_kratos:
secrets:
default: secret
cookie: secret
cipher: secret (must be 32 characters long)

radar_hydra:
secrets:
system: secret
cookie: secret

Database migration

Important: before database migration, the steps in the sections above must have been completed successfully.

Notes:

  • Database passwords can be found in the etc/secrets.yaml file.
  • Unless customized the username for all databases is postgres.

1. Disable services that write to the databases

To prevent any changes to the databases during the migration, disable all services that write to the databases ( see Disable database changes).

2. Manual import of upload-connector and/or app-server databases

App-server database import

Perform when using the app-server service. The username and password for the app-server database are indicated with <user> and <password>, respectively. The username and password for the management_portal database are indicated with <mp-user> and <mp-password>, respectively.

kubectl exec radar-appserver-postgresql-0 -- bash -c "PGPASSWORD=<password> pg_dump -U <user> appserver" > appserver.sql
kubectl exec -i postgresql-0 -- bash -c "PGPASSWORD=<mp-password> psql -U <mp-user> -t -c 'CREATE DATABASE appserver'"
cat appserver.sql | kubectl exec -i postgresql-0 -- bash -c "PGPASSWORD=<mp-password> psql -U <mp-user> -d appserver"
Upload-connector database import

Perform when using the upload-connector service. The username and password for the upload-connector database are indicated with <user> and <password>, respectively. The username and password for the management_portal database are indicated with <mp-user> and <mp-password>, respectively.

kubectl exec radar-upload-postgresql-0 -- bash -c "PGPASSWORD=<password> pg_dump -U <user> uploadconnector" > uploadconnector.sql
kubectl exec -i postgresql-0 -- bash -c "PGPASSWORD=<mp-password> psql -U <mp-user> -t -c 'CREATE DATABASE uploadconnector'"
cat uploadconnector.sql | kubectl exec -i postgresql-0 -- bash -c "PGPASSWORD=<mp-password> psql -U <mp-user> -d uploadconnector"

3. Create users in the postgres database

Log into the Postgresql database using the psql utility. The username and password for the Postgresql database are indicated with <user> and <password>, respectively. Note that the --pset pager=off option is used to disable paging in the output to prevent problems with certain statements.

kubectl exec postgresql-0 -it -- sh -c 'PGPASSWORD=<password> psql -U <user> --pset pager=off'

Create database users and set ownership of the databases (remove any database that is not needed):

CREATE USER managementportal;
CREATE USER restsourceauthorizer;
CREATE USER appconfig;
CREATE USER kratos;
CREATE USER appserver;
CREATE USER uploadconnector;
ALTER DATABASE managementportal OWNER to managementportal;
ALTER DATABASE restsourceauthorizer OWNER to restsourceauthorizer;
ALTER DATABASE appconfig OWNER to appconfig;
ALTER DATABASE kratos OWNER to kratos;
ALTER DATABASE appserver OWNER to appserver;
ALTER DATABASE uploadconnector OWNER to uploadconnector;

Transfer ownership of all tables in respective databases to the new users. Ignore sections of the command for any database that is not used.

\c managementportal
CREATE
OR REPLACE FUNCTION exec(text) returns text language plpgsql volatile
AS
$f$
BEGIN
EXECUTE $1;
RETURN $1;
END;
$f$;
SELECT exec ( 'ALTER TABLE ' || table_name || ' OWNER TO ' || table_catalog )
FROM information_schema.tables
WHERE table_schema = 'public';
SELECT exec ( 'ALTER SEQUENCE ' || sequence_name || ' OWNER TO ' || sequence_catalog )
FROM information_schema.sequences
WHERE sequence_schema = 'public';

\c restsourceauthorizer
CREATE
OR REPLACE FUNCTION exec(text) returns void language plpgsql volatile
AS
$f$
BEGIN
EXECUTE $1;
END;
$f$;
SELECT exec ( 'ALTER TABLE ' || table_name || ' OWNER TO ' || table_catalog )
FROM information_schema.tables
WHERE table_schema = 'public';
SELECT exec ( 'ALTER SEQUENCE ' || sequence_name || ' OWNER TO ' || sequence_catalog )
FROM information_schema.sequences
WHERE sequence_schema = 'public';

\c appconfig
CREATE
OR REPLACE FUNCTION exec(text) returns text language plpgsql volatile
AS
$f$
BEGIN
EXECUTE $1;
RETURN $1;
END;
$f$;
SELECT exec ( 'ALTER TABLE ' || table_name || ' OWNER TO ' || table_catalog )
FROM information_schema.tables
WHERE table_schema = 'public';
SELECT exec ( 'ALTER SEQUENCE ' || sequence_name || ' OWNER TO ' || sequence_catalog )
FROM information_schema.sequences
WHERE sequence_schema = 'public';

\c kratos
CREATE
OR REPLACE FUNCTION exec(text) returns text language plpgsql volatile
AS
$f$
BEGIN
EXECUTE $1;
RETURN $1;
END;
$f$;
SELECT exec ( 'ALTER TABLE ' || table_name || ' OWNER TO ' || table_catalog )
FROM information_schema.tables
WHERE table_schema = 'public';
SELECT exec ( 'ALTER SEQUENCE ' || sequence_name || ' OWNER TO ' || sequence_catalog )
FROM information_schema.sequences
WHERE sequence_schema = 'public';

\c appserver
CREATE
OR REPLACE FUNCTION exec(text) returns text language plpgsql volatile
AS
$f$
BEGIN
EXECUTE $1;
RETURN $1;
END;
$f$;
SELECT exec ( 'ALTER TABLE ' || table_name || ' OWNER TO ' || table_catalog )
FROM information_schema.tables
WHERE table_schema = 'public';
SELECT exec ( 'ALTER SEQUENCE ' || sequence_name || ' OWNER TO ' || sequence_catalog )
FROM information_schema.sequences
WHERE sequence_schema = 'public';

\c uploadconnector
CREATE
OR REPLACE FUNCTION exec(text) returns text language plpgsql volatile
AS
$f$
BEGIN
EXECUTE $1;
RETURN $1;
END ;
$f$;
SELECT exec ( 'ALTER TABLE ' || table_name || ' OWNER TO ' || table_catalog )
FROM information_schema.tables
WHERE table_schema = 'public';
SELECT exec ( 'ALTER SEQUENCE ' || sequence_name || ' OWNER TO ' || sequence_catalog )
FROM information_schema.sequences
WHERE sequence_schema = 'public';

4. Automated import of the Postgresql database

Update environments.yaml file

Add the mods/migration/1.3.0.yaml file to the values: section, like so:

environments:
default:
values:
- ../etc/base.yaml
- ../etc/production.yaml
- ../etc/production.yaml.gotmpl
- ../etc/secrets.yaml
- ../mods/migration/1.3.0.yaml

Start the database migration of management_portal and Timescaledb databases by using the auto-migration feature of the CloudNativePG operator. Run the helmfile sync command once with the mods/migration/1.3.0.yaml modification file. Omit any service that is not in use from the command below:

helmfile sync -lname=cloudnative-postgresql -lname=radar-jdbc-connector-grafana -lname=radar-jdbc-connector-data-dashboard -lname=radar-jdbc-connector-realtime-dashboard

Confirm that all database services initialize successfully.

5. Manual import of Timescaledb databases

For the grafana and realtime-dashboard databases, the migration is performed manually due to current problems with the migration of Timescaledb hypertables. The perform the following steps for any of these services when in use:

  1. attach shell to grafana-timescaledb pod:
kubectl exec grafana-timescaledb-1 -it -- bash

and export/import the database into the new grafana-timescale cluster:

export source=postgres://<user>:<password>@grafana-metrics-timescaledb-postgresql/grafana-metrics
pg_dump -d "$source" -fc -f /controller/grafana.bak
psql -d grafana -t -c 'select timescaledb_pre_restore();'
pg_restore -d grafana -fc /controller/grafana.bak
psql -d grafana -t -c 'select timescaledb_post_restore();'
  1. Attach shell to realtime-dashboard-timescaledb pod:
kubectl exec realtime-dashboard-timescaledb-1 -it -- bash

and export/import the database into the new realtime-dashboard timescaledb cluster:

export SOURCE=postgres://<user>:<password>@realtime-dashboard-timescaledb-postgresql/realtime-dashboard
pg_dump -d "$SOURCE" -Fc -f /controller/realtime-dashboard.bak
psql -d realtime-dashboard -t -c 'SELECT Timescaledb_pre_restore();'
pg_restore -d realtime-dashboard -Fc /controller/realtime-dashboard.bak
psql -d realtime-dashboard -t -c 'SELECT Timescaledb_post_restore();'

5. Re-enable services that write to the databases

Re-enable all services that were disabled in the Disable services that write to databases section above (see Disable database changes).

Redis migration

This migration replaces Redis standalone with an instance of Redis cluster (radar-redis).

In production.yaml set values for the desired PVC size of radar-redis. For instance:

radar_redis:
redis-cluster:
redisCluster:
clusterSize: 3
leader:
replicas: 1
follower:
replicas: 1
storageSpec:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 1Gi

2. Installation of Redis cluster

Once the correct values have been set, radar-redis can be installed in parallel together with redis-operator. For this run:

helmfile sync -lname=radar-redis name=redis-operator

3. Migration of Redis standalone data to cluster

The migration of the redis data to radar-redis is performed by running the following command in the shell of radar-redis-leader:

redis-cli --cluster import radar-redis-leader-headless:6379 --cluster-from redis-master:6379

4. Uninstall standalone Redis

Once you have confirmed a successful migration of the redis data to radar-redis, as well as the functionality of radar-redis you can safely uninstall redis.

helm uninstall redis

Post-migration cleanup

Perform these steps when the database migration is successful.

  1. Remove any database passwords from the secrets.yaml file. An easy way to do this is to compare your secrets.yaml file to base-secrets.yaml file and remove any entries not present in base-secrets.yaml.

  2. Turn off any deployed legacy database service. For this update the production.yaml file like so:

postgresql:
_install: false
...

radar_appserver_postgresql:
_install: false
...

data_dashboard_timescaledb:
_install: false
...

grafana_metrics_timescaledb:
_install: false
...

realtime_dashboard_timescaledb:
_install: false
...

radar_upload_postgresql:
_install: false
...

For esthetics, you can also remove any configuration passed under any of these sections. For instance, remove any of indicated lines:

grafana_metrics_timescaledb:
_install: true
_extra_timeout: 210
replicaCount: 1
postgresql: <-- remove this section
replication:
enable: false
applicationName: radarGrafanaMetrics
auth:
database: grafana-metrics
primary:
persistence:
size: 8Gi
  1. Remove the mods/migration/1.3.0.yaml file reference from the environments.yaml file.

  2. Update the deployment:

helmfile sync
  1. Remove any pvc resource on the Kubernetes cluster associated with the old databases.

    Important: this step will permanently delete the data! Only perform this step when sure the migration completed successfully.

    The relevant pvc names are:

  • data-postgresql-0
  • data-data-dashboard-timescaledb-postgresql-0
  • data-grafana-metrics-timescaledb-postgresql-0
  • data-radar-appserver-postgresql-0
  • data-radar-upload-postgresql-0
  • data-realtime-dashboard-timescaledb-postgresql-0

Upgrade to RADAR-Kubernetes version 1.2.0

Update production.yaml file

  1. Remove any line beginning with _chart_version:.
  2. Remove any line beginning with imageTag:.
  3. Add email server config to management_portal and radar_appserver sections analogous to:
management_portal:
smtp:
enabled: true
host: smtp
port: 25
from: noreply@example.com
starttls: false
auth: true
radar_appserver:
smtp:
enabled: true
host: smtp
port: 25
from: noreply@example.com
starttls: false
auth: true
  1. Update Timescaledb database configuration:
  • Rename timescaledb_username to grafana_metrics_db_username
  • Remove grafana_metrics_username and timescaledb_db_name variables.
  • When using realtime-dashboard, add realtime_dashboard_db_username that points to the current value of timescaledb_username.
  1. Refactor the Timescaledb database configuration:

Rename timescaledb: to grafana_metrics_timescaledb:.When using data-dashboard-backend or realtime-dashboard-backend, copy the grafana_metrics_timescaledb: entry to realtime_dashboard_timescaledb: and data_dashboard_timescaledb: similar to:

grafana_metrics_timescaledb:
_install: true
...
postgresql:
replication:
...

data_dashboard_timescaledb:
_install: true
...
postgresql:
replication:
...

realtime_dashboard_timescaledb:
_install: true
...
postgresql:
replication:
...

IMPORTANT: For databases where data should persist after the update uncomment the respective existingClaim field. For example:

grafana_metrics_timescaledb:
postgresql:
primary:
persistence:
existingClaim: "data-timescaledb-postgresql-0"
  1. Add postgresql: indent to Postgresql and Timescaledb related database configurations like so:

postgresql:
_install: true
...
postgresql:
replication:
...

grafana_metrics_timescaledb:
_install: true
...
postgresql:
replication:
...

data_dashboard_timescaledb:
_install: true
...
postgresql:
replication:
...

realtime_dashboard_timescaledb:
_install: true
...
postgresql:
replication:
...

radar_appserver_postgresql:
_install: true
...
postgresql:
replication:
...

radar_upload_postgresql:
_install: true
...
postgresql:
replication:
...
  1. Rename radar_jdbc_connector: to radar_jdbc_connector_grafana:.

  2. Add:

kratos:
_install: false
...

kratos_ui:
_install: false
...

Update secrets.yaml file

  1. Add the following new secrets to the secrets.yaml file to corresponding sections:
management_portal:
oauth_clients:
radar_data_dashboard_backend:
client_secret: <add your own random secret here>
radar_appserver:
smtp:
username: <your smtp username>
password: <your smtp password>
data_dashboard_db_password: <same password as Timescaledb_password>
realtime_dashboard_db_password: <same password as Timescaledb_password>
  1. Rename the grafana_metrics_password secret to grafana_metrics_db_password and timescaledb_password to data_dashboard_db_password.

  2. Add postgresql: indent to radar_appserver_postgresql: like so:

radar_appserver_postgresql:
postgresql:
global:
postgresql:
auth:
postgresPassword: <password>
auth:

MongoDB

MongoDB has been updated to a new version and it's not compatible with the current version that has been installed in the cluster. There are three paths forward:

  • Keeping the current version. Add legacy chart version to your production.yaml:
mongodb:
_install: true
# Keep the current version of MongoDB in order to be compatible with the stored data.
_chart_version: 11.1.10
  • Deleting the MongoDB and its volumes and then installing it and configuring Graylog again. This is the recommended approach since usually there is no important data is stored in MongoDB and the Graylog stack will be replaced in the next release.
  • Upgrading MongoDB cluster. If you have configured Graylog significantly, then it might be better to upgrade the MongoDB instead of reinstalling it. Options are:
    • Manually exporting the databases and doing a clean reinstall as stated in the last step and the importing the data again.
    • Following official MongoDB upgrade instructions for version 5.0, 6.0 and 7.0.

Upgrade to RADAR-Kubernetes version 1.1.x

Before running the upgrade make sure to copy environments.yaml.tmpl to environments.yaml and if you've previously changed environments.yaml apply the changes again. This is necessary due to addition of helmDefaults and repositories configurations to this file.

Upgrade to RADAR-Kubernetes version 1.0.0

Before running the upgrade, compare etc/base.yaml and etc/base.yaml.gotmpl with their production.yaml counterparts. Please ensure that all properties in etc/base.yaml are overridden in your production.yaml or that the base.yaml default value is fine, in which case no value needs to be provided in production.yaml.

To upgrade the initial services, run

kubectl delete -n monitoring deployments kube-prometheus-stack-kube-state-metrics
helm -n graylog uninstall mongodb
kubectl delete -n graylog pvc datadir-mongodb-0 datadir-mongodb-1

Note that this will remove your graylog settings but not your actual logs. This step is unfortunately needed to enable credentials on the Graylog database hosted by the mongodb chart. You will need to recreate the GELF TCP input source as during install.

Then run

helmfile -f helmfile.d/00-init.yaml apply --concurrency 1
helmfile -f helmfile.d/10-base.yaml --selector name=cert-manager-letsencrypt apply

To update the Kafka stack, run:

helmfile -f helmfile.d/10-base.yaml apply --concurrency 1

After this has succeeded, edit your production.yaml and change the cp_kafka.customEnv.KAFKA_INTER_BROKER_PROTOCOL_VERSION to the corresponding version documented in the Confluent upgrade instructions of your Kafka installation. Find the currently installed version of Kafka with kubectl exec cp-kafka-0 -c cp-kafka-broker -- kafka-topics --version. When the cp_kafka.customEnv.KAFKA_INTER_BROKER_PROTOCOL_VERSION is updated, again run

helmfile -f helmfile.d/10-base.yaml apply

To upgrade to the latest PostgreSQL helm chart, in production.yaml, uncomment the line postgresql.primary.persistence.existingClaim: "data-postgresql-postgresql-0" to use the same data storage as previously. Then run

kubectl delete secrets postgresql
kubectl delete statefulsets postgresql-postgresql
helmfile -f helmfile.d/10-managementportal.yaml apply

If installed, radar-appserver-postgresql, uncomment the production.yaml line radar_appserver_postgresql.primary.existingClaim: "data-radar-appserver-postgresql-postgresql-0". Then run

kubectl delete secrets radar-appserver-postgresql
kubectl delete statefulsets radar-appserver-postgresql-postgresql
helmfile -f helmfile.d/20-appserver.yaml apply

If installed, to upgrade TimescaleDB, uncomment the production.yaml line timescaledb.primary.existingClaim: "data-timescaledb-postgresql-0". Then run

kubectl delete secrets Timescaledb-postgresql
kubectl delete statefulsets Timescaledb-postgresql
helmfile -f helmfile.d/20-grafana.yaml apply

If installed, to upgrade radar-upload-postgresql, uncomment the production.yaml line radar_upload_postgresql.primary.existingClaim: "data-radar-upload-postgresql-postgresql-0". Then run

kubectl delete secrets radar-upload-postgresql
kubectl delete statefulsets radar-upload-postgresql-postgresql
helmfile -f helmfile.d/20-upload.yaml apply

If minio is installed, upgrade it with the following instructions:

# get minio PV and PVC
kubectl get pv | grep export-minio- | tr -s ' ' | cut -d ' ' -f 1,6 | tr '/' ' ' | cut -d ' ' -f 1,3 | tee minio-pv.list
# Uninstall the minio statefulset
helm uninstall minio
# Associate PV with the new PVC name
while read -r pv pvc
do
# Don not delete PV
kubectl patch pv $pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
# Delete PVC
kubectl delete pvc $pvc
# Name of the new PVC
newpvc=$(echo $pvc | sed 's/export-/data-/')
# Associate PV with the new PVC name
kubectl patch pv $pv -p '{"spec":{"claimRef":{"name": "'$newpvc'", "namespace": "default", "uid": null}}}'
# Create new PVC
cat <<EOF | sed "s/data-minio-i/$newpvc/" | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app.kubernetes.io/instance: minio
app.kubernetes.io/name: minio
name: data-minio-i
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
EOF
done < minio-pv.list
# Do the new helm install.
helmfile -f helmfile.d/20-s3.yaml apply

Delete the redis stateful set (this will not delete the data on the volume)

kubectl delete statefulset redis-master
helmfile -f helmfile.d/20-s3.yaml sync --concurrency 1

Supporting tasks

Disable data ingestion

To disable data ingestion, temporarily disable gateway, rest-connector (Fitit, Garmin, Oura, ...) and upload-connector services in the production.yaml file: This is accomplished by redeploying while setting the replicaCount to 0 for these services. For instance:

radar_gateway:
replicaCount: 0
...
radar_fitbit_connector:
replicaCount: 0
...
radar_oura_connector:
replicaCount: 0
...
radar_upload_source_connector:
replicaCount: 0
...

Followed by:

helmfile sync

BEWARE: the list of services that is scaled down in this way may vary depending on the services that are used in your RADAR-Kubernetes deployment.The example above is not exhaustive and will not be updated with future versions of RADAR-Kubernetes.

In order to re-enable data ingestion, set the replicaCount back to the original value and redeploy.

Disable database changes

In prevent databases from receiving any updates (e.g., when performing an off-line database update), we should disable all services that perform write operations. This is accomplished by redeploying while setting the replicaCount to 0 for these services. At the moment of this writing this would include:

management_portal:
replicaCount: 0
...
radar_appserver:
replicaCount: 0
...
radar_appconfig:
replicaCount: 0
...
radar_rest_sources_backend:
replicaCount: 0
...
radar_upload_connect_backend:
replicaCount: 0
...
kratos:
replicaCount: 0
...
hydra:
replicaCount: 0
...
radar_jdbc_connector_grafana:
replicaCount: 0
...
radar_jdbc_connector_data_dashboard_backend:
replicaCount: 0
...
radar_jdbc_connector_realtime_dashboard:
replicaCount: 0
...

followed by:

helmfile sync

BEWARE: the list of services that is scaled down in this way may vary depending on the services that are used in your RADAR-Kubernetes deployment.The example above is not exhaustive and will not be updated with future versions of RADAR-Kubernetes.

In order to re-enable services, set the replicaCount back to the original value and redeploy.