Procedure

1. Edit the cluster-monitoring-config ConfigMap:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Put your PVC configuration for the component under data/config.yaml :

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       <component>:
         volumeClaimTemplate:
           metadata:
             name: <PVC_name_prefix>
           spec:
             storageClassName: <storage_class>
             resources:
               requests:
                 storage: <amount_of_storage>

   See the Kubernetes documentation on PersistentVolumeClaims for information on how to specify volumeClaimTemplate . For example, to configure a PVC that claims local persistent storage for Prometheus, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         volumeClaimTemplate:
           metadata:
             name: localpvc
           spec:
             storageClassName: local-storage
             resources:
               requests:
                 storage: 40Gi

   In the above example, the storage class created by the Local Storage Operator is called local-storage . To configure a PVC that claims local persistent storage for Alertmanager, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       alertmanagerMain:
         volumeClaimTemplate:
           metadata:
             name: localpvc
           spec:
             storageClassName: local-storage
             resources:
               requests:
                 storage: 40Gi

3. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically and the new storage configuration is applied.

Procedure

1. Start editing the cluster-monitoring-config ConfigMap:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Put your retention time configuration under data/config.yaml :

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         retention: <time_specification>

   Substitute <time_specification> with a number directly followed by ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks), or y (years). For example, to configure retention time to be 24 hours, use:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         retention: 24h

3. Save the file to apply the changes. The pods affected by the new configuration are restarted automatically.

Procedure

1. Print the currently active Alertmanager configuration into file alertmanager.yaml :

   $ oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' |base64 -d > alertmanager.yaml

2. Change the configuration in file alertmanager.yaml to your new configuration:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: default
     routes:
     - match:
         alertname: Watchdog
       repeat_interval: 5m
       receiver: watchdog
     - match:
         service: <your_service> 1
       routes:
       - match:
           <your_matching_rules> 2
         receiver: <receiver> 3
   receivers:
   - name: default
   - name: watchdog
   - name: <receiver>
     <receiver_configuration>

   1

   service specifies the service that fires the alerts. <your_matching_rules> specify the target alerts. receiver specifies the receiver to use for the alert. For example, this listing configures PagerDuty for notifications:

   global:
     resolve_timeout: 5m
   route:
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: default
     routes:
     - match:
         alertname: Watchdog
       repeat_interval: 5m
       receiver: watchdog
     - match: service: example-app routes: - match: severity: critical receiver: team-frontend-page
   receivers:
   - name: default
   - name: watchdog
   - name: team-frontend-page pagerduty_configs: - service_key: "your-key"

   With this configuration, alerts of critical severity fired by the example-app service are sent using the team-frontend-page receiver, which means that these alerts are paged to a chosen person. Apply the new configuration in the file:

   $ oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run -o=yaml |  oc -n openshift-monitoring replace secret --filename=-

Procedure

1. Configure the necessary port forwarding:

   $ oc -n openshift-monitoring port-forward svc/prometheus-operated 9090

2. Fetch the JSON object containing acting alerting rules and their properties:

   $ curl -s http://localhost:9090/api/v1/rules | jq '[.data.groups[].rules[] | select(.type=="alerting")]'
       "name": "ClusterOperatorDown",
       "query": "cluster_operator_up{job=\"cluster-version-operator\"} == 0",
       "duration": 600,
       "labels": {
         "severity": "critical"
       "annotations": {
         "message": "Cluster operator {{ $labels.name }} has not been available for 10 mins. Operator may be down or disabled, cluster will not be kept up to date and upgrades will not be possible."
       "alerts": [],
       "health": "ok",
       "type": "alerting"
       "name": "ClusterOperatorDegraded",
       ...

Procedure

1. Open the OpenShift Container Platform web console and navigate to the Monitoring → Alerting → Alerts page. Optional: Filter the alerts by name using the Filter Alerts by name field. Optional: Filter the alerts by state using one or more of the state buttons Firing , Silenced , Pending , Not firing . Optional: Sort the alerts by clicking one or more of the Name , State , and Severity column headers. After you see the alert, you can see either details of the alert or details of its governing alerting rule. To see alert details, click on the name of the alert. This is the page with alert details: The page has the graph with timeseries of the alert. It also has information about the alert, including: A link to its governing alerting rule Description of the alert To see alerting rule details, click the button in the last column and select View Alerting Rule . This is the page with alerting rule details: The page has information about the alerting rule, including: Alerting rule name, severity, and description The expression that defines the condition for firing the alert The time for which the condition should be true for an alert to fire Graph for each alert governed by the alerting rule, showing the value with which the alert is firing Table of all alerts governed by the alerting rule

Procedure

1. Open the OpenShift Container Platform web console and navigate to the Monitoring → Alerting → Silences page. Optional: Filter the silences by name using the Filter Silences by name field. Optional: Filter the silences by state using one or more of the state buttons Active , Pending , Expired . Optional: Sort the silences by clicking one or more of the Name , State , and Firing alerts column headers. After you see the silence, you can click its name to see the details, including: Alert specification State Start time End time Number and list of firing alerts

Procedure

1. Navigate to the Monitoring → Alerting → Silences page. For the silence you want to modify, click the button in the last column and click Edit silence . Alternatively, you can click Actions → Edit Silence in the Silence Overview screen for a particular silence. In the Edit Silence screen, enter your changes and click the Save button. This will expire the existing silence and create one with the chosen configuration.

Procedure

1. Navigate to the Monitoring → Alerting → Silences page. For the silence you want to expire, click the button in the last column and click Expire Silence . Alternatively, you can click the Actions → Expire Silence button in the Silence Overview page for a particular silence. Confirm by clicking Expire Silence . This expires the silence.

Procedure

1. Open the OpenShift Container Platform web console and navigate to the Monitoring → Metrics page. In the query field, enter your PromQL query. To show all available metrics and PromQL functions, click Insert Metric at Cursor . For multiple queries, click Add Query . For deleting queries, click for the query, then select Delete query . For keeping but not running a query, click the Disable query button. Once you finish creating queries, click the Run Queries button. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message. Queries that operate on large amounts of data might timeout or overload the browser when drawing timeseries graphs. To avoid this, hide the graph and calibrate your query using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs. Optional: The page URL now contains the queries you ran. To use this set of queries again in the future, save this URL.

Procedure

1. Initially, all metrics from all enabled queries are shown on the plot. You can select which metrics are shown. To hide all metrics from a query, click for the query and click Hide all series . To hide a specific metric, go to the query table and click the colored square near the metric name. To zoom into the plot and change the shown time range, do one of the following: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu in the left upper corner to select the time range. To reset the time range, click Reset Zoom . To display outputs of all queries at a specific point in time, hold the mouse cursor on the plot at that point. The query outputs will appear in a pop-up box. For more detailed information about metrics of a specific query, expand the table of that query using the drop-down button. Every metric is shown with its current value. To hide the plot, click Hide Graph .

Procedure

1. Navigate to the OpenShift Container Platform web console and authenticate. To access Prometheus, navigate to the "Monitoring" → "Metrics" page. To access the Alerting UI, navigate to the "Monitoring" → "Alerting" page. To access Grafana, navigate to the "Monitoring" → "Dashboards" page.

Procedure

1. $ oc -n openshift-monitoring get routes
   NAME                HOST/PORT                                                     ...
   alertmanager-main   alertmanager-main-openshift-monitoring.apps._url_.openshift.com ...
   grafana             grafana-openshift-monitoring.apps._url_.openshift.com           ...
   prometheus-k8s      prometheus-k8s-openshift-monitoring.apps._url_.openshift.com    ...

2. Prepend https:// to the address, you cannot access web UIs using unencrypted connection. For example, this is the resulting URL for Alertmanager:

   https://alertmanager-main-openshift-monitoring.apps._url_.openshift.com

3. Navigate to the address using a web browser and authenticate.