OpenShift Container Platform Installation
One agent needs to be deployed on each OpenShift node to monitor all the containers and services in the cluster.
The agent will automatically collect all labels applied to containers running on OpenShift. You can use these labels as selectors to run plugins and schedule checks against services running in your cluster.
To deploy the Outlyer agent on OpenShift Container Platform, you will need to install oc and log in to your OpenShift cluster as admin or any user who has the
First, create a secret in your cluster, in which to store your Agent key. The outlyer-agent container will be deployed into the
oc create secret generic outlyer-agent-key --namespace=kube-system --from-literal=accountKey='<PUT YOUR ACCOUNT AGENT KEY HERE>'
Because Kubernetes doesn’t provide a cluster name by default, and you could potentially add multiple OpenShift clusters to your
Outlyer account, you will need to set a cluster name in a configmap that the agent daemonset can read to ensure all your instances
and metrics can be distinguished in your account using the
oc create configmap outlyer-config --namespace=kube-system --from-literal=k8s.cluster='<PUT A UNIQUE CLUSTER NAME HERE, i.e. openshift.prod.acme.com>'
Next, apply the RBAC (Role Based Access Control) permissions required for the Outlyer Agent to run:
oc apply -f https://www.outlyer.com/assets/files/rbac.yaml
Create a secret containing the Red Hat Customer Portal credentials:
oc create secret docker-registry rh-container-catalog \ --docker-server=registry.connect.redhat.com \ --docker-username=<USERNAME HERE> \ --docker-password=<PASSWORD HERE> \ --docker-email=<EMAIL HERE> \ --namespace=kube-system
Associate this secret with the outlyer-agent service account:
oc secrets link outlyer-agent rh-container-catalog --for=pull --namespace=kube-system
Apply the DaemonSet to launch Outlyer across all nodes:
oc apply -f https://www.outlyer.com/assets/files/daemonset-openshift.yaml
To confirm the DaemonSet is deployed properly:
oc get pods --namespace=kube-system | grep outlyer-agent
You will see one running for each schedulable node you have in your cluster. Make sure your OpenShift master and infra nodes configuration allow the running of Outlyer pods.
kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. The default Kubernetes dashboards for Outlyer use the metrics from kube-state-metrics to give an overview of the cluster. You will need to deploy kube-state-metrics on your Kubernetes cluster following the instructions here:
git clone https://github.com/kubernetes/kube-state-metrics.git cd kube-state-metrics oc apply -f kubernetes
Once successfully deployed, you will have access to a lot of useful metrics that can be used to see the state of your cluster.
Editing Outlyer DaemonSet and RBAC Configuration
The configuration for the Outlyer agent DaemonSet and RBAC can be downloaded from here for local changes:
curl -O https://www.outlyer.com/assets/files/rbac.yaml curl -O https://www.outlyer.com/assets/files/daemonset-openshift.yaml
The DaemonSet manifest daemonset-openshift.yaml can be edited to add additional deployment configuration options:
- If you want to monitor Java services via JMX you will need to deploy a different Docker image that has Java installed alongside the Outlyer agent. This is included seperately as Java increases the image size so should only be downloaded to all your hosts if needed. To pull the Java version of the Docker image edit the daemonset-openshift.yaml file container image to add ‘-java’ at the end of the image tag. i.e. ‘registry.connect.redhat.com/outlyer/agent2:1.4.6-java’
- You can add additional environment variables to pass to the agent configuration when it starts up. For full details of what environment settings are available, please refer to our docker instructions.
- You can add nodeSelectors to only run the Daemonset on specific nodes.
Role-Based Access Control (“RBAC”)
Role-Based Access Control (“RBAC”) uses the “rbac.authorization.k8s.io” API group to drive authorization decisions, allowing admins to dynamically configure policies through the Kubernetes API. You can read more about using RBAC here.
To simplify deployment of the Outlyer DaemonSet, a service account with cluster role policies is included alongside the Daemonset manifest and deployed automatically when running the instructions above. If your cluster doesn’t have RBAC enabled, these additional manifests will not affect the running of the Outlyer DaemonSet.
Updating and Upgrading the DaemonSet
If you want to deploy a newer version of the agent, you can update the container image version, or apply new settings to the DaemonSet. Once your manifest is updated with the changes you can use the following command to update the DaemonSet on Kubernetes:
oc apply -f <path-to>/daemonset.yaml
Or to simply deploy a newer image version to the DaemonSet:
oc --namespace=kube-system set image ds/outlyer-agent outlyer-agent=registry.connect.redhat.com/outlyer/agent2:1.4.6
If you do not see your OpenShift cluster Nodes appear in your account’s Host list after running the above instructions successfully you can check if the DaemonSet is running using the following command:
oc get daemonsets --namespace=kube-system
This will return a message like below if the DaemonSet is successfully deployed and running on a 5 Node cluster:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE outlyer-agent 5 5 5 5 5 <none> 4s
If you see any of the numbers are not the same as expected this should indicate there was an issue deploying the DaemonSet across the cluster. You will then need to list the Kubernetes Pods and get the logs for the outlyer-agent pods via oc to see exactly what the issue was:
$ oc get pods --namespace=kube-system NAME READY STATUS RESTARTS AGE outlyer-agent-c2lqg 1/1 Running 0 10m $ oc logs outlyer-agent-c2lqg --namespace=kube-system 2018-03-02 23:22:47 INFO apscheduler.executors.default - Running job "local_sync (trigger: interval[0:00:10], next run at: 2018-03-02 23:22:57 UTC)" (scheduled at 2018-03-02 23:22:47.591177+00:00) ......
Removing the Outlyer Agent
You can remove the Outlyer agent with the following commands:
oc delete --namespace=kube-system daemonset outlyer-agent oc delete --namespace=kube-system serviceaccount outlyer-agent oc delete --namespace=kube-system clusterrole outlyer-agent oc delete --namespace=kube-system secret outlyer-agent-key rh-container-catalog oc delete --namespace=kube-system configmap outlyer-config