Knowledgebase

Introduction

MongoDB is an open-source database program. Classified as a NoSQL database program, the data stored in MongoDB is in JSON-like format. Instead of tables and rows, it stores data in collections of documents. Each collection contains related documents, and each document is a JSON-like object. The data model of MongoDB provides the flexibility in storing data with an optional schema.

The MongoDB Kubernetes Operator is an extension for Kubernetes. It allows deployment and management of MongoDB server on a Kubernetes cluster. It supports the deployment of replica sets in which a group of MongoDB servers stores the same data providing redundancy and high availability for production usage.

This guide walks you through the deployment of a MongoDB replica set on RCS Kubernetes Engine (VKE) using the MongoDB Kubernetes Operator, securing the deployment with a TLS certificate generated using cert-manager, and deploying RCS Load Balancers for external access.

Prerequisites

• Deploy a RCS Ubuntu 20.04 instance to use as a management workstation.
• Deploy a Kubernetes cluster at RCS with at least three or more nodes with 2GB RAM and 2 CPUs.

On the management workstation:

• Install kubectl.
• Download your VKE configuration and configure kubectl.
• Install RCS Container Storage Interface.
• Install RCS Cloud Controller Manager.

You'll perform the rest of the steps in this guide from your management workstation.

Install the MongoDB Kubernetes Operator

The MongoDB Kubernetes Operator is a custom resource definition and a controller that simplifies the MongoDB server's deployment and management. This section explains the steps to install the custom resource definition, the role bindings, and the operator.

Clone the MongoDB Kubernetes Operator repository.

# git clone https://github.com/mongodb/mongodb-kubernetes-operator.git

Switch to the repository directory.

# cd mongodb-kubernetes-operator

Install the custom resource definition.

# kubectl apply -f config/crd/bases/mongodbcommunity.mongodb.com_mongodbcommunity.yaml

The above command installs the MongoDBCommunity resource type in the cluster. This resource allows the deployment and scaling of the MongoDB replica set. You use this resource for defining the configuration of the MongoDB replica set.

Create a new namespace.

# kubectl create namespace mongodb

A namespace isolates the resources related to a single application in the cluster. The above command creates a new namespace named mongodb that you use in the next steps to contain all the resources related to the MongoDB replica set.

Install the necessary roles and role bindings.

# kubectl apply -k config/rbac -n mongodb

The above command installs a specific set of permissions required by the MongoDB Kubernetes Operator to interact with the resources in the cluster. The -n flag specifies the namespace to apply the changes.

Install the MongoDB Kubernetes Operator.

# kubectl apply -f config/manager/manager.yaml -n mongodb

The above command installs the MongoDB Kubernetes Operator in the cluster. It creates a new pod in the mongodb namespace and watches for state changes related to the MongoDBCommunity resource type.

Verify the installation.

# kubectl get pods -n mongodb

Output.

NAME                                           READY   STATUS    RESTARTS   AGE
mongodb-kubernetes-operator-648bf8cc59-mc5sm   1/1     Running   0          10s

Install the cert-manager Add-On

The cert-manager add-on for Kubernetes provides custom resource definitions that allow issuing TLS certificates. It installs three custom resources in the cluster: Issuer, ClusterIssuer, and Certificate. The Certificate resource requires an issuer parent resource, and it generates a Kubernetes secret which contains a pair of the TLS certificate and the private key.

Install the cert-manager add-on.

# kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml

The above command installs all the Kubernetes resources provided by the cert-manager add-on in the cluster.

Create a new configuration directory.

# mkdir ~/mongodb-replica-set

Switch to the configuration directory.

# cd ~/mongodb-replica-set

This guide uses a self-signed TLS certificate to secure the MongoDB replica set. A valid TLS certificate requires a certificate authority signature. To sign the TLS certificate for the MongoDB replica set, you create an issuer that generates a self-signed certificate authority.

Using a text editor, create a new file named certificate.yaml.

# nano certificate.yaml

Add the following contents to the file.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: mongodb-ca-issuer
spec:
  selfSigned: {}

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mongodb-ca-cert
spec:
  isCA: true
  commonName: mongodb-ca-cert
  secretName: mongodb-ca-cert-key-pair
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: mongodb-ca-issuer
    kind: Issuer

---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: mongodb-issuer
spec:
  ca:
    secretName: mongodb-ca-cert-key-pair

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: mongodb-cert
spec:
  commonName: '*.mongodb-svc.mongodb.svc.cluster.local'
  secretName: mongodb-cert-key-pair
  dnsNames:
  - '*.mongodb-svc.mongodb.svc.cluster.local'
  - 'mongodb-0.example.com'
  - 'mongodb-1.example.com'
  - 'mongodb-2.example.com'
  issuerRef:
    name: mongodb-issuer
    kind: Issuer

The above configuration creates four resources: the mongodb-ca-issuer Issuer resource allows the creation of the mongodb-ca-cert Certificate resource, which is a self-signed certificate authority. It uses the self-signed certificate authority to create the mongodb-issuer Issuer resource, which allows creating the mongodb-cert Certificate resource, which you use to secure the MongoDB replica set.

The commonName and the dnsName attributes in the mongodb-cert Certificate resource contain the certificate's hostnames. The commonName attribute is a wildcard followed by the syntax of the hostname used by the MongoDB Kubernetes Operator to provision the servers locally. The syntax of the hostname is as follows:

<metadata.name>-<member-number>.<metadata.name>-svc.<namespace>.svc.cluster.local

The <metadata.name> is the name of the MongoDBCommunity resource. In the next steps, you build the configuration for the MongoDB replica set using mongodb as the metadata name and the number of MongoDB replica set members you want to spawn in the cluster.

Replace and add a list of additional hostnames for external access in the dnsName attribute according to the number of members you want to spawn. You use these additional hostnames as replica set horizons in the next steps.

Apply the configuration.

# kubectl apply -f certificate.yaml -n mongodb

Verify the certificate state.

# kubectl get certificate -n mongodb

Output.

NAME              READY   SECRET                     AGE
mongodb-ca-cert   True    mongodb-ca-cert-key-pair   11s
mongodb-cert      True    mongodb-cert-key-pair      10s

Deploy MongoDB Replica Set

A MongoDB replica set is a group of two or more instances of the MongoDB server that store the same data to enable redundancy and high availability for production usage. The MongoDB Kubernetes Operator enables you to deploy a MongoDB replica set on your VKE cluster using a single declarative configuration file instead of deploying and configuring multiple servers. This section explains the configuration and deployment of a MongoDB replica set using the MongoDB Kubernetes Operator.

Create a new file named replica-set.yaml.

# nano replica-set.yaml

Add the following contents to the file.

apiVersion: v1
kind: Secret
metadata:
  name: demo-admin-password
type: Opaque
stringData:
  password: 'your_password'

---
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: mongodb
spec:
  members: 3
  type: ReplicaSet
  version: "5.0.5"
  replicaSetHorizons:
  - horizon: mongodb-0.example.com:27017
  - horizon: mongodb-1.example.com:27017
  - horizon: mongodb-2.example.com:27017
  security:
    authentication:
      modes: ["SCRAM"]
    tls:
      enabled: true
      certificateKeySecretRef:
        name: mongodb-cert-key-pair
      caCertificateSecretRef:
        name: mongodb-cert-key-pair
  users:
    - name: demo-admin
      db: admin
      passwordSecretRef:
        name: demo-admin-password
      roles:
        - name: clusterAdmin
          db: admin
        - name: userAdminAnyDatabase
          db: admin
      scramCredentialsSecretName: my-scram
  statefulSet:
      spec:
        volumeClaimTemplates:
          - metadata:
              name: data-volume
            spec:
              accessModes: ["ReadWriteOnce"]
              storageClassName: RCS-block-storage
              resources:
                requests:
                  storage: 20Gi
          - metadata:
              name: logs-volume
            spec:
              accessModes: ["ReadWriteOnce"]
              storageClassName: RCS-block-storage-hdd
              resources:
                requests:
                  storage: 40Gi

The above configuration creates a MongoDBCommunity resource in the cluster and a Kubernetes secret containing a password for the administrator user. When you apply the configuration, the MongoDB Kubernetes Operator creates a Kubernetes StatefulSet to provision the MongoDB replica set. Each replica set member is a pod with two containers: a container for the mongod process binary and a container for the MongoDB agent, which handles the provisioning of the mongod process. Each pod consists of 2 volumes: a volume for storing the database and a volume for storing logs.

The following are the highlights of the configuration:

• metadata.name - The name of the MongoDBCommunity resource, used as prefix of related resources.

• spec.version - The version of MongoDB community server installed in the replica set.

• spec.members - The number of replica set members.

• spec.replicaSetHorizons - The DNS hostnames used for external access. They should match the number of replica set members. Ensure that the TLS certificate includes these hostnames.

• spec.security.tls - Enable TLS security and define the Kubernetes secret created by cert-manager containing the TLS certificate key-pair.

• spec.users - The name of the administrator user, the Kubernetes secret referring to the administrator user's password, and the name of the collection for storing authentication information.

• spec.statefulSet.spec.volumeClaimTemplates - The storage class and the storage size for volumes created for each member in the replica set. The above configuration uses the storage classes defined by RCS Container Storage Interface (CSI) to provision RCS Block Storage for each volume. Refer to the RCS Container Storage Interface documentation for more information.

If you want to set non-default values for the Kubernetes StatefulSet deployed by the MongoDB Kubernetes Operator, you can use the spec.statefulSet attribute. It provides direct access to the Kubernetes Staefulset configuration. Refer to the configuration samples for more information.

Apply the configuration.

# kubectl apply -f replica-set.yaml -n mongodb

Follow the replica set deployment progress.

# watch -n 0.1 kubectl get pods -n mongodb

It takes around 5 to 10 minutes to finish the deployment.

Output.

Every 0.1s: kubectl get pods -n mongodb

NAME                                           READY   STATUS    RESTARTS   AGE
mongodb-0                                      2/2     Running   0          3m6s
mongodb-1                                      2/2     Running   0          102s
mongodb-2                                      2/2     Running   0          55s
mongodb-kubernetes-operator-648bf8cc59-mc5sm   1/1     Running   0          10m

Verify the deployment.

# kubectl get mongodbcommunity -n mongodb

Output.

NAME      PHASE     VERSION
mongodb   Running   5.0.5

If the deployment stops making any progress, you can use the following commands to troubleshoot and find the issue causing the delay.

# kubectl describe mongodbcommunity mongo -n mongodb
# kubectl describe pod mongo-0 -n mongodb
# kubectl get pvc -n mongodb
# kubectl get pv -n mongodb

The following are the issues you might face during the deployment of the replica set:

• Not having enough CPU.
• Not having enough Memory.
• RCS Block Storage limit exceeded.

Ensure that your VKE cluster has enough resources available and you do not exceed the total number of RCS Block Storage volumes limit.

Access MongoDB Shell

You can fetch the connection string and the credentials to connect to the MongoDB replica set inside the cluster with the credentials stored by the MongoDB Kubernetes Operator in the Kubernetes secrets.

This is the syntax for the name of the Kubernetes secrets resource that contains the credentials:

<metadata.name>-<auth-db>-<username>

Fetch the connection string.

# kubectl get secret mongodb-admin-demo-admin -n mongodb -o json

The above command returns JSON response with BASE64 encoded values.

Install the jq package.

# apt install jq

The above package lets you decode the BASE64 values in the JSON response.

Fetch the usable connection string.

# kubectl get secret mongodb-admin-demo-admin -n mongodb -o json | jq -r '.data | with_entries(.value |= @base64d)'

Output.

{
  "connectionString.standard": "mongodb://demo-admin:your_password@mongodb-0.mongodb-svc.mongodb.svc.cluster.local:27017,mongodb-1.mongodb-svc.mongodb.svc.cluster.local:27017,mongodb-2.mongodb-svc.mongodb.svc.cluster.local:27017/admin?ssl=true",
  "connectionString.standardSrv": "mongodb+srv://demo-admin:your_password@mongodb-svc.mongodb.svc.cluster.local/admin?ssl=true",
  "password": "your_password",
  "username": "demo-admin"
}

You use the connectionString.standardSrv value and the TLS certificate to connect to the MongoDB replica set inside the cluster.

Enter the mongod container.

# kubectl exec -it mongodb-0 -c mongod -n mongodb -- /bin/bash

Connect to the MongoDB replica set using mongosh.

$ mongosh --tls --tlsCAFile /var/lib/tls/ca/*.pem --tlsCertificateKeyFile /var/lib/tls/server/*.pem "mongodb+srv://demo-admin:your_password@mongodb-svc.mongodb.svc.cluster.local/admin?ssl=true"

Fetch all databases.

mongodb [primary] admin> show dbs

Output.

admin   176 kB
config  172 kB
local   545 kB

Exit the mongosh session.

mongodb [primary] admin> quit()

Exit the mongod container.

$ exit

Fetch the TLS Certificate

You secured the MongoDB replica set deployed in the cluster using the TLS certificate key pair. You need that certificate and the private key to connect to the MongoDB replica set. Every pod provisioned by the MongoDB Kubernetes Operator contains the certificate and the private key in /var/lib/tls in the mongod container.

You can manually enter the mongod container of the first pod using kubectl exec -it mongodb-0 -c mongod -n mongodb -- /bin/bash to copy and paste the certificate and the private key files or use the following shortcuts to fetch them.

Fetch the certificate file.

# kubectl exec mongodb-0 -c mongod -n mongodb -- /bin/bash -c "cat /var/lib/tls/ca/*.pem" > crt.pem

The above command creates a file named crt.pem in the working directory containing the certificate contents.

Fetch the private key file.

# kubectl exec mongodb-0 -c mongod -n mongodb -- /bin/bash -c "cat /var/lib/tls/server/*.pem" > key.pem

The above command creates a file named key.pem in the working directory containing the private key contents.

Set Up External Access

The RCS Cloud Controller Manager provides the ability to provision RCS Load Balancer with Kubernetes. It binds the LoadBalancer service type with the RCS Load Balancer service. When you create a new LoadBalancer service, the RCS Cloud Controller Manager provisions a new RCS Load Balancer in your account and configures it according to the object specifications and the annotations. This section explains how to set up external access for the MongoDB replica using load balancers and DNS records.

Create a new file named external-access.yaml.

# nano external-access.yaml

Add the following contents to the file.

apiVersion: v1
kind: Service
metadata:
  name: mongodb-lb-0
  namespace: mongodb
  annotations:
    service.beta.kubernetes.io/RCS-loadbalancer-protocol: "tcp"
spec:
  type: LoadBalancer
  ports:
  - name: mongodb
    port: 27017
    protocol: TCP
  selector:
    app: mongodb-svc
    statefulset.kubernetes.io/pod-name: mongodb-0

---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-lb-1
  namespace: mongodb
  annotations:
    service.beta.kubernetes.io/RCS-loadbalancer-protocol: "tcp"
spec:
  type: LoadBalancer
  ports:
  - name: mongodb
    port: 27017
    protocol: TCP
  selector:
    app: mongodb-svc
    statefulset.kubernetes.io/pod-name: mongodb-1

---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-lb-2
  namespace: mongodb
  annotations:
    service.beta.kubernetes.io/RCS-loadbalancer-protocol: "tcp"
spec:
  type: LoadBalancer
  ports:
  - name: mongodb
    port: 27017
    protocol: TCP
  selector:
    app: mongodb-svc
    statefulset.kubernetes.io/pod-name: mongodb-2

The above configuration creates three LoadBalancer service objects in the cluster. You can adjust the number of load balancers according to the number of members in the replica set. Each LoadBalancer service object provisions a RCS Load Balancer, which forwards TCP traffic on port 27017 (default MongoDB port) to individual members in the replica set.

Apply the configuration.

# kubectl apply -f external-access.yaml -n mongodb

Follow the load balancer deployment progress.

# watch -n 0.1 kubectl get service -n mongodb

It takes around 5 minutes to finish the deployment.

Output.

NAME              TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)           AGE
mongodb-arb-svc   ClusterIP      None             <none>            27017/TCP         155m
mongodb-lb-0      LoadBalancer   10.110.133.138   104.207.153.207   27017:31808/TCP   2m7s
mongodb-lb-1      LoadBalancer   10.98.170.249    45.32.70.189      27017:32151/TCP   2m7s
mongodb-lb-2      LoadBalancer   10.97.161.16     66.42.101.239     27017:30036/TCP   2m7s
mongodb-svc       ClusterIP      None             <none>            27017/TCP         155m

Configure the following DNS records in your domain to set up the A records and the Service (SRV) records for each load balancer.

Verify DNS records.

# nslookup mongodb-0.example.com
# nslookup mongodb-1.example.com
# nslookup mongodb-2.example.com
# nslookup -q=SRV _mongodb._tcp.mongodb.example.com

The following is the connection string for external access.

mongodb+srv://demo-admin:your_password@mongodb.example.com/admin

You must pass the TLS certificate fetched in the previous section along with the connection string.

Verify the connection using the MongoDB Shell tool on your local system.

mongosh --tls --tlsCAFile /path/to/certificate/file --tlsCertificateKeyFile /path/to/private_key/file "mongodb+srv://demo-admin:your_password@mongodb.example.com/admin"

Output.

Current Mongosh Log ID: 62a71d81196f9e1982eefd7a
Connecting to:          mongodb+srv://<credentials>@mongodb.example.com/admin?appName=mongosh+1.3.1
Using MongoDB:          5.0.5
Using Mongosh:          1.3.1

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

mongodb [primary] admin>

Conclusion

You deployed a MongoDB replica set on a RCS Kubernetes Engine (VKE) cluster, which stores data on the RCS Block Storage service using the MongoDB Kubernetes Operator. You secured the replica set using a TLS certificate created using the cert-manager add-on. You used the RCS Load Balancer service for setting up external access to the MongoDB replica set.

More Information

• RCS Kubernetes Engine
• MongoDB Kubernetes Operator
• RCS Container Storage Interface
• RCS Cloud Controller Manager