Ubuntu 14.04 - Deploy a Ceph Cluster (Part 1)

Introduction

This tutorial is going to show you how to deploy a Ceph cluster. A Ceph cluster can currently provide object or block storage over a network, and will offer a mountable filesystem in future. In layman's terms, this is equivalent to deploying your own Amazon simple storage solution (S3) or elastic block storage (EBS). This is particularly useful if you want to:

• host your own data due to performance/security/costs reasons.
• increase the services available in your cloud offering.
• provide a value added service to your dedicated server business.

Layout

We're going to deploy a Ceph cluster across 3 virtual machines, with the help of 4 Virtualbox instances. The cluster will constist of 2 storage nodes and a single monitoring node. The "extra" virtual machine being used that is not part of the cluster is just a node we use to deploy the cluster in the first place.

In a production environment you would want at least 3 OSD nodes and at least 3 monitoring nodes. Due to the monitoring service being so lightweight, it can be run on the same host as the OSDs but it is better not to. I will show you how to add these nodes to your cluster later.

Steps

First, install ubuntu 14.04 server on a single Virtual Machine and ensure it is fully up-to-date, before cloning it 3 times so that you have 4 virtual box instances. We will refer to these new clones as ceph admin, ceph mon, ceph osd1, and ceph osd2.

We need to configure each of the nodes with a static IP and update our DNS server to point to each of these nodes. If you do not have a DNS server, then you will have to rely on their IPs, but I recommend deploying a simple DNS server in 6 easy steps with the help of Docker, which is what I do.

Configure Ceph Deployer

Use the commands below to install the ceph-deploy tool (1.5.22trusty) on the deployment/admin node. This tool will help us to turn the other virtual machines into a Ceph cluster.

wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo apt-key add -
echo deb http://download.ceph.com/debian-jewel/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
sudo apt update
sudo apt install ceph-deploy

This tutorial will install the Jewel LTS release, which is the latest LTS at the time of writing this tutorial. Check the releases page, and if there is a later LTS, replace jewel in the script above with the new release's name.

This will instal ceph-deploy 1.5.33, but if you just run sudo apt install ceph-deploy instead of the commands above, then you will end up retrieving ceph-deploy 1.4.0 instead.

All Nodes - NTP & OpenSSH Server

All nodes need NTP and openssh server so that they can be connected to by the deployment tool, and so that the Ceph cluster doesn't get 'confused' due to time differences between nodes in the cluster.

sudo apt-get install ntp openssh-server -y

Configuring The Ceph User

The admin node must have password-less SSH access to all the other Ceph nodes, with sudo privileges. This is because it needs to be able to install software and configuration files without prompting for passwords. We need to create a Ceph user on ALL Ceph nodes in the cluster. A uniform user name across the cluster may improve ease of use but we don't want to use obvious user names to protect agatinst brute force hacks.

Run the following procedure on every node, substituting [username] for the user name you define, describes how to create a user with passwordless sudo. I recommend putting in something like ceph_random$uffix

USERNAME="[username]"
sudo useradd -d /home/$USERNAME -m $USERNAME
sudo passwd $USERNAME

Now that we have created the user on the server, we need to allow that user to execute sudo commands without being prompted for a password. We do this by running:

sudo visudo -f /etc/sudoers

Add the following line to the end of the file.

[USERNAME] ALL=(ALL) NOPASSWD:ALL

If you don't put the line at the end, but right under the line for root, it doesn't work for some reason, so be sure to put the line at the bottom of the file.

If you prefer to use vim over nano, then you may want to set your default editor.

Once you have created the ceph user on every node, log into the ceph user on the deployment/admin node and generate an ssh key, making sure not to set a passphrase.

ssh-keygen

Now add yourself to each of the other nodes:

ssh-copy-id [ceph user]@[hostname]

Conclusion

Congratulations, you've finished the first stage of deploying your cluster. You will now be able to continue to part 2 to finish deploying the cluster.

References

• Ceph Documentation - Installation (Quick)
• Ask Ubuntu - How to run sudo command with no password?
• Ceph Storage on Proxmox