Deploying a GlusterFS Cluster

In this tutorial, we are going to configure three Debian 8 servers as a GlusterFS cluster with replication such that we still have access to our data if any of the servers fail.

For this tutorial, as with most tutorials, I will be running commands as a non-root user with sudo privileges. I recommend that you do the same.

DNS

The first thing you should do is configure the DNS so that each host can be referred to by a name, such as gluster1-internal which will point to each nodes internal IP address (if there is one). We want to be using internal networking for GlusterFS to improve replication performance. If you don't have an internal DNS server (you can deploy one quickly with docker), then you can just edit the /etc/hosts file which will be tedious but worth it.

Install GlusterFS

On each of the nodes, execute:

sudo apt update
sudo apt install glusterfs-server -y

On Debian 8 this will install version 3.5.2 If you need to make use of heterogeneous bricks, then you need 3.6 or above. Click here to install 3.8 instead.

Start Peering

On one of your nodes, run the command below to peer with each of the other nodes.

sudo gluster peer probe [other node internal hostname]

For me, I ran the command on gluster1.programster.org so to peer with the other two nodes I ran:

sudo gluster peer probe gluster2.programster.org
sudo gluster peer probe gluster3.programster.org

You can now test that this worked by running the following command on any/each of the nodes:

sudo gluster peer status

You should get output similar to below:

Number of Peers: 2

Hostname: 10.1.0.88
Uuid: e2cff423-8cca-4a09-9f77-ef2a9840c84e
State: Peer in Cluster (Connected)

Hostname: gluster3.programster.org
Uuid: 3660bcb2-dd84-42f0-886e-4f880d3a0d01
State: Peer in Cluster (Connected)

Create a Storage Volume

Now we are going to create our storage volume. You need to decide where all the data should go on each server. You can pick a path that is different on each, but I find it simpler to have the same path on each server and went with /gluster/volume1.

Run the following command on each of your servers (changing the path if you wish).

VOLUME_DIR_PATH="/gluster/volume1"
sudo mkdir -p $VOLUME_DIR_PATH

Now run the following script on just one of your servers (any of them). Change the settings as you wish.

# Settings
VOLUME_NAME="volume1"
NUMBER_OF_SERVERS=3
VOLUME_DIR_PATH="/gluster/volume1"
HOST1="gluster1.programster.org"
HOST2="gluster2.programster.org"
HOST3="gluster3.programster.org"

sudo gluster volume create \
$VOLUME_NAME \
replica $NUMBER_OF_SERVERS \
transport tcp \
$HOST1:$VOLUME_DIR_PATH \
$HOST2:$VOLUME_DIR_PATH \
$HOST3:$VOLUME_DIR_PATH \
force

This should output something like:

volume create: volume1: success: please start the volume to access data

Start the volume

Now we need to start the volume with the following command:

sudo gluster volume start volume1

This should output something like:

volume start: volume1: success

Conclusion

You now have a GlusterFS cluster created. Now you probably want to actually use it. For this you will need to set up some GlusterFS clients which may or may not be the same servers in the cluster.

References

• DigitalOcean - How To Create a Redundant Storage Pool Using GlusterFS on Ubuntu Servers
• ServerFault - GlusterFS: Replication Over Odd Numbers Of Nodes
• Github - docker-volume-glusterfs