Programster's Blog

Tutorials focusing on Linux, programming, and open-source

Backing Up Your Git Repositories Through Mirroring

Git

When Micro$oft aquired Github for several billion dollars, it dawned on me how dumb it was that I had parked a lot of my hard work on somebody else's land with no backup solution in place. If I was to lose access to the code I put on Github, I would be lucky if I still had half of it still checked out on my local computers, and would have to spend quite a bit of time hunting it all down. Now was the time to put an easy backup solution in place.

Why Not Just Use GitLab?

I have already posted about setting up your own GitLab server, and had thought about just moving all my code across. However, that would be extremely tedious, and Github is still "the place to be" that most people recognize and I like the fact that if I am committing code to Github, it shows up in my profile under activity, so people can see how active/inactive I am. I also like knowing that everything on my GitLab server is private and everything on Github is public. Thus I came up with a solution that allows me to keep using Github, but easily recover should something go wrong.

Steps

Deploy Git Server

Deploy a linux server (I just have a local computer using KVM to dploy a VPS, but you could use something like DIgitalOcean if you dont have computers lying around). For this tutorial, I am using Debian 9 but the steps are mostly the same across all distros.

Install git on it.

sudo apt-get install git

Congrats, you now have a git server.

Create The Mirrors

Mirroring a repository so that you get the full commit history, branches, tags, etc is as easy as:

git clone --mirror remote-repository-you-wish-to-backup

For example

git clone --mirror https://github.com/programster/zfs-balancer

Once you have cloned the repository, you need to keep it up-to-date with any changes that have been made to the original. This is as easy as navigating to within the repository and running

git remote update --prune

The --prune results in branches that were eventually deleted on the original to be deleted in the mirror.

Update Script

Manually going into each repository and running the update command would be very tedious. Hence I built a script to do this for me.

I created a folder at $HOME/repo-mirrors and performed all the cloning within there. Thus it is just a folder of all the repository mirrors. I then created a bash script at $HOME/update-mirrors.sh with the following contents which will update all of the mirrors in the repos folder.

#!/bin/bash
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

for f in $DIR/repo-mirrors/*; do
    if [ -d "$f" ]; then
       ( cd $f && echo "updating $f ..." && git remote update --prune && echo "")
    fi
done

Add A Cron

We don't want to have to manually call that script to update all the mirrors. Instead we should have the server do this automatically at a set point in time which is what a cron job is perfect for. Since I only feel the need to do this daily (and midnight is good for me), I will edit my crontab with:

crontab -e

... and add the following line:

@daily /bin/bash $HOME/update-mirrors.sh 2> $HOME/update-errors.log

...however, it you could do it hourly with:

@hourly /bin/bash $HOME/update-mirrors.sh 2>$HOME/update-errors.log

... or every minute with:

* * * * * /bin/bash $HOME/update-mirrors.sh 2> $HOME/update-errors.log

How To Use The Backup

If the original repository goes away or ends up disappearing behind a "paywall", you can just clone and work from your mirror. E.g.

git clone programster@git.programster.org:/home/programster/repo-mirrors/my-repo-name

However, you don't want to make changes to the mirror and have it try to mirror the remote repository still. Thus I would move it out of the repo-mirrors directory into a separate repos directory. Thus the update script/cron will no longer try to update it.

Backing Up The Mirrors

If you're worried, that history may get deleted from the original repository, and thus filter through to your mirrors, then you may wish to backup your mirrors. You can easily do this with duplicity.

Install duplicity...

sudo apt-get install duplicity -y

Create a folder for your backups to go within...

mkdir $HOME/backups

Create a script with the necessary command to backup your mirrors:

#!/bin/bash
export PASSPHRASE="my-passphrase-here"
duplicity $HOME/repo-mirrors file:$HOME/backups

Have that script executed via cron:

crontab -e

Add a line for executing the backup script. I will have mine run at 1 AM.

0 1 * * * /bin/bash $HOME/backup-script.sh

References