Yesterday, I upgraded three sites running SMF (Simple Machines Forum) software to v2.0RC5, upgrading them from v2.0RC4. Some of these sites are more than just a simple implementation of the base software. All of them have some modifications added, one (Melinda’s Backups forum) has a considerable degree of modification, and unlike WordPress, where plugins are pretty much stand alone, relying on well defined hooks in the software to get into the path, modifications in SMF tend to be intrusive. Despite all the modifications, I upgraded all the sites over just a few hours.
I believe that is because of the approach I have taken to managing the configurations in Git. In this post I want to explain how I do it.
Before I do, let me explain how SMF and modifications are structured. SMF is a large body of software with over a 1000 files in its base install package. A few files are modified as a result of administrative activity – the obvious Settings.php file (and its backup Settings_bak.php) for basic configuration, some files (install.php or upgrade.php and their helpers) are deleted after use, and some (such as agreement.txt or the Smiley’s installed) are possibly edited by the admin during site setup. But the vast majority remains stable except when modifications are involved or a new release of SMF is made.
There are some directories that are volatile but contain nothing much from the installation (cache) or are really an extension of the operational database ( the user attachment directory, storing files users have uploaded), and these are therefore dealt with slightly differently.
Modifications (aka mods) are generally packaged in a fairly standard way so that they can be managed within SMF via a package manager. They consist of an archive of files with an xml information file as one of them, which describe how the rest should be deployed. Some files are just installed into the installation as they come. One file generally handles updating the database to include extra database fields, or settings and one file, also xml based, describes how each of the raw SMF files should be modified in order to incorporate the mod. Generally it describes a series of edits to files in either the Sources directory of the SMF install or one or more of the default language files. Because of this editing capability a mod is pretty closely tied to the version of SMF its run against and makes the assumption (which isn’t always true) that its modifications do not intrude into areas where other modifications are being made.
Because of the intimacy described above and because mods are generally relatively simple changes I have taken the strategic decision to take responsibility for the modifications myself once they are installed in my system. In other words, I regard most mods as though they are changes I personally have made to SMF which I will maintain through SMF upgrades. That is not to say I would use the work that the modification author has done to upgrade it, nor that I don’t feedback issues to him that I find in using it, just that I don’t use any automated way of incorporating modification changes into my installation.
Apart from the default themes, SMF provides a way to allow user developed themes to provide a look and feel for each site. These are installed in their own independent sub-directory and of consequence are relatively stand alone. Each area of the display uses a template, and some themes only provide a template for the overall look of the site, relying the SMF to select a template from a default theme if the supplied theme doesn’t have one. This does provide one place where some manual intervention maybe necessary as versions are changed – new themes have to replicate the css/index.css file from the default theme – so changes in this file in the default theme may need to be copied into the themes version on upgrade.
Now lets take a closer look at Git. I don’t aim to provide a git tutorial in this post, even for a small area of its functionality, but we do need to understand the concept of branches and commit history. Git “commits” are snapshots of the state of the software at a certain point in time and are represented in the diagram on the right as blue dots. A series of commits going up the page in the diagram show the history as its is being created. Some changes are made to the software and a new commit is made. You will notice that at some points the path between two commits branches. This is because a new git “branch” is made and development splits. The heads of these branches (ie the lastest commit that has been made) is represented by the green boxes. These have names (the names of the branch) and using git you can switch between them to get to the state you were at when you last left that branch.
You will also notice that the lines between the commits merge. That is because it is possible to merge one git branch into another – picking up the changes in that branch and merging it into the current one.
There are just two other important characteristics of git that are important. The first is, that it maintains a list of files which it is looking after. It is possible to have files not being tracked by git, and and when you switch branches (provided you don’t use a hook – see below – to do something strange) the un-tracked files are not touched. In managing SMF, I leave the cache and attachment directories, and the files modified by the user (e.g. agreement.txt) un-tracked. I do however keep the files that get deleted tracked as we explain below.
The other important characteristic of git is the use of hooks. These are scripts that are run automatically at certain stages of the process. In managing SMF I make use of a script that is run when either a commit is made or a merge of branches is performed.
When I first install a site, I plan of having two versions of it. One is on my development machine and where I run the version of git I am describing, generally the other is on a hosted solution somewhere. I like to have access via SSH using a public/private key pair – where my public key is stored on the server allowing me access. I also need to have rsync available on the server which I use for copying files to it (via SSH). If you only have ftp access you will need to modify my scripts accordingly.
So my first step is to create a local directory on my development machine to manage the installation. In reality I create several directories, only one of which is the web root. The others hold supporting files, such as raw pictures or licence text and sometimes even the apache configuration. For the sake of simplicity of this article I will assume a single directory holding the forum software in the root. My first step is to unpack the distribution into this directory and to then create my git repository thus:
git add avatars Smileys Sources Themes
git add favicon.ico index.php install* licence.txt news_readme.html readme.html
git add Settings.php SSI.php ssi_examples.php ssi_examples.shtml subscriptions.php
git branch distribution
git tag v2.0RC4
git branch base
Note: the “git tag”command would use the actual version number of the distribution I am using. I haven’t used the latest here because I want to show the upgrade to v2.0RC5. Also, the list of files added represents the current release – but avoids directories which I don’t want to track, even though they have potentially trackable files in them (each one has an index.php that redirects attempts to list them back to the main root’s index.php). This is a compromise to avoid long lists of untracked files (by default “git status” shows the whole directory as untracked if no files are tracked within it but lists each individual untracked file if only one is tracked).
Note:also I don’t bother tracking Settings_bak.php. This is a useful trick to avoid problems when Settings.php gets overwritten and you want to see what your database password was.
The important activity here was the creation of the “distribution” branch. Using git’s independent treatment of branches, I can maintain the correct state of the distribution before any mods have been applied through each release history. I also create branch “base”. This is a temporary branch that I need to create to save the state of the master branch just prior to running any installation or upgrade script on the test system. SMF deletes some files during this process that I will need for the production system.
I now run the installation process on the development/test system. This will cause SMF to alter Settings.php and delete the install.php and related files. I commit the result. I will need to install the software on site, so I need to go back to our base system (so I still have all the installation files) and install so:
git checkout base
rsync -a ./ me@mysite:public_html/
### RUN INSTALLATION ON SITE
git checkout master
git checkout -b site
rsync me@mysite:public_html/Settings.php .
git add Settings.php
git checkout master
git branch -D base
Note: obviously I replace me@mysite:public_html with the appropriate username,domain and directory for my site.
Notice how I create a new branch off of “master” called “site” which will hold the site copy the system. By reading back and then committing Settings.php I keep a local copy of this file which is derived after the version in the “master” branch. This subtle ordering of updates means subsequent merging of the master branch into the site branch will not overwrite the key details that are likely to be different between the development and production systems.
I delete branch base as I have finished with it in relation to this release. I will use it again during the next version upgrade.
I now install any modifications, make a new theme etc as I wish. I use git as it should to commit changes or sometimes I make a small branch to develop a change on and merge it when complete – until I have a solution that works on my development system which I am ready to install on site. I have this ready on the “master” branch.
Now is the time to look how I use a git hook to help me with the on site installation process. I create an executable file .git/hooks/post-commit and make a symbolic link to it in .git/hooks/post-merge. This is what post-commit contains:
# Output a version file that we can include at the bottom of the page
branch=$(git branch | sed -n s/^\*\ //p)
version=$(git describe –tags)
cd "$(git rev-parse –show-cdup)"
if [ "$branch" == "site" ]; then
git clean -X -f
echo "<?php echo ‘$version’;?>" > version.inc
rsync -aqz –delete –exclude=/.git –exclude /attachments –exclude /avatars/uploads –exclude /cache ./ me@mysite:public_html/
You will see that this file checks if the user is on the “site” branch and if he is cleans up a bit and copies the files to site. I have added the generation of a version.inc file which contains the version of software being installed – which can be included in other php files, such as your theme. Alter that how you want to make it work for you.
Once I have that hook in place – and whenever I have a version of software that is ready to be installed on site, I do the following
git checkout site
git merge master
git checkout master
Occassionally I might be faced with a merge failure which i have to tidy up using git commands. If you are following you will need to read the git documentation to see how.
I always switch back to the master branch straight after the merge (unless there is a failure). It means if I jump in to make a quick change without thinking too carefully about which branch I should be on, I am always on the right one.
Now to the process I went through yesterday. SMF released version 2.0RC5 and I wanted to install it on my production site.
Firstly I switched to my “distribution” branch, copied all the files from the upgrade version of SMF into my directory for this particular site. I used git status to check for any new files that I wasn’t yet tracking and used git add to add them. I then commited the result and tagged it (using git tag) with the release name (ie v2.0RC5).
I then switched to my “master” branch and used git merge to merge the updated distribution in to the master. This threw up a few merge conflicts (places where modifications I made came in the same place as changes in the distribution). This was not surprising – some of the changes I had made were fixing bugs that I had found in the earlier release, and my solution had not always been identical to how the SMF developers had fixed it. However it was fairly easy to complete. Note: it is important you do all of this without running the upgrade. You will see why in a minute.
One file needed special attention. Themes/default/css/index.css . By doing the following
git diff master v2.0RC4 Themes/default/css/index.css
I could see the changes made to this file in this release. I needed to go through each one and compare with the same part of Themes/mytheme/css/index.css to see if a change was needed in my theme. A found several that were there – some of which I accepted, some of which I decided where not appropriate to my theme and I preferred to leave like it was. If these were changes after the merge, I committed them separately.
At this point I needed to do the following
git branch base #creates a new base branch (but don’t switch to it)
### RUN UPGRADE on my development system
git commit -a #updates master branch with the fact that the upgrade process deleted some files
git checkout site
git merge base #get site to the pre-update stage
### RUN UPGRADE on production system
git merge master #updates site branch with special changes for theme, and marks the upgrade files deleted
git checkout master
git branch -D base
This updated both my development system and then later the production system with the new release. Note: I spent a short while after upgrading my development system to check that all was still well. In particular it was during this part of the process I noticed errors in layout due to changes made in the release that needed a corresponding change in my index.css file in the theme. It meant that by the time I came to update the production system I was pretty confident the upgrade would work.
I trust this has been useful. I find that I have been able to maintain an SMF forum for over 3 years this way – including a major upgrade from version 1.x to 2.0RC3.
As a postscript. I starting using a similar technique to mange this WordPress installation. However, there is a different approach to modifications in WordPress, in that generally speaking they sit in an independent directory and therefore do not interfere with the main code as much. Using the method described above would prevent me installing plugins automatically or doing the version upgrade of the main software automatically. I have therefore limited the use of a git repository to manage the files in my bespoke theme (and separate repositories for other parts of the site (which is about to include an SMF forum managed specifically as this post suggests).