Creating an SPA with Polymer

I maintain a medium sized Microsoft Access application that is the operational system for a small business.  This business has offices in London, Birmingham Leeds and a copy of the Access application runs on PCs within the office system of these offices, allowing the staff to run the business.  A SQL Server database runs in the London office and all the Microsoft Access clients connect to it.

For reasons of confidentiality I wont give too many more details about the business, but this article and those that follow will concentrate on the technical aspects of creating a web based replacement that can sit alongside (but maybe eventually replace) the existing Access client.  The reasons for taking this path are both business driven and technical.  The key drivers are:-

  • Access is Windows only, but there is a desire to connect from other devices (such as iPads and Macintosh laptops).
  • Difficulties in distributing each new release of the Access client to all offices.  A web application doesn’t have this problem
  • Difficulties in doing multiple changes to the Access client simultaneously through lack of version control. The web application is essentially textual source code and can be version controlled (I will use git).
  • Ability to explore different UI approaches to existing processes to improve usability.

I am undertaking this project in the background, and expect it to take quite a while to complete.  For this reason, a key underlying driver is that the new and old applications sit side by side.

I have decided to build the web application as a Single Page Application (SPA) .  I did start this project before, about two years ago, and at that time started with JQuery Mobile.  A characteristic of this was I started to see quite large and complex files that would have become increasingly difficult to maintain.  The essence of using Polymer is that it possible to build relatively small self contained custom web elements, that comprise both the html user interface, but also the control logic surrounding it.  This encapsulation is what I am looking for.

The SPA will communicate with the database with small self contained PHP scripts called via AJAX requests.  They will use PDO to access the SQL Server database.  I could have chosen from a wide range of approaches, but this one is one I know and feel confident with.

My efforts at writing a series of posts to cover a project have, in the past, led to nothing.  I write the first post like this, then get dragged into the project and by the time I have emerged from the other side I have no desire to talk about it further.  This may go the same way, or maybe not.  Polymer, and web components in general, is a new technology that has yet to be fully explored.  I have some opinions about application structure and approaches that I hope to discuss in further articles.  Whether they see the light of day is another matter.  Lets just wait and see.

 

 

 

 

 

Adding custom buttons with your theme in an SMF forum

I recently wanted to update a theme I was developing for an SMF forum so it could add the “unread messages” and “replies to my posts” links as a menu items on the main menu.

Since the menu is placed on every page all this takes place in the main index.template.php for your theme.

With the integration hooks introduced in SMF v2.0 it becomes a very easy process. First during the theme initialisation (function template_init) when need to add a temporary (ie it only lasts for the display of the page) integration hook for the menu buttons

	add_integration_function('integrate_menu_buttons', 'mytheme_button_add',FALSE);

Then we just add a function at the end of the template page

function mytheme_button_add(&$buttons) {
	global $context,$txt,$scripturl;

	//only do this if user is logged in
	if ($context['user']['is_logged'])
	{
		//this is our new entry
		$newbuttons = array('unread' => array(
					'title' => $txt['view_unread_category'],
					'href' => $scripturl . '?action=unread',
					'show' => true,
					'sub_buttons' => array()
					),
				'replies' => array(
					'title' => $txt['replies'],
					'href' => $scripturl, '?action=unreadreplies',
					'show' => true,
					'sub_buttons' => array()
					)
				);
		//now insert just after home.
		$buttons = array_slice($buttons,0,1,true) + $newbuttons + array_slice($buttons,1,count($buttons)-1,true);
	}
	return;
}

Note how we slice the button array into two halves (in this case 1 before the break and count -1 after, but you could put them at any position), and insert the new buttons in between.

Managing SMF software in GIT

Yesterday, I upgraded three sites running SMF (Simple Machines Forum) software to v2.0RC5, upgrading them from v2.0RC4.  Some of these sites are more than just a simple implementation of the base software.  All of them have some modifications added, one (Melinda’s Backups forum) has a considerable degree of modification, and unlike WordPress, where plugins are pretty much stand alone, relying on well defined hooks in the software to get into the path, modifications in SMF tend to be intrusive.  Despite all the modifications, I upgraded all the sites over just a few hours.

I believe that is because of the approach I have taken to managing the configurations in Git.  In this post I want to explain how I do it.

Continue reading “Managing SMF software in GIT”

Displaying my RSS feed on Hartley Consultants’ front page

I had always planned to replace the feed on the front page of Hartley Consultants web site with a summary of recent posts from this blog.  I had previously sucked in news articles from the local SMF Forum that is on site, but I intend to remove the forum shortly because I am setting up the equivalent on this site, where it is more appropriate.

It turned out to be remarkably easy to achieve.  A call to “file_get_contents” with the url of my feed returned the raw xml into a variable, whilst a call to “preg_match_all” enabled me to parse the feed content and find the relevant information to display.  The formatting of the articles was already achieved by matching the same format as I had used for the forum feed.

Here is the code I used

	$feed = file_get_contents('http://www.chandlerfamily.org.uk/feed/');
	//having got the xml feed from chandler's zen we parse the xml looking for all the entries
	$pat = '#<item>.*?<title>(.*?)</title>.*?<link>(.*?)</link>.*?<pubDate>[^,]+,\s+(\d+)\s+([[:alpha:]]+).*?</pubDate>'; 
	$pat .= '.*?<description><!\[CDATA\[(.*?)\[\.\.\.\]\]\]></description>.*?</item>#sm';
	if (preg_match_all($pat,$feed, $topics,PREG_SET_ORDER)){
		$item = 0;
		foreach($topics as $newsitem) {
			$item++;
?>    <div class="newsitem">
        <div class="date"><h4><?php echo $newsitem[4]; ?></h4><h5><?php echo $newsitem[3]; ?></h5></div><h3><?php echo $newsitem[1]; ?></h3>
        <p><?php echo $newsitem[5];?></p>
        <p><a href="<?php echo $newsitem[2]?>" title="view full post at www.chandlerfamily.org.uk (in a new window)" target="_blank">More ...</a></p> 
    </div>
<?php
			if($item > 2) break;
		}
	} else {
		echo '<div class="newsitem">Sorry No Data</div>';
	}
	unset ($feed);
	unset ($topics);

The most important part is the pattern to match. But even that turned out to be quite straight forward, even in it did take a bit of trial and error to get it exactly right. I hope someone trying to do the same thing can use this as a starting point.

Allowing Syntax Highlighting in Comments

I have installed the SyntaxHighlighter Evolved plugin to allow me to post snippets of code to my blog.  However (as far as I can tell) it does not natively support allowing syntax highlighting in comments.  As I was reviewing the old posts pulled in from my Drupal site, and in particular the comments that I had made to update some of the older posts, I realised that I had quoted code in them which would be useful to also highlight.  I decided that I would try and add the facility for syntax highlighting to be added to comments in a simple way as possible.

I noticed that that way highlighting seemed to work is that it adds a construct to the <pre> tags that delineate the code that should be highlighted of “class=brush:lang;” where lang is the particular language we are highlighting for.  I felt this was a little complex for comments to include in their “semi” html markup,  but that it should be simple enough for them to add <pre lang=”lang”> as a construct.

Continue reading “Allowing Syntax Highlighting in Comments”

Integrating Gitweb output into a WordPress page

One of my known stumbling blocks to converting my site to WordPress was how was I going to manage to incorporate gitweb so that its output appears within the middle of a page, formatted to at least look like a normal page. The problem occurs because gitweb takes over managing the output and writes an html page complete with headers and footer. You are allowed to provide your own site_header.html and site_footer.html files, but these are inserted just after the opening tag and just before the closing tag. There is no way you can add to the header.

One approach to this, and the one I took on my previous version of this site, was to embed gitweb within an <iframe>. Effectively gitweb’s output is sandboxed within the <iframe> context. Unfortunately, despite some javascript trying to detect it, when the quantity of output changed the size of the iframe needed to change and it didn’t seem to do that automatically. I often found the output was chopped off (with a scroll bar provided) even when my full page was nowhere near full screen.

I have taken a different approach this time. In outline I have created a holding page with permalink “/software/” to hold the repository code and then created a special template for my site theme which when called embeds output from gitweb into the page. There are a few tweaks to the header is moved to the correct place and that links within the output are correct.  Down to the detail.

Continue reading “Integrating Gitweb output into a WordPress page”

Switching to WordPress

As will shortly become apparent as this blog goes live (I am currently editing posts on my test system, but intend to include the posts I am making whilst I am setting things up), I have switched my personal web site from Drupal to WordPress.  I thought I would explain why.

Over the years I have tried a number of different platforms for my web site.  My last but one implementation was WordPress, but I decided on my previous implementation to use Drupal.  My decision then was based on the fact that I had decided that wanted something more complex than a simple blog and Drupal looked to allow that possibility.  So I did set about building the site and managed to achieve a passable result.  However I didn’t find I was updating the site all that much with new ideas.

Continue reading “Switching to WordPress”

The making of MBChat v3 – part 2 coordinating multiple activities

In the previous blog article I discuss a process which would take several seconds to run. Obviously it is started as soon in the load process for the initial web page as possible.

Several other activities have to happen, such as loading the sound libraries, the dom has to be ready, the user has to have entered his username and password and it has to have been validated before the user can enter chat.

In the previous blog article I discuss a process which would take several seconds to run. Obviously it is started as soon in the load process for the initial web page as possible.

Several other activities have to happen, such as loading the sound libraries, the DOM has to be ready, the user has to have entered his username and password and it has to have been validated before the user can enter chat.

So several of the start up steps in getting the user properly logged on to chat and with the sound system fully working have to be co-ordinated. In order to avoid having to poll some flag to see if an item had completed, I decided to write a mootools class to handle this situation.

In order to create a coordination point for several activities, the programmer declares a new instance of the coordinator class, thus:-

var coordinator = new Coordinator(['rsa','login','dom','verify'],function(activity){
    loginRequestOptions.e = activity.get('rsa').e.toString();
    loginRequestOptions.n = activity.get('rsa').n.toString(10);
    loginRequestOptions.msg = 'MBChat version:'+MBChatVersion+'   using:'+Browser.Engine.name+Browser.Engine.version;
    loginRequestOptions.msg += ' on:'+Browser.Platform.name;
MBchat.init(loginRequestOptions,activity.get('rsa'));
    window.addEvent('beforeunload', function() {
         MBchat.logout(); //Will send you back from whence you came (if you are not already on the way)
    });
    soundcoord.done('chat',{});
});

In this instance, I am creating a coordinator which will coordinate the activities with the names of the first array parameter. When all the activities are complete the second parameter is a function which gets called with the activity object. This consists of a set of objects keyed by the activity name which are parameters which can be used. In this instance we are getting the public key components of the ‘rsa’ activity to pass to loginRequestOptions.

Note also at the end of the function a call to soundcoord.done. This is another coordinator I set up to coordinate the initialisation of the sound system. We can’t do anything with the sound system until both it is ready, and the user has logged into the chat (through this current activity). The done method is called with the name of the activity and an object (in this case empty) to pass into the final function in the activity parameter.

Just for completeness, this is the call to coordinate.done for the rsa activity so you can see how parameters (in this case the RSA key pair) are passed

var rsa = new RSA();
function genResult (key,rsa) {
    coordinator.done('rsa',key);
};

/*
We are kicking off a process to generate a rsa public/private key pair.  Typically this
takes about 1.2 seconds or so to run to completion with this key length, so should be done
before the user has completed his input - which is when we will need the result.  The genResult
function will be called when complete.
*/

rsa.generateAsync(64,65537,genResult);

This class itself is very very simple. You can download the latest release from my git repository as shown here. Although you might like to just copy and paste it from here

/*
 	Copyright (c) 2011 Alan Chandler
    This file is part of Coordinator.

    Coordinator is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    Coordinator is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with Coordinator(file COPYING.txt).  If not, 
    see <http://www.gnu.org/licenses/>.

*/

var Coordinator = new Class({
    initialize: function(activities,callback) { //an array of activity names
        this.activities = new Hash();
        activities.each(function(activity) {
            this.activities.set(activity,false);
        }.bind(this));
        this.callback = callback;
    },
    done: function(activity,parameters) {
        this.activities.set(activity,parameters);
        if (this.activities.every(function(activity) {
            return activity;
            })) this.callback(this.activities);
    }
});

The making of MBChat v3 – part 1 introducing Nested Closure Programming

MB chat is an Ajax chat application that I built for Melinda’s Backups. It was originally so that the leadership team could hold meetings without resorting to e-mail, but it became more of a general social thing for the members of the web site. In order for the software to work out if someone had said anything new, it polled a database every two seconds. Not something that would scale particularly but OK for v1. That was until Melinda herself came to chat one time, and a large number of people tried to come in at the same time.

MB chat is an Ajax chat application that I built for Melinda’s Backups. It was originally so that the leadership team could hold meetings without resorting to e-mail, but it became more of a general social thing for the members of the web site. In order for the software to work out if someone had said anything new, it polled a database every two seconds. Not something that would scale particularly but OK for v1. That was until Melinda herself came to chat one time, and a large number of people tried to come in at the same time. We found out that the scale limit was about 20 people. Beyond that and chat fell over.

In v2 I explored techniques for avoiding the need to poll a database and developed a solution based around the same technique as I mention in providing a high speed link between web clients using named pipes. Unfortunately the slight variation on this technique where essentially a message generated by one user has to be passed to all other users rather than just the one other, meant that you couldn’t wait for the two sides to synchronise (as is important in that solution). This means in some circumstances a message could get missed – so even with this technique a level of database interaction was necessary. I therefore also explored switching to the SQLite database as it seemed to offer a potential speed advantage. However I soon discovered that whilst it was almost faster the lock out mechanisms when multiple users attempted to access the database at the same time were crude and actually slowed things down. I got to 50 people, but that was the best I could achieve.

One thing I did discover in this test was that although SQLite was fairly fast with a single transaction per user interaction, it was lighteningly fast if multiple accesses could be done within the same transaction. I started thinking about a solution that could make that happen.

During this time, my daughter told me how they were considering a secure version of a chat program at her work. This got me thinking about how I could make MBChat secure. I set myself the following challenge:-

  • The chat server and the clients had to communicate with each other over the open internet
  • Only authorised users should be able to access the functionality or see any of the conversations
  • Clients can be sure they are talking to the server they think they are (to prevent an imposter capturing the messages)

A moments thought will immediately dictate that we should be encrypting at least the conversational messages with a key, but that since the software is in essence being downloaded from the server, it cannot contain that key within it as anyone watching the traffic would immediately be able to sniff out the key and use it to decrypt the conversation that was supposed to be secure .

In fact I decided that the only way to do this properly is for the client to generate a public/private key pair dynamically once it has started and transmit the public key of that pair to the server. The server will have to encrypt the key used to encrypt messages with that public key and pass it back to the client.  The client (and only that client) who will be able to decrypt it with his private key and start using it in the subsequent client server communication.

Generating a public/private key pair turns out to be a non trivial task.  If we are performing the task at the client end, inside the browser, then Javascript will most likely have to be the programming language which will be used to run the software which generates the public/private key pair.  Even with a reasonably fast processor, it takes several seconds to calculate an appropriate 64bit key (which we use).  If this was a normal script the browser would lock up solid, and would soon report a script failure.

I found Atsushi Oka had already solved this problem and provided a set of javascript routines that could generate the RSA public/private key pair. His solution he called “Nested Closure Oriented Progarmming”. If you take a look at the Nested Closure Oriented Programing module he has produced, you will see that part of his approach is to modify the base Object prototype to add the additional routines in. It was only after I built my first attempt to use his software to produce the key did I discover that there was major conflict with the mootools framework library that was an integral base for the rest of MBchat.

However after a detailed analysis of how the RSA key generation routines where built, I believed it was possible to make a mootools class that did almost the same job, but avoiding the need to alter the object prototype. I developed my own Nested Closure class which turned out to be considerably simpler that the library it replaced.

The key approach of nested closure programming is to have STEPS which are encapsulated within a function. A function can have part of its process broken into sub-steps, the key being that at the lowest level of steps the code can complete within the time allowed by the browser. Then all that is needed is an engine that runs one step, waits for a tick (a millisecond) and then executes the next step. Finally we need a mechanism to provide loops, for steps to indicate when they complete whether to exit this loop or to repeat the step or to continue to the next step (or loop if it was the last step of the loop), and a mechanism for passing results between steps.

Lets look at some small fragments of the RSA generation which show how it is used:-

Firstly – declaring the generator class, dealing with the result and starting the processes

RSA.prototype.generateAsync = function(keylen,exp,result) {
 var self=this;
 var generator = new NCOP( stepping_generate,true);
 var _result = function(keyPair) {
  result( self );
 };
 generator.EXEC([keylen,exp],_result);
...

We declare an instance of class NCOP (generator in this case) with the name of the highest level function used in the process, together with a flag which indicates that this instance of the NS class is unique. This allows the class to bind its key functions to the window object – making them easier to call (so you call can DO() rather than this.DO() or generator.DO()).

We start the process going and pass the parameters (as an array – which is an easy text transformation from the original code). We also pass the function that is called when the process terminates (_result).

Looking at the head of the stepping_generate function we can see how the parameters are picked up, and with a simple text transformation returned to two variables that were originally the functional parameters of that function

// Generate a new random private key B bits long, using public expt E
function stepping_generate (myArgs) {
 var B,E;
 B = myArgs[0];
 E = myArgs[1];
...

Once this high level routine has done some simple initialisation of variables, it starts the stepping activity (the main process) by returning but calling the DO method (it could have alternatively called the STEP method) of the main NCOP class. We see here how the first few steps are declared by declaring an array (indicated by the “[” at the start of the function parameters) of functions. Notice also a nested arrays starting at step 1.3. Arrays represent loops with a list of steps that are repeated until an action is taken to exit the array. Finally notice in step 1.3.1 that in certain instances the code executes return AGAIN(). This causes that whole loop to be started from the beginning rather than the (default) action of proceeding to the next step.

return DO([
 // Step1.ver2
 function () {
  RSA.log("RSAEngine:1.1");
  self.p = new BigInteger();
  rng = new SecureRandom();
 },
 function () {
  RSA.log("RSAEngine:1.2");
  // return self.p.stepping_fromNumber1( B-qs, 1, rng ).BREAK();
  return self.p.stepping_fromNumber1( qs[0], 1, rng );
 },
 [
  // Step1.3 ver3
  function () {
   RSA.log("RSAEngine:1.3.1");
   if ( self.p.subtract(BigInteger.ONE).gcd(ee).compareTo(BigInteger.ONE) != 0 ) return AGAIN();
  },
  function () {
   RSA.log("RSAEngine:1.3.2 : calling stepping_isProbablePrime");
   return self.p.stepping_isProbablePrime(10);
  },
...

It is also possible to pass the result from one function into the next step. This is shown in step 2.3.2 where a function “stepping_isProbablePrime” is called (it itself is using the NS methods to split its action into steps). The result of this step is then passed into the next step as the (only) parameter to the function which uses it to determine the flow of the following step (either exit the loop by calling DONE() or repeating the loop by calling AGAIN())

function() {
 RSA.log("RSAEngine:2.3.2");
 return self.q.stepping_isProbablePrime(10);
},
function(result) {
 RSA.log( "RSAEngine:2.3.3:result="+result );
 if ( result ) {
  RSA.log("RSAEngine:2.3.3=>EXIT");
  return DONE();
 } else {
  RSA.log("RSAEngine:2.3.3=>AGAIN");
  return AGAIN();
 }
},
...

This works very well. Dependant on the speed of the computer, it can generate a 64bit RSA public/private key pair in 1 to 2 seconds. This happens in the background – in some configurations of chat whilst the user is entering his username and password. In this instance that time delay is not even noticed. More importantly the browser never pops up a message to say that the script is taking too long to run.

Obviously when several things are happening at once (generating the key and awaiting a user to login) there is need to coordinate the completion. A simple mootools class to do that will be the subject of my next blog