Next Steps with UserMgr

As the last article hinted, I had hoped to move forward with the web site. The next stage is an update to my UserMgr application.

As the last article hinted, I had hoped to move forward with the web site. The next stage is an update to my UserMgr application.

As you can see if you attempt to access certain areas of the web site you are required to provide a user name and password. Currently those usernames and passwords are stored in a database, and behind the (username and password protected) Site Administration tab there is an application that allows me to edit this database. Also, the Family Area is also protected and there is part of the application on this page that allows members of the family to change their own user information. The code from all this can be found in the Software Projects area.

Continue reading “Next Steps with UserMgr”

My new Blog – with Lucene

Today is the day – my new blog has been made live

It has taken a while, because I got side tracked with getting my AKCMoney application working but I have completed my blog.

Today is the day – my new blog has been made live

It has taken a while, because I got side tracked with getting my AKCMoney application working but I have completed my blog.

I had been wanting to replace textpattern for a while, as I was unhappy with running two simulteneous database engines at the same time, and I had already decided to standardise on Postgres for my other applications. I did make a start on attempting to convert Textpattern over to Postgres, but once my experience of getting up to speed with Java took over, I wanted to use Java for my new blog.

The original idea was to just use a standard database (just as the old TextPattern application had) to store the articles, but I also needed to figure out a way to search the articles. It was then that I decided to try out lucence. The more I started to understand how to use it for searching, the more I realised that it would be just as good a tool to store the articles as well. So thats how I proceeded.

I also needed a “language” to write my articles in – Textpattern has Textile, and I needed something similar. As you can see in my article WYSIWYG v WYSIWYM we used to have a system based on nroff macros to make very high quality documentation. Using that as an inspiration, Roffer was born.

More use of interesting open source libraries, I built the language using JavaCC. and it seems to work very nicely indeed. All the articles in this site have been written using it. I made roffer act as the tokenizer in Lucene, so the combination of the two can act as a complete database system.

So I built a database layer, that is then used by the rest of the code to work with the basic concepts of articles, categories and some simple searching.

Using tapestry for the basic application completes the picture.

Source Code for the application will be released soon under the GPL.

UPDATE: 26th February 2011. The software has been retired some time ago, and I have since used WordPress, Drupal, and now back to WordPress for this software.  The original Java application is available in my old-apps repository

AKC Money is born

At work I got my new laptop last October. But I couldn’t get my Palm Pilot to Sync with it, so was getting increasingly worried that my data, particularly my diary, was getting out of date. Also, I had an application for managing my accounts (Micromoney) that contains absolutely critical financial data holding the state of my personal finances.

Near the end of November I decided to try and sync with firstly my Linux Desktop, and then when that didn’t work my old laptop. None of them worked anylonger. It looks as though something has gone wrong with the Palm.

At work I got my new laptop last October. But I couldn’t get my Palm Pilot to Sync with it, so was getting increasingly worried that my data, particularly my diary, was getting out of date. Also, I had an application for managing my accounts (Micromoney) that contains absolutely critical financial data holding the state of my personal finances.

Near the end of November I decided to try and sync with firstly my Linux Desktop, and then when that didn’t work my old laptop. None of them worked anylonger. It looks as though something has gone wrong with the Palm.

A bit of searching on the internet found that a possible cause is static charge buildup, but the only way out of this is to let the battery run down and destroy the contents.

So I decided on a different approach. Build an application to run on my web site that replicated the functionality. From a security point of view that required both login and https – so that I could get at the data from work when filling in my expenses, but I also felt I could improve the application in a few areas (such as the management of foreign currency) that the existing application was poor in.

So I have taken a step sideways and stopped working on my blog, to urgently build this new application.

Its taken about one month – which was longer than I had hoped, but its working. Of course I don’t want you all to see my financial data, so I also installed demo copy with its own database and not protected via login and https.

Things I added

  • The ability to work in Foreign Currencies for transactions. I must admit the way I have done this is slightly strange, but it matches my own workflow, in which I claim my expenses either as
    • the result of transactions where I have paid for cash (and therefore need to remember the exchange rate at the last converion, or
    • the result of credit card transactions where I will initially guess at the conversion rate using the same rate as the cash transactions, but which will be accurately reflected in actual GBP figures when the statement comes through
  • Items have to be separately cleared from both the source and destination accounts. My old version tried to implement this with a single cleared flag and it didn’t always work. I have two specific flags for each end of the transaction
  • recognise that some accounts normally have the majority of transactions as debit (e.g. my bank account where there is a single salary credit, and then lots and lots of debit transactions. On the other hand my Company Expenses account is a credit account (I credit it with debits from other accounts for every expense, its debited – and normally by bank acount is credited when the company repays me those expenses.

I will release source code soon (under the GPL) – as soon as I can get the site tidied up

A little test Java v PHP

I was curious as to the relative speeds on Java versus PHP. So I created a couple of simple applications (see below).

PHP took approximately 2.5 seconds to execute the core part of executing a loop a million times.

Java took about 0.045 seconds to do the same thing, over 50 times faster.

I was curious as to the relative speeds on Java versus PHP. So I created a couple of simple applications (see below).

PHP took approximately 2.5 seconds to execute the core part of executing a loop a million times.

Java took about 0.045 seconds to do the same thing, over 50 times faster.

I know its a very crude measure, but it did suprise me how much better java is at executing than php. Of course the overhead of getting the results to and from tomcat sitting behind apache, compared to running php within the apache process might be pretty significant, but it does encourage me not to worry so much about whether I did the right thing building my application in Java.

For PHP


<html>
    <head>
        <title>PHP speed test</title>
   </head>
   <body>
        <p> This is a test to see how good php is in executing a loop a million times</p>
        <p>Times are: 
<?php  

/* Copyright (c) 2005 Alan Chandler, licenced under the GPL */

define (LOOPS,1000);
$x=3.0;
$start=microtime();
for ($i=0;$i&lt;LOOPS;$i++) {	
    for ($j=0;$j&lt;LOOPS;$j++) {
        $y=$x*3.0;
        $x=$y/3.0;
    }
}
$end=microtime();

list($su,$ss) = explode(" ",$start);

list($eu,$es) = explode(" ",$end);

$start= date('H:i:s',(float)$ss).":".(int)((float)$su*1000);
$end =  date('H:i:s',(float)$es).":".(int)((float)$eu*1000);

echo $start." ".$end;

?>     </p>

    </body>
</html>

For Java, the tapestry page

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>Java Speed Test</title>
</head>
<body>
<!-- Copyright (c) 2005 Alan Chandler, licenced under the GPL -->
<p> This is a test to see how good java is in executing a loop a million times</p>
<p>Times are: <span jwcid="@Insert" value="ognl:times">hh:mm:ss:mmm to hh:mm:ss:mmm</span></p>
</body>
</html>

and the Java

package uk.org.chandlerfamily.tapestry.speed;

/* Copyright (c) 2005 Alan Chandler, licenced under the GPL */

import org.apache.tapestry.html.BasePage;

import java.util.Date;import java.text.SimpleDateFormat;

public abstract class Home extends BasePage {	
    private static final int LOOPS=1000;	
    public String getTimes() {
        int i;	
        int j;
        float x=3;
        float y=7;
    
        Date start = new Date();
        for (i=0; i&lt;LOOPS;i++) {
            for (j=0;j&lt;LOOPS;j++) {
                y=x*3;
                x=y/3; // Waste time
            }
        }

        Date end = new Date();
        SimpleDateFormat fmt = new SimpleDateFormat("HH:mm:ss:SSS");
        return fmt.format(start) +" to " + fmt.format(end);	
    }

}

iBatis v Hiberbate

For those that have been following the “Taking the Java plunge” series, you will note that I have chosen iBatis as the framework to use the database. By reading the various mailing lists that I do, I get the distinct impression that Hibernate is the flavour of the month and this was worrying me, in that I couldn’t understand why.

For those that have been following the “Taking the Java plunge” series, you will note that I have chosen iBatis as the framework to use the database. By reading the various mailing lists that I do, I get the distinct impression that Hibernate is the flavour of the month and this was worrying me, in that I couldn’t understand why.

A brief discussion on the iBatis mailing list has just shed an interesting perspective on the differences between the iBatis and Hibernate, and since this made the light of understanding pop up in my head, I thought I would share it here.

In simple terms, Hibernate maps Java Objects to database tables. iBatis maps Java Objects to SQL statements.

Therefore, if you use Hibernate, as a programmer you don’t have to know SQL – it is generated for you. This is fine, and particularly useful if you are doing a insert or delete operations, but you loose the power that SQL gives you in the SELECT parts of your application.

SQL was developed specifically as a set processing language, and the database engines that use it have optimised their access of data around those concepts. Therefore if you need to do any sort of complex processing it is much more efficient to do it in the SQL rather than do it by processing in Java.

Generally speaking applications, particularly anything more than simple CRUD updates are going to do far more SELECTs than anything else.

So now I know why I was so attracted to iBatis. It stems from the fact that I tend to automatically normalise the database design as I do it. I knew that my database design for the family tree application was going to involve a single table called persons that would hold information about complete families, and that I was going to do want to do fairly complex joins on verious versions of this table. One of the simpler examples is something like :-

SELECT f.*,c.* from persons as f join persons as c on c.father = f.id where f.id = 23;

So iBatis will naturally map the results of a query like this into a complex object (an object which has got attributes for the father and a collections object such as a list for the children). I never got far enough into Hibernate to know whether you can do this or not. I am led to believe its a bit difficult.

So I was attracted to iBatis for the control that it gave me over the database processing I was doing.

This leads to one element of the issue that I think I need to address. It has been argued that in developing a web project you don’t want the programmers to have to come to grips with SQL. Therefore Hibernate is the way to go.

To me that is nonsense. To do a good job in developing any application you need to use the right tools for the job. In the corporate sense, if you want to avoid your web application programmers having to also learn SQL, then choose a tool that allows a separation of responsibility between those that need to know Java and those that need to know SQL.

Tapestry provides for the separation of HTML coding from the java, use iBatis to provide the separation of the Java from the SQL.

Taking the Java Plunge – release 1.0 reached

Well I made it. It was early in the weekend that the code was working, but its taken a couple more days of struggle to get it so I could put it into production. Learning how to get tomcat to authorize the users is hell to debug when it doesn’t seem to do what the book says it should.

Looking back at what I have done surprises me. It was a simple application (just 5 web pages) but the number of separate technologies that I had to learn (or in some cases perfect) were many. Here is a list (on top of Debian Unstable/Stable)

Well I made it. It was early in the weekend that the code was working, but its taken a couple more days of struggle to get it so I could put it into production. Learning how to get tomcat to authorize the users is hell to debug when it doesn’t seem to do what the book says it should.

Looking back at what I have done surprises me. It was a simple application (just 5 web pages) but the number of separate technologies that I had to learn (or in some cases perfect) were many. Here is a list (on top of Debian Unstable/Stable)

  • download and conversion to debian package of sun sdk
  • set up and configuration of Tomcat
  • complilation of jk_mod from source
  • installation and configuration of jk_mod into Apache2
  • download, installation and use of Eclipse with Web Tools (including debugging)
  • download, installation and use of git (and cogito) scm system
  • Java Programming – with special attention having to be paid to the Collections classes
  • download, installation of Tapestry web framework
  • learning all of the Tapestry concepts (basic, forms with translaors and application state objects in particular)
  • download, installation of iBatis database framework
  • learning the use of iBatis concepts (with complex joins and how they populated Java collection classes)
  • set up of postgres access controls so that code and myself (using psql) could create and then access the database
  • SQL for key transactions, and in particular complex joins in order to extract the family relationships from the database
  • set up of jdbc driver for Postgres and configuration in Tomcat so that could access family tree database
  • set up of a datasource realm in Tomcat and a users database in Postgres to provide access controls to application
  • html encoding
  • css encoding
  • javascript and how to use within the Tapestry framework
  • Povray to create the chandlerfamily logo (not in the timeframe of this project – was done earlier)
  • inkscape to draw graphic items
  • gimp to convert graphic items to gif

Taking the JAVA plunge – into the last mile

After some hard work at getting my ideas together and then some marathon debugging sessions, my application is almost working. It has taken me a lot longer than I envisioned – for two real reasons.

Firstly, for a long time I had an approximate idea of what screens and what workflow I wanted in the application but I was finding it difficult to make it concrete. When I didn’t have a clear picture of what I was aiming at, my development didn’t proceed.

After some hard work at getting my ideas together and then some marathon debugging sessions, my application is almost working. It has taken me a lot longer than I envisioned – for two real reasons.

Firstly, for a long time I had an approximate idea of what screens and what workflow I wanted in the application but I was finding it difficult to make it concrete. When I didn’t have a clear picture of what I was aiming at, my development didn’t proceed.

In the end, I found that to draw out the screen layouts and then merge them onto a single page as the best thing to do. I then printed them out and annotated the copies with the actions and how they were going to control variables across the application. This picture shows the layout that I drew (which is close to what I will end up with).

Unfortunately the picture that I had placed here in this blog entry has been lost

I found Inkscape www.inkscape.org under both Windows and Linux as a really good tool to do this. I first drew each page as a separate drawing, and then used the import tool to pull them all together in a single drawing.

Unfortunately, I found Inkscape cumbersome for drawing arrows to represent the transitions between the pages. Initially I just used marks on the printed copies, but these got difficult to see with so many scribbles, so in the end I used KIVIO under kde to draw single simple boxes for the pages and coloured arrows for the flows between them.

update I now use DIA for this, and export the result as an SVG which I can then size and export as a PNG image using Inkscape

The second reason for lack of progress was trying to be too clever with not creating a session objects to store persistance data in. I was trying to carry the data between pages, and getting screwed up when the page was being recreated without access to the information I wanted. As soon as I created a few Tapesty “Application State Objects” for my session, and just got standard forms to load data to and from these objects everything became a lot simpler.

Now just trying to complete the javascript for a dynamic piece on my Edit page, looking at using CSS for control of the formats on all pages (and using the same to provide mouse hover tool tips) and I will be moving on to production deployment.

I’ll report on that soon.

Taking the JAVA plunge – some weeks later

It has been a while since my last entry, but here I am sitting in a hotel room in Instanbul, and discovered that it has a wireless lan capability.

I have started my first application having picked Tapestry 4.0 (currently at beta 5, but upgrading as soon as new versions come out) and iBATIS to connect to the database. I have been able to access my database and display pages with the data installed.

It has been a while since my last entry, but here I am sitting in a hotel room in Instanbul, and discovered that it has a wireless lan capability.

I have started my first application having picked Tapestry 4.0 (currently at beta 5, but upgrading as soon as new versions come out) and iBATIS to connect to the database. I have been able to access my database and display pages with the data installed.

I sort of stuck with Eclipse, but used the version bundled with Web Tools Platform. This is combination seems to be giving me the ability to run and then test (under Tomcat controlled by Eclipse) my application is run in debug mode.

Now that I have got to use tapestry in practice it is very easy to use, and have been able to make a good start on my “Family Tree” application.

Having looked at Hibernate and Cayenne as database frameworks, I decided to stick with iBATIS as my database access framework. It seems to be that right combination of simple to understand yet powerful in use.

Taking the Java Plunge – Selecting and using an IDE

It has taken a my about 3 weeks of exploring to know what my next step in taking the Java Plunge should be.

I have set up both Netbeans (4.1) and Eclipse (3.1) IDEs and tried to get both of them to create simple Hello World JSP applications.

Debian packages for both of them seem out of date, so I have downloaded both from their respective web sites.

Netbeans initially seemed extremely simple, as I immediately understood all the windows, what they did and how to create an application.

It has taken a my about 3 weeks of exploring to know what my next step in taking the Java Plunge should be.

I have set up both Netbeans (4.1) and Eclipse (3.1) IDEs and tried to get both of them to create simple Hello World JSP applications.

Debian packages for both of them seem out of date, so I have downloaded both from their respective web sites.

Netbeans initially seemed extremely simple, as I immediately understood all the windows, what they did and how to create an application.

However, at the same time I have been exploring exactly how I was going to construct my web applications, and am currently favouring Tapestry (3.0.3) as the correct framework to use. This has a plugin for Eclipse called Spindle which should aid development, and so I have decided to invest the time and effort in working my way through the Eclipse tutorials.

The more I get into it, the more I understand how to create and debug java applications and what all the different windows mean. Confusing at first – but very useful once you understand some of the concepts.

I think I am going to stick with Eclipse.

Taking the Java Plunge – Finding Out What To Do

Thats it – I just have to start it. 5 years after adding it to my TODO list for my home computer network I have started my first java web application. I want to try and build a “family tree” application to capture and record in a database my family tree. I will then provide a tool for dumping this data into an ascii format (probably xml) so that it will not be lost to future generations.

Asking what I need on the Debian User mailing list gathered the usual helpful responses (particularly because Debian can’t support the non free Sun licences).

Thats it – I just have to start it. 5 years after adding it to my TODO list for my home computer network I have started my first java web application. I want to try and build a “family tree” application to capture and record in a database my family tree. I will then provide a tool for dumping this data into an ascii format (probably xml) so that it will not be lost to future generations.

Asking what I need on the Debian User mailing list gathered the usual helpful responses (particularly because Debian can’t support the non free Sun licences).

The general message I got was

  • Use the Sun run time and sdk libraries – they are the best
  • Choose Netbeans or Eclipse as the IDE
  • Server side you use Apache, with Tomcat to serve JSP and servlets

There is a helpful web site which says how to make debian packages out of the sun downloads, so they get installed according to Debian standards

UPDATE: The link in the previous paragraph doesn’t work, and is anyway irrelevant since Sun open sourced Java and Debian put the Sun runtime into its repositories

Next job – learn the language