Progress reports really do slow things down!

This is something I’ve known for quite a long time, but if you’re ever thinking “it’s not that much extra power to add progress reports”, just make a mental note of exactly how many progress reports you’re making, and try commenting them all out.

BeebMaze generated a new maze in mere seconds, but at the same time, that was seconds too long. As part of refactoring, I took out the progress reports with the intent of putting them back in, since there wasn’t that many.

It now generates a new maze in about the time it takes to paint the screen once. :D

(Download this build from the build server now: Build #18)

Multi-lingual interfaces for a website

One of the uni projects I’m doing is a Hotel Management system for “The Blackfish Hotel”, as part of the third-year group project. Part of this is an online booking system, where customers can book rooms in the hotel.

Our “customer”, the manager of the hotel, has asked for the website to have a version in Finnish as well as English, as apparently a large percentage of his customers are from Finland. This posed a challenging problem for us to solve… well… sort of.
Continue reading

Pigs do actually fly!

Well, we finished the Computer Graphics project!

Take a look at some of the photos on the page about it!

It took more than a couple of nights in the Uni Linux labs to complete, but we managed it.

This was Vasileios’ first time using Git, and to be quite honest, I don’t think we could have done this without using some form of version control software. Other groups were messing around trying to dig the latest files out of each other’s home directories, which is a rather bad way of doing it. Git handled all of that for us automatically.

OpenGL is quite an interesting bit of software too, I’d not expected it to be quite so straight-forward. It’s simply a case of saying what you want to draw, giving it a set of vertices, and then telling it to draw them. You then can apply mathematical transformations to the set of vertices, and draw more vertices and transform those to build your final model. By using the matrix stack and some clever modelling techniques such as scene graphs, you can (for example) draw an apple, move it onto a tree, then move the entire tree plus apple to where you want it.

It’s been a very interesting project, even if the final rush and the heaps of documentation (read: 78 pages) that needed to be written almost overnight made me hate the module.

Regular Expressions Tester Engine

I’ve actually taken a bit of time to fix up, make user friendly, and publish a tool that I wrote a little while ago, and has been very useful to me.

The regular expressions tester engine basically provides a front-end to the Microsoft .NET Framework Regular Expressions engine, allowing you to test regular expressions you’ve written to your hearts content.

As usual, I’m releasing it as-is under the MIT licence, and as usual the source can be found on GitHub.

For a page which might have more information on it at some distant point in the future, but more importantly has a download and bugtracker link, see this page on my main website.

ACCBot’s recent breakage

A couple of weeks ago, the IRC bot that we use over at Wikipedia’s Account Creation Assistance project decided it would stop giving notifications to the IRC channel. Previously, it used UDP as the transport between the web interface and the IRC bot. However, for some unknown reason, this stopped working.

After seemingly endless messing around with PHP, netcat, more PHP and a bit of telnet, I came to the conclusion that it was fucked, and there was no way to recover it with any ease.

Previously, the code looked something like this:

For receiving notifications over UDP, that was all we needed – it worked and was semi-secure. However, when it stopped working, I took a more radical approach.

You’ve probably heard of Amazon Web Services (AWS) by now, if you haven’t then I recommend you take a look.

One of the AWS services is something called the Simple Notification Service, which seems to be exactly what I want – a notification system. However, the only notification endpoints are HTTP pings, e-mail, or an SQS endpoint.

SQS is what I chose eventually – it’s another of Amazon’s services, the Simple Queue Service. This has the “advantage” of queuing all the notifications so if you’re offline you will still get them all. However, for our case this isn’t ideal, but the bot isn’t usually down for long if it goes down. So, I decided to go for SNS->SQS over HTTPS as the transport for the notifications, rather than UDP.

Of course, code needed changing – at first I thought drastically, but it turned out to be a much smaller change than I anticipated:

It looks small, just another explicit check to see if we actually received anything. That’s until you realise that I wrote another function to take some of the work off to one side.

There’s not much that’s changed, but it was an interesting technical challenge :P The only thing that has noticeably changed is the lag from notification generation to display on IRC – can be anywhere up to about 5s if you’re unlucky!

Strobe Light

Growing tired of the poor strobe light applications available on Android Market, I decided to build my own.

Lots of the so-called strobe light apps which already exist on Market seem to be of the type which use the phone screen as the “strobe” – this isn’t good cos it takes a while to update the screen, and it isn’t as bright as the camera flash led.

My app is a very simple one – a quick loop to set the flash on, wait a while, set the flash off, and wait a while.

The two waits are configurable with sliders, with values of anything from 0 (theoretically) to 500ms.

The only problem is the fact that developing camera-based applications on Android is a bit of a dark art – not all manufacturers seem to use the Camera API to access their camera. Therefore, camera-based apps can be a bit hit-and-miss as to whether or not they will work. As I only have the HTC Desire, I can’t test it on anything other than the Android Emulators, and my actual phone. I have, however, had reports of it working on a number of devices, including the Samsung Galaxy S2, HTC Incredible, and the HTC Evo 4G. There have also been reports of it NOT working on the Galaxy Tab, Droid X, and the XT720 Milestone.

As all the flashing takes place in an infinite loop, there’s no time to handle GUI events. Therefore I branch out the flashing bit to a separate thread, and use Handlers to make the necessary calls back to the main thread to update the GUI as necessary.

The main reason to update the GUI is to make sure that should the strobe thread get stopped in any way, such as on activity changes or screen re-orients (currently disabled) etc, the GUI “enable” toggle button actually shows the correct status, as well as to easily display any error messages as necessary.

There’s still a few outstanding bugs (screen orientation!), and there’s a few changes I’d love to make to it, but it’s coming along quite nicely, I just need more time to actually work on it!

Technology updates

Quite a bit has happened recently!

I’ll try and give a bit of info about each, there’s quite a lot to write about though!

EyeInTheSky

An IRC bot which stalks changes to Wikipedia based on regex matches. Read more

caterpie beedrill as minecraft server

Those of you who use my Minecraft server will know that the server known as “caterpie” was bad. Any more than one person connected, and it would lag to the depths of hell. I’ve replaced it with a 1.7G beast of a server, still running as an Amazon server instance.

stwalkerster.co.uk

I registered a new domain! I’m still in the process of setting the thing up though, but that will become my main website, not quite sure what I’m gonna put there yet, but it’ll probably replace what webspace I had at http://helpmebot.org.uk/~stwalkerster/

ACC‘s IRC Bot, ACCBot

A couple of weeks ago, the IRC bot that we use over at Wikipedia’s Account Creation Assistance project decided it would stop giving notifications to the IRC channel.

I’ll write more about this at a later time, hopefully soon, but probably when I’ve finished messing with my new site :P

strobe light

Growing tired of the poor strobe light applications available on Android Market, I decided to build my own.

I’ll also write more about this too at a later time – I think this bit and the last one need their own posts.

EyeInTheSky

EyeInTheSky is one of my newer projects, an idea which I’ve stolen from two other people.

Wikipedia has so many modifications being made to it that it’s just not possible to keep an eye on everything you want to watch. While the MediaWiki software has a feature known as a watchlist, it’s neither flexible nor easy to use in my opinion.

EyeInTheSky is an IRC bot (seems to be my speciality!) which monitors the Wikimedia IRC recent changes feed, compares every entry to a set of regular expressions and reports them to a different network.

It’s possible to set up the bot with an entire XML tree of regular expressions matching on the username, edit summary, and page title. There are also logical constructs which allow more-or-less unlimited regexes to specify what exactly you want to watch.

For example, with this tree, I could specify I wanted to stalk all the edits which are made by someone with “the” in their name, “and” or “or” but not “xor” in the page title, and with “train” in the edit summary:

I also can set a flag, something that I can then set my IRC client to respond to, and it will speak that flag for every stalked edit.

Of course, it’s not just edits that can be stalked this way – log entries are sent to the IRC RC feed in the exact same way. It’s just a case of specifying Special:Log/delete as the page title to get the deletion log, for example. The entire log entry except for the time/user is sent in the edit summary field. This means the same system can be used to stalk log entries as well.

The bot logs all stalked edits, and is capable of emailing the entire log to me, so I can clear the log when I disconnect from IRC, and when I get back on, I can email the log, go through what I’ve missed, and catch up.

I’m planning on making it multi-channel too, with probably multiple people able to command it to email them the log. I can already tell it to not email certain stalks, especially as some of the stalks that have been set up are not things that interest me, but rather interest other people. I just ignore those when it reports them, and have it set not to email me for those stalks.

There’s quite a lot this little bot can do, if you want to learn more, I’d recommend taking a look over the source code, and see what you think!

The source code is available on GitHub here.

Wikipedia Account Request System – Password Storage

The current ACC system has some really useless bits which are hard to change, such as the password storage system. At the moment, the database is filled with “securely” stored passwords, such as “5f4dcc3b5aa765d61d8327deb882cf99”. Any quick Google search will quickly tell you exactly how the passwords are currently stored, a simple MD5 hash. This is quite clearly inadequate, so as part of the rewrite I’ve been aiming to store the passwords much more securely.

In all the examples, I’m going to use the password “password”.

At the moment, it’s simple to set a password, just store

md5("password");

into the database. It’s also simple to check the password, just check

md5($suppliedpassword) === $storedpassword

However, I was wanting to store the passwords with a salt, a different salt for each user – hence making cracking the MD5 hash much less feasible.

The function I’m now using to encrypt a password is this:

The $2$ at the front indicates the version of the password hash for later use. For a password “password” and a username “username”, this gives the encrypted result $2$8c6e7b658b4be4bb325870a1764ca4fb

When a password is checked, the code looks at the first three chars of the stored password, and determines if it matches $2$ or not. If it does, the provided password is encrypted with the new hashing function, and compared to the stored password. If they match, it’s the right password.

If the first three chars are not $2$, then it hashes the password using the old method, compares it, and if it matches, takes the provided password, hashes it with the new function, saves it to the database, and returns that it’s the right password.

This has the effect of being transparent to the user, but increasing the security of their password the first time they log in to the new system.

Wikipedia Account Request System

I thought it was about time I did a bit of a technical post on the new Wikipedia Account Request System that’s been sat around slowly being worked on over what’s nearly a year(!) now.

It’s still a long way off, but I’ve not had time to actually buckle down and do work on it, so I’m hoping that I’ll be able to spend a bit more time with it in the near future.

Since the migration to GitHub, I’ve been doing quite a bit of development work on it, and have recently (semi) finalised the database, which will hopefully speed things up a bit, and stop me from saying “ooh, let’s do this with the database”, “nah, nevermind”, “ooh, let’s do this instead”, etc.

The database finalisation comes after writing the conversion script to convert the database from the current format into the new format – there’s roughly 35 operations to be done to make the database sort-of OK, 28 of which are done on one single database table.

I’m taking this opportunity to make these somewhat huge database changes to the core of the system as there’s not much that’s using the database at the moment in the new system, and a huge migration would have to happen in order to swap from one system to another anyway, so I’m not too fussed about making more changes like this.

As the developers of the current system will know, the code is quite frankly shocking. I’m pretty certain that SQL injection and XSS attacks are prevented, but only because we apply about 15000 sanitisation operations to the input data, mangling anything that’s remotely cool such as unicode chars – to cite a recent example: • – into a mess that MIGHT be displayed correctly on the tool, but any other areas just don’t work. In this case, MediaWiki rejected it as a bad title, because it was passed • instead of •.

The new system should hopefully solve some of these issues.

For starters, all the database quote escaping is going – I’m not even going to do database input sanitising – and I’m going to actively reject any change that adds it.

There’s a reason for this, and that is because of the database abstraction layer I’m using for this new system – PDO.

PDO handles all the database connection details for me automatically, and supports both raw SQL queries, and prepared statements. Where the former requires sanitisation to be secure, the latter doesn’t. You simply pass in place-holders (called parameters) to the query where your input goes. You can then bind values or variables to the parameters, and execute the query. Because the query and parameters are passed separately to the server, no sanitisation ever needs to happen because it’s just impossible to inject anything in the first place.

The really cool thing that I’m planning to (ab)use a lot is the ability to retrieve a database row from the database as an instance of a class you’ve previously defined.

The above is an actual excerpt from the User class of WARS at the moment, and the database structure of the acc_user table.

As you can see, the class has a set of fields which exactly match the names of the columns in the table. This is a key part of making the code work – all you need to do is create a query which pulls out all the columns for one row in the database, pass it the parameter which tells it which row to return, and then tell it to fetch an object, telling it which class to instantiate. A simple four-line function dealing with the searching and retrieval from the database, and instantiating a class with the relevant data – it’s actually beautiful! :D

My plan is to use this structure of data access objects for all the other database tables, and then I should be able to deal with the entire system on a purely object-based level, rather than constantly mashing in database queries here and there.