Archive for June, 2007

The Art of Re-installing

June 25, 2007

So, the new hard drive is in; and right off the bat, its not crashing as much.. well, it crashed once so far, but I’m not sure if thats out of sheer luck or something. But an interesting 4 hours of headache with some Unicode encoding and PyXML.

It was bizzare, I started off the day going “ok, some stuff seems to be broken, just simply reinstall it”; since I had my share of playing around with Ubuntu and linux by now, I figure this was going to be a easy task, except when I tried installing TurboGears.. apparently, python complained and won’t let me.. Sure, I checked my Macbook and my macbook has Python 2.4.4 and my desktop has 2.4.3, so I might as well upgrade it… and thats when all hell went broke loose.

I’ve gotten used to building projects from source now (something which I wasn’t when I first started), So I downloaded Python 2.4.4 source and build it from scratched; worked fine. Installed pyXML, and TurboGears and all lights are green. Then I tried running OLM…

ImportError: From some path:  undefined symbol: PyUnicodeUCS4_Decode

Ok… some googling later and it tells me to build the source using the option -enable-unicode=UCS4

So I did… and it had a hell of a time installing.. or lack of, it didn’t install at all.. always breaking halfway. 2 hours of headache later, I googled up 2.4.3, which wasn’t obvious on the Python website, but it is there. Installed it and and wow.. everything worked… I’m relieved, but dammit.. why the hell didn’t it work earlier!

Going in circles with source compiling, never versions aren’t always the better version.

Tony

Paths paths paths…

June 21, 2007

So my work machine has a tendancy of crashing.. so I’m relying less on it right now (since my productivity hits a wall when I’m trying to debugg and everything dies). So I went on a quest… (at least, until my machine is fixed) to install OLM on my Apple Laptop.. needless to say, it was a lot more work and headscratching than i anticipated.

Karen managed to get it running relatively quickly, and I figured out after a day, that I needed to set a few more paths and copy some files over for TurboGears to see SQLAlchemy. Of course, I don’t believe a day of playing around should really discourage anyone from installing OLM on OS X, so I wrote an entry on the wiki, hopefully speeding up the installation 😛

HTML encoding function thingy…

June 19, 2007

Well, I would first like to say, thank you to everyone who answered my last blog entry. CONCAT was a good idea but we are trying to keep it valid within the SQL standard, anything specific to postgresql or such we would like to avoid.

Well, I manage to stumble onto another problem. This time, we are at the javascripts calling my python function (if that makes any sense).

My function in python takes an element, which is either a String, or a List itself. It first check to see if it is a list, doing this with type(args) == type([]). If its not, then it assumes its a string, and processes it.

To my knowledge, i can call my function “release” by doing:

self.user.goto(“/release?myList=something”)

which will return a string, If i want a list, I would do:

self.user.goto(“/release?myList=somethign&myList=somethingelse”)

The problem arises when I try to input an empty list… I thought about trying:

self.user.goto(“/release?myList=&myList=”)

but that returns a list of 2 empty strings inside 😦

UPDATE: So I figured it out.. made me almost smack myself for not thinking of it.. apparently, passing in nothing will give me an empty list, exactly what I’m looking for 🙂

SQL Woes.. Multi-fetch!

June 10, 2007

The problem:

We are refactoring and cleaning up on OLM , an online marking tool. As part of that, we’ve switched from SQLObject to SQLAlchemy, but we are having some performance problems related to the number of queries. Our navigation page displays students’ names, grades, and so on. It also shows the TA who is marking their assignment. The problem arises when there is more than one TA marking one student’s assignment.

For example, suppose we have a student named Bob, who is being marked by Jane and Liang (both of whom are also students). We have one query which gets all the TA’s marking Bob. In the navigate page, under the markers for Bob, we wish to put a list of TAs, like this:

Name Grade Marker
Bob B Jane, Liang

Unfortunately, we can’t think of a way to get all of the TAs
marking a particular student as a single string with a single query.
Instead, we are looping, which hurts performance a lot.

The Schema:

User
user_id int, primary key
user_name char
student_number char
display_name char
created date
groups foreign key to Groups

Status
status_id int, primary key
state string
grade float
assignment foreign key to Assignment
author foreign key to User
overall_comment char
lock_holder foreign key to User
lock_timestamp date

PermissionMap
permission_id int, primary key
author foreign key to User
marker foreign key to User
assignment foreign key to Assignment
permission foreign key to Permission

The Query:

To acquire the student and his grade we do:

select distinct d.user_name, d.status.grade
from User natural join Status as d
where d.user_name = d.author

To acquire the markers list:

select PermissionMap.marker
from PermissionMap
where PermissionMap.author = our_student

This would give us all the markers, and then we would parse it and manually put it into the table.

Well, as you can see, our goal is to remove the code required to shove the marker list into the marker column, is it possible to do the whole thing within SQL? Also, if we have 300 students, would be nice if we didn’t have to do 300 different queries to the database.. is there an elegant solution to this that we are missing?

Tony

Programming.. in the real world *gasp*

June 9, 2007

Me as a Programmer
My experience as a programmer was mainly as a random hobby or doing it as part of an assignment for school. While I have a strong interest in programming, I admit I haven’t indulge myself into any big projects where I have to do a lot of poking around. Most of the courses I’ve taken as an undergraduate student so far involves little or no starter code, and you build the thing from scratch.

Introduction to the world of a programmer, OLM
It didn’t take me long to realize how lost I was when I started with OLM. My first task was a simple one, update the wiki and include information on any tools needed for the developers. A guide was already there for checking out and setting up the project, though it was missing a few tools like the python profiler and the development kits of a few packages.

It was when I got OLM up and running and started adding stuff to the code, was where my lack of experience searching through things kicked in. I didn’t know where everything was, the tools were all foreign to me and I certainly have no idea how everything gets called. All this while Martin was blazing away installing the profiler and finding out why our navigate page took 3 seconds to load.

The first line of code I added was about 4 – 5 hours into the analyzing. By the time I finally finished my 2nd warm up (which was augmenting to the navigate page to show the rubric breakdown), it took about 8 hours and maybe 40 lines of code.

It took me about a week before I was comfortable using javascript and SQLObject, which then made augmenting the code a lot speedier. I was closing tickets a lot faster (the easy ones anyway), though I might have gotten carried away with some things and committed too early without really testing out all possibilities. At first, I submitted really buggy code… further playing around revealed random bugs that I didn’t expect to happen when I written them. Although I thought this is where I can finally start blazing through the code, changing things left and right; Martin had come up with some interesting discoveries

Speed Issues
It turns out that the template engine we were using (kid) was rendering pretty slowly, having only 100 students, rendering the page with their names, marks and such took 3 seconds.. adding the rubric augmentation only exploded this result, making it into 10 seconds. With this, we fired off an email to the creators of TurboGears hoping they might know why our pages were rendering so slowly.

Switching to SQLAlchemy
It was pretty bad, we got an email back from the creator of TurboGears and they tell us that they don’t even use SQLObject anymore, they switched over to SQLAlchemy. Through a series of more meetings, we decided to switch over to SQLAlchemy.

The syntax of SQLAlchemy was a lot different than SQLObject, it also didn’t help that the documentation online wasn’t as helpful with very limited examples. A lot of head scratching and a week later, I’m still a little shaky working with SQLAlchemy, though it is getting better. In the end, this is far better than than typing out raw SQL queries.

I have learnt a lot so far and done a lot of relearning. I haven’t touched python in a year but after using it, never programmed with Javascripts before and didn’t know what a template engine was. I am doing a bit better now, though I still have lots to learn, especially with using debugging tools.

What is OLM?
http://www.drproject.org/olm

Tony