How keeping code in the right layer saved a website

At my job at e-sites, one of the projects I worked on was the new Sport 1 website. The website went live yesterday morning. About 20 minutes later, I was asked to look at why it was so slow.

For the website, we had a layered setup. In our ‘domain layer’, we had models and entities, where a model understood how to work with an entity. So, say I want all the news articles for a specific sport, I can do this:

$aNewsArticles = $oNewsModel->getForSport($iSportId);

Where $oNewsModel is an instance of our news model. After the call, $aNewsArticles would be an array, containing instances of our news entity class. Simple, right?

Well, apparently not. Since I setup the groundwork for the project and (basically) then went on vacation with little time to explain it to others, this is what I ran into:

Class NewsModel {
	public function getForSport($iSportId) {
	    $aArticles = query("SELECT newsId from news where sportId = %d", $iSportId);
	    $aArticleList = array()
	    foreach($aArticles AS $aArticle) {
	        $aArticleList[] = new NewsEntity($aArticle['newsId']);
	    }
	    return $aArticleList;
	}
}

Do you see where this is going? Yes, thats where

Class NewsEntity {
	public function __construct($iArticleId) {
	    $aData = query("SELECT * from news where newsId = %d", $iSportId);
	    $this->iId = $aData['newsId'];
	    $this->sTitle = $aData['newsTitle'];
	    /* ..... */
	}
}

Taking into account that on the frontpage we a bunch of lists with a total of 80 articles that where all retreived the same way, we where executing at least 80 queries to much. And all that, just because somebody put the query in the wrong place.

In the end, it was a fairly easy fix, but it shouldn’t have been done this way in the first place. Now, I understand how this would have happened. Somebody had to do the first thing with news and, in that particular case, knew the ID of the news-item, since it was part of the URL. Well, if you already know the ID, whats easier then this:

$oNewsItem = new NewsEntity($iId);

Yea, that must be the easiest way to make that work. Feed the ID into the entity object and have that get the rest. And then, when you need a bunch of those objects, well, just selecting the IDs of the items you need and creating new objects, which then get the rest of the data, thats pretty easy, too. To bad its also not so fun for the database server.

So now we know how it happened, how do we make sure it doesn’t happen again? Obviously, making sure everybody has the same idea of what should go where is a (good) start. Ideally, there would be time to write some documentation, and you’d be able to oversee (and participate in) the further development, meaning you can put a stop to something while its happening instead of after the website went live. But, like I said, I was going on vacation, so I did neither. 🙂

Posted in Random blah | Leave a comment

PHP’s SoapServer and generating WSDL files

Since PHP5, doing stuff with Soap became way easier. SoapServer and SoapClient where added, giving pretty much everybody the ability to create a simple SOAP provider and consumer. But, as always, there is a but. And the ‘but’ in this case, is that if you go to server.php?wsdl, you get an error stating “WSDL generation isn’t supported.” And anybody who has ever written a WSDL by hand knows that it sucks. So, my mission of tonight: to generate a WSDL file from my PHP code.

Obviously, I started with a search for something like “php generate wsdl from code”.  That exact search gives my over 2m results on Google. To bad non of them had the answer for me. Then, I read somebody directing somebody else in the direction of Zend_Soap_AutoDiscover and decided to take a look myself. It seemed to do what I wanted it to, so I figured, why not have a look at it.

After downloading ZendFramework to my laptop and setting up autoloading for it, the first thing I tried was creating a script separate from my server.php called generate.php. In that script, I included my service class, told Zend_Soap_AutoDiscover I wanted to use that and to handle() requests. Browsed to the file, and voila, I had my WSDL file. So, I saved the file. Then, I found out Zend_Soap_AutoDiscover has a dump($filename) method that can do that for me. Instructed SoapServer and SoapClient to use that WSDL, and it worked!

Well, somewhat. Calling $client->__getFunctions() showed me my functions. But if I tried to call one of them, it threw an exception. After some working, I noticed that the targetNamespace was http://localhost/soap/generate.php, which is plain wrong as I’m calling http://localhost/soap/server.php. Ah, that makes sence – Zend_Soap_AutoDiscover couldn’t have known I wanted to use server.php now could it?

So then I thought, why not put the Zend_Soap_AutoDiscover code in my server.php, detect ?wsdl and give the WSDL in those cases. After first having a small codesnippet in server.php that detects ?wsdl and lets Zend_Soap_AutoDiscover do its thing -which worked- I decided the final step was to extend SoapServer so that it would, in fact, support WSDL generation. (Sure, not native, but for native support it’d need to be done in C, which I don’t know anywhere near well enough to make an appempt.)

So I went ahead and wrote my class, WsdlGeneratingSoapServer (describing names for the win). It didn’t need to do much – just overwrite setClass, setObject and addFunction so that it can keep track of what the SoapServer can do (and then passing on the call to parent::, obviously) and overwrite handle, which should detect ?wsdl and, if found, have Zend_Soap_AutoDiscover do the generating, calling parent::handle otherwise.

Sounds easy enough. And it was, I had the class done in about 10 minutes. And then, I tested it. WSDL generation worked perfectly, so goal accomplished! Or well, not really. Normal soapcalls now result in a SoapFault. So I started to try changing things. Long story short, this simple code caused a soapfault. And since thats crazy, copied that script from my laptop (which has PHP Version 5.3.2-1ubuntu4.2 + Suhosin Patch 0.9.9.1) to my VPS (which has PHP Version 5.3.3RC4-dev) and what do you know, it worked.

I didn’t feel like doing any more PHP tonight, so decided to write this blogpost instead. I’ll try the code at work tomorrow, hopefully it’ll work there as well. 🙂

In the mean time, here is the code. If you change new SoapServer(…); to new WsdlGeneratingSoapServer(…); (and have Zend_Soap_AutoDiscover, which is part of Zend Framework), you’ll have WSDL generating abillities, too!

Posted in PHP, Programming, Web | Tagged , , | 1 Comment

Updates that bite

Every year, there is the JOTI (See http://www.joti.org/en/index.html and http://jota-joti.scoutingrijen.nl/en/home ) event. We (Scouting Rijen, the local Scouting branch) join the event every year. The last 4 years, I was a part of the team organising that. (But I stopped after last year, as the work that I find interesting is all done.) Simple put, we have about 30 workstations, and a bunch of systems that act as servers for one thing or another. (Diagram)

Since we’re working with kids, and there are always kids that try to screw something up, we want to be able to stop that kid, before we’re kicked off of the servers by the people who manage things world wide. (Which can happen if you spam to much, swear, that type of thing.) But to stop the kid, we need to know he’s screwing up. And with 30 kids at a time and just 4 of us, you can’t do that unless you can centralise everything.

So, thats where I came in. In a few iterations over the years, we started with a simple C program that dumped all the outgoing IRC messages to a console, and ended up with a Java program that scans the outgoing IRC messages, nickname changes, etc. on our gateway server, gives it a spam score based on some criteria, and stores it in a MySQL database on a different server. The second server is a simple LAMP (Fedora to be exact), plus that is has two screens on which we see the newest outgoing messages, colour coded based on the spam score that was given. This happens on a different server, because in our situation (with a budget of €100~€150 a year, mostly relying on donations of hardware) our internet gateway server isn’t the most powerfull machine, and we can’t get a more powerfull machine, but we can get another not so powerfull machine. So different things get different servers. (These are SRS1 (gateway) and SRS3 (LAMP) on the diagram.)

So, that was last year. Everything worked without a problem, traffic logging happened the way we wanted it to, and it proved to be a very usefull tool for us. This year, the guys wanted to change some things. To see if they didn’t break anything, they wanted to first build up the network as it existed last year. Everything looked like it worked, but as soon as the Java program needed to log something, it crashed with a database connection exception. They double-checked the config, they tried again, and it crashed again. Then, they called me. Since I didn’t have anything better to do (this was last night) and I wanted to head over to where they where anyway, as they where going to a bar afterwards, I came over to see what was wrong.

So, I took a look. The Java program was running correctly, it was indeed an issue connecting to the database. Going to that server, the database was running. The output was working properly, I could look into the database using phpMyAdmin, everything was fine. The configuration for the Java program was fine, as well. In fact, nothing had changed, except for one thing: The harddisk of SRS3 was moved to a new system that was much smaller and more powerfull. But the system was running without an issue.

Oh, yea, and it installed some updates. Well, if you start a system for the first time in about 10 months, you can expect a few updates. What we didn’t expect, was that one of those updates would cause the firewall to stop letting traffic trough on port 3306 (which MySQL uses). Problem solved. 🙂 (And luckily it wasn’t somewhere in a production environment – not that I’d use Fedora there.)

Bah, this was supposed to be short.

Posted in Linux, Scouting | Leave a comment

Usability

jory@jory-xubuntu:/$ htop
The program ‘htop’ is currently not installed.  You can install it by typing:
sudo apt-get install htop
jory@jory-xubuntu:/$ sudo apt-get install htop
Setting up htop (0.8.3-1ubuntu1) …
jory@jory-xubuntu:$ htop

And people say Linux isn’t user friendly. 🙂

Posted in Computing, Linux | Leave a comment