John Newcombe

This week my computational intelligence soiree introduced me to computer vision systems. This is all about computers recognising things by their attributes. Typically these could be a combination of colour size shape etc. and could involve a bit of AI and so on. To keep things simple I decided to see if I could get my Mac to detect a lemon.

So with a bit of jiggery pokery I managed to compile and link to the Open Computer Vision library (OpenCV). This simplifies things somewhat and allows us to capture images, manipulate them and display them.

A lemon detector turns out to be quite simple really. The steps we take are as follows;

• capture a frame from a web cam;
• copy this to a second image;
• remove everything that isn’t yellow;
• create a black and white (binary) image where white represents the lemon;
• calculate the shape around the white bit (convex hull);
• draw the shape on the original image.

Pretty simple really, oh… then we do it all again for the next frame and so on. The code is at the bottom of the post:

This worked like a charm. The video displays in realtime and, as long as it is in shot, the red line picks out the lemon even when thrown across the room.

OK so spotting lemons is not that useful, but what about something that can highlight that the person wandering off with your bike, is not the person who parked it? Similarly useful would be  a system that could identify people entering a railway station with luggage, and leaving without it? Of course these sorts of things are already being done and I suppose it is a step in the right direction being able to witness my bike being stolen, even if I still can’t catch the thief or get the bike back. Perhaps, an example of something more useful would be the latest Sony Playstation, with its ‘lemon topped’ controllers.

My interest of course is for Herbert my robot (see earlier post), this technology could give him some form of vision. Therefore it is my intention to build the lemon detector to run on an iPhone and attach this to Herbert. I am not sure where his search for lemons will take him but, should he get lost, at least he will be able to phone home.

The Code

/*
This program has been designed to recognise a Lemon. The program processes images captured from an attached camera a frame at a time. Each frame is copied (cvCopy) in order that manipulation can be applied to one image without affecting the original image.

The copied image is first converted to HSV colour space before being smoothed using a guassian filter. The resultant image is then restricted in range to the colour yellow and converted to a binary image.

Morthological operators are applied in order to clean the image further and the convex hull is determined. The convex hull is then drawn on the normal image.

Helper functions have been used to determine the convex hull an to display a convex hull on an image and care has been taken to release memory, both within the functions and within the main loop, once it is no longer required.

The option to display both the normal and binary image is given with the DISPLAY_BINARY definition.
*/

#include <cv.h>
#include <highgui.h>
#include <math.h>

//Set to 1 to show both the binary and normal image windows.
//Set to 0 to show the normal image only.
#define DISPLAY_BINARY 1

//declare functions
void DrawConvexHull(IplImage *dstImage, CvSeq *hull, CvScalar colour, int width);
CvSeq* GetConvexHullOfImage(IplImage* srcBinaryImage);

//main entry point int main(int argc, char *argv[]) {

//declarations
int key = 0;
IplImage* imageTemp;
IplImage* currentFrame;
IplImage* currentFrameHSV;
IplImage* currentFrameBinary;

//Create and Open camera window
char* windowTitleCamera = “Camera Image”;
char* windowTitleBinary = “Binary Image”;

cvNamedWindow(windowTitleCamera, CV_WINDOW_AUTOSIZE );

#if DISPLAY_BINARY
cvNamedWindow(windowTitleBinary, CV_WINDOW_AUTOSIZE );
#endif

//define the capture device
CvCapture* captureDevice = cvCaptureFromCAM(-1);

//initialise images
currentFrame = cvQueryFrame(captureDevice);
currentFrameHSV = cvCreateImage(cvSize(currentFrame->width, currentFrame->height) , IPL_DEPTH_8U, 3);
currentFrameBinary = cvCreateImage(cvSize(currentFrame->width, currentFrame->height) , IPL_DEPTH_8U, 1);

imageTemp = cvCreateImage(cvSize(currentFrameHSV->width, currentFrameHSV->height) , IPL_DEPTH_8U, 1);

//main loop
while (key != 27) {

//get the current frame from the capture device
currentFrame = cvQueryFrame(captureDevice);

//create a copy of the frame for manipulation
cvCopy(currentFrame, currentFrameHSV);

//use ‘inline’ function to convert to HSV
cvCvtColor(currentFrameHSV, currentFrameHSV, CV_BGR2HSV);

//usea guausian filter to help remove noise
cvSmooth( currentFrameHSV, currentFrameHSV, CV_GAUSSIAN, 7, 7 );

//convert the image to binary whilst setting the range to detect yellow
cvInRangeS(currentFrameHSV, cvScalar(20,100,100), cvScalar(30,255,255), currentFrameBinary);

//clean up the binary image
//use morthological operators (combination of erode and dilate) to clean the iamge somewhat

cvDilate(currentFrameBinary, currentFrameBinary, NULL, 5); cvErode(currentFrameBinary, currentFrameBinary, NULL, 5);

//could use something like this
//cvMorphologyEx(currentFrameBinary, currentFrameBinary,imageTemp, NULL, CV_MOP_GRADIENT, 1);

// Could add outlier removal here…
//

//use convex hull algorithm to find shape CvSeq *hull = GetConvexHullOfImage(currentFrameBinary);

//draw convex hull on to normal frame
if(hull) {
DrawConvexHull(currentFrame, hull, CV_RGB( 255, 0, 0 ), 2);

//tidy up
cvClearSeq(hull);

}

//show the normal captured image in a window
cvShowImage(windowTitleCamera, currentFrame);

#if DISPLAY_BINARY
//show the binary image in a window
cvShowImage(windowTitleBinary, currentFrameBinary);
#endif

//get any key presses
key = cvWaitKey(10);

}
}
}

// Clean up cvDestroyWindow(windowTitleCamera);
cvDestroyWindow(windowTitleBinary);

//need to release images and memory storage here cvReleaseImage(&currentFrame);
cvReleaseImage(&currentFrameHSV); cvReleaseImage(&currentFrameBinary);

return 0;

//Simple function to create a connvex hull of an image.
//The caller is responsible for releasing the memory of the returned CvSeq. CvSeq* GetConvexHullOfImage(IplImage* srcBinaryImage) {

//use convex hull algorithm to find shape CvPoint point; CvMemStorage
*pointStorage = cvCreateMemStorage(0); CvSeq *ptseq = cvCreateSeq( CV_SEQ_ELTYPE_POINT|CV_32SC2,sizeof(CvContour), sizeof(CvPoint), pointStorage );

//wander through the image identifying points that are set to 1
for(int i=0;i<srcBinaryImage->width;i++) {
for(int j=0;j<srcBinaryImage->height;j++) {
if(((uchar*)(srcBinaryImage->imageData + srcBinaryImage->widthStep*j))[i] == 255) {

//we have a pixel set to 1 point.x = i; point.y = j;
cvSeqPush(ptseq, &point);

}
}
}

//create the hull
CvSeq *hull = cvConvexHull2(ptseq, 0, CV_COUNTER_CLOCKWISE, 0 );

//tidy up
cvClearMemStorage(pointStorage); cvClearSeq(ptseq);

return hull;

}

//Helper function to draw a convex hull on an image
void DrawConvexHull(IplImage *dstImage, CvSeq *hull, CvScalar colour, int width) {

if(hull) {

CvPoint pt0 = **CV_GET_SEQ_ELEM( CvPoint*, hull, hull->total – 1 );

//wander through the hull drawing from point to point
for( int i = 0; i < hull->total ; i++ ) {

CvPoint pt = **CV_GET_SEQ_ELEM( CvPoint*, hull, i );
cvLine(dstImage, pt0, pt, colour, width);
pt0 = pt;

}
}
}

For the last few months I have been looking at Fuzzy Logic and Fuzzy Sets. These things are great, they allow you to build software systems that are based on linguistic rules such as , “If the car in front is close, then brake”. The fuzzy bit is the term ‘Close’, what is close? Certainly my idea of close is different to many others on the road.

It’s this ability to program for vaguness that’s cool. Following this I started looking at Type 2 Fuzzy Logic systems, this adds a third dimension to the processing and to be honest gets a bit silly in the amount of computing power its uses, but there are simplifications that make it workable. Ok so where is this leading, to be honest I have no idea, I shall just enjoy the ride.

I have been looking at behavioural robotics these last few weeks and the result, Herbert, has been wandering around our living room, much to the amusement of our family and friends.

Herbert is made of Lego and doesn’t really do too much. He wanders around the room avoiding obstacles and, when he finds some food, or more acurately something green on the floor, he slows down as if to eat it. The point of course, is that it is an exercise in programming not one of application. For me, this exercise in programming has taken me back to basics. This week for example, Herbert been used to demonstrate the effectiveness of the ‘finite state machine’.

Finite state machines are incredibly simple to code and very effective. They go something like this:

//(pseudo code)
while(true){
  switch(currentState)
  {
  case stateForward:
    if(obstacleDetected)
    {
      setReversingTimer()
      currentState = stateReversing
    }
    else
    {
      forward()
    }
  case stateReversing:
    if(reversingTimeElapsed)
    {
      setTurningTimer()
      currentState = stateTurning
    }
    else
    {
      reverse()
    }
  case stateTurning:
    if(turningTimeElapsed)
    {
      currentState = stateForward
    }
    else
    {
     turn()
    }
  }
}

Ok so now we have a state machine handling our robots meandering. We can easily add extra states e.g. ‘Feeding’ and so on. However, as all of the wandering and avoiding is done on a single thread of execution, what we could do is add the extra behaviours to additional threads. This way we end up with a whole stack of state machine controlled behaviours all accessing the same motors and sensors. We need a bit of arbitration to control access to the motors and sensors but this approach (Subsumption; Brooke, 1986) is great for our Herbert.

The thing is that should anyone ask how he works, I can answer by describing the above with large helpings of geekish enthusiasm. The trouble is that all I ever get asked is ‘What does it do?’ In which case I end up replying, ‘Not much really’.