CMUcam4

Hi Patrick,

The CMUcam4 is a black box color tracking sensor. So... to answer your questions:

1) The Hough transform is not built into the software. The CMUcam4 does not have a buffer to process the image it is working on. It loads 160 16-bit pixels at a time from the image in a temporary storage buffer and preforms color thresholding on those pixels with a few other calculations. The CMUcam4 firmware never sees more than 160 pixels of the image at a time and has a VERY limited time to preform all calculations.

2) You have the right idea about what you need to program the chip. If you head over to the firmware source code and binaries page you'll find information on how to program the chip. All the image processing stuff is hand coded assembly language. Depending on where you want to insert yourself in the image processing pipeline you'll either need to dive into the assembly or not.

Basically, there's a dual core image processing driver that segments the image the CMUcam4 sees at 30 FPS. One processor samples 160x60 pixels of the image and the other processor samples 160x60 of the image. Both processors interleave their sample points to process a 160x120 image. The 160x120 image comes from a down-sampled 640x480 image that the image sensor is sending to the main processor at 30 FPS.

You do not want to touch the assembly code... it is very tightly coded and meets some very tight deadlines. Messing with it is possible... but, it will be way harder than its worth. The assembly code uploads a 160x120 1-bit per pixel segmented image to main memory that is double buffered. From here you have a bit more time to breath and can copy the image to some other point in memory and do whatever you want to it.

3) The cable you are asking about is correct.

4) The CMUcam4 cannot record video. It can only take long exposure images... meaning it can take a picture by building the image out of vertical lines of pixels captured per frame at whatever the frame rate is. So at 30 FPS it can capture 30 columns of an image a second.

...

So, I think you will be disappointed by the CMUcam4 for use as a general purpose vision system. It doesn't use the most powerful processor available. The reason for this is that it's hard to design a system for relatively cheap that needs external DRAM or SRAM to hold an image. When you get into having a processor with a general purpose frame buffer you need to design the board to have multiple layers and use specialized and expensive processors and pin packages. Couple this with the project only being mainly worked on by one person in their free time and it becomes more effort than it's worth. This is because if I build a more powerful system I have to provide more features for it - too much for one guy :).

...

Now, what the CMUcam4 can do is track colored object blobs very well. If you use it as a black box sensor and ignore all the complex things going on in the firmware that makes it run on a low cost processor (in low volume) it will preform very nicely for you.

So, you mention that you want to track lines... does this mean you want to use it in a line following application? Because the CMUcam4 does have features designed in it for that purpose.

Using the "line mode" command the CMUcam4 can send you host processor a 80x60 binary bitmap of whatever color it is segmenting the image by at 30 FPS. You can then compute the center of mass of all the pixels on each row in the binary bitmap image. This will then give you 60 centroids which will tell you the path of the line in an image that you are looking at. Assuming you choose the color space (RGB/YUV) and color thresholds appropriately the only 1-bits (tracked pixels) in the 80x60 binary image will be from whatever line you wanted to follow.

The next step to make the robot follow the line is to feed the 60 centroids to some kinda of filter that outputs a number telling the robot if it needs to turn or not. That number will then be feed into a PID loop for motor control and that's it. The CMUcam4 has these features so that it can be used to compete in the mobot competition - an outdoor line following contest.

If you need to be able to handle decision points for line following the binary 80x60 image can also be used for that to. In fact, I suppose you could preform the Hough Transform on the 80x60 binary image also. The CMUcam4 internal works on a 160x120 image. The reason why it sends only a 80x60 image out is that you cannot push a 160x120 image down a serial wire at 30 FPS at 250K baud.

...

So... we have an Arduino interface library that takes care of all the communication stuff for you so that you can just use the Arduino for everything. However, the Arduino and Arduino Mega (preferred because is has multiple serial ports) may lack the processing power to process the 80x60 binary bitmap sent at 30 FPS. I designed the interface library to be portable, however. If whatever system you have can implement a certain list of primitive operations then you can use the CMUcam4 interface library on any processor you want.

If you have more questions about this feel free to ask. Right now I'm finishing up the finalized version of the interface library. We have a beta version online right now.

Thanks,

Wow!
Thanks for the quick followup. This still sounds an awful lot like I should just get one of these and start playing with it for real (instead of playing with it in my head). I'll work on ordering one of them tomorrow. I might be hooking this up to something a lot more like a Beagleboard than an Arduino. Still trying to figure out that end. (And perhaps this won't be the best solution, but it's captured my imagination at the moment, perhaps I should have searched for "Beagleboard+camera" instead of "Arduino+Camera", which is how I found this... hmmm...).

In the mean time...Back to question 3,

Should I order the 3.3V or the 5V variant of the FTDI cable? It's not clear from the schematics alone whether Vcc is 5V or 3.3V. I'm assuming 3.3V, but since I have the opportunity to ask, I'll go ahead and ask :-)

I do want to use this in a line following application, but I still haven't the foggiest idea what I don't know that I don't know yet. I'm curious about your statement regarding the "mobot competition - an outdoor line following contest" vis-a-vis a statement I read somewhere on the wiki about the camera not functioning well in direct sunlight and that perhaps a filter might be needed. Could you elaborate more on that?

Thanks again for the quick (and very informative) response. I'm looking forward to playing (errr, "working") with this!

--wpd

It works outside fine. Lots of IR light from the sun might cause you to need an IR filter (like lens from a pair of sunglasses). I copied that over from the CMUcam1 wiki when I was porting the web pages (hurriedly) and didn't think much about the implication that statement has. I think I've updated that now.

Order the 5V FTDI cable. Truly, it doesn't matter because both of the cables supply 5V VCC which is what the CMUcam4 needs. The 3.3V cable just has 3.3V I/Os while the 5V cable has 5V I/Os. The CMUcam4 is 3.3V and 5V tolerant.

Thanks,

ok, I just pulled the trigger. I should get some toys sometime next week I hope :-)

I guess I should have asked this before I spent my $$$. But better late than never. Have you any experience using the CMUcam to recognize self-similar patterns? What sort of roadblocks do you anticipate I will run into if/when I try this myself? I am wondering if following fiducials might be better/easier than following a line.

As always, thanks for the tips, and for your work on this project.

Oh yeah, on a completely unrelated tangent... is there any way to configure your redmine server to send email replies? It would be much easier for me to track threads that way.

Thanks again.

--wpd

You can use the histogram feature and the window size feature to look at the histogram of parts of an image. Different textures have different histograms.

And... I think I found the option to enable email notifications.

Thanks,

OK, I just got my shiny new CMUCam4 and a 5V FTDI cable (as per your suggestion). When I plug things in green lights come on, red lights go on and off, and blink, and I see a bounding box vaguely around a red lego brick I wiggle in front of the camera, all according to what I've read on the Quick Start page.

Hooray.

Now what...

I'd like to update the software on the device. But I just noticed the caveat on the BST section (since I have a Mac) on the Quick Start page that says "NOTE: You cannot use the FTDI 5V Cables with BST!". I'm hoping that (like last time I got worried about something in the wiki) this is an artifact of an older revision of the CMUCam, or BST, but I thought I should ask.

Also, I followed the BST instructions on the Quick Start page to set the baud, number of bits, parity, etc... but am not sure what to do when it says "Go to Communicate and select Connect, Terminal Echo, and Reset Propeller from the drop-down menu", since I don't have a "Communicate" menu in my version of BST (0.19.3).

So, can I use the 5V FTDI cable with my shiny new CMUCam4? Do I need to recompile BST to wiggle RTS correctly? (Is that even possible?)

I'm gonna go eat my supper now and think on this some more...

--wpd

FWIW, I pulled the green wire as described on the Quick Start guide and discovered that I needed to select the port in BST in order to enable the communication menu, and now I can communicate with the CMUCam4.

I do wonder, however, if I will be able to load code on the device without the ability to reset it. I expect it will be a somewhat manual, laborious process of getting the timing right between pressing the reset button and clicking on whatever "download code" option is available in BST.

We'll see.

--wpd

Your only option is to recompile BST to wiggle to RTS line or look into more open source solutions. Parallax finished Prop GCC after I wrote that article. So... there are other options now.

Look into Prop GCC. In particular the propeller loader can load SPIN bin files onto the propeller chip. So, you will be able to use this to reload the system.

Thanks,