Adding to the firmware
Added by Len Marinaccio 11 months ago
Developers must hate this kind of question but here goes:
I understand that the CMUcam4 is supposed to be a "black box" which processes images and returns data. Very nice, really. However, it seems short in its motion detection capabilities. One could set up a "grid" and pass each cell coordinate set to the CMUcam4 as a tracking window, get statistics and save them for later comparison. Do this for all cells in the grid and you have a statistical map of the image and by sweeping it again and again looking for variances, you could figure out what might be moving.
However, it seems this would be very time consuming! It relies on constant polling and I'm not sure the speed would be acceptable. I would like to know if I can add capability to the existing firmware to do frame comparison that actively identifies motion areas. This is important because color tracking can easily fail when the subject being tracked is colored similarly to the background. But with temporal motion detection, you can still catch the subject as it moves, even a little.
How many cores of the Propeller are in use? Is it possible to requisition one (even if it means dismissing the FAT functions or something) to do this detection?
On top of that, my entire application could be run simply in a single core, alleviating the need for an Arduino altogether. Since the hardware I was planning to build is almost exactly what the CMUcam4 is (less the SD card port) it would be nice just to modify the firmware to suit my needs.
Thanks!
Len
Replies (4)
RE: Adding to the firmware - Added by Kwabena Agyeman 11 months ago
Sure, if you know how to program in SPIN you'll be fine. In fact, if you know how to program in C you can program in SPIN. Just make sure to have the reference data sheet for SPIN operators handy. The code has a number of tricks in it so... if you see something really weird consult the reference manual. You'll need to use the Propeller Tool since I use some operations that don't seem to compile in BST. If you feel like translating the non compiling operators in BST to something that compiles then you can then use BST if you want. BST is a cross platform IDE for the propeller chip.
I'm using 7 cores nominally, and then when the DF, SF, DB, or SB functions are run I use all 8. So... if you don't have need for the file system stuff you can reclaim about 8 KB for code, data and a core. Then the file system functions in the main firmware can also be cut out for more space... let's say 1 KB. Then since you won't need the the DF or DB drivers also and their stack space... you can get another 1 KB and a core.
So, 10 KB and 2 free cores to work with.
Please understand that SPIN is slow. Do not use SPIN for anything other than business logic and byte/word/long moves/fills. If you need fast processing speed you'll have to write the code in PASM. This is easy for me since I'm an expert. But, it can be difficult for anyone who does not know how to program in assembly. If you do know how to program in assembly then you'll find the propeller chip assembly to be extremely peasant to work with.
If you need clarification on anything just ask. For general purpose help please register at the parallax forums and use the collective body of understanding there. Questions on the parallax forum are usually answered in hours.
Thanks,
PS: Parallax just got GCC working for the propeller chip. If you feel adventurous, then you could code the image processing code in C. Compile it using GCC to run in COG mode where the C code runs natively at 24 MIPs, then include the assembled binary blob in the SPIN code using the "file" directive. As long as you pass all the pointers the cog needs to run using the PAR register this technique will work. It may take a few tries to get it operational but it's theoretically and practically possible.
RE: Adding to the firmware - Added by Len Marinaccio 11 months ago
The Propeller is relatively new to me but C is no problem. I've long since lost count of the number of different assembly languages I've used over the years so I can literally get up to speed with that in a couple days.
The challenges will be getting the hang of the Propeller environment and spending the time to get acquainted with the CMUcam4 code.
How much RAM do you use and what's left over?
Len
RE: Adding to the firmware - Added by Kwabena Agyeman 10 months ago
Hi, the propeller has a shared code and data space. If you make the changes I suggested, this frees you 10 KB for code and data.
I'm using 99.999% of the memory by default for the application. Basically, we just kept feature creeping until we ran out of RAM (code and data space).
Thanks,
RE: Adding to the firmware - Added by Kwabena Agyeman 10 months ago
The Line Grabber Driver code is crazy code. I had to do a lot of optimizations and tricks to fit the code in the cogs that do all the image processing. I used every register available and then I even used shadow registers in the COG RAM space for variable storage. If you search in the manual for shadow registers you'll see what I am talking about. All the image processing is done in hand coded ASM and is optimized to the limit. It's actually a dual processor solution too. I have two processors running at once working on the image to get 160x120 30 FPS. Once core does 160x60 30 FPS and another core does 160x60 @ 30 FPS.
Also, you'll notice a "LNSwitch" variable in the code. The way things work is that all the data structures are double buffered. The LNSwitch variable is either 0 or -1. When it's 0 it says that the Line Grabber Driver is currently modifying the first half of any double buffer. When it's -1 it means that the Line Grabber Driver is currently modifying the second half of any double buffer.
So, what's double buffered is:
The tracking data which is an 8 byte structure.
The segmented image which is an 2,400 byte structure.
The 128 bin histogram which is a 258 byte structure.
Since all of these are double buffered they take double the space. The Line Grabber driver just runs in the background updating these variable data structures continuously. So, you can tap into the data being output by just copying whatever half of the double buffer the Line Grabber Driver just got done producing. By looking for when the LNSwitch variable changes you can see when new data just got produced for all three of the double buffers. This means the LNSwitch variable should be toggling between 0 and -1 at 30 HZ. I built timeouts into all my code that looks for this 30 HZ toggling so that the Servo Driver and TV Driver code does not behave weirdly if you were to remove the camera module in operation and have the Line Grabber Driver hang looking for the next frame from a missing camera.
There are plenty of examples in the code on how to copy whatever part of the buffer the Line Grabber Driver just got done producing.
If you need to modify the line grabber driver... understand that you need to remove some feature from it first. Also understand that the timing is really tight. So, you can't do whatever you want in the line grabber driver's central processing loop.
Thanks,
(1-4/4)