Augmenting Natural User Interface
One hot summer in College Station, I received Undergraduate Summer Research Grant (USRG) from Texas A&M University to perform research.
I settled down in the Embedded Systems and CoDesign Lab under Dr. Rabi Mahapatra as my advisor
to learn about the embedded system.
Back then, Microsoft Kinect had recently been unvieled so, we decided on augmenting some of it's capability with a System on Chip embedded system.
Hence, I got a real first hand experience of making an embedded system work for me. It started like any other project
where I had to learn the basic functionality by running through the demo. The system that was available at the time was
Programmable System on Chip (PSoC) supplied by Cypress Semiconductor.
Here is the setup for both the systems:
At that time, Microsoft had not opened their API for Kinect so, I had to use the OpenNI interface that people had created to
communicate with Kinect. But, it was a good learning experience to see the inner workings of how the framework was laid out.
I had some problem installing the software and get it working. After some searching the internet, going through the google group page
and blogs from other enthusiasts I was able to get over the hump.
I was able to write a painting application to take input from the hands via Kinect and draw on the canvas.
Another part of the system was the PSoC which was going to talk to Kinect via my code which listened on the serial port. I had to open
a serial communication port and listen on it. I had two pieces of PSoC working in tandem. One was the wireless reciver which was hooked up to the computer
and other one was a controller which had capacitors on it. Since, it had three capacitors it could be used for selecting from 3 variables and would talk to the other PSoC over
the air. I had some difficulty setting it up but, after some help from other students I was able to get it working.
Once it was all setup I could move my hand around and paint on the computer via Kinect and select the paints and colors based on the PSoC I was wearing
on my hand. Since, it had three capacitors I was able to use it as Red, Green and Blue (RGB) values and select the colors based off of that.
It was interesting and challenging project as I did not have a lot of programming experience with the embedded systems. However, I was
able to complete the task and present my research work. I also had to write a paper on that which has been included below.
Check out the code