Windows 7 Camera Fundamentals

News

Extraordinary Robot
Robot
This video covers the basics of reading camera data from the Kinect sensor. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides.
  • [00:25] Camera data information
  • [03:30] Creating the UI
  • [04:48] Initializing the Kinect runtime
  • [07:18] Reading values from the RGB camera
  • [11:26] Reading values from the Depth camera
  • [13:06] Adjusting camera tilt
[h=3]Setup[/h]The steps below assume you have setup your development environment as explained in the "Setting Up Your Development Environment" video.
[h=2]To get a depth or color image you must do the following[/h]
  1. Create an instance of a runtime object
  2. Initialize the instance with the correct options
  3. Subscribe to an event to receive camera data
  4. Open the video stream
  5. Convert the raw camera data into an image
[h=1]Task: Display the RGB Camera Image[/h][h=2]Designing your UI[/h]We’ll add two 320x240 Image controls to the MainWindow XAML file as shown in the following XAML:

XAML

http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="
image%5B2%5D.png

[h=3]Initializing the runtime[/h]In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseColor to use the RGB camera:

C#

nui.Initialize(RuntimeOptions.UseColor);
Visual Basic

nui.Initialize(RuntimeOptions.UseColor)[h=3]Subscribe to the VideoFrameReady event[/h]To read values from the RGB camera, add an event handler for the VideoFrameReady event on the Kinect Runtime in the Window_Loaded event as shown below:
Tip: You can have Visual Studio automatically build the event handler for you by typing "nui.VideoFrameReady +=" and hitting the tab key twice.
C#

nui.VideoFrameReady += new EventHandler(nui_VideoFrameReady);void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e){ // event code goes here}
Visual Basic

AddHandler nui.VideoFrameReady, AddressOf nui_VideoFrameReadyPrivate Sub nui_VideoFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs) ' event code goes hereEnd SubUnderstanding the video frame ready returned values
The video frame returns an ImageFrameReadyEventArgs that contains an ImageFrame class. As shown below, the ImageFrame class contains two things:
  • Metadata about the image, such as ImageType to know if it’s a depth or color image, and resolution to know the size of the image
  • Image – the actual image data, which is stored in the Bits byte[] array

[h=2]Converting the byte[] array to an image[/h]To convert the byte[] array that represents the camera image to display it in an Image control (ex: image1 below), call the BitmapSource.Create method as shown below. The last parameter is stride. The stride is the number of bytes from one row of pixels in memory to the next row of pixels in memory. Stride is also called pitch. For more information, go to http://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx:

C#

PlanarImage imageData = e.ImageFrame.Image;image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, PixelFormats.Bgr32, null, imageData.Bits, data.Width * imageData.BytesPerPixel);
Visual Basic

Dim imageData As PlanarImage = e.ImageFrame.Imageimage1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, _ PixelFormats.Bgr32, Nothing, data.Bits, imageData.Width * imageData.BytesPerPixel)The Coding4Fun Kinect Toolkit has an extension method built into the ImageFrame class that simplifies creating the bitmap:

C#

image1.Source = e.ImageFrame.ToBitmapSource();
Visual Basic

image1.Source = e.ImageFrame.ToBitmapSource()[h=3]Getting Supported Resolutions[/h]The Kinect cameras can support different resolutions depending on what ImageType is passed in. To determine these resolutions, call the GetValidResolutions on the ImageStream object:

C#

var x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex);
Visual Basic

Dim x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex)Open the VideoStream
Before the event will fire, you must call the Open method on VideoStream as shown below. This opens a video stream, sets the PoolSize to 2, sets the resolution to 640x480, and sets the return image as a color image:

C#

nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
Visual Basic

nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color)Understanding PoolSize
Pool Size is the number of buffers you want queued up. Usually, you want one displaying, and one being filled. The minimum is two but specifying more than two will give you slightly more latency in the frames you display, and so you’ll be more likely to have one waiting for you. The trade off, then, is smoother playback but higher latency.
[h=1]Task: Displaying Depth Data[/h]To add depth data, we use the same steps as above.
[h=3]Update the Runtime Initialization[/h]First, we’ll update the Initialize method to include RuntimeOptions.UseDepth to receive depth data from Kinect:
C#


nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);
Visual Basic

nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepth)[h=3]Subscribe to the DepthFrameReady event[/h]Next, add the event handler to receive the Depth information. For simplicity, we’ll use the Coding4Fun Kinect toolkit to display the depth image in the image2 Image control:
C#


nui.DepthFrameReady += new EventHandler(nui_DepthFrameReady);void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e){ image2.Source = e.ImageFrame.ToBitmapSource(); }
Visual Basic

AddHandler nui.DepthFrameReady, AddressOf nui_DepthFrameReadyPrivate Sub nui_DepthFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs) image2.Source = e.ImageFrame.ToBitmapSource()End Sub[h=3]Open the Depth Stream[/h]Finally, we’ll call DepthStream.Open to open a 320x240 depth stream:
C#


nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);
Visual Basic

nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth)The result is shown below:

[h=1]Task: Adjust the Kinect Camera Tilt[/h]
Warning: The tilt mechanism in the sensor array is not rated for frequent use. Your code should not make calls to tilt the device more than 15 times in any two-minute window. Changing the tilt too often results in an error message from the function.
[h=3]Setup[/h]Like before, create an instance of the Kinect Runtime class and call the Initialize method:
C#


Runtime nui = new Runtime();nui.Initialize(RuntimeOptions.UseColor);
Visual Basic

nui.Initialize(RuntimeOptions.UseColor)[h=3]Adjusting the tilt[/h]To adjust the tilt or pitch, set the ElevationAngle property to a value between –27 (Camera.ElevationMinimum) and +27 (Camera.ElevationMaximum). The code below sets the camera to the maximum elevation:
C#


nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum;
Visual Basic

nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum
njs.gif


More...
 
Back
Top