|OpenCV for 3D, the Point Cloud Library (PCL)|
"The Point Cloud Library (or PCL) is a large scale, open project  for point cloud processing. The PCL framework contains numerous state-of-the art algorithms including filtering, feature estimation, surface reconstruction, registration, model fitting and segmentation. These algorithms can be used, for example, to filter outliers from noisy data, stitch 3D point clouds together, segment relevant parts of a scene, extract keypoints and compute descriptors to recognize objects in the world based on their geometric appearance, and create surfaces from point clouds and visualize them -- to name a few."
PCL is middleware that "plugs in" to the OpenNI framework. "The OpenNI Framework provides the interface for physical devices and for middleware components. The API enables modules to be registered in the OpenNI framework and used to produce sensory data. " http://www.openni.org/
Basically, the PCL library is to 3D sensors what OpenCV is to 2D sensors (regular cameras). Where OpenCV processes a Mat (2D surface) PCL processes a point cloud. A 2D surface is made up of pixels, a point cloud is made up of points (x,y,z).
The PCL library is an open source library released under the BSD license (it is free for commercial and research work). The project is supported by some heavy hitters like: Toyota, Google, Intel and NVIDIA.
To learn more about some of the cool things you can do with PCL and your Kinect, read the tutorials: