Bookmark and Share

Software Topics

Is OpenCV being deployed in any high-volume products?
18 replies [Last post]
Scott Gardner
Scott Gardner's picture
Offline
Last seen: 6 years 37 weeks ago
Level 9: Necromancer
Joined: 2011-02-09
Points: 366

Hopefully everyone saw the interesting video of our conversation with Gary Bradski.  I asked Gary this question, but he said that he doesn’t usually hear much about where his library is being deployed.  Join the discussion, earn some experience points, and give Gary some feedback about how OpenCV is being used.

squreshi
Offline
Last seen: 1 year 41 weeks ago
Level 4: Thaumaturgist
Joined: 2011-05-31
Points: 95

At NVIDIA GTC 2012, it was mentioned that Google Maps, Google Street View, and Google Earth all OpenCV.

NelsonBridwell (not verified)

I am glad to see that current OpenCV supports integer image formats!  I was very perplexed many years ago when I read documentation indicating that image pixels were represented as 4-channel 32b FP numbers.

An additional consideration is the extent to which OpenCV tends to require floating point computations, which might not be well-supported on embedded processors.

In the example below it took OpenCV 31 seconds to perform a face-recognition task on one ARM processor, using floating point emulation.  On another processor, it took 448 seconds.

On industrial machine vision smart cameras, all of the processing is performed using DSP integer math, which results in sub-millisecond typical execution times for filtering or pattern matching functions.  

http://www.computer-vision-software.com/blog/2009/03/arm-wrestling-with-opencv/

David Thompson
Offline
Last seen: 6 years 40 weeks ago
Level 4: Thaumaturgist
Joined: 2011-06-03
Points: 65

Since 1.1 OpenCV has supported many pixel formats (I believe channels where always interlaced).  In particular 1-4 channel 8-32 bit pixel formats are part of the IplImage format.  Most algorithms branch immediately based on the image format.  For embedded development I am aware of one framework that already integrates OpenCV.  Arago, a subproject of OpenEmbedded supporting many architectures include OMAP and Davinci, includes OpenCV in it's distribution.

Scott Gardner
Scott Gardner's picture
Offline
Last seen: 6 years 37 weeks ago
Level 9: Necromancer
Joined: 2011-02-09
Points: 366

David Thompson wrote:

For embedded development I am aware of one framework that already integrates OpenCV.  Arago, a subproject of OpenEmbedded supporting many architectures include OMAP and Davinci, includes OpenCV in it's distribution.

David, thanks for the info.  I'll take a look at Arago Project.  I did find that they have a 4-part demo on YouTube.

Scott

 

NelsonBridwell (not verified)

It looks like OpenCV has come back to life since it was adopted by Willow Garage a few years ago.  There appear to be significant development efforts and a number of improvements.  Because it is entirely free, support, however, could be problematic.  I am not sure that there is anyone that you will be able to call to request an urgent bug fix, particularly for that module that was written by a grad student who graduated 5 years ago, and now works as a DBA at Merrill Lynch...

My limited exposure to it was while looking at an attempt to organize a mobile robot software competition that included a visual odometry event where robots needed to use vision to accurately track their positions.

http://www.mobilerobot.org/MRSC/Competition.htm

As a test, I successfully tried out the camera calibration function that uses multiple 3D images of a checkerboard, but I decided to give it a pass for an serious uses because of efficiency concerns.

 

Scott Gardner
Scott Gardner's picture
Offline
Last seen: 6 years 37 weeks ago
Level 9: Necromancer
Joined: 2011-02-09
Points: 366

NelsonBridwell wrote:

It looks like OpenCV has come back to life since it was adopted by Willow Garage a few years ago.  There appear to be significant development efforts and a number of improvements.  Because it is entirely free, support, however, could be problematic.  I am not sure that there is anyone that you will be able to call to request an urgent bug fix, particularly for that module that was written by a grad student who graduated 5 years ago, and now works as a DBA at Merrill Lynch...

Nelson, I don't know if you watched the entire video of our discussion with Gary, but I asked him the question about support for open-source.  It's really not a unique problem, since a cottage industry of consultants have sprung up to support the OpenCV library -- just like Linux and other open-source code.  It's obviously not the same as just calling up Mathworks for support by a person who is paid to help you, but it sounds like OpenCV has really taken off in a lot of products.  I'm glad Willow Garage is letting Gary continue to support it, but I share a bit of your skepticism for code written by volunteers.  Hopefully others will join this discussion to offer some insight.

NelsonBridwell (not verified)

All of my comments are based upon my experiences a few years ago when it was unsupported.  Circumstances may have recently changed, in which case some of my observations (but probably not all) might no longer apply.

 

NelsonBridwell (not verified)

When I was trying out OpenCV several years ago (2007) the only documented image format employed quad 32 bit floating point numbers for each pixel.  When I mentioned this to a university professor he replied that he had alwasy wondered why an image processing project that they had worked on had such slow frame rates.

OpenCV has also not been recently developed or supported by Intel, other than the performance primitives optimization.  Any support for OpenCV is purely voluntary and support forums tend to be clogged by beginning students who are working on class projects.  New functions have been the product of university researchers, who do not routinely tend to focus on performance.

If I was interested in an embedded 2D vision application, I would seriously look at porting other image libraries.  If I needed some of the unique OpenCV capabilities, I would look at porting that particular module from OpenCV, rather than all of OpenCV.

David Thompson
Offline
Last seen: 6 years 40 weeks ago
Level 4: Thaumaturgist
Joined: 2011-06-03
Points: 65

As of 2007, OpenCV supported many, many image formats including 1-4 channel 8-bit formats, as well as 1-4 channel 32-bit formats. 

Scott Gardner
Scott Gardner's picture
Offline
Last seen: 6 years 37 weeks ago
Level 9: Necromancer
Joined: 2011-02-09
Points: 366

NelsonBridwell wrote:

OpenCV has also not been recently developed or supported by Intel, other than the performance primitives optimization.  Any support for OpenCV is purely voluntary and support forums tend to be clogged by beginning students who are working on class projects.  New functions have been the product of university researchers, who do not routinely tend to focus on performance.

Nelson, ahh, now you highlight the reason for the existence of the Embedded Vision Alliance. laugh

As you point out, most of the knowledge base about computer vision has come from Academia, and there are few resources for embedded system designers who aren't interested in publishing a research paper, while also needing to build a real system that actually works in a constrained power budget.

As you say, while Intel's IPP was the basis for OpenCV, it doesn't seem like Intel is working much on optimizations anymore.  However, Gary said that nVidia has a team that is working to optimize the library for GPUs.

I also think you highlight the overall limitation of open-source, since it is mostly voluntary support.  Mathworks has joined EVA as a Member, and I'm working to get more technical content that takes a deep dive into their library and perhaps a few of the custom Matlab libraries people have developed.

If you have a few samples of your code or some detailed performance numbers or charts, feel free to share.  I've set the file attachment limit to 20MB, so you could even upload some images.

Thanks again,

Scott

david moloney
Offline
Last seen: 1 year 34 weeks ago
Level 2: Evoker
Joined: 2011-06-01
Points: 18

OpenCV images are NOT stored as floating-point numbers.

The primary image data-structure in OpenCV is Iplimage:

http://opencv.willowgarage.com/documentation/basic_structures.html

In that data-structure pixels are stored as three 8-bit unsigned values for each of the Blue, Green and Red channels:

 

char *imageData;

 

There is no reason why OpenCV can't be used in embedded applications and in fact it is.

 

Scott Gardner
Scott Gardner's picture
Offline
Last seen: 6 years 37 weeks ago
Level 9: Necromancer
Joined: 2011-02-09
Points: 366

david moloney wrote:

There is no reason why OpenCV can't be used in embedded applications and in fact it is.

David, thanks for joining the conversation.  Do you know of a few example OpenCV-based embedded products that ship in decent volume?  It would be great to get some feedback to Gary Bradski.  Perhaps we can get him on camera again to see how his project is going.

NelsonBridwell (not verified)

Direct porting of OpenCV code is not well suited for real embedded industrial or consumer applications because it uses a higly inefficient data format for image representation.  The intensity of each pixel is represented by 4 floating point numbers.  Such a format, although possibly useful for a few limited applicaitons, is significantly slower and takes up much more memory than traditional integer DSP algorithms.

squreshi
Offline
Last seen: 1 year 41 weeks ago
Level 4: Thaumaturgist
Joined: 2011-05-31
Points: 95

I believe others have already responded to this, but it is not true that OpenCV represents images a 4 floating-point numbers. Perhaps the poster's confusion arises from the fact that if you use the OpenCV macros/functions like cvGetM() etc. then the resulting return type is cvScalar which is a C union of an array of four elements. But for efficient access into the IplImage, you don't use that and the native format can be one of IPL_DEPTH_8U, IPL_DEPTH_16U, IPL_DEPTH_32F, or a few others. And you can have multiple channels.

David Thompson
Offline
Last seen: 6 years 40 weeks ago
Level 4: Thaumaturgist
Joined: 2011-06-03
Points: 65

This is incorrect information, as a user of OpenCV.  I can attest to the fact that the iplimage format support many, many different image packing formats including 1-4 channel with 8 or 32 bit  pixel formats (I believe the channels are always interlaced).  The iplimage format provides enough generality for iterating over all of these different formats using pointer arithmetic.  In addition, many algorithms inside of OpenCV have a primary branch on the image type (I would suggest downloading the source code for OpenCV 1.1+ if you doubt this information).  In addition, it does compile and run for Arago ( a subproject of OpenEmbedded), which runs on many TI omap and davinci series processors.

Scott Gardner
Scott Gardner's picture
Offline
Last seen: 6 years 37 weeks ago
Level 9: Necromancer
Joined: 2011-02-09
Points: 366

NelsonBridwell wrote:

The intensity of each pixel is represented by 4 floating point numbers.  Such a format, although possibly useful for a few limited applicaitons, is significantly slower and takes up much more memory than traditional integer DSP algorithms.

Nelson, I'm no expert on OpenCV, but I did play around with the code for a bit.  While FP is an option, the current version allows you to specify the pixel format.  Here's the reference from the docs:

"If CV_LOAD_IMAGE_ANYDEPTH is passed the pixel format can be 8 bit unsigned, 16 bit unsigned, 32 bit signed or 32 bit floating point."

Anyone else in the (rapidly-growing) community have a comment?  I'd be really interested in hearing what format nVidia is supporting in their optimized library.  I suspect they run floating-point really fast.

Scott
 

RobbySun
Offline
Last seen: 2 weeks 5 days ago
Level 3: Conjurer
Joined: 2011-05-27
Points: 42

That's an interesting question.

rahulp
Offline
Last seen: 7 years 14 weeks ago
Level 2: Evoker
Joined: 2011-06-10
Points: 18

Based on our understanding of implementing OpenCV on embedded devices, we recently put out an white paper that looks  into mitigating the challenges of porting OpenCV to embedded platform and looks into implementation of OpenCV on TI`s heterogeneous ARM+DSP platforms. 

Refer : http://focus.ti.com/lit/wp/spry175/spry175.pdf