Back in mid-April, I shared with you the video of the March 2012 Embedded Vision Alliance Member Summit keynote from Jim Donlon, a Program Manager for DARPA (the U.S. Defense Advanced Research Projects Agency). Donlon's specific responsibility is the "Mind's Eye" program, which strives to harness computer vision intelligence as an alternative to human video observers in identifying objects and their interactions. Specifically, Mind's Eye "seeks to develop the capability for visual intelligence by automating the ability to learn generally applicable and generative representations of action between objects in a scene directly from visual inputs, and then reason over those learned representations."
One month later, I contrasted DARPA's completely automated Mind's Eye approach with that of the Descriptive Camera, developed by New York University's Interactive Telecommunications Program (NYU ITP). The Descriptive Camera harnesses human beings to create the associated metadata for an image. As I wrote, quoting from the developer's documentation, "The technology at the core of the Descriptive Camera is Amazon's Mechanical Turk API. It allows a developer to submit Human Intelligence Tasks (HITs) for workers on the Internet to complete. The developer sets the guidelines for each task and designs the interface for the worker to submit their results. The developer also sets the price they're willing to pay for the successful completion of each task. An approval and reputation system ensures that workers are incented to deliver acceptable results.
"For faster and cheaper results, the camera can also be put into "accomplice mode," where it will send an instant message to any other person. That IM will contain a link to the picture and a form where they can input the description of the image… After the shutter button is pressed, the photo is sent to Mechanical Turk for processing and the camera waits for the results. A yellow LED indicates that the results are still "developing" in a nod to film-based photo technology. With a HIT price of $1.25, results are returned typically within 6 minutes and sometimes as fast as 3 minutes. The thermal printer outputs the resulting text in the style of a Polaroid print."
Now back to DARPA, and with thanks to Extremetech for the initial coverage that caught my eye, is a surveillance approach which intriguingly blends the Mind's Eye and Descriptive Camera schemes The intent of the Cognitive Technology Threat Warning System (CT2WS) is to improve threat detection in combat settings. Quoting from the press release:
For warfighters operating in the field, the ability to detect threats from standoff distances can be life-saving. When advanced radar and drone coverage is not available, warfighters typically rely on their own vision to scan their surroundings. Scanning over a wide area, though, is challenging because of the amount of territory that must be reviewed, the limited field of view of the human eye, and the effects of fatigue. Current technologies like binoculars, cameras, and portable radars can help to improve visibility and increase the threat detection rate. Unfortunately, current miss rates of 47 percent or greater using these technologies leave warfighters unprepared and vulnerable.
DARPA launched the Cognitive Technology Threat Warning System (CT2WS) program in 2008 with the goal of maximizing warfighters’ awareness of their surroundings by developing man-portable visual threat detection devices. CT2WS succeeded in creating a technology kit capable not only of identifying up to 91 percent of targets during testing with extremely low false-alarm rates, but also widening a warfighter’s field of view to 120 degrees when all components of the kit are used in tandem. By incorporating a commercial radar (the Cerberus Scout surveillance system), target detection reached 100 percent…
…The CT2WS system includes three component technologies: a 120-megapixel, tripod-mounted, electro-optical video camera with a 120-degree field of view; cognitive visual processing algorithms that can be run on laptops or more powerful computers to identify potential targets and cue images for operator review; and an electroencephalogram (EEG) cap that monitors the operator’s brain signals and records when the operator detects a threat. The components can be configured as necessary to work with existing systems and meet specific mission requirements.
CT2WS built on the concept that humans are inherently adept at detecting the unusual. Even though a person may not be consciously aware of movement or of unexpected appearance, the brain detects it and triggers the P-300 brainwave, a brain signal that is thought to be involved in stimulus evaluation or categorization. By improving the sensors that capture imagery and filtering results, a human user who is wearing an EEG cap can then rapidly view the filtered image set and let the brain’s natural threat-detection ability work. Users are shown approximately ten images per second, on average. Despite that quick sequence, brain signals indicate to the computer which images were significant.
The use of EEG-based human filtering significantly reduces the amount of false alarms. The cognitive algorithms can also highlight many events that would otherwise be considered irrelevant but are actually indications of threats or targets, such as a bird flying by or a branch’s swaying. In testing of the full CT2WS kit, absent radar, the sensor and cognitive algorithms returned 810 false alarms per hour. When a human wearing the EEG cap was introduced, the number of false alarms dropped to only five per hour, out of a total of 2,304 target events per hour, and a 91 percent successful target recognition rate.
For more information, check out the following additional coverage:
