Use your webcam with JavaFX and openCV - Part II

In case you want to use your webcam as input device for grabbing and processing images, this blog post has some pieces of information which might be valuable for you.

Last week I've discovered that you can use OpenCV to grab images from your webcam quite easily, all you need is the starting point project on github and a valid openCV installation for your system.  Maybe you have a look at my previous posting which documents my struggle to get it up and running.

I've learned that in the end it is quite easy to set it up.

On windows you'll just need to download the openCV archive and use the appropriate dll and jar file. On MacOsX you can either compile openCV yourself or use the premade scripts from the macports or homebrew installation systems.

The github project gives you a full setup and self contained program which demonstrates how to use JavaFX to create a simple gui for a image processing application. It shows also how to convert a "Mat" datastructure to a format suitable for JavaFX (without using temporary files), and a Service implementation for the image source. Bottom line is that the picture taken from the webcam will be grabbed by openCV, processed by openCV and displayed with JavaFX.

With this you have the starting point to do more image processing using desktop Java facilities. 

Use your webcam with JavaFX and openCV - Part I

This time I want to show you how to use your Webcam with JavaFX and OpenCV. 

Attention: I've revisited this topic some years later, see for example javacv-webcam with GraalVM

There are quite some approaches to use the MacBook Pro webcam Isight camera in a Java application, but embarrassingly enough I couldn't get them to work.

Attempt #1 : rococoa

After setting up the project with a simple hello world example, I always got a nullpointer when trying to    load a qt movie. When checking out the sources and building them myself I had some troubles with failing tests, looking at the developer mailing list I saw that this project is pretty "dormant" to say the least. All of those points don't say anything about that it is not possible with rococoa and mountain lion to take snapshots of the screen camera, but I didn't have a good feeling and thus I searched on for another solution.

Attempt #2 : vlcj

The well known vlc project has also java bindings, but the website says

... it does also work just fine on Windows and should work on Mac - but for Mac you likely need a bleeding-edge release of vlc...

This didn't sound too promising. At least I've tried and I run into this issue. At least it seems to work with a specific version of the vlc media player and a specific version of the vlcj wrapper. Maybe I'll return to this library when I need more than just a snapshot picture of my camera.

Solution: 3rd party tool

The solution I came up with was to just use the imagesnap program, which can be installed via macports by issuing

sudo port install imagesnap

This puts a little helper program in your path which enables you to take pictures from your webcam.

As a Java guy, I'm not really satisfied with this, as a pragmatic programmer I would say:

Anyhow, the aspect "how to get the image from a source" should be encapsulated anyway in an application, so maybe in the future I'll come up with a more adequate way avoiding the 3rd party dependency. The main motivation for me to use the webcam as input source is to do some image processing with it, and this is now possible.

Executing a 3rd party application and grabbing its output

After the decision to go with the imagesnap program, it is more or less standard procedure to get to the image data. All you need is to execute the application and give it suitable command line parameters.

For example, like this:

	package net.ladstatt.apps.isight
	import java.io.InputStream
	import javafx.application.Application
	import javafx.scene.Group
	import javafx.scene.Scene
	import javafx.scene.image.Image
	import javafx.scene.image.ImageView
	import javafx.stage.Stage
	object HelloIsight {
	def main(args: Array[String]): Unit = {
	Application.launch(classOf[HelloIsight], args: _*)
	}
	}
	class HelloIsight extends javafx.application.Application {
	// this can get more complicated if you use a "native" approach
	// using a jni based solution
	// see http://iharder.sourceforge.net/current/macosx/imagesnap/
	def snapIt: InputStream = {
	val runtime = Runtime.getRuntime()
	val process = runtime.exec(Array("/opt/local/bin/imagesnap", "-"))
	process.getInputStream()
	}
	override def start(stage: Stage): Unit = {
	stage.setTitle("Webcam snapshot")
	val group = new Group
	val imageView = new ImageView(new Image(snapIt))
	group.getChildren.add(imageView)
	val scene = new Scene(group)
	stage.setScene(scene)
	stage.show()
	}
	}

view raw HelloIsight.scala hosted with ❤ by GitHub

You can see that you can use the input stream directly from the imagesnap program, which comes in handy for reusing it for an Image object in JavaFX.

To make it a little more interesting, you can now combine the opencv hello world code and you will get a nice setup for further image processing experiments with yourself in front row.

In order to be able to use maven as dependency management system, you will have to install the opencv.jar in your local maven repository. This can be done like this:

mvn install:install-file -Dfile=/opt/local/share/OpenCV/java/opencv-244.jar \
                         -DgroupId=org.opencv \
                         -DartifactId=opencv-java \
                         -Dtype=jar \
                         -Dversion=2.4.4 \
                         -Dpackaging=jar

Still, the native libs have to be in  /opt/local/share/OpenCV/java/.

And here is the slightly modified code for running opencv with your isight camera using JavaFX Image:

	package net.ladstatt.apps.isight
	import java.io.File
	import java.io.FileInputStream
	import org.opencv.core.Core
	import org.opencv.core.MatOfRect
	import org.opencv.core.Point
	import org.opencv.core.Scalar
	import org.opencv.highgui.Highgui
	import org.opencv.objdetect.CascadeClassifier
	import javafx.application.Application
	import javafx.scene.Group
	import javafx.scene.Scene
	import javafx.scene.image.Image
	import javafx.scene.image.ImageView
	import javafx.stage.Stage
	object HelloOpenCVUsingIsight {
	def main(args: Array[String]): Unit = {
	Application.launch(classOf[HelloOpenCVUsingIsight], args: _*)
	}
	}
	// this can get more complicated if you use a "native" approach
	// using a jni based solution
	// see http://iharder.sourceforge.net/current/macosx/imagesnap/
	trait ImageSource {
	def sourceImage: File = {
	val runtime = Runtime.getRuntime()
	// without options, it will just put the image snapshot to a file named "snapshot.jpg"
	// we use it in quiet mode
	val process = runtime.exec(Array("/opt/local/bin/imagesnap", "-q"))
	assert(process.waitFor() == 0)
	val input = new File("snapshot.jpg")
	input.deleteOnExit()
	input
	}
	}
	trait FaceScanner {
	def scanFace(inputFile: File): File = {
	// Create a face detector from the cascade file in the resources
	// directory.
	val faceDetector = new CascadeClassifier(getClass().getResource("/lbpcascade_frontalface.xml").getPath())
	val image = Highgui.imread(inputFile.getPath())
	// Detect faces in the image.
	// MatOfRect is a special container class for Rect.
	val faceDetections = new MatOfRect()
	faceDetector.detectMultiScale(image, faceDetections)
	// Draw a bounding box around each face.
	for (rect <- faceDetections.toArray()) {
	Core.rectangle(image, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), new Scalar(0, 255, 0))
	}
	// Save the visualized detection.
	val fileName = "faceDetection.png"
	Highgui.imwrite(fileName, image)
	val f = new File(fileName)
	f.deleteOnExit()
	f
	}
	}
	class HelloOpenCVUsingIsight extends javafx.application.Application with ImageSource with FaceScanner {
	override def init(): Unit = {
	// important to have this statement on the "right" thread
	System.load(new File("/opt/local/share/OpenCV/java/libopencv_java244.dylib").getAbsolutePath())
	}
	override def start(stage: Stage): Unit = {
	stage.setTitle("Webcam snapshot with face detection")
	val group = new Group
	val imageView = new ImageView(new Image(new FileInputStream(scanFace(sourceImage))))
	group.getChildren.add(imageView)
	val scene = new Scene(group)
	stage.setScene(scene)
	stage.show()
	}
	}

view raw OpenCVISight.scala hosted with ❤ by GitHub

Here is an example result of me hiding behind a book about the beautiful country of Bhutan with face detection applied.

Check out the whole project including pom.xml on the github repository.

Note that this project is pretty much mac only, since it depends on the native library location of opencv, opencv itself, and the native image grabber. It shouldn't be much of a problem to use the same concepts with linux or windows, though.

Update (the day afterwards):

A better solution: just use OpenCV!

After having some sleep and a lot of try and error, I found a solution which doesn't depend on the 3rd party tool but only uses OpenCV to create snapshots of the video input source, the ISight webcam. In fact, it is very easy using the new Desktop Java Bindings after all.

Here is a tiny code snippet to grab images using only OpenCV:

	package net.ladstatt.apps.isight
	import org.opencv.highgui.VideoCapture
	import java.io.File
	import org.opencv.core.Mat
	import org.opencv.highgui.Highgui
	import scala.collection.JavaConversions._
	import org.opencv.core.CvType
	import java.util.Date
	import java.util.UUID
	object VideoCaptureWithOpenCV {
	def main(args: Array[String]) {
	System.load(new File("/opt/local/share/OpenCV/java/libopencv_java244.dylib").getAbsolutePath())
	val videocapture = new VideoCapture(0)
	assert(videocapture.isOpened())
	while (videocapture.grab) {
	val image = new Mat()
	while (videocapture.read(image) == false) { println("waiting for successful grab") }
	val fn = "image_%s.png".format(UUID.randomUUID.toString())
	Highgui.imwrite(fn, image)
	println("grabbing photo ...")
	}
	println("now you have many photos of yourself :)")
	}
	}

view raw VideoCaptureWithOpenCV.scala hosted with ❤ by GitHub

That's it!

This solution is far superior than the 3rd party tool, you can grab more images in a shorter time, it is better integrated and easier to deploy. (The deployment of such applications is still a bit of magic  since you need native libraries which have to reside somwhere on your desktop system and not in the distributed jar....  More on this maybe in a follow up posting).

What about windows?

I tried the solution also on Windows8, the code above works without change also on this platform. All you need is to include the proper DLL for your architecture and of course the openCV jars. Both are provided in the openCV distribution archive in the subfolders build/java.


Thanks for reading :)

OpenCV on MacOSX - with Java support

You surely know that OpenCV has now first class java support since version 2.4.4. What you may not know is that literally since yesterday it is quite easy to install it on MacOsX, given that you use MacPorts.

Factory Floor

box:lad$ sudo port selfupdate
Password:
--->  Updating MacPorts base sources using rsync
MacPorts base version 2.1.3 installed,
MacPorts base version 2.1.3 downloaded.
--->  Updating the ports tree
--->  MacPorts base is already the latest version

The ports tree has been updated. To upgrade your installed ports, you should run
  port upgrade outdated
box:lad$ sudo port install opencv +java
--->  Computing dependencies for opencv
--->  Dependencies to be installed: apache-ant cmake pkgconfig
--->  Fetching archive for apache-ant
--->  Attempting to fetch apache-ant-1.9.0_0.darwin_12.noarch.tbz2 from http://lil.fr.packages.macports.org/apache-ant
--->  Attempting to fetch apache-ant-1.9.0_0.darwin_12.noarch.tbz2.rmd160 from http://lil.fr.packages.macports.org/apache-ant
--->  Installing apache-ant @1.9.0_0
--->  Activating apache-ant @1.9.0_0
--->  Cleaning apache-ant
--->  Fetching archive for cmake
--->  Attempting to fetch cmake-2.8.10_1.darwin_12.x86_64.tbz2 from http://lil.fr.packages.macports.org/cmake
--->  Attempting to fetch cmake-2.8.10_1.darwin_12.x86_64.tbz2.rmd160 from http://lil.fr.packages.macports.org/cmake
--->  Installing cmake @2.8.10_1
--->  Activating cmake @2.8.10_1
--->  Cleaning cmake
--->  Fetching archive for pkgconfig
--->  Attempting to fetch pkgconfig-0.27.1_2.darwin_12.x86_64.tbz2 from http://lil.fr.packages.macports.org/pkgconfig
--->  Attempting to fetch pkgconfig-0.27.1_2.darwin_12.x86_64.tbz2.rmd160 from http://lil.fr.packages.macports.org/pkgconfig
--->  Installing pkgconfig @0.27.1_2
--->  Activating pkgconfig @0.27.1_2
--->  Cleaning pkgconfig
--->  Fetching archive for opencv
--->  Attempting to fetch opencv-2.4.4_3+java.darwin_12.x86_64.tbz2 from http://lil.fr.packages.macports.org/opencv
--->  Attempting to fetch opencv-2.4.4_3+java.darwin_12.x86_64.tbz2 from http://mse.uk.packages.macports.org/sites/packages.macports.org/opencv
--->  Attempting to fetch opencv-2.4.4_3+java.darwin_12.x86_64.tbz2 from http://packages.macports.org/opencv
--->  Fetching distfiles for opencv
--->  Attempting to fetch OpenCV-2.4.4a.tar.bz2 from http://ignum.dl.sourceforge.net/project/opencvlibrary/opencv-unix/2.4.4
--->  Verifying checksum(s) for opencv
--->  Extracting opencv
--->  Applying patches to opencv
--->  Configuring opencv
--->  Building opencv
--->  Staging opencv into destroot
--->  Installing opencv @2.4.4_3+java
--->  Deactivating opencv @2.4.4_2
--->  Cleaning opencv
--->  Activating opencv @2.4.4_3+java
--->  Cleaning opencv
--->  Updating database of binaries: 100.0%
--->  Scanning binaries for linking errors: 100.0%
--->  No broken files found.
box:lad$ port contents opencv | grep java
  /opt/local/share/OpenCV/java/libopencv_java244.dylib
  /opt/local/share/OpenCV/java/opencv-244.jar
box:lad$ 

Some Scala code to use it:

	package net.ladstatt.apps
	import java.io.File
	import org.opencv.core.Core
	import org.opencv.core.MatOfRect
	import org.opencv.core.Point
	import org.opencv.core.Scalar
	import org.opencv.highgui.Highgui
	import org.opencv.objdetect.CascadeClassifier
	/**
	* Scala version of the introductory tutorial on opencv.org
	*/
	object HelloOpenCV {
	System.load(new File("/opt/local/share/OpenCV/java/libopencv_java244.dylib").getAbsolutePath())
	def main(args: Array[String]): Unit = {
	new DetectFaceDemo().run()
	}
	}
	class DetectFaceDemo {
	def run(): Unit = {
	println("\nRunning DetectFaceDemo")
	// Create a face detector from the cascade file in the resources
	// directory.
	val faceDetector = new CascadeClassifier(getClass().getResource("/lbpcascade_frontalface.xml").getPath())
	val image = Highgui.imread(getClass().getResource("/person.png").getPath())
	// Detect faces in the image.
	// MatOfRect is a special container class for Rect.
	val faceDetections = new MatOfRect()
	faceDetector.detectMultiScale(image, faceDetections)
	println("Detected %s faces".format(faceDetections.toArray().length))
	// Draw a bounding box around each face.
	for (rect <- faceDetections.toArray()) {
	Core.rectangle(image, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), new Scalar(0, 255, 0))
	}
	// Save the visualized detection.
	val filename = "faceDetection.png"
	println(String.format("Writing %s", filename))
	Highgui.imwrite(filename, image)
	}
	}

view raw HelloOpenCV.scala hosted with ❤ by GitHub

Many thanks to Andrew Stromnov to make this possible, since compiling yourself OpenCV with Java Support is not something the average Java guy will do. (I did it. It was a pleasure. ;-) )

Update:

Keep in mind that the port command compiles the jar file with the currently available JDK. If you run the port command in verbose mode you'll see that the jar file is assembled using ant. In order to force the port command to use a certain JDK you can patch the ant script:

 80 # OS specific support.  $var _must_ be set to either true or false.
 81 cygwin=false;
 82 darwin=false;
 83 mingw=false;
 84 case "`uname`" in
 85   CYGWIN*) cygwin=true ;;
 86   Darwin*) darwin=true
 87            if [ -z "$JAVA_HOME" ] ; then
 88                if [ -x '/usr/libexec/java_home' ] ; then
 89                    JAVA_HOME=`/usr/libexec/java_home -v 1.7`
 90                elif [ -d "/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home" ]; then
 91                    JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home
 92                fi
 93            fi
 94            ;;
 95   MINGW*) mingw=true ;;
 96 esac

Like this, the JDK 1.7 on my machine will be used for generating the jar file. 

Upgrading gcc to gcc48 on MacOsX

This post describes how to update your gcc installation to gcc48 on MacOsX.

Short:

sudo port install gcc48 +universal

You can follow the instructions found here to update your installation of gcc. At the time of writing, gcc in version 4.8 is the current (experimental) version.

Be sure to change the default gcc command to the newly installed by issuing

sudo port select --set gcc mp-gcc48

and then, afterwards

hash gcc 

(to rehash it, see this link)

Test your gcc installation by issuing

gcc --version

which should give you an output like this.

gcc (MacPorts gcc48 4.8-20130328_0+universal) 4.8.1 20130328 (prerelease)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Your gcc installation on the command line should now be current.