Use your webcam with JavaFX and openCV - Part I

This time I want to show you how to use your Webcam with JavaFX and OpenCV. 

Attention: I've revisited this topic some years later, see for example javacv-webcam with GraalVM

There are quite some approaches to use the MacBook Pro webcam Isight camera in a Java application, but embarrassingly enough I couldn't get them to work.

Attempt #1 : rococoa

After setting up the project with a simple hello world example, I always got a nullpointer when trying to    load a qt movie. When checking out the sources and building them myself I had some troubles with failing tests, looking at the developer mailing list I saw that this project is pretty "dormant" to say the least. All of those points don't say anything about that it is not possible with rococoa and mountain lion to take snapshots of the screen camera, but I didn't have a good feeling and thus I searched on for another solution.

Attempt #2 : vlcj

The well known vlc project has also java bindings, but the website says

... it does also work just fine on Windows and should work on Mac - but for Mac you likely need a bleeding-edge release of vlc...

This didn't sound too promising. At least I've tried and I run into this issue. At least it seems to work with a specific version of the vlc media player and a specific version of the vlcj wrapper. Maybe I'll return to this library when I need more than just a snapshot picture of my camera.

Solution: 3rd party tool

The solution I came up with was to just use the imagesnap program, which can be installed via macports by issuing

sudo port install imagesnap

This puts a little helper program in your path which enables you to take pictures from your webcam.

As a Java guy, I'm not really satisfied with this, as a pragmatic programmer I would say:

Anyhow, the aspect "how to get the image from a source" should be encapsulated anyway in an application, so maybe in the future I'll come up with a more adequate way avoiding the 3rd party dependency. The main motivation for me to use the webcam as input source is to do some image processing with it, and this is now possible.

Executing a 3rd party application and grabbing its output

After the decision to go with the imagesnap program, it is more or less standard procedure to get to the image data. All you need is to execute the application and give it suitable command line parameters.

For example, like this:

	package net.ladstatt.apps.isight
	import java.io.InputStream
	import javafx.application.Application
	import javafx.scene.Group
	import javafx.scene.Scene
	import javafx.scene.image.Image
	import javafx.scene.image.ImageView
	import javafx.stage.Stage
	object HelloIsight {
	def main(args: Array[String]): Unit = {
	Application.launch(classOf[HelloIsight], args: _*)
	}
	}
	class HelloIsight extends javafx.application.Application {
	// this can get more complicated if you use a "native" approach
	// using a jni based solution
	// see http://iharder.sourceforge.net/current/macosx/imagesnap/
	def snapIt: InputStream = {
	val runtime = Runtime.getRuntime()
	val process = runtime.exec(Array("/opt/local/bin/imagesnap", "-"))
	process.getInputStream()
	}
	override def start(stage: Stage): Unit = {
	stage.setTitle("Webcam snapshot")
	val group = new Group
	val imageView = new ImageView(new Image(snapIt))
	group.getChildren.add(imageView)
	val scene = new Scene(group)
	stage.setScene(scene)
	stage.show()
	}
	}

view raw HelloIsight.scala hosted with ❤ by GitHub

You can see that you can use the input stream directly from the imagesnap program, which comes in handy for reusing it for an Image object in JavaFX.

To make it a little more interesting, you can now combine the opencv hello world code and you will get a nice setup for further image processing experiments with yourself in front row.

In order to be able to use maven as dependency management system, you will have to install the opencv.jar in your local maven repository. This can be done like this:

mvn install:install-file -Dfile=/opt/local/share/OpenCV/java/opencv-244.jar \
                         -DgroupId=org.opencv \
                         -DartifactId=opencv-java \
                         -Dtype=jar \
                         -Dversion=2.4.4 \
                         -Dpackaging=jar

Still, the native libs have to be in  /opt/local/share/OpenCV/java/.

And here is the slightly modified code for running opencv with your isight camera using JavaFX Image:

	package net.ladstatt.apps.isight
	import java.io.File
	import java.io.FileInputStream
	import org.opencv.core.Core
	import org.opencv.core.MatOfRect
	import org.opencv.core.Point
	import org.opencv.core.Scalar
	import org.opencv.highgui.Highgui
	import org.opencv.objdetect.CascadeClassifier
	import javafx.application.Application
	import javafx.scene.Group
	import javafx.scene.Scene
	import javafx.scene.image.Image
	import javafx.scene.image.ImageView
	import javafx.stage.Stage
	object HelloOpenCVUsingIsight {
	def main(args: Array[String]): Unit = {
	Application.launch(classOf[HelloOpenCVUsingIsight], args: _*)
	}
	}
	// this can get more complicated if you use a "native" approach
	// using a jni based solution
	// see http://iharder.sourceforge.net/current/macosx/imagesnap/
	trait ImageSource {
	def sourceImage: File = {
	val runtime = Runtime.getRuntime()
	// without options, it will just put the image snapshot to a file named "snapshot.jpg"
	// we use it in quiet mode
	val process = runtime.exec(Array("/opt/local/bin/imagesnap", "-q"))
	assert(process.waitFor() == 0)
	val input = new File("snapshot.jpg")
	input.deleteOnExit()
	input
	}
	}
	trait FaceScanner {
	def scanFace(inputFile: File): File = {
	// Create a face detector from the cascade file in the resources
	// directory.
	val faceDetector = new CascadeClassifier(getClass().getResource("/lbpcascade_frontalface.xml").getPath())
	val image = Highgui.imread(inputFile.getPath())
	// Detect faces in the image.
	// MatOfRect is a special container class for Rect.
	val faceDetections = new MatOfRect()
	faceDetector.detectMultiScale(image, faceDetections)
	// Draw a bounding box around each face.
	for (rect <- faceDetections.toArray()) {
	Core.rectangle(image, new Point(rect.x, rect.y), new Point(rect.x + rect.width, rect.y + rect.height), new Scalar(0, 255, 0))
	}
	// Save the visualized detection.
	val fileName = "faceDetection.png"
	Highgui.imwrite(fileName, image)
	val f = new File(fileName)
	f.deleteOnExit()
	f
	}
	}
	class HelloOpenCVUsingIsight extends javafx.application.Application with ImageSource with FaceScanner {
	override def init(): Unit = {
	// important to have this statement on the "right" thread
	System.load(new File("/opt/local/share/OpenCV/java/libopencv_java244.dylib").getAbsolutePath())
	}
	override def start(stage: Stage): Unit = {
	stage.setTitle("Webcam snapshot with face detection")
	val group = new Group
	val imageView = new ImageView(new Image(new FileInputStream(scanFace(sourceImage))))
	group.getChildren.add(imageView)
	val scene = new Scene(group)
	stage.setScene(scene)
	stage.show()
	}
	}

view raw OpenCVISight.scala hosted with ❤ by GitHub

Here is an example result of me hiding behind a book about the beautiful country of Bhutan with face detection applied.

Check out the whole project including pom.xml on the github repository.

Note that this project is pretty much mac only, since it depends on the native library location of opencv, opencv itself, and the native image grabber. It shouldn't be much of a problem to use the same concepts with linux or windows, though.

Update (the day afterwards):

A better solution: just use OpenCV!

After having some sleep and a lot of try and error, I found a solution which doesn't depend on the 3rd party tool but only uses OpenCV to create snapshots of the video input source, the ISight webcam. In fact, it is very easy using the new Desktop Java Bindings after all.

Here is a tiny code snippet to grab images using only OpenCV:

	package net.ladstatt.apps.isight
	import org.opencv.highgui.VideoCapture
	import java.io.File
	import org.opencv.core.Mat
	import org.opencv.highgui.Highgui
	import scala.collection.JavaConversions._
	import org.opencv.core.CvType
	import java.util.Date
	import java.util.UUID
	object VideoCaptureWithOpenCV {
	def main(args: Array[String]) {
	System.load(new File("/opt/local/share/OpenCV/java/libopencv_java244.dylib").getAbsolutePath())
	val videocapture = new VideoCapture(0)
	assert(videocapture.isOpened())
	while (videocapture.grab) {
	val image = new Mat()
	while (videocapture.read(image) == false) { println("waiting for successful grab") }
	val fn = "image_%s.png".format(UUID.randomUUID.toString())
	Highgui.imwrite(fn, image)
	println("grabbing photo ...")
	}
	println("now you have many photos of yourself :)")
	}
	}

view raw VideoCaptureWithOpenCV.scala hosted with ❤ by GitHub

That's it!

This solution is far superior than the 3rd party tool, you can grab more images in a shorter time, it is better integrated and easier to deploy. (The deployment of such applications is still a bit of magic  since you need native libraries which have to reside somwhere on your desktop system and not in the distributed jar....  More on this maybe in a follow up posting).

What about windows?

I tried the solution also on Windows8, the code above works without change also on this platform. All you need is to include the proper DLL for your architecture and of course the openCV jars. Both are provided in the openCV distribution archive in the subfolders build/java.


Thanks for reading :)