2D Image Filters with OpenCV

In this blog post I'm giving you an example on how to do basic 2D image filtering using OpenCV and displaying the result instantly using JavaFX.




Image filtering means that you apply various transformations on a given image. Of course, image processing is math, and I'll assume since you stumbled by this blog you are familiar with the basic concepts of image processing - if not there are plenty of articles in the web which can give you a good overview. Wikipedia will always give you a broader view on the topic.

Like you've noticed in the past few posts on this blog I'm making myself familiar with the OpenCV library, and the best way to learn a new API is of course to read whats available and make your own experiments. In my case, I've made a JavaFX application which makes it easy to explore the different effects you can achieve by changing the kernel values and getting instant feedback.

Warning: This blog post is just about very basic filtering, and chances are high that some of the operations deriving from parameterizing the kernels have their own names and/or have more efficient implementations in OpenCV.

As a sidenote, if you don't already know Bret Victors talk on 'inventing on principle' you should definitely visit his web site. I've tried to make the program given below in a way that the user can experiment and maybe get new ideas about the whole problem, invent their own kernel for example. It's fun to change some values here and there ...

Like this you get an idea what's behind words like 'blurring' or 'sharpening', and find out that "finding edges" means nothing more than apply simple yet powerful mathematical operations on an 2D matrix.

I have provided some example kernels along with the application to give you some starting points - but feel free to explore the effects. Check out this page for an explanation of the used kernels in the application.

Another motivation for this blog post is to explore the feasibility of using Scala along with OpenCV and - I'm biased - I find it a very good match. Even more so if you use JavaFX to implement the GUI.

Below is the code, this is all you need for the video above. (yes - I've discovered iMovie! ;-) ).

Using Scala Futures and OpenCV together with JavaFX

In this post I want to show you how you can improve the performance of your application by using datastructures and approaches which make use of non blocking parallelism.

Recently I digged into the API's of OpenCV, which is a great image processing library. I wrote several blog posts about it, and this is sort of a follow up on these posts. However, this time I want to improve the project by introducing Scala Futures into the codebase. (Why? because its there!)

Scala Futures are an integral part of Scala 2.10 and are explained here in more detail. I want to show you how the readability of applications can benefit - as well as their performance. The latter will be more important for your managers, but using Futures combined with Scala's for comprehension have their own aesthetic appeal.

If you compare the isight-java project from this and this commit, you'll find that not very much changed for the end user, in fact it is more or less the same end user experience. However the version I'm describing here makes heavy use of the 'Futures' concept.

In short, the application is based on a filtering pipeline, starting on the grabbed image several different algorithms are applied to it, passing and mutating the Mat datastructure from one operation to the other.

Using Scala Futures and for comprehension, this translates to a code like this:

You'll recognize the pipeline structure in the code above. The neat thing is that you'll get error handling for free using the recover combinator. If you compare the code above with the one of the previous post, you'll notice that it looks much clearer and the intention of the code is really apparent. (even though IMO the last version wasn't too bad either :))

Of course, the image processing functions needed some adaptions to return futures:

You can see that using futures is quite easy and feels somewhat natural when you combine it with the for comprehensions. For a more detailed discussion on what happens under the covers, please read the article on futures on scala-lang.org.

Disclaimer: I've mixed up some concepts of the Scala libraries and JavaFX parallelism (I'm using JavaFX's  Service and Task concepts along with Scala Futures) - some may argue that this is not necessary or even dangerous. Quoting this guy here: TMTOWTDI. Be aware that mixing different approaches of parallelism can lead to confusion of the poor guy inheriting your code, or may result in unwanted effects (?). One side effect I noted when using Scala Futures was the necessity to use the Platform.runLater( ... ) trick to make sure the image service runs on the gui thread.

Anyway, if you go that road with Scala Futures, with small tweaks to the source code (at least on the surface) you'll get a parallelized version which, when used in conjunction with the for comprehension, looks like a sequential code.

If your result consists of several, independent sub problems which you combine in a final step you'll get the best results when parallelizing your app.

Even if in this application this is not the case, I've nevertheless noticed a considerable improvement in responsiveness and speed (whatever reasons this had: either wishful thinking or just bad implementation beforehand ;-) ) as well as readability of the code.

It could well be that the guys over at the scalaFX camp have done something to use concurrency as convenient as the scala team did for scala futures - if not: this would be a great idea.

me blurred beyond recognition

Check out the full source code for this blog post here.

Use your webcam with JavaFX and OpenCV - Part III

In this blog post I want to show you how you can use your webcam to grab pictures and build a GUI using JavaFX to show the video stream filtered by OpenCV algorithms.

Starting with the application I've developed for the last post I've added several new features to it, which I'm going to explain here.

First, I've added two sliders to the application which control the width and height of the image. There are pre - made controls for that (Slider) which are quite easy to use. Combining those with a BorderPane you already get what you need to create functional UI.

On the openCV side, we just need to slice the grabbed Mat data structure with Range objects, and thats it. It is interesting to see the difference in speed (and thus, how fast openCV and your webcam can provide data) when changing the size of the grabbed image.

Here is the source for the application shown above.

Converting an image taken from the webcam  to grayscale using OpenCV and Java

You all know that pictures of yourself look better if you do it in grayscale. This is easy to accomplish using openCV, since there is the very handy Imgproc class which provides several nice static methods like Imgproc.cvtColor(...).

Once again, this method operates on the Mat datastructure:

So far, we've pretty much completed the same like  this introductory tutorial here using JavaFX and Scala. Here is the commit for further reference.

... then some days later ...

I've overhauled the code and made it more interesting also seen from the Scala and the JavaFX point of view. Furthermore I've introduced a feature to blur the captured image as another example for using the OpenCV API using the Java bindings. I tried to group the code in different traits so you can quickly reuse them if you find them useful.

Here is a screenshot of the main program logic which uses all parts:

The following code shows how to create a combobox containing custom objects using the helper functions introduced in the small project:

Using the approach to put everything needed to build a combobox into its own scope makes the code more readable since you don't have to bother with namespace pollution. Speaking of this - on the mailinglist there is also an ongoing discussion to deprecate and then remove the builders for the various visual components. I tend to create helper functions which can be parameterized:

This "mkFoo" approach helps a lot to structure your code.

Finally, you'll get a screencast of the running application showing my desktop while my webcam is filming my TV Set with airplay turned on.

The source code for this little application is available on my github site.

Use your webcam with JavaFX and openCV - Part II

In case you want to use your webcam as input device for grabbing and processing images, this blog post has some pieces of information which might be valuable for you.

Last week I've discovered that you can use OpenCV to grab images from your webcam quite easily, all you need is the starting point project on github and a valid openCV installation for your system.  Maybe you have a look at my previous posting which documents my struggle to get it up and running.

I've learned that in the end it is quite easy to set it up.

On windows you'll just need to download the openCV archive and use the appropriate dll and jar file. On MacOsX you can either compile openCV yourself or use the premade scripts from the macports or homebrew installation systems.

The github project gives you a full setup and self contained program which demonstrates how to use JavaFX to create a simple gui for a image processing application. It shows also how to convert a "Mat" datastructure to a format suitable for JavaFX (without using temporary files), and a Service implementation for the image source. Bottom line is that the picture taken from the webcam will be grabbed by openCV, processed by openCV and displayed with JavaFX.

With this you have the starting point to do more image processing using desktop Java facilities. 

Use your webcam with JavaFX and openCV - Part I

This time I want to show you how to use your Webcam with JavaFX and OpenCV.

There are quite some approaches to use the MacBook Pro webcam Isight camera in a Java application, but embarrassingly enough I couldn't get them to work.

Attempt #1 : rococoa

After setting up the project with a simple hello world example, I always got a nullpointer when trying to    load a qt movie. When checking out the sources and building them myself I had some troubles with failing tests, looking at the developer mailing list I saw that this project is pretty "dormant" to say the least. All of those points don't say anything about that it is not possible with rococoa and mountain lion to take snapshots of the screen camera, but I didn't have a good feeling and thus I searched on for another solution.

Attempt #2 : vlcj

The well known vlc project has also java bindings, but the website says

... it does also work just fine on Windows and should work on Mac - but for Mac you likely need a bleeding-edge release of vlc...

This didn't sound too promising. At least I've tried and I run into this issue. At least it seems to work with a specific version of the vlc media player and a specific version of the vlcj wrapper. Maybe I'll return to this library when I need more than just a snapshot picture of my camera.

Solution: 3rd party tool

The solution I came up with was to just use the imagesnap program, which can be installed via macports by issuing

sudo port install imagesnap

This puts a little helper program in your path which enables you to take pictures from your webcam.

As a Java guy, I'm not really satisfied with this, as a pragmatic programmer I would say:

Anyhow, the aspect "how to get the image from a source" should be encapsulated anyway in an application, so maybe in the future I'll come up with a more adequate way avoiding the 3rd party dependency. The main motivation for me to use the webcam as input source is to do some image processing with it, and this is now possible.

Executing a 3rd party application and grabbing its output

After the decision to go with the imagesnap program, it is more or less standard procedure to get to the image data. All you need is to execute the application and give it suitable command line parameters.

For example, like this:


You can see that you can use the input stream directly from the imagesnap program, which comes in handy for reusing it for an Image object in JavaFX.

To make it a little more interesting, you can now combine the opencv hello world code and you will get a nice setup for further image processing experiments with yourself in front row.

In order to be able to use maven as dependency management system, you will have to install the opencv.jar in your local maven repository. This can be done like this:

mvn install:install-file -Dfile=/opt/local/share/OpenCV/java/opencv-244.jar \
                         -DgroupId=org.opencv \
                         -DartifactId=opencv-java \
                         -Dtype=jar \
                         -Dversion=2.4.4 \
                         -Dpackaging=jar

Still, the native libs have to be in  /opt/local/share/OpenCV/java/.

And here is the slightly modified code for running opencv with your isight camera using JavaFX Image:

Here is an example result of me hiding behind a book about the beautiful country of Bhutan with face detection applied.

Check out the whole project including pom.xml on the github repository.

Note that this project is pretty much mac only, since it depends on the native library location of opencv, opencv itself, and the native image grabber. It shouldn't be much of a problem to use the same concepts with linux or windows, though.

Update (the day afterwards):

A better solution: just use OpenCV!

After having some sleep and a lot of try and error, I found a solution which doesn't depend on the 3rd party tool but only uses OpenCV to create snapshots of the video input source, the ISight webcam. In fact, it is very easy using the new Desktop Java Bindings after all.

Here is a tiny code snippet to grab images using only OpenCV:


That's it!

This solution is far superior than the 3rd party tool, you can grab more images in a shorter time, it is better integrated and easier to deploy. (The deployment of such applications is still a bit of magic  since you need native libraries which have to reside somwhere on your desktop system and not in the distributed jar....  More on this maybe in a follow up posting).

What about windows?

I tried the solution also on Windows8, the code above works without change also on this platform. All you need is to include the proper DLL for your architecture and of course the openCV jars. Both are provided in the openCV distribution archive in the subfolders build/java.


Thanks for reading :)

OpenCV on MacOSX - with Java support

You surely know that OpenCV has now first class java support since version 2.4.4. What you may not know is that literally since yesterday it is quite easy to install it on MacOsX, given that you use MacPorts.

Factory Floor

box:lad$ sudo port selfupdate
Password:
--->  Updating MacPorts base sources using rsync
MacPorts base version 2.1.3 installed,
MacPorts base version 2.1.3 downloaded.
--->  Updating the ports tree
--->  MacPorts base is already the latest version

The ports tree has been updated. To upgrade your installed ports, you should run
  port upgrade outdated
box:lad$ sudo port install opencv +java
--->  Computing dependencies for opencv
--->  Dependencies to be installed: apache-ant cmake pkgconfig
--->  Fetching archive for apache-ant
--->  Attempting to fetch apache-ant-1.9.0_0.darwin_12.noarch.tbz2 from http://lil.fr.packages.macports.org/apache-ant
--->  Attempting to fetch apache-ant-1.9.0_0.darwin_12.noarch.tbz2.rmd160 from http://lil.fr.packages.macports.org/apache-ant
--->  Installing apache-ant @1.9.0_0
--->  Activating apache-ant @1.9.0_0
--->  Cleaning apache-ant
--->  Fetching archive for cmake
--->  Attempting to fetch cmake-2.8.10_1.darwin_12.x86_64.tbz2 from http://lil.fr.packages.macports.org/cmake
--->  Attempting to fetch cmake-2.8.10_1.darwin_12.x86_64.tbz2.rmd160 from http://lil.fr.packages.macports.org/cmake
--->  Installing cmake @2.8.10_1
--->  Activating cmake @2.8.10_1
--->  Cleaning cmake
--->  Fetching archive for pkgconfig
--->  Attempting to fetch pkgconfig-0.27.1_2.darwin_12.x86_64.tbz2 from http://lil.fr.packages.macports.org/pkgconfig
--->  Attempting to fetch pkgconfig-0.27.1_2.darwin_12.x86_64.tbz2.rmd160 from http://lil.fr.packages.macports.org/pkgconfig
--->  Installing pkgconfig @0.27.1_2
--->  Activating pkgconfig @0.27.1_2
--->  Cleaning pkgconfig
--->  Fetching archive for opencv
--->  Attempting to fetch opencv-2.4.4_3+java.darwin_12.x86_64.tbz2 from http://lil.fr.packages.macports.org/opencv
--->  Attempting to fetch opencv-2.4.4_3+java.darwin_12.x86_64.tbz2 from http://mse.uk.packages.macports.org/sites/packages.macports.org/opencv
--->  Attempting to fetch opencv-2.4.4_3+java.darwin_12.x86_64.tbz2 from http://packages.macports.org/opencv
--->  Fetching distfiles for opencv
--->  Attempting to fetch OpenCV-2.4.4a.tar.bz2 from http://ignum.dl.sourceforge.net/project/opencvlibrary/opencv-unix/2.4.4
--->  Verifying checksum(s) for opencv
--->  Extracting opencv
--->  Applying patches to opencv
--->  Configuring opencv
--->  Building opencv
--->  Staging opencv into destroot
--->  Installing opencv @2.4.4_3+java
--->  Deactivating opencv @2.4.4_2
--->  Cleaning opencv
--->  Activating opencv @2.4.4_3+java
--->  Cleaning opencv
--->  Updating database of binaries: 100.0%
--->  Scanning binaries for linking errors: 100.0%
--->  No broken files found.
box:lad$ port contents opencv | grep java
  /opt/local/share/OpenCV/java/libopencv_java244.dylib
  /opt/local/share/OpenCV/java/opencv-244.jar
box:lad$ 

Some Scala code to use it:

Many thanks to Andrew Stromnov to make this possible, since compiling yourself OpenCV with Java Support is not something the average Java guy will do. (I did it. It was a pleasure. ;-) )

Keep in mind that the opencv will get compiled with the JDK configured at the time of installing as far as I could tell.

Upgrading gcc to gcc48 on MacOsX

sudo port install gcc48 +universal

You can follow the instructions found here to update your installation of gcc. At the time of writing, gcc in version 4.8 is the current (experimental) version.

Be sure to change the default gcc command to the newly installed by issuing

sudo port select --set gcc mp-gcc48

and then, afterwards

hash gcc 

(to rehash it, see this link)

Test your gcc installation by issuing

gcc --version

which should give you an output like this.

gcc (MacPorts gcc48 4.8-20130328_0+universal) 4.8.1 20130328 (prerelease)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Your gcc installation on the command line should now be current.