javacv-webcam: GraalVM edition

TLDR: java-webcam can be compiled with GraalVM to a native binary, source code available here. 

Tyne & Wear Archives & Museums

Recently, I managed to find some time to get a custom java runtime image to work with JavaCV and a simple webcam application, which opens nice possibilities to play around doing image processing with Java.

On the other hand, if you have followed the Java community lately, you've been surely pointed several times to the current big thing Panama and Loom, ah I mean of course GraalVM. 

I'm very fond of the vision of GraalVM, which tries to provide a common platform for several different language eco systems (among other things), in my view a meta supplier for many business opportunities, not clear at the moment what the implications are really. 

It goes without saying that it would be great to get JavaCV and JavaFX to run with GraalVM as well.

Disclaimer: I'm just an interested developer who applies mentioned technologies, more or less successfully.  I cannot stress enough how grateful one can be for enabler technologies like JavaFX, JavaCV or GraalVM. 

Ok. What is necessary for a webcam binary, written in Java, at the time of writing? 

The good news is: not much. In retrospect (after hours - I still can't believe it works now, at least at the moment, for me, in my small bubble).

After much fiddling around the only thing really necessary was bringing together the right tools. I'm using GraalVM 2020.2.0 and Maven 3.6.2 at the moment, the most recent XCode, brew ... well ok ... probably much libraries and toolchains I'm not aware of (neither I want to be - to be honest). 

But the good news is that apart from setting up your environment "properly" and checking out javacv-webcam 2020.3.1,  all you have to do is to enter following command:

mvn clean client:build client:run -Djavacpp.platform.custom -Djavacpp.platform.host -Dgraalvm.home=/path/to/graalvm

After downloading all dependencies (can take some time!) and compiling (forget about download times, this is the real deal), you'll get a huge binary AND a running webcam application (most probably after having been asked to grant permissions by your friendly OS). ... which made my day!

Alternatively, you can start javacv-webcam binary then like this:

target/client/x86_64-darwin/net.ladstatt.javacv.javacv-webcam

Currently, this binary is huge (>500MB!), which is a little too much for my taste ;-) - a simple Hello World without JavaCV has around 60 MB (try compiling HelloFX in gluon-samples project). 

In this regard I'm sure it is possible to reduce the size by applying a more specific configuration. The best - only - chance I see here is to tinker around with JavaCV's dependencies, be more explicit with exclusions, hope that still everything you need works. To be honest, I already had to exclude one transitive dependency, otherwise linking didn't work with GraalVM. 

I just had luck my example app didn't need this specific library I guess. 

Anyway, the road for an easier deployment of Java image processing applications is paved, which is simply exciting! Slap like now!

SudokuFX Version 2020.3.0

After successfully prototyping a webcam image stream with latest JavaFX goodies my next step for my hobby project SudokuFX is clear - port everything to JavaCV and use it's implementation to speedup SudokuFX.

With version 2020.3.0, I rewrote almost everything; conceptually my application follows the original approach, but the implementation is simpler and yields itself better for future enhancements.

Some highlights:

- SudokuFX uses JavaCV

- got rid of overcomplicated usage of Scala Future's

- improved image recognition by tweaking used templates

- improved visual appearance by a smarter approach to overlaying empty cells with numbers, not replacing them

- better memory and CPU utilisation

- Changed versioning scheme to use revision / changelist
- Upgrade to JavaFX 15-ea+4

- Upgrade to Scala 2.13.3

- many other small improvements and bug fixes ... 

There is one drawback - currently there is no android support anymore in the current version, but I'm trying to reactivate it again (maybe more, but you know how it works for hobby software side projects ... )

I'll describe some of the mentioned points above now in more detail ...

SudokuFX uses JavaCV now ... why?

Let me start with basic project setup, that is to say maven dependencies and stuff you need to get SudokuFX to work. 

Speaking of maven dependencies, this is the major reason why I'm switching from 'pure' OpenCV to JavaCV, since it is somehow - magically - fully mavenized and I can use OpenCV indirectly in my application by using JavaCV APIs. 

No need to compile OpenCV myself, no fiddling around with CMake, just staying in my cozy maven Java dependency world, which should lower the bar to start with this project substantially.

Of course JavaCV imposes certain new rules and approaches for the programmer, you have to adapt a little bit and make yourself acquainted with another API. For this project however it payed off. 

I discovered some spots to improve the image processing. It runs way faster and more reliable now.

On the other hand, one has to be careful not to run in subtle bugs which arise from off heap memory management which is a fancy way to say that you'll get exposed to native code behaviour. 

Anyway, starting point for the rewrite to JavaCV was to use raw data from Mat via ByteBuffers, which I only found in JavaCV's API, not in OpenCV. Maybe I'm right, there is no equivalent for this in pure OpenCV C++ / Java API, maybe someone points me to it. With my current approach I'm quite happy, 2020.3 release runs way faster than any other version of SudokuFX so far :)

No more Scala Futures

Scala has a Future implementation in it's standard lib, but it doesn't have the best reputation. All discussions in the Scala community lately made me rethink my approach, and in the end I realized that this pervasive usage of Scala's Future in my code didn't really speed up things, but only bloated the whole source and made it less understandable. As such I refactored this construct out of the source, with planning to later reintroduce it again after cleaning up structure. But I learned that my problem didn't really profit from Future's, as such they are left out completely at the current state of affairs. I can imagine that I'll reintroduce some parallelism again, but at the moment the algorithm runs faster than before. Maybe I'll come back to it and optimise it more, I don't know.

Improved image processing by adapting templates

I'm using template matching for recognising numbers, no third party library like tesseract for example. This keeps it simple, but is certainly a weak point in the approach. Here the project could profit by using a simple neural net maybe. By tweaking and disambiguating the used templates however SudokuFX is now capable of recognising quite a lot of different fonts, which is great (some fonts however will fail miserably ;-) )

Improved visual appearance

One thing I'm quite fond of is that SudokuFX now overlays the computed solution with the original image, yielding a much better visual appearance of the solved Sudoku. It looks far more advanced than before, which is great. 

Improved resource handling

CPU as well as memory footprint are now far better than they used to be. This was also one of the goals of the refactoring, SudokuFX uses less energy, memory and utilizes CPU far better than before.

Versioning Scheme

A blog post from Axel Fontaine made me curious as to not use maven release plugin anymore, but an approach far simpler - SudokuFX is using this approach now too.

JavaFX 15-ea+4 / Scala 2.13.3

Like I reported in a recent blogpost, I migrated SudokuFX to Java11, which is a LTS version of Java and one of the first releases which where JavaFX was decoupled. JavaFX now lives it's own life - but accidentally - or rather planned - has the same release cycles like Java. 

BUT in my usecase scenario I directly profit from the fact that I can use a 'newer' JavaFX with the Java11 LTS version.

Why? Because at the moment of writing this lines here, I have troubles getting Scala to work with Java13, whereas with Java11 my project runs out of the box. 

Scala 2.13.3 is serving as a good starting point to finally change to Scala3, which will also be an interesting endeavour in the future.

So, to sum up, try it out yourself, use SudokuFX to solve your Sudoku problems, have a look at the code to learn something about Scala or JavaCV or report bugs you'll easily find and participate at github.

Source code which is referred to by this article can be found here.

DevFest Vienna 2013

Not long ago I had the opportunity to present my Sudoku2go project at the DevFest Vienna 2013, which was recorded and is available on Youtube.

I switch to english after the introduction, so skip the first minutes.

Among other things I present the basics to build a JavaFX application, show how easy it is to create user interfaces with Scene Builder, how to connect a native library (OpenCV) with the Java ecosystem, and above all how to use Scala to glue it all together.

The code is available on github, and on this blog there are several entries which describe certain aspects of the application.

Thanks +DevFest Vienna for giving me the opportunity to present the project :)

Use your webcam with JavaFX and OpenCV - Part III

In this blog post I want to show you how you can use your webcam to grab pictures and build a GUI using JavaFX to show the video stream filtered by OpenCV algorithms.

Starting with the application I've developed for the last post I've added several new features to it, which I'm going to explain here.

First, I've added two sliders to the application which control the width and height of the image. There are pre - made controls for that (Slider) which are quite easy to use. Combining those with a BorderPane you already get what you need to create functional UI.

On the openCV side, we just need to slice the grabbed Mat data structure with Range objects, and thats it. It is interesting to see the difference in speed (and thus, how fast openCV and your webcam can provide data) when changing the size of the grabbed image.

Here is the source for the application shown above.

Converting an image taken from the webcam  to grayscale using OpenCV and Java

You all know that pictures of yourself look better if you do it in grayscale. This is easy to accomplish using openCV, since there is the very handy Imgproc class which provides several nice static methods like Imgproc.cvtColor(...).

Once again, this method operates on the Mat datastructure:

So far, we've pretty much completed the same like  this introductory tutorial here using JavaFX and Scala. Here is the commit for further reference.

... then some days later ...

I've overhauled the code and made it more interesting also seen from the Scala and the JavaFX point of view. Furthermore I've introduced a feature to blur the captured image as another example for using the OpenCV API using the Java bindings. I tried to group the code in different traits so you can quickly reuse them if you find them useful.

Here is a screenshot of the main program logic which uses all parts:

The following code shows how to create a combobox containing custom objects using the helper functions introduced in the small project:

Using the approach to put everything needed to build a combobox into its own scope makes the code more readable since you don't have to bother with namespace pollution. Speaking of this - on the mailinglist there is also an ongoing discussion to deprecate and then remove the builders for the various visual components. I tend to create helper functions which can be parameterized:

This "mkFoo" approach helps a lot to structure your code.

Finally, you'll get a screencast of the running application showing my desktop while my webcam is filming my TV Set with airplay turned on.

The source code for this little application is available on my github site.

Use your webcam with JavaFX and openCV - Part II

In case you want to use your webcam as input device for grabbing and processing images, this blog post has some pieces of information which might be valuable for you.

Last week I've discovered that you can use OpenCV to grab images from your webcam quite easily, all you need is the starting point project on github and a valid openCV installation for your system.  Maybe you have a look at my previous posting which documents my struggle to get it up and running.

I've learned that in the end it is quite easy to set it up.

On windows you'll just need to download the openCV archive and use the appropriate dll and jar file. On MacOsX you can either compile openCV yourself or use the premade scripts from the macports or homebrew installation systems.

The github project gives you a full setup and self contained program which demonstrates how to use JavaFX to create a simple gui for a image processing application. It shows also how to convert a "Mat" datastructure to a format suitable for JavaFX (without using temporary files), and a Service implementation for the image source. Bottom line is that the picture taken from the webcam will be grabbed by openCV, processed by openCV and displayed with JavaFX.

With this you have the starting point to do more image processing using desktop Java facilities. 

Use your webcam with JavaFX and openCV - Part I

This time I want to show you how to use your Webcam with JavaFX and OpenCV. 

Attention: I've revisited this topic some years later, see for example javacv-webcam with GraalVM

There are quite some approaches to use the MacBook Pro webcam Isight camera in a Java application, but embarrassingly enough I couldn't get them to work.

Attempt #1 : rococoa

After setting up the project with a simple hello world example, I always got a nullpointer when trying to    load a qt movie. When checking out the sources and building them myself I had some troubles with failing tests, looking at the developer mailing list I saw that this project is pretty "dormant" to say the least. All of those points don't say anything about that it is not possible with rococoa and mountain lion to take snapshots of the screen camera, but I didn't have a good feeling and thus I searched on for another solution.

Attempt #2 : vlcj

The well known vlc project has also java bindings, but the website says

... it does also work just fine on Windows and should work on Mac - but for Mac you likely need a bleeding-edge release of vlc...

This didn't sound too promising. At least I've tried and I run into this issue. At least it seems to work with a specific version of the vlc media player and a specific version of the vlcj wrapper. Maybe I'll return to this library when I need more than just a snapshot picture of my camera.

Solution: 3rd party tool

The solution I came up with was to just use the imagesnap program, which can be installed via macports by issuing

sudo port install imagesnap

This puts a little helper program in your path which enables you to take pictures from your webcam.

As a Java guy, I'm not really satisfied with this, as a pragmatic programmer I would say:

Anyhow, the aspect "how to get the image from a source" should be encapsulated anyway in an application, so maybe in the future I'll come up with a more adequate way avoiding the 3rd party dependency. The main motivation for me to use the webcam as input source is to do some image processing with it, and this is now possible.

Executing a 3rd party application and grabbing its output

After the decision to go with the imagesnap program, it is more or less standard procedure to get to the image data. All you need is to execute the application and give it suitable command line parameters.

For example, like this:


You can see that you can use the input stream directly from the imagesnap program, which comes in handy for reusing it for an Image object in JavaFX.

To make it a little more interesting, you can now combine the opencv hello world code and you will get a nice setup for further image processing experiments with yourself in front row.

In order to be able to use maven as dependency management system, you will have to install the opencv.jar in your local maven repository. This can be done like this:

mvn install:install-file -Dfile=/opt/local/share/OpenCV/java/opencv-244.jar \
                         -DgroupId=org.opencv \
                         -DartifactId=opencv-java \
                         -Dtype=jar \
                         -Dversion=2.4.4 \
                         -Dpackaging=jar

Still, the native libs have to be in  /opt/local/share/OpenCV/java/.

And here is the slightly modified code for running opencv with your isight camera using JavaFX Image:

Here is an example result of me hiding behind a book about the beautiful country of Bhutan with face detection applied.

Check out the whole project including pom.xml on the github repository.

Note that this project is pretty much mac only, since it depends on the native library location of opencv, opencv itself, and the native image grabber. It shouldn't be much of a problem to use the same concepts with linux or windows, though.

Update (the day afterwards):

A better solution: just use OpenCV!

After having some sleep and a lot of try and error, I found a solution which doesn't depend on the 3rd party tool but only uses OpenCV to create snapshots of the video input source, the ISight webcam. In fact, it is very easy using the new Desktop Java Bindings after all.

Here is a tiny code snippet to grab images using only OpenCV:


That's it!

This solution is far superior than the 3rd party tool, you can grab more images in a shorter time, it is better integrated and easier to deploy. (The deployment of such applications is still a bit of magic  since you need native libraries which have to reside somwhere on your desktop system and not in the distributed jar....  More on this maybe in a follow up posting).

What about windows?

I tried the solution also on Windows8, the code above works without change also on this platform. All you need is to include the proper DLL for your architecture and of course the openCV jars. Both are provided in the openCV distribution archive in the subfolders build/java.


Thanks for reading :)