javacv-webcam: GraalVM edition

TLDR: java-webcam can be compiled with GraalVM to a native binary, source code available here. 

Tyne & Wear Archives & Museums

Recently, I managed to find some time to get a custom java runtime image to work with JavaCV and a simple webcam application, which opens nice possibilities to play around doing image processing with Java.

On the other hand, if you have followed the Java community lately, you've been surely pointed several times to the current big thing Panama and Loom, ah I mean of course GraalVM. 

I'm very fond of the vision of GraalVM, which tries to provide a common platform for several different language eco systems (among other things), in my view a meta supplier for many business opportunities, not clear at the moment what the implications are really. 

It goes without saying that it would be great to get JavaCV and JavaFX to run with GraalVM as well.

Disclaimer: I'm just an interested developer who applies mentioned technologies, more or less successfully.  I cannot stress enough how grateful one can be for enabler technologies like JavaFX, JavaCV or GraalVM. 

Ok. What is necessary for a webcam binary, written in Java, at the time of writing? 

The good news is: not much. In retrospect (after hours - I still can't believe it works now, at least at the moment, for me, in my small bubble).

After much fiddling around the only thing really necessary was bringing together the right tools. I'm using GraalVM 2020.2.0 and Maven 3.6.2 at the moment, the most recent XCode, brew ... well ok ... probably much libraries and toolchains I'm not aware of (neither I want to be - to be honest). 

But the good news is that apart from setting up your environment "properly" and checking out javacv-webcam 2020.3.1,  all you have to do is to enter following command:

mvn clean client:build client:run -Djavacpp.platform.custom -Djavacpp.platform.host -Dgraalvm.home=/path/to/graalvm

After downloading all dependencies (can take some time!) and compiling (forget about download times, this is the real deal), you'll get a huge binary AND a running webcam application (most probably after having been asked to grant permissions by your friendly OS). ... which made my day!

Alternatively, you can start javacv-webcam binary then like this:

target/client/x86_64-darwin/net.ladstatt.javacv.javacv-webcam

Currently, this binary is huge (>500MB!), which is a little too much for my taste ;-) - a simple Hello World without JavaCV has around 60 MB (try compiling HelloFX in gluon-samples project). 

In this regard I'm sure it is possible to reduce the size by applying a more specific configuration. The best - only - chance I see here is to tinker around with JavaCV's dependencies, be more explicit with exclusions, hope that still everything you need works. To be honest, I already had to exclude one transitive dependency, otherwise linking didn't work with GraalVM. 

I just had luck my example app didn't need this specific library I guess. 

Anyway, the road for an easier deployment of Java image processing applications is paved, which is simply exciting! Slap like now!

javacv-webcam: a skeleton project to do image processing with java

Recently I've simplified JavaCV Webcam project for an easy start with JavaCV and JavaFX.

If you follow me on this blog or my occasional posts on twitter, you surely know that I like to explore new features for JavaFX in combination with Scala. Lately, I had migrated SudokuFX to JavaCV. With this release SudokuFX doesn't use OpenCV directly anymore but uses JavaCV's wrapper which provided at the point of writing a more low level API to the Mat data structure. 

SudokuFX provides some examples on how to use JavaCV, however if you search for a simple example on how to use JavaCV and JavaFX and you want to access your webcam, javacv-webcam is the right project for you. 

javacv-webcam is written in 100% Java. It shows how to configure javafx-maven-plugin to create a custom java runtime image with jlink. 

Another aspect which is worth mentioning is that javacv-webcam is an example for a modular java application. Here it is worth mentioning that IntelliJ in a recent version has good support for writing module-info.java and compiler errors are helpful as well. 

I tried to make javacv-webcam a project with zero configuration, you should just be able to check out the project and just run it ... 

Note: on windows or linux you have to apply small changes in module-info.java, but this should pose no problem. On MacOsX you may have to fiddle around with privacy / security settings for java. Speaking of which - you have to use Java11 and a recent maven distribution.

For build instructions, just checkout github readme page. This post refers to javacv-webcam version 2020.3.0.

Note: There is an older version for javacv-webcam which is implemented in Scala and does some comparisons to other ways how to use JavaCV. Also, there is a newer version for javacv-webcam which contains configuration to compile it to a native binary using GraalVM.

SudokuFX Version 2020.3.0

After successfully prototyping a webcam image stream with latest JavaFX goodies my next step for my hobby project SudokuFX is clear - port everything to JavaCV and use it's implementation to speedup SudokuFX.

With version 2020.3.0, I rewrote almost everything; conceptually my application follows the original approach, but the implementation is simpler and yields itself better for future enhancements.

Some highlights:

- SudokuFX uses JavaCV

- got rid of overcomplicated usage of Scala Future's

- improved image recognition by tweaking used templates

- improved visual appearance by a smarter approach to overlaying empty cells with numbers, not replacing them

- better memory and CPU utilisation

- Changed versioning scheme to use revision / changelist
- Upgrade to JavaFX 15-ea+4

- Upgrade to Scala 2.13.3

- many other small improvements and bug fixes ... 

There is one drawback - currently there is no android support anymore in the current version, but I'm trying to reactivate it again (maybe more, but you know how it works for hobby software side projects ... )

I'll describe some of the mentioned points above now in more detail ...

SudokuFX uses JavaCV now ... why?

Let me start with basic project setup, that is to say maven dependencies and stuff you need to get SudokuFX to work. 

Speaking of maven dependencies, this is the major reason why I'm switching from 'pure' OpenCV to JavaCV, since it is somehow - magically - fully mavenized and I can use OpenCV indirectly in my application by using JavaCV APIs. 

No need to compile OpenCV myself, no fiddling around with CMake, just staying in my cozy maven Java dependency world, which should lower the bar to start with this project substantially.

Of course JavaCV imposes certain new rules and approaches for the programmer, you have to adapt a little bit and make yourself acquainted with another API. For this project however it payed off. 

I discovered some spots to improve the image processing. It runs way faster and more reliable now.

On the other hand, one has to be careful not to run in subtle bugs which arise from off heap memory management which is a fancy way to say that you'll get exposed to native code behaviour. 

Anyway, starting point for the rewrite to JavaCV was to use raw data from Mat via ByteBuffers, which I only found in JavaCV's API, not in OpenCV. Maybe I'm right, there is no equivalent for this in pure OpenCV C++ / Java API, maybe someone points me to it. With my current approach I'm quite happy, 2020.3 release runs way faster than any other version of SudokuFX so far :)

No more Scala Futures

Scala has a Future implementation in it's standard lib, but it doesn't have the best reputation. All discussions in the Scala community lately made me rethink my approach, and in the end I realized that this pervasive usage of Scala's Future in my code didn't really speed up things, but only bloated the whole source and made it less understandable. As such I refactored this construct out of the source, with planning to later reintroduce it again after cleaning up structure. But I learned that my problem didn't really profit from Future's, as such they are left out completely at the current state of affairs. I can imagine that I'll reintroduce some parallelism again, but at the moment the algorithm runs faster than before. Maybe I'll come back to it and optimise it more, I don't know.

Improved image processing by adapting templates

I'm using template matching for recognising numbers, no third party library like tesseract for example. This keeps it simple, but is certainly a weak point in the approach. Here the project could profit by using a simple neural net maybe. By tweaking and disambiguating the used templates however SudokuFX is now capable of recognising quite a lot of different fonts, which is great (some fonts however will fail miserably ;-) )

Improved visual appearance

One thing I'm quite fond of is that SudokuFX now overlays the computed solution with the original image, yielding a much better visual appearance of the solved Sudoku. It looks far more advanced than before, which is great. 

Improved resource handling

CPU as well as memory footprint are now far better than they used to be. This was also one of the goals of the refactoring, SudokuFX uses less energy, memory and utilizes CPU far better than before.

Versioning Scheme

A blog post from Axel Fontaine made me curious as to not use maven release plugin anymore, but an approach far simpler - SudokuFX is using this approach now too.

JavaFX 15-ea+4 / Scala 2.13.3

Like I reported in a recent blogpost, I migrated SudokuFX to Java11, which is a LTS version of Java and one of the first releases which where JavaFX was decoupled. JavaFX now lives it's own life - but accidentally - or rather planned - has the same release cycles like Java. 

BUT in my usecase scenario I directly profit from the fact that I can use a 'newer' JavaFX with the Java11 LTS version.

Why? Because at the moment of writing this lines here, I have troubles getting Scala to work with Java13, whereas with Java11 my project runs out of the box. 

Scala 2.13.3 is serving as a good starting point to finally change to Scala3, which will also be an interesting endeavour in the future.

So, to sum up, try it out yourself, use SudokuFX to solve your Sudoku problems, have a look at the code to learn something about Scala or JavaCV or report bugs you'll easily find and participate at github.

Source code which is referred to by this article can be found here.

Webcam Image Stream with JavaCV

JavaCV?

JavaCV is a very cool library which empowers Java users to access native libraries like OpenCV or FFmpeg and many others.

Among other things, it prepackages OpenCV with maven, as such you can jumpstart in OpenCV development in mere minutes. Even if it downloads the internet without special tuning, it is as far as I know the simplest and fastest way to try out computer vision algorithms on the Java platform.

As a bonus, it enables also writing applications for Android, not to mention the usual desktop platforms like Windows / Mac and Linux.

In the past I've created some applications using OpenCV and it's official Java API, which is also available via JavaCV as far as I know. But JavaCV additionally provides API's which give you an added value as it makes using OpenCV APIs more idiomatic for a Java user, which is always a good thing for adoption.

In short, you save some time setting up your system if you just decide to use JavaCV.

Motivation

The motivation for devoting some time to JavaCV was to see if I could take advantage of the new PixelBuffer API which can use since JavaFX13 native arrays directly.

It seems clear that this should boost performance considerably and improves framerates for applications like my SudokuFX project. I've created a proof of concept  which compares different ways of using a webcam image stream based on the OpenCV stack.

In this project there are three implementations given:

• a swing based, officially endorsed way to show a webcam / image
• the classic JavaFX variant
• the DirectBuffer approach

I've measured how fast each approach is, and like expected, on my machine it turns out that the last one using JavaFX with DirectBuffer is the fastest way to show a webcam image stream.

Here is the output for running the three different versions, with the swing approach being the slowest one.

home:target lad$ java -jar javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar 
Mean: 719434
Mean: 628738
Mean: 601112
Mean: 579823
Mean: 565205
Mean: 552922

home:target lad$ java -jar javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar classic
Mean: 9508516
Mean: 9461551
Mean: 9423204
Mean: 9419029
Mean: 9415144

home:target lad$ java -jar javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar swing  
Mean: 35786482
Mean: 35410293
Mean: 35220229
Mean: 35092391
Mean: 35030641

Above numbers say that DirectBuffer approach is the way to go, no discussion about that! I didn't measure CPU load or memory behaviour, but I'm sure those numbers are better as well.

Conclusion

If you are using OpenCV and JavaFX, you should definitely give JavaCV and its enhanced API's for OpenCV a try.

Check out the source code for this blog post on github.

I would like to thank Samuel Audet who does a great job maintaining JavaCV and who also helped me get my webcam image to fit to JavaFX's PixelBuffer.

Version 2020.3.0 of javacv-webcam uses the same techniques, but is implemented in 100% Java. Check it out.