JNI, Python and _Py_Zero that was not there

Recently, I had this issue with JNI code calling Python script using shared libraries provided by Python itself.

The schema is quite easy to follow, and you can easily apply it inside C based code.

// Initialize Python (can be done once - for a whole life of JVM)

// Remember to reserve PyGILState
// It is not thread safe	
gstate = PyGILState_Ensure();

// c_str will contain content of java.lang.String passed
// from Java
const char *c_str;
PyRun_SimpleString( c_str );

// once finished remember to release GIL


You can find fully working sample (Linux/macOS) here – python via JNA. In one of our projects we had to go via JNI for some post processing computations done by Python, otherwise, we had huge performance drawbacks. In fact, we have noticed ~100 times faster executions of the code while using Python with JNI together with some additional caching – reference. Note that we were running code using 256 CPUs, where both Java and Python were running on all CPUs. As for passing java.lang.String from Java to C, in case you are not familiar with it, take a look here – passing java.lang.String value from Java to C.

Usually, this approach works perfectly fine. Unless something is not quite OK. Last week I got really strange error coming from very simple Python script with numpy installed. And the error was:

File “../python2.7/site-packages/numpy/core/__init__.py”, line 24, in
raise ImportError(msg)
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you’re working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.

which is not quite promising. It looks like something is really broken. And it looks really serious. Now, let’s check what exactly is happening there. Let’s take a look at __init__.pyt inside core.

    from . import multiarray
except ImportError:
    msg = """
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.
    raise ImportError(msg)

What we can see here is that multiarray is one to blame. We need to find out what multiarray is.

> find . -name "multiarray*"

In fact, it’s a shared library that is loaded from Python. We need to know why does it fail inside JVM. Do you remember the execution schema?

JVM -> SomeJavaClass -> JNI -> Python library (libpython2.7) -> numpy

Let’s see what happens if we try to load this library inside JVM. We can use supper simple code for that

public class TestMulti {
  static {
    // System.loadLibrary takes name of library as argument
    // System.load takes full path of file
  public static void main(String[] args) {}

And now you are talking! It looks like multiarray.so is not able to find some symbols.

> java -Djava.library.path=... -cp . TestMulti
Exception in thread "main" java.lang.UnsatisfiedLinkError: .../multiarray.so: undefined symbol: _Py_ZeroStruct
  at java.lang.ClassLoader$NativeLibrary.load(Native Method)
  at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
  at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
  at java.lang.Runtime.load0(Runtime.java:809)
  at java.lang.System.load(System.java:1086)
  at python.PythonJustLoad.(PythonJustLoad.java:14)

Symbol _Py_ZeroStruct comes from libpython2.7. You can check it quite easily

> nm .../libpython2.7.so | grep _Py_Zero
00000000003c6cc0 D _Py_ZeroStruct

It looks like location of libpython2.7 is specified on LD_LIBRARY_PATH.

> env | grep LD_LIBRARY_PATH | tr ':' '\n' | grep python

Digging through StackOverflow lead me to people who had similar issues: Java: load shared librariees with dependencies. And I know that JNA solves the issue. However, I prefered to stick to JNI. So, we need to make sure that symbols from libpython2.7 are loaded properly while still using JNI. We have two options here

> export LD_PRELOAD=.../libpython2.7.so

This approach will enforce loading library priori to any other library loaded by process

> man ld.so
              A list of additional, user-specified, ELF shared libraries to be loaded before all others.
              The items of the list can be separated by  spaces  or  colons. This  can  be  used  to 
              selectively override functions in other shared libraries.  The libraries are searched for 
              using the rules given under DESCRIPTION. For set-user-ID/set-group-ID ELF binaries, preload 
              pathnames containing slashes are ignored, and libraries in the standard search directories  
              are  loaded only if the set-user-ID permission bit is enabled on the library file.

However, this is not quite what we want. This way, we will load Python library all over the place. Each and every process will load it. Not exactly what we are looking for. What we can do, instead, is to enforce loading of Python library directly within JNI code

void loadLibrary() {
  if(handle == NULL) {
    handle = dlopen("libpython2.7.so", RTLD_LAZY | RTLD_GLOBAL);
    if (!handle) {
      fprintf(stderr, "%s\n", dlerror());



// Initialize Python (can be done once for whole live of JVM)

// Remember to reserve PyGILState
// It is not thread safe	
gstate = PyGILState_Ensure();

// c_str will contain content of java.lang.String passed
// from Java
const char *c_str;
PyRun_SimpleString( c_str );

// once finished remember to release GIL


Two things here. First, we enforce RTLD_GLOBAL and make symbols from libpython2.7 visible for multiarray.so (note that this is not default behavior of JNIJVM loads libs using dlopen(filename, RTLD_LAZY). Second, we are inside JVM itself and we enforce loading of library only in case the code is actually executed. And that’s what solves the issue.

Have fun with JNI!


1. OpenJDK – loading libraries
2. Java: load shared librariees with dependencies
3. man ld.so
4. Embedding Python code

Remove redundant imports in Java file in CLI

cat MyClass.java | grep ^import | sort | uniq

Multiple threads and JVM access from C code

In case you are looking for sample code related to JVM and accessing it from C (like this one):

take a look here: recipe № 027.

requested Java version ((null)) not available. Using Java at “” instead

If you are getting following errors while trying to install xlsx package in R (macOS)

> library(xlsx)
Loading required package: rJava
Error : .onLoad failed in loadNamespace() for 'rJava', details:
> library(xlsx)
JavaVM: requested Java version ((null)) not available. Using Java at "" instead.
JavaVM: Failed to load JVM: /bundle/Libraries/libserver.dylib
JavaVM FATAL: Failed to load the jvm library.
Error : .onLoad failed in loadNamespace() for 'xlsx', details:
  call: .jinit()
  error: JNI_GetCreatedJavaVMs returned -1

Error: package or namespace load failed for ‘xlsx’

try following

> sudo R CMD javareconf -n
> R
> install.packages('rJava', type='source')
> install.packages('xlsx')
> library(xlsx)

Enforce LANG settings in R – macOS

defaults write org.R-project.R force.LANG en_US.UTF-8

What if you want to modify Makefile content while building your code

Well, it’s possible. Let’s say we have directory with source code: main.cpp

├── Makefile
└── main.cpp

Source code is supper simple.

int main() { return 0; }

What we want to do is to run Makefile and make sure that all occurrences of main will be changed to directory name (e.g. myCode).

All we have to do is to get location of Makefile, update it, store it inside other file and call that other file. In the meantime we can perform some source code manipulation as well.

makefilelocation = $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
makefiletarget = $(makefilelocation).target

BASENAME := $(notdir $(CURDIR))

all: call
        $(warning "I will call myself")

call: copy
        $(MAKE) -f $(makefiletarget) exe

exe: main.cpp
        gcc -o main main.cpp

        @cp main.cpp $(BASENAME).cpp
        @sed -e "s/main/$(BASENAME)/g" $(makefilelocation) > $(makefiletarget)

        -rm $(BASENAME).cpp $(BASENAME) $(makefiletarget)

After execution of this Makefile we will have something like this

├── Makefile
├── Makefile.target
├── main.cpp
├── myCode
└── myCode.cpp

Change file name to directory name

OK, I am not sure whether anybody really needs something like this. But, in case you want to change your executable name to the same name as your directory name, try this

BASENAME := $(notdir $(CURDIR))

all: copy exe

        @cp main.cpp $(BASENAME).cpp

exe: copy $(BASENAME).cpp
        gcc -o $(BASENAME).o $(BASENAME).cpp

/bin/sh: @echo: command not found

If you get this one, remember about missing new line after shell call that spans over multiple lines that are split with \. You will get this error in case you don’t add explicit new line and you want to silence the command using @ symbol at the same time.

LIST=a b c d

        @echo "Starting!"
        @echo "Loop over values"

        @for target in `echo ${LIST}`; do \
                echo "Value: $$target"; \
        done; \
        @echo "Done!"

Below, you have new line after last \ – this one will work without any problems.

LIST=a b c d

        @echo "Starting!"
        @echo "Loop over values"

        @for target in `echo ${LIST}`; do \
                echo "Value: $$target"; \
        done; \

        @echo "Done!"

Alternatively, you can make last echo to be part of the multiline command. Here, you will get the expected result as well.

LIST=a b c d

        @echo "Starting!"
        @echo "Loop over values"

        @for target in `echo ${LIST}`; do \
                echo "Value: $$target"; \
        done; \
        echo "Done!"

Whether to use this approach or not (I mean, to call shell from Makefile) is the topic for a completely different story.

More about commands modifiers: here

Running Open MPI on macOS

I am, generally, against all these automation stuff (brew, etc.) at macOS. I prefer to install everything from sources (as long as it is possible).

So, here are few steps to get Open MPI up and running at macOS. Please make sure to download source package from: https://www.open-mpi.org/software/ompi/v2.0/downloads/openmpi-2.0.2.tar.gz.

# put this file somewhere inside your $HOME
# and untar it
> tar xf openmpi-2.0.2.tar
> cd openmpi-2.0.2/

# Make sure you have XCode and command line tools
# installed - all of these if free of charge
> ./configure --prefix=$HOME/opt/usr/local
> make all
> make install

# After installation is done, you can 
# verify it
> $HOME/usr/local/bin/mpirun --version
mpirun (Open MPI) 2.0.2

Report bugs to http://www.open-mpi.org/community/help/

Now, it’s time for Hello world! code.

/* Put this text inside hello.c file */

int main(int argc, char** argv) {
    int rank;
    int world;

    MPI_Init(NULL, NULL);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &world);
    printf("Hello: rank %d, world: %d\n",rank, world);

And, eventually, we can compile and run it

$HOME/opt/usr/local/bin/mpicc -o hello ./hello.c
$HOME/opt/usr/local/bin/mpirun -np 2 ./hello
Hello: rank 0, world: 2
Hello: rank 1, world: 2

That’s all.

Get all hidden files inside $HOME

Simple one liner that helps to find all config locations inside your HOME directory

ls -a1 $HOME | grep '^\.'

In case you like more bass, the same way I do ;)

iTunes -> Windows -> Equaliser

# alternatively: Option + Command + E

apps keep asking for key chain access

1. Open Key Chain Access and unlock all key chains

2. use security utility to set timeout for keychain

security -v set-keychain-settings -t 72000 login.keychain

You can read more about security using man

man security

Scratch Coding Cards by Natalie Rusk

Summary: So far, the best Scratch Coding resource I have found :)

“Old school” way of learning (one that industrial era developed) is based on reading. To know something, you have to read something. Well, that’s not always the best idea. And let me remind you, that an old philosopher, long, long time ago in a land far, far away, said:

You know, Phaedrus, writing shares a strange feature with painting. The offsprings of painting stand there as if they are alive, but if anyone asks them anything, they remain most solemnly silent. The same is true of written words. You’d think they were speaking as if they had some understanding, but if you question anything that has been said because you want to learn more, it continues to signify just that very same thing forever

– Socrates

And that’s exactly what it is whet it comes to learning from books. You are always left with the content as it is, and you can hardly discuss anything with it.

Let’s be honest, reading is really boring nowadays. You have so many, various, sources of knowledge: youtube, online courses, tutorials, podcasts, screencasts, etc.

Even more, sometimes you don’t need them at all. All you need is just an inspiration. Idea of what can be done, and how can you achieve it. You don’t need all the steps, described one by one. You don’t need full recipe. General idea is more than enough. After you know what you are looking for, all you need to do is to experiment and try to achieve it.

This is exactly the way you can learn using Scratch Coding Cards. By far, this is the most engaging and most powerful tool to teach Scratch I was dealing with. I have some experience with teaching Scratch and I know that finding good way of engaging kids into Scratch is quite challenging. There are few things I done in past. Like, fighting very bad and powerful Knight – you can find him at youtube. Or, some ideas quickly sketched on a piece of paper: here (my beloved Moleskine notebooks ;) ). However, they require either following the instructions, strictly, when you watch video or they require someone extra comment from teacher – like in case of Scratch UML.

When it comes to Scratch Coding Cards, it’s all different. All you need to do, is to give one of the sets to kids, and they will solve it. That’s the beauty of this Box full of cards. Kids are simply shuffling cards, looking at them to check what can be done and which blocks are important, and few minutes later solution is there.

What you get inside the Box (which is firm and solid) are 75 cards.

Cards are grouped into activities (challenges you are supposed to code). Each challenge can be solved by following steps presented on cards. Together with my kids, we have found that strict following of the order is not required.


Each card describes some activity and show you how to achieve that. It’s supper simple. Note that my kids are non english speakers and they were still able to follow cards by simply looking and images. This was really cool!

And, one more comment. There was no cheating at all. Kids were given cards and I slowly shaded away :)

If you are looking for some ideas and new, fresh way of teaching Scratch, make sure to check this one. Really worth it.

Have fun!!

Printed: Scratch Coding Cards

ldd in macOS

otool -L libFile.dylib

Redirecting http traffic

Sometimes, simple solutions are the best

> iptables -A PREROUTING -t nat -p tcp --dport 80 -j REDIRECT --to-port 8888
> iptables -t nat -I OUTPUT -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8888

# in case we want to list what we have

> iptables -t nat -nvL --line-numbers

# if we decide to remove the rule - ## stands for rule No.

> iptables -D PREROUTING ## -t nat
> iptables -D OUTPUT ## -t nat

Mr. Robot – some thoughts about bugs

Most coders think debugging software is about fixing a mistake, but that’s bullshit.
Debugging’s actually all about finding the bug.

Coding apprentice – TODOs

Coding apprentice:
– Hi there. Well, you know, there are lots of TODOs inside code.
What does it mean, really? Is it: This One Datelessly On-site?

If you have makefile that has weird name but you still want syntax highlighting

:set syntax=make

makefile – all you wanted to know about variable but were afraid to ask

Sometimes, all you need is a simple solution for checking whether variable you really (really!) need is already defined.

assert = $(if $2,$(if $1,,$(error $2)))

    $(call assert,$(VARIABLE),VARIABLE is not defined)

Stolen from: http://gmsl.sourceforge.net and found here The GNU Make Book

makefile – know your location

If you want to know where you really are during Make

location = $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
$(warning $(location))

Source: unknown Stack Overflow user