C and controlling debug stuff (something, almost, like Log4j) ;)

In case you are not aware of the macros ;) I am sure you are aware, but, just in case :)

int f(int a, int b) {
  #ifdef PROFILER
  BeginProfiling("f");
  #endif
  ...
  #ifdef PROFILER
  EndProfiling();
  #endif
}

Then, you can control it by doing this

// when you want to profile
#define PROFILER

// when you don't want to profile
//#define PROFILER

In case you are not aware of function pointers ;)

#include <stdio.h>

void (*doprofiling)(void);

void profile()
{
    printf("I am profiling\n");
}

void no_profile()
{
    printf("I am not profiling\n");
}

void fun() {
  doprofiling();
}

int main()
{
    doprofiling = profile;

    fun();

    doprofiling = no_profile;

    fun();

    return 0;
}

Then, you can switch in the code dynamically

gcc -o profile ./profile.c
./profile
I am profiling
I am not profiling

Or, you can use something like this, and you can apply different decorators to different functions

#include <stdio.h>

void doprofiling(void (*proffun)(void)) {
  proffun();
}

void profile()
{
    printf("I am profiling\n");
}

void no_profile()
{
    printf("I am not profiling\n");
}

void fun_prof() {
  void (*decorator)(void) = profile;
  doprofiling(decorator);
}

void fun_no_prof() {
  void (*decorator)(void) = no_profile;
  doprofiling(decorator);
}

int main()
{
    fun_prof();
    fun_no_prof();
    return 0;
}

And, you can still dynamically apply it in the code.

    
> gcc -o ./profile ./profile.c
> ./profile
I am profiling
I am not profiling

Compiling Slatec on macOS

# SLATEC Common Mathematical Library, Version 4.1, July 1993
# a comprehensive software library containing over
# 1400 general purpose mathematical and statistical routines
# written in Fortran 77.

If you want to install SLATEC, you need to make sure to install gfortran.

Take a look here for a brief instructions, somewhere in the middle of shell code: http://www.owsiak.org/?p=3492

Make sure to download sources and linux makefile and put all files at the same level. By saying at the same level, I mean that all *.f files from slatec_src.tgz and makefile and dynamic and static from slatec4linux.tgz are in the same dir.

SLATEC sources: http://www.netlib.org/slatec/slatec_src.tgz
SLATEC makefile: http://www.netlib.org/slatec/slatec4linux.tgz

Before building library, make sure to export FC variable (it is needed by makefile)

export FC=gfortran

Make sure to change this line inside dynamic/makefile

libslatec.so: $(OBJ)
    $(CC) -shared -o $@ $(OBJ)

to

libslatec.so: $(OBJ)
    $(FC) -shared -o $@ $(OBJ)

Call make

make

Wait a little bit. Take a look inside static and dynamic, files should be there.

find . -name "libslatec*"
./dynamic/libslatec.so
./static/libslatec.a

Now, you can try to perform make install (pay attention here as it will overwrite hardcoded locations). Alternatively, you can use

-L${WHERE_YOUR_BUILD_WAS_DONE}/dynamic -lslatec

SLATEC refers to symbols that can be found inside LAPACK package. If you don’t have it installed, take a look here

mkdir lapack
cd lapack
curl "http://www.netlib.org/lapack/lapack-3.7.1.tgz" -o lapack-3.7.1.tgz
tar zxf lapack-3.7.1.tgz
cd lapack-3.7.1
ln -s make.inc.example make.inc
make

After compilation is done, you can find liblapack.a inside lapack-3.7.1.


Jekyll on macOS

1. Install Ruby

I prefer to install from sources: https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.1.tar.gz

> ./configure --prefix=$HOME/opt/ruby
> make
> make install

2. Install RubyGems

I prefer to install from sources: https://rubygems.org/pages/download

> export PATH=$HOME/opt/ruby/bin:$PATH
> ruby setup.rb

3. Change repo location

I compile Ruby without ssh support. I have to change the ruby gems repo

> gem sources -r https://rubygems.org/
> gem sources -a http://rubygems.org/

(if you want to build with OpenSSL, take a look here: http://www.owsiak.org/?p=3262 There is a sample related to building OpenSSL at macOS)

4. You can install Jekyll

> gem install jekyll
> jekyll --version
jekyll 3.5.2

That’s it.

Source: Originally, posted here: https://stackoverflow.com/questions/46019526/


git, github and multiple users

Different solutions for working with github repos when you have multiple accounts

If you have two user accounts at github: {user_a} and {user_b}. Sometimes, you may encounter issues while pushing changes back

remote: Permission to {user}/{repo}.git denied to {user_a}.
fatal: unable to access 'https://github.com/{user}/{repo}.git/': The requested URL returned error: 403

If you are using https, you can play with .git/config

> git clone https://github.com/{user}/{repo}.git
> vi .git/config

_-= Inside VIM =-_
# you may enforce user here
[remote "origin"]
        url = https://{user_a}@github.com/{user}/{repo}.git

If you are using ssh, you can play with .ssh/config

Host {user_a}@github.com
  IdentityFile ~/.ssh/id_rsa_a

Host {user_b}@github.com
  IdentityFile ~/.ssh/id_rsa_b

Host github.com-{user_a}
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_a

Host github.com-{user_b}
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_b

Host github.com-{user}
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_rsa_a

Host bitbucket.org
    User git
    IdentityFile ~/.ssh/id_rsa_c

SVN and meld

> svn diff --diff-cmd='meld' file_you_have_changed.c

SVN and multiline log comments

> svn commit -m"We tend to forget\
that multiline comments, in SVN, are supper easy.\
And, they can help to introduce multiple changes\
inside one commit. Like:\
- improvements,\
- bug fixes,\
- motivations behind the code." my_super_file.c

Parameter Expansion – POSIX (bash)

                     +----------------------+-----------------+-----------------+
                     |       parameter      |    parameter    |    parameter    |
                     |   Set and Not Null   |   Set But Null  |      Unset      |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter:-word} | substitute parameter | substitute word | substitute word |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter-word}  | substitute parameter | substitute null | substitute word |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter:=word} | substitute parameter | assign word     | assign word     |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter=word}  | substitute parameter | substitute null | assign word     |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter:?word} | substitute parameter | error, exit     | error, exit     |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter?word}  | substitute parameter | substitute null | error, exit     |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter:+word} | substitute word      | substitute null | substitute null |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter+word}  | substitute word      | substitute word | substitute null |
+--------------------+----------------------+-----------------+-----------------+

source: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02


R 3.4, rJava, macOS and even more mess ;)

So, you want to have rJava (e.g. rJava_0.9-8.tar.gz) inside your fresh, new installation of R 3.4 and you are running macOS. There are bad news and good news ;)

Bad news. It will fail with default clang that comes with XCode. You need something better here, something with support for OpenMP. And you can get it following way

# make sure to create some place where you want to have it (I, personally, put stuff into ~/opt)
> mkdir ~/opt
> cd ~/opt
> curl http://releases.llvm.org/4.0.1/clang+llvm-4.0.1-x86_64-apple-darwin.tar.xz \
-o clang+llvm-4.0.1-x86_64-apple-darwin.tar.xz
> tar xf clang+llvm-4.0.1-x86_64-apple-darwin.tar.xz

Now, make sure to install gfortran. I am using version from this location: GFortran.

Once you have it, make sure to install most recent JDK from Oracle. You can find it here: JavaSE.

After you have your Java installed, make sure to do following (double check that everything is OK so far)

> R --version
R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin15.6.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
http://www.gnu.org/licenses/.
> /usr/libexec/java_home -V
Matching Java Virtual Machines (3):
    1.8.0_144, x86_64:	"Java SE 8"	/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
    1.8.0_111, x86_64:	"Java SE 8"	/Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home
    1.7.0_80, x86_64:	"Java SE 7"	/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home

# Make sure to put following line inside you ~/.profile
# export JAVA_HOME=$(/usr/libexec/java_home -v 1.8.0_144)
# close and re-open Terminal session. Now, you should be able to see:
> echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
/usr/libexec/java_home
/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home

Make sure to enable your JDK for JNI

> sudo vi /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Info.plist

# make sure to replace
#<array>
#  <string>CommandLine</string>
#</array>
# with
#<array>
#  <string>CommandLine</string>
#  <string>JNI</string>
#</array>

Now, it’s time to configure R. Make sure to run following command.

!! (note that we change JAVA_HOME to JRE) !!

> JAVA_HOME=${JAVA_HOME}/jre
> sudo R CMD javareconf \
JAVA_HOME=${JAVA_HOME} \
JAVA=${JAVA_HOME}/../bin/java \
JAVAC=${JAVA_HOME}/../bin/javac \
JAVAH=${JAVA_HOME}/../bin/javah \
JAR=${JAVA_HOME}/../bin/jar \
JAVA_LIBS="-L${JAVA_HOME}/lib/server -ljvm" \
JAVA_CPPFLAGS="-I${JAVA_HOME}/../include -I${JAVA_HOME}/../include/darwin"

Note! It looks like R CMD javareconf doesn’t update all the flags, make sure that file: /Library/Frameworks/R.framework/Versions/3.4/Resources/etc/Makeconf contains following entries:

JAVA_LIBS="-L${JAVA_HOME}/lib/server -ljvm" \
JAVA_CPPFLAGS="-I${JAVA_HOME}/../include -I${JAVA_HOME}/../include/darwin"

You should have your Java and R linked together. It’s time to add support for clang that you have just installed few steps above.

Let’s say your clang is in your home dir. Here: /Users/user_name/opt/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/

Make sure, to add following file (~/.R/Makevars – more info can be found here: R for Mac OS X FAQ and here: R Installation and Administration)

CLANG=/Users/user_name/opt/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0
CC=$(CLANG)/bin/clang
CXX=$(CLANG)/bin/clang++
CXX1X=$(CLANG)/bin/clang++
CXXFLAGS=-I$(CLANG)/include

Also, make sure to modify file:

/Library/Frameworks/R.framework/Versions/3.4/Resources/etc/Makeconf

inside this file, make sure that line “LDFLAGS” reads

LDFLAGS = -L/usr/local/lib -L/Users/user_name/opt/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/lib -lomp

You can confirm that it works by calling:

# you can confirm that by calling
> R CMD config --ldflags
-fopenmp -L/usr/local/lib 
-L/Users/user_name/opt/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/lib 
-lomp -F/Library/Frameworks/R.framework/.. -framework R 
-lpcre -llzma -lbz2 -lz -licucore -lm -liconv

Now, it’s time to get rJava. Simply download it, and install it:

> curl "https://cran.r-project.org/src/contrib/rJava_0.9-8.tar.gz" -o rJava_0.9-8.tar.gz
> R CMD INSTALL rJava_0.9-8.tar.gz
* installing to library ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library’
* installing *source* package ‘rJava’ ...
...
...
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (rJava)

And you can now test it. Let’s say you have (in your working directory) following layout

utils/
└── RUsingStringArray.java

and file RUsingStringArray.java contains

package utils;

public class RUsingStringArray {
  public String [] createArray() {
    System.out.println("Creating empty array");
    return new String[0];
  }
}

You can compile it and run it in R

> javac utils/*.java
> export CLASSPATH=`pwd`
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.8.0_144)/jre
> R
> library(rJava)
> .jinit()
> obj <- .jnew("utils.RUsingStringArray")
> s <- .jcall(obj, returnSig="[Ljava/lang/String;", method="createArray")
> .jcall(obj, returnSig="I", method = "arrayLen", s)
Class: class [Ljava.lang.String;
[1] 0
>

Retro Like (8bit) Terminal Game – playing for Fun and Profit :)

So, what do you think about playing retro style, terminal based, game where you can gain some real life experience as well. If you are interested, take a look below (click the image for the full experience – or click here for full screen size).

I find Cathode to be supper fancy terminal emulator with element of surprise. Whenever you want to gain some attention during presentations it comes handy. There is nothing as good as “Back to the Future” effect when you want to gain some attention from the audience. Especially, when your terminal session itself is quite boring. On regular basis, I definitely prefer iTerm2. But for the looks, Cathode plays it’s role.

As for the resources.

Music: https://youtu.be/hTZ-zQInMsk

Cathode: http://www.secretgeometry.com/apps/cathode/

Type speed instructions for macOS: Building GNU Typist for Mac OS

Sounds: AtomSplitter


How to share enum between C++ and Java

There was this question at Stack Overflow. How to share enums. And you want to share them between C++ and Java. Well, there is one issue here. You can’t base on comments (to trick compiler) as Java and C++ have the same comment schema. How to use comments to make cross language coding? Take a look here.

You can use a nasty hack with “disabling” public keyword in C++ using macro.

#include <stdio.h>

#define public
#include "A.java"
;
#undef public

int main() {
  A val = a;
  if(val == a) {
    printf("OK\n");
  } else {
    printf("Not OK\n");
  }
}

Then, inside Java class you can define enum

public
enum A {
  a,
  b
}

And you can use it in Java later on. As you can see above, it is used in C++ as well.

public class B {
  public static void main(String [] arg) {
    A val = A.a;
    if(val == A.a) {
      System.out.println("OK");
    } else {
      System.out.println("Not OK");
    }
  }
}

Seems to work fine :)

> javac *.java
> java B
OK
> g++ -o main ./main.cc
> ./main
OK

Running GDB in macOS sierra

At some point Apple dropped GNU based toolchain in favor of LLVM. However, sometimes you need these tools. One of such examples is debugging Fortran codes. lldb is not able to do it properly, yet. You can trace the execution of code, you can step over, but you can’t examine variables.

In that case, you need GDB. You can install it following way.

First, get the GDB sources

mkdir src
cd src
curl "http://ftp.gnu.org/gnu/gdb/gdb-8.0.tar.gz" -o gdb-8.0.tar.gz
tar zxf gdb-8.0.tar.gz
cd gdb-8.0
./configure --prefix=$HOME/opt/usr/local
make
make install

This way, you will install GDB inside $HOME/opt/usr/local. However, installing GDB is not enough. You need to Code Sign it in order to make it possible to debug codes using GDB. To make it possible, do following. If you don’t have Code Signed GDB, it will end with the error.

You can find detailed description of how to Code Sign GDB here.

After you are done, you can test your GDB installation and run the code inside debugger. Lets say we have following code:

! program.f90
program prog
  integer variable
  variable = 1
  write(*,*) variable
end

After compilation with “-g” flag

gfortran -g -o ./program ./program.f90

we can simply run it inside gdb

# Make sure to export location of gdb
export PATH=${PATH}:$HOME/opt/usr/local/bin/
gdb ./program
GNU gdb (GDB) 8.0
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin16.5.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./program...Reading symbols from 
...
done.
(gdb) list
1	program prog
2	  integer variable
3	  variable = 1
4	  write(*,*) variable
5	end
(gdb) break 4
Breakpoint 1 at 0x100000e2d: file ./program.f90, line 4.
(gdb) run
Starting program: ./program
[New Thread 0x1403 of process 1234]
warning: unhandled dyld version (15)

Thread 2 hit Breakpoint 1, prog () at ./program.f90:4
4	  write(*,*) variable
(gdb) print variable
$1 = 1
(gdb)

R, Java, rJava and macOS adventures

For updated info (for R3.4) take a look HERE.

R, rJava, Java and macOS is a mess. In fact, Java itself is a mess when it comes to macOS. First of all, you can have different Java installations: Oracle based or system based.

# this is the place where Java Framework resides
/System/Library/Frameworks/JavaVM.framework

# this is the place where Oracle based Java will be installed
/Library/Java/JavaVirtualMachines

Apple’s layout is quite different comparing to what you expect from typical JDK installation.

tree -d
.
├── Frameworks -> Versions/Current/Frameworks
├── Resources -> Versions/Current/Resources
└── Versions
    ├── A
    │   ├── Commands
    │   ├── Frameworks
    │   │   ├── JavaNativeFoundation.framework
    │   │   │   ├── Resources -> Versions/Current/Resources
    │   │   │   └── Versions
    │   │   │       ├── A
    │   │   │       │   ├── Resources
    │   │   │       │   │   └── English.lproj
    │   │   │       │   └── _CodeSignature
    │   │   │       └── Current -> A
    │   │   └── JavaRuntimeSupport.framework
    │   │       ├── Resources -> Versions/Current/Resources
    │   │       └── Versions
    │   │           ├── A
    │   │           │   ├── Resources
    │   │           │   │   └── English.lproj
    │   │           │   └── _CodeSignature
    │   │           └── Current -> A
    │   ├── JavaPluginCocoa.bundle
    │   │   └── Contents
    │   │       ├── MacOS
    │   │       ├── Resources
...

Anyway, if you want to configure R such way it works properly with JVM, you have to make some tweaking. First of all, you have to play with JAVA_HOME. In macOS you have this special command: java_home

> /usr/libexec/java_home -V
Matching Java Virtual Machines (2):
    1.8.0_11, x86_64:	"Java SE 8"	/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
    1.8.0_05, x86_64:	"Java SE 8"	/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home

# if you run it without any switches it will show currently used Java
/usr/libexec/java_home
/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home

so far, so good. If you tree this one, you will see that structure is quite similar to other systems (e.g. Linux, Windows).

tree -d `/usr/libexec/java_home`
/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
├── bin
├── db
│   ├── bin
│   └── lib
├── include
│   └── darwin
├── jre
│   ├── bin
│   └── lib
│       ├── applet
│       ├── cmm
│       ├── ext
│       ├── fonts
│       ├── images
│       │   └── cursors
│       ├── jfr
│       ├── jli
│       ├── management
│       ├── security
│       └── server
...

However, if you try to use JDK with R, there is a small surprise behind the corner. To get Java running in R, you call javareconf as follows

sudo R CMD javareconf

But, there is a small but. javareconf assumes that whenever you JAVA_HOME matches “/Home$” using grep, it points to system based Java installation – instead of Oracle’s one. This makes things weird. Half of your settings point to JDK and other half to JVM installed by Apple. Things can go wrong, and will.

To overcome this issue, you need to create fake Java home directory – without “/Home” in the name.

cd /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents
ln -s Home RHome

then, you need to export it as JAVA_HOME

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/RHome

and run javareconf with settings allowing to compile the code

sudo R CMD javareconf \
JAVA_HOME=${JAVA_HOME} \
JAVA=${JAVA_HOME}/bin/java \
JAVAC=${JAVA_HOME}/bin/javac \
JAVAH=${JAVA_HOME}/bin/javah \
JAR=${JAVA_HOME}/bin/jar \
JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/server \
JAVA_CPPFLAGS="-I${JAVA_HOME}/include -I${JAVA_HOME}/include/darwin"

Now, you are able to test whether rJava compiles fine. Download rJava package and try to configure it manually (outside R) from sources.

rJava can be downloaded here: https://cran.r-project.org/web/packages/rJava/index.html

curl https://cran.r-project.org/src/contrib/rJava_0.9-8.tar.gz -o rJava_0.9-8.tar.gz
tar zxf rJava_0.9-8.tar.gz
cd rJava
./configure

if it passes, start R and install package from sources

> install.packages("rJava", type="source")

Note that with messy Java installations (Apple’s JREs, Oracle’s JDKs) you might need to play with setting up Java env properly in a first place. It might be needed to update file

`/usr/libexec/java_home`/../Info.plist

if you want to edit the file, make sure to use some editor you prefer (e.g. VI)

sudo vi `/usr/libexec/java_home`/../Info.plist

and make sure your JVM can handle JNI. You need to change:

<key>JVMCapabilities</key>
<array>
  <string>CommandLine</string>
</array>

to

<key>JVMCapabilities</key>
<array>
  <string>CommandLine</string>
<string>JNI</string>
</array>

Note!

rJava requires XQuartz for compilation, make sure to install it as well: https://www.xquartz.org


There are daemons in Java ;)

Take a look here, for more details: recipe № 029


JNI, Python and _Py_Zero that was not there

Recently, I had this issue with JNI code calling Python script using shared libraries provided by Python itself.

The schema is quite easy to follow, and you can easily apply it inside C based code.

// Initialize Python (can be done once - for a whole life of JVM)
Py_Initialize();

// Remember to reserve PyGILState
// It is not thread safe	
gstate = PyGILState_Ensure();

// c_str will contain content of java.lang.String passed
// from Java
const char *c_str;
PyRun_SimpleString( c_str );

// once finished remember to release GIL
PyGILState_Release(gstate);

Py_Finalize();

You can find fully working sample (Linux/macOS) here – python via JNA. In one of our projects we had to go via JNI for some post processing computations done by Python, otherwise, we had huge performance drawbacks. In fact, we have noticed ~100 times faster executions of the code while using Python with JNI together with some additional caching – reference. Note that we were running code using 256 CPUs, where both Java and Python were running on all CPUs. As for passing java.lang.String from Java to C, in case you are not familiar with it, take a look here – passing java.lang.String value from Java to C.

Usually, this approach works perfectly fine. Unless something is not quite OK. Last week I got really strange error coming from very simple Python script with numpy installed. And the error was:


File “../python2.7/site-packages/numpy/core/__init__.py”, line 24, in
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you’re working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.

which is not quite promising. It looks like something is really broken. And it looks really serious. Now, let’s check what exactly is happening there. Let’s take a look at __init__.pyt inside core.

try:
    from . import multiarray
except ImportError:
    msg = """
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.
"""
    raise ImportError(msg)

What we can see here is that multiarray is one to blame. We need to find out what multiarray is.

> find . -name "multiarray*"
./numpy/core/multiarray_tests.so
./numpy/core/multiarray.so

In fact, it’s a shared library that is loaded from Python. We need to know why does it fail inside JVM. Do you remember the execution schema?

JVM -> SomeJavaClass -> JNI -> Python library (libpython2.7) -> numpy

Let’s see what happens if we try to load this library inside JVM. We can use supper simple code for that

public class TestMulti {
 
  static {
    // System.loadLibrary takes name of library as argument
    // System.load takes full path of file
    System.load("../numpy/core/multiarray.so");
  }
 
  public static void main(String[] args) {}
}

And now you are talking! It looks like multiarray.so is not able to find some symbols.

> java -Djava.library.path=... -cp . TestMulti
Exception in thread "main" java.lang.UnsatisfiedLinkError: .../multiarray.so: undefined symbol: _Py_ZeroStruct
  at java.lang.ClassLoader$NativeLibrary.load(Native Method)
  at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
  at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
  at java.lang.Runtime.load0(Runtime.java:809)
  at java.lang.System.load(System.java:1086)
  at python.PythonJustLoad.(PythonJustLoad.java:14)

Symbol _Py_ZeroStruct comes from libpython2.7. You can check it quite easily

> nm .../libpython2.7.so | grep _Py_Zero
00000000003c6cc0 D _Py_ZeroStruct

It looks like location of libpython2.7 is specified on LD_LIBRARY_PATH.

> env | grep LD_LIBRARY_PATH | tr ':' '\n' | grep python
.../python/2.7

Digging through StackOverflow lead me to people who had similar issues: Java: load shared librariees with dependencies. And I know that JNA solves the issue. However, I prefered to stick to JNI. So, we need to make sure that symbols from libpython2.7 are loaded properly while still using JNI. We have two options here

> export LD_PRELOAD=.../libpython2.7.so

This approach will enforce loading library priori to any other library loaded by process

> man ld.so
...
...
LD_PRELOAD
              A list of additional, user-specified, ELF shared libraries to be loaded before all others.
              The items of the list can be separated by  spaces  or  colons. This  can  be  used  to 
              selectively override functions in other shared libraries.  The libraries are searched for 
              using the rules given under DESCRIPTION. For set-user-ID/set-group-ID ELF binaries, preload 
              pathnames containing slashes are ignored, and libraries in the standard search directories  
              are  loaded only if the set-user-ID permission bit is enabled on the library file.
...
...

However, this is not quite what we want. This way, we will load Python library all over the place. Each and every process will load it. Not exactly what we are looking for. What we can do, instead, is to enforce loading of Python library directly within JNI code

void loadLibrary() {
  if(handle == NULL) {
    handle = dlopen("libpython2.7.so", RTLD_LAZY | RTLD_GLOBAL);
    if (!handle) {
      fprintf(stderr, "%s\n", dlerror());
      exit(EXIT_FAILURE);
    }
  }
}

...
...

loadLibrary();

// Initialize Python (can be done once for whole live of JVM)
Py_Initialize();

// Remember to reserve PyGILState
// It is not thread safe	
gstate = PyGILState_Ensure();

// c_str will contain content of java.lang.String passed
// from Java
const char *c_str;
PyRun_SimpleString( c_str );

// once finished remember to release GIL
PyGILState_Release(gstate);

Py_Finalize();

Two things here. First, we enforce RTLD_GLOBAL and make symbols from libpython2.7 visible for multiarray.so (note that this is not default behavior of JNIJVM loads libs using dlopen(filename, RTLD_LAZY). Second, we are inside JVM itself and we enforce loading of library only in case the code is actually executed. And that’s what solves the issue.

Have fun with JNI!

References:

1. OpenJDK – loading libraries
2. Java: load shared librariees with dependencies
3. man ld.so
4. Embedding Python code


Remove redundant imports in Java file in CLI

cat MyClass.java | grep ^import | sort | uniq

Multiple threads and JVM access from C code

In case you are looking for sample code related to JVM and accessing it from C (like this one):

take a look here: recipe № 027.


requested Java version ((null)) not available. Using Java at “” instead

If you are getting following errors while trying to install xlsx package in R (macOS)

> library(xlsx)
Loading required package: rJava
Error : .onLoad failed in loadNamespace() for 'rJava', details:
> library(xlsx)
JavaVM: requested Java version ((null)) not available. Using Java at "" instead.
JavaVM: Failed to load JVM: /bundle/Libraries/libserver.dylib
JavaVM FATAL: Failed to load the jvm library.
Error : .onLoad failed in loadNamespace() for 'xlsx', details:
  call: .jinit()
  error: JNI_GetCreatedJavaVMs returned -1

Error: package or namespace load failed for ‘xlsx’

try following

> sudo R CMD javareconf -n
> R
> install.packages('rJava', type='source')
> install.packages('xlsx')
> library(xlsx)

Enforce LANG settings in R – macOS

defaults write org.R-project.R force.LANG en_US.UTF-8

What if you want to modify Makefile content while building your code

Well, it’s possible. Let’s say we have directory with source code: main.cpp

tree
.
├── Makefile
└── main.cpp

Source code is supper simple.

int main() { return 0; }

What we want to do is to run Makefile and make sure that all occurrences of main will be changed to directory name (e.g. myCode).

All we have to do is to get location of Makefile, update it, store it inside other file and call that other file. In the meantime we can perform some source code manipulation as well.

makefilelocation = $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
makefiletarget = $(makefilelocation).target

BASENAME := $(notdir $(CURDIR))

all: call
        $(warning "I will call myself")

call: copy
        $(MAKE) -f $(makefiletarget) exe

exe: main.cpp
        gcc -o main main.cpp

copy:
        @cp main.cpp $(BASENAME).cpp
        @sed -e "s/main/$(BASENAME)/g" $(makefilelocation) > $(makefiletarget)

clean:
        -rm $(BASENAME).cpp $(BASENAME) $(makefiletarget)

After execution of this Makefile we will have something like this

tree
.
├── Makefile
├── Makefile.target
├── main.cpp
├── myCode
└── myCode.cpp

Change file name to directory name

OK, I am not sure whether anybody really needs something like this. But, in case you want to change your executable name to the same name as your directory name, try this

BASENAME := $(notdir $(CURDIR))

all: copy exe

copy:
        @cp main.cpp $(BASENAME).cpp

exe: copy $(BASENAME).cpp
        gcc -o $(BASENAME).o $(BASENAME).cpp

←Older