If you start your experience with jshell, you will notice that passing command line arguments to your script may be a struggle. Typically, you would expect something like this
> jsell my_script.jsh arg1 'some other arg' yet_another arg
to be working such way, arguments are passed to your script. This is not the case here. The reason for this is, jshell takes a list of files as arguments and parses them for it’s own.
However, you can overcome this issue. And, you can even make it very flexible thanks to Apache ANT. Make sure to get ANT (e.g. 1.10) and put it somewhere. Also, make sure to set ANT_HOME such way it points to your ANT installation.
Then, you can do following inside script
8< -- CUT HERE --- CUT HERE ---- jshell_script_file ---- CUT HERE -- CUT HERE ---
import org.apache.tools.ant.types.Commandline;
{
class A {
public void main(String args[]) {
for(String arg : args) {
System.out.println(arg);
}
}
}
new A().main(Commandline.translateCommandline(System.getProperty("args")));
}
/exit
8< -- CUT HERE --- CUT HERE --- CUT HERE --- CUT HERE --- CUT HERE -- CUT HERE ---
and you can call it like this
# -R will pass arguments for runtime. In this sample we pass -D and it sets system property "args"
# to value 'Some arg with spaces' $SHELL $TERM some_other_arg
#
> jshell --class-path $ANT_HOME/lib/ant.jar \
-R-Dargs="'Some arg with spaces' $SHELL $TERM some_other_arg" \
./jshell_script_file
Some args with spaces
/bin/bash
xterm-256color
some_other_arg
As usual, there are no easy solutions when it comes to R and mac ;)
First of all, I suggest to get clean, isolated copy of OpenMPI so you can be sure that your installation has no issues with mixed libs. To do so, simply compile OpenMPI 3.0.0
# Get OpenMPI sources
mkdir -p ~/opt/src
cd ~/opt/src
curl "https://www.open-mpi.org/software/ompi/v3.0/downloads/openmpi-3.0.0.tar.gz" \
-o openmpi-3.0.0.tar.gz
tar zxf openmpi-3.0.0.tar.gz
# Create location for OpenMPI
mkdir -p ~/opt/openmpi/openmpi-3.0.0
./configure --prefix=$HOME/opt/openmpi/openmpi-3.0.0
make
make install
It’s time to verify that OpenMPI works as expected. Put content (presented below) into hello.c and run it.
/* Put this text inside hello.c file */
#include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv) {
int rank;
int world;
MPI_Init(NULL, NULL);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &world);
printf("Hello: rank %d, world: %d\n",rank, world);
MPI_Finalize();
}
To compile and run it make sure to do following
export PATH=$HOME/opt/openmpi/openmpi-3.0.0/bin:${PATH}
mpicc -o hello ./hello.c
mpirun -np 2 ./hello
If you get output as below – it’s OK. If not – “Huston, we have a problem”.
Hello: rank 0, world: 2
Hello: rank 1, world: 2
Now, it’s time to install Rmpi – unfortunately, on macOS, you need to compile it from sources. Download source package and build it
mkdir -p ~/opt/src/Rmpi
cd ~/opt/src/Rmpi
curl "https://cran.r-project.org/src/contrib/Rmpi_0.6-6.tar.gz" -o Rmpi_0.6-6.tar.gz
R CMD INSTALL Rmpi_0.6-6.tar.gz \
--configure-args="--with-Rmpi-include=$HOME/opt/openmpi/openmpi-3.0.0/include\
--with-Rmpi-libpath=$HOME/opt/openmpi/openmpi-3.0.0/lib\
--with-Rmpi-type=OPENMPI"
As soon as it is ready, you can try whether everything works fine. Try to run it outside R. Just to make sure everything was compiled and works as expected:
mkdir -p ~/tmp/Rmpi_test
cp -r /Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rmpi ~/tmp/Rmpi_test
cd ~/tmp/Rmpi_test/Rmpi
mpirun -np 2 ./Rslaves.sh \
`pwd`/slavedaemon.R \
tmp needlog \
/Library/Frameworks/R.framework/Versions/3.4/Resources/
# If it works, that's fine. Nothing will happen in fact, it will simply run.
# Now, you may be tempted to run more instances (you will probably get error)
mpirun -np 4 ./Rslaves.sh \
`pwd`/slavedaemon.R \
tmp needlog \
/Library/Frameworks/R.framework/Versions/3.4/Resources/
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 4 slots
that were requested by the application:
./Rslaves.sh
Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
# You can increase number of slots by putting
# localhost slots=25
# inside ~/default_hostfile and running mpirun following way
mpirun --hostfile=~/default_hostfile -np 4 \
./Rslaves.sh \
`pwd`/slavedaemon.R \
tmp \
needlog \
/Library/Frameworks/R.framework/Versions/3.4/Resources/
Now, we can try to run everything inside R
R
...
...
> library(Rmpi)
> mpi.spawn.Rslaves()
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 4 slots
that were requested by the application:
/Library/Frameworks/R.framework/Versions/3.4/Resources/library/Rmpi/Rslaves.sh
Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
Error in mpi.comm.spawn(slave = system.file("Rslaves.sh", package = "Rmpi"), :
MPI_ERR_SPAWN: could not spawn processes
>
Ups. The issue here is that Rmpi runs MPI code via MPI APIs and it doesn’t call mpirun. So, we can’t pass hostfile directly. However, there is hope. Hostfile is one of ORTE parameters (take a look here for more info: here and here).
This way, we can put location of this file here: ~/.openmpi/mca-params.conf. Just do following:
mkdir -p ~/.openmpi/
echo "orte_default_hostfile=$HOME/default_host" >> ~/.openmpi/mca-params.conf
Now, we can try to run R once more:
R
...
...
> library(Rmpi)
> mpi.spawn.Rslaves()
4 slaves are spawned successfully. 0 failed.
master (rank 0, comm 1) of size 5 is running on: pi
slave1 (rank 1, comm 1) of size 5 is running on: pi
slave2 (rank 2, comm 1) of size 5 is running on: pi
slave3 (rank 3, comm 1) of size 5 is running on: pi
slave4 (rank 4, comm 1) of size 5 is running on: pi
This time, it worked ;) Have fun with R!
In case you are not aware of the macros ;) I am sure you are aware, but, just in case :)
int f(int a, int b) {
#ifdef PROFILER
BeginProfiling("f");
#endif
...
#ifdef PROFILER
EndProfiling();
#endif
}
Then, you can control it by doing this
// when you want to profile
#define PROFILER
// when you don't want to profile
//#define PROFILER
In case you are not aware of function pointers ;)
#include <stdio.h>
void (*doprofiling)(void);
void profile()
{
printf("I am profiling\n");
}
void no_profile()
{
printf("I am not profiling\n");
}
void fun() {
doprofiling();
}
int main()
{
doprofiling = profile;
fun();
doprofiling = no_profile;
fun();
return 0;
}
Then, you can switch in the code dynamically
gcc -o profile ./profile.c
./profile
I am profiling
I am not profiling
Or, you can use something like this, and you can apply different decorators to different functions
#include <stdio.h>
void doprofiling(void (*proffun)(void)) {
proffun();
}
void profile()
{
printf("I am profiling\n");
}
void no_profile()
{
printf("I am not profiling\n");
}
void fun_prof() {
void (*decorator)(void) = profile;
doprofiling(decorator);
}
void fun_no_prof() {
void (*decorator)(void) = no_profile;
doprofiling(decorator);
}
int main()
{
fun_prof();
fun_no_prof();
return 0;
}
And, you can still dynamically apply it in the code.
> gcc -o ./profile ./profile.c
> ./profile
I am profiling
I am not profiling
# SLATEC Common Mathematical Library, Version 4.1, July 1993
# a comprehensive software library containing over
# 1400 general purpose mathematical and statistical routines
# written in Fortran 77.
If you want to install SLATEC, you need to make sure to install gfortran.
Take a look here for a brief instructions, somewhere in the middle of shell code: http://www.owsiak.org/?p=3492
Make sure to download sources and linux makefile and put all files at the same level. By saying at the same level, I mean that all *.f files from slatec_src.tgz and makefile and dynamic and static from slatec4linux.tgz are in the same dir.
SLATEC sources: http://www.netlib.org/slatec/slatec_src.tgz
SLATEC makefile: http://www.netlib.org/slatec/slatec4linux.tgz
Before building library, make sure to export FC variable (it is needed by makefile)
export FC=gfortran
Make sure to change this line inside dynamic/makefile
libslatec.so: $(OBJ)
$(CC) -shared -o $@ $(OBJ)
to
libslatec.so: $(OBJ)
$(FC) -shared -o $@ $(OBJ)
Call make
make
Wait a little bit. Take a look inside static and dynamic, files should be there.
find . -name "libslatec*"
./dynamic/libslatec.so
./static/libslatec.a
Now, you can try to perform make install (pay attention here as it will overwrite hardcoded locations). Alternatively, you can use
-L${WHERE_YOUR_BUILD_WAS_DONE}/dynamic -lslatec
SLATEC refers to symbols that can be found inside LAPACK package. If you don’t have it installed, take a look here
mkdir lapack
cd lapack
curl "http://www.netlib.org/lapack/lapack-3.7.1.tgz" -o lapack-3.7.1.tgz
tar zxf lapack-3.7.1.tgz
cd lapack-3.7.1
ln -s make.inc.example make.inc
make
After compilation is done, you can find liblapack.a inside lapack-3.7.1.
1. Install Ruby
I prefer to install from sources: https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.1.tar.gz
> ./configure --prefix=$HOME/opt/ruby
> make
> make install
2. Install RubyGems
I prefer to install from sources: https://rubygems.org/pages/download
> export PATH=$HOME/opt/ruby/bin:$PATH
> ruby setup.rb
3. Change repo location
I compile Ruby without ssh support. I have to change the ruby gems repo
> gem sources -r https://rubygems.org/
> gem sources -a http://rubygems.org/
(if you want to build with OpenSSL, take a look here: http://www.owsiak.org/?p=3262 There is a sample related to building OpenSSL at macOS)
4. You can install Jekyll
> gem install jekyll
> jekyll --version
jekyll 3.5.2
That’s it.
Source: Originally, posted here: https://stackoverflow.com/questions/46019526/
Different solutions for working with github repos when you have multiple accounts
If you have two user accounts at github: {user_a} and {user_b}. Sometimes, you may encounter issues while pushing changes back
remote: Permission to {user}/{repo}.git denied to {user_a}.
fatal: unable to access 'https://github.com/{user}/{repo}.git/': The requested URL returned error: 403
If you are using https, you can play with .git/config
> git clone https://github.com/{user}/{repo}.git
> vi .git/config
_-= Inside VIM =-_
# you may enforce user here
[remote "origin"]
url = https://{user_a}@github.com/{user}/{repo}.git
If you are using ssh, you can play with .ssh/config
Host {user_a}@github.com
IdentityFile ~/.ssh/id_rsa_a
Host {user_b}@github.com
IdentityFile ~/.ssh/id_rsa_b
Host github.com-{user_a}
HostName github.com
User git
IdentityFile ~/.ssh/id_rsa_a
Host github.com-{user_b}
HostName github.com
User git
IdentityFile ~/.ssh/id_rsa_b
Host github.com-{user}
HostName github.com
User git
IdentityFile ~/.ssh/id_rsa_a
Host bitbucket.org
User git
IdentityFile ~/.ssh/id_rsa_c
> svn diff --diff-cmd='meld' file_you_have_changed.c
> svn commit -m"We tend to forget\
that multiline comments, in SVN, are supper easy.\
And, they can help to introduce multiple changes\
inside one commit. Like:\
- improvements,\
- bug fixes,\
- motivations behind the code." my_super_file.c
+----------------------+-----------------+-----------------+
| parameter | parameter | parameter |
| Set and Not Null | Set But Null | Unset |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter:-word} | substitute parameter | substitute word | substitute word |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter-word} | substitute parameter | substitute null | substitute word |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter:=word} | substitute parameter | assign word | assign word |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter=word} | substitute parameter | substitute null | assign word |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter:?word} | substitute parameter | error, exit | error, exit |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter?word} | substitute parameter | substitute null | error, exit |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter:+word} | substitute word | substitute null | substitute null |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter+word} | substitute word | substitute word | substitute null |
+--------------------+----------------------+-----------------+-----------------+
source: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02
So, you want to have rJava (e.g. rJava_0.9-8.tar.gz) inside your fresh, new installation of R 3.4 and you are running macOS. There are bad news and good news ;)
Bad news. It will fail with default clang that comes with XCode. You need something better here, something with support for OpenMP. And you can get it following way
# make sure to create some place where you want to have it (I, personally, put stuff into ~/opt)
> mkdir ~/opt
> cd ~/opt
> curl http://releases.llvm.org/4.0.1/clang+llvm-4.0.1-x86_64-apple-darwin.tar.xz \
-o clang+llvm-4.0.1-x86_64-apple-darwin.tar.xz
> tar xf clang+llvm-4.0.1-x86_64-apple-darwin.tar.xz
Now, make sure to install gfortran. I am using version from this location: GFortran.
Once you have it, make sure to install most recent JDK from Oracle. You can find it here: JavaSE.
After you have your Java installed, make sure to do following (double check that everything is OK so far)
> R --version
R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin15.6.0 (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under the terms of the
GNU General Public License versions 2 or 3.
For more information about these matters see
http://www.gnu.org/licenses/.
> /usr/libexec/java_home -V
Matching Java Virtual Machines (3):
1.8.0_144, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
1.8.0_111, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home
1.7.0_80, x86_64: "Java SE 7" /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home
/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
# Make sure to put following line inside you ~/.profile
# export JAVA_HOME=$(/usr/libexec/java_home -v 1.8.0_144)
# close and re-open Terminal session. Now, you should be able to see:
> echo $JAVA_HOME
/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
/usr/libexec/java_home
/Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home
Make sure to enable your JDK for JNI
> sudo vi /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Info.plist
# make sure to replace
#<array>
# <string>CommandLine</string>
#</array>
# with
#<array>
# <string>CommandLine</string>
# <string>JNI</string>
#</array>
Now, it’s time to configure R. Make sure to run following command.
!! (note that we change JAVA_HOME to JRE) !!
> JAVA_HOME=${JAVA_HOME}/jre
> sudo R CMD javareconf \
JAVA_HOME=${JAVA_HOME} \
JAVA=${JAVA_HOME}/../bin/java \
JAVAC=${JAVA_HOME}/../bin/javac \
JAVAH=${JAVA_HOME}/../bin/javah \
JAR=${JAVA_HOME}/../bin/jar \
JAVA_LIBS="-L${JAVA_HOME}/lib/server -ljvm" \
JAVA_CPPFLAGS="-I${JAVA_HOME}/../include -I${JAVA_HOME}/../include/darwin"
Note! It looks like R CMD javareconf doesn’t update all the flags, make sure that file: /Library/Frameworks/R.framework/Versions/3.4/Resources/etc/Makeconf contains following entries:
JAVA_LIBS="-L${JAVA_HOME}/lib/server -ljvm" \
JAVA_CPPFLAGS="-I${JAVA_HOME}/../include -I${JAVA_HOME}/../include/darwin"
You should have your Java and R linked together. It’s time to add support for clang that you have just installed few steps above.
Let’s say your clang is in your home dir. Here: /Users/user_name/opt/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/
Make sure, to add following file (~/.R/Makevars – more info can be found here: R for Mac OS X FAQ and here: R Installation and Administration)
CLANG=/Users/user_name/opt/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0
CC=$(CLANG)/bin/clang
CXX=$(CLANG)/bin/clang++
CXX1X=$(CLANG)/bin/clang++
CXXFLAGS=-I$(CLANG)/include
Also, make sure to modify file:
/Library/Frameworks/R.framework/Versions/3.4/Resources/etc/Makeconf
inside this file, make sure that line “LDFLAGS” reads
LDFLAGS = -L/usr/local/lib -L/Users/user_name/opt/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/lib -lomp
You can confirm that it works by calling:
# you can confirm that by calling
> R CMD config --ldflags
-fopenmp -L/usr/local/lib
-L/Users/user_name/opt/clang+llvm-4.0.1-x86_64-apple-macosx10.9.0/lib
-lomp -F/Library/Frameworks/R.framework/.. -framework R
-lpcre -llzma -lbz2 -lz -licucore -lm -liconv
Now, it’s time to get rJava. Simply download it, and install it:
> curl "https://cran.r-project.org/src/contrib/rJava_0.9-8.tar.gz" -o rJava_0.9-8.tar.gz
> R CMD INSTALL rJava_0.9-8.tar.gz
* installing to library ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library’
* installing *source* package ‘rJava’ ...
...
...
*** installing help indices
** building package indices
** testing if installed package can be loaded
* DONE (rJava)
And you can now test it. Let’s say you have (in your working directory) following layout
utils/
└── RUsingStringArray.java
and file RUsingStringArray.java contains
package utils;
public class RUsingStringArray {
public String [] createArray() {
System.out.println("Creating empty array");
return new String[0];
}
}
You can compile it and run it in R
> javac utils/*.java
> export CLASSPATH=`pwd`
> export JAVA_HOME=$(/usr/libexec/java_home -v 1.8.0_144)/jre
> R
> library(rJava)
> .jinit()
> obj <- .jnew("utils.RUsingStringArray")
> s <- .jcall(obj, returnSig="[Ljava/lang/String;", method="createArray")
> .jcall(obj, returnSig="I", method = "arrayLen", s)
Class: class [Ljava.lang.String;
[1] 0
>
So, what do you think about playing retro style, terminal based, game where you can gain some real life experience as well. If you are interested, take a look below (click the image for the full experience – or click here for full screen size).
I find Cathode to be supper fancy terminal emulator with element of surprise. Whenever you want to gain some attention during presentations it comes handy. There is nothing as good as “Back to the Future” effect when you want to gain some attention from the audience. Especially, when your terminal session itself is quite boring. On regular basis, I definitely prefer iTerm2. But for the looks, Cathode plays it’s role.
As for the resources.
Music: https://youtu.be/hTZ-zQInMsk
Cathode: http://www.secretgeometry.com/apps/cathode/
Type speed instructions for macOS: Building GNU Typist for Mac OS
Sounds: AtomSplitter
There was this question at Stack Overflow. How to share enums. And you want to share them between C++ and Java. Well, there is one issue here. You can’t base on comments (to trick compiler) as Java and C++ have the same comment schema. How to use comments to make cross language coding? Take a look here.
You can use a nasty hack with “disabling” public keyword in C++ using macro.
#include <stdio.h>
#define public
#include "A.java"
;
#undef public
int main() {
A val = a;
if(val == a) {
printf("OK\n");
} else {
printf("Not OK\n");
}
}
Then, inside Java class you can define enum
public
enum A {
a,
b
}
And you can use it in Java later on. As you can see above, it is used in C++ as well.
public class B {
public static void main(String [] arg) {
A val = A.a;
if(val == A.a) {
System.out.println("OK");
} else {
System.out.println("Not OK");
}
}
}
Seems to work fine :)
> javac *.java
> java B
OK
> g++ -o main ./main.cc
> ./main
OK
At some point Apple dropped GNU based toolchain in favor of LLVM. However, sometimes you need these tools. One of such examples is debugging Fortran codes. lldb is not able to do it properly, yet. You can trace the execution of code, you can step over, but you can’t examine variables.
In that case, you need GDB. You can install it following way.
First, get the GDB sources
mkdir src
cd src
curl "http://ftp.gnu.org/gnu/gdb/gdb-8.0.tar.gz" -o gdb-8.0.tar.gz
tar zxf gdb-8.0.tar.gz
cd gdb-8.0
./configure --prefix=$HOME/opt/usr/local
make
make install
This way, you will install GDB inside $HOME/opt/usr/local. However, installing GDB is not enough. You need to Code Sign it in order to make it possible to debug codes using GDB. To make it possible, do following. If you don’t have Code Signed GDB, it will end with the error.
You can find detailed description of how to Code Sign GDB here.
After you are done, you can test your GDB installation and run the code inside debugger. Lets say we have following code:
! program.f90
program prog
integer variable
variable = 1
write(*,*) variable
end
After compilation with “-g” flag
gfortran -g -o ./program ./program.f90
we can simply run it inside gdb
# Make sure to export location of gdb
export PATH=${PATH}:$HOME/opt/usr/local/bin/
gdb ./program
GNU gdb (GDB) 8.0
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin16.5.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./program...Reading symbols from
...
done.
(gdb) list
1 program prog
2 integer variable
3 variable = 1
4 write(*,*) variable
5 end
(gdb) break 4
Breakpoint 1 at 0x100000e2d: file ./program.f90, line 4.
(gdb) run
Starting program: ./program
[New Thread 0x1403 of process 1234]
warning: unhandled dyld version (15)
Thread 2 hit Breakpoint 1, prog () at ./program.f90:4
4 write(*,*) variable
(gdb) print variable
$1 = 1
(gdb)
For updated info (for R3.4) take a look HERE.
R, rJava, Java and macOS is a mess. In fact, Java itself is a mess when it comes to macOS. First of all, you can have different Java installations: Oracle based or system based.
# this is the place where Java Framework resides
/System/Library/Frameworks/JavaVM.framework
# this is the place where Oracle based Java will be installed
/Library/Java/JavaVirtualMachines
Apple’s layout is quite different comparing to what you expect from typical JDK installation.
tree -d
.
├── Frameworks -> Versions/Current/Frameworks
├── Resources -> Versions/Current/Resources
└── Versions
├── A
│ ├── Commands
│ ├── Frameworks
│ │ ├── JavaNativeFoundation.framework
│ │ │ ├── Resources -> Versions/Current/Resources
│ │ │ └── Versions
│ │ │ ├── A
│ │ │ │ ├── Resources
│ │ │ │ │ └── English.lproj
│ │ │ │ └── _CodeSignature
│ │ │ └── Current -> A
│ │ └── JavaRuntimeSupport.framework
│ │ ├── Resources -> Versions/Current/Resources
│ │ └── Versions
│ │ ├── A
│ │ │ ├── Resources
│ │ │ │ └── English.lproj
│ │ │ └── _CodeSignature
│ │ └── Current -> A
│ ├── JavaPluginCocoa.bundle
│ │ └── Contents
│ │ ├── MacOS
│ │ ├── Resources
...
Anyway, if you want to configure R such way it works properly with JVM, you have to make some tweaking. First of all, you have to play with JAVA_HOME. In macOS you have this special command: java_home
> /usr/libexec/java_home -V
Matching Java Virtual Machines (2):
1.8.0_11, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
1.8.0_05, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home
/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
# if you run it without any switches it will show currently used Java
/usr/libexec/java_home
/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
so far, so good. If you tree this one, you will see that structure is quite similar to other systems (e.g. Linux, Windows).
tree -d `/usr/libexec/java_home`
/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
├── bin
├── db
│ ├── bin
│ └── lib
├── include
│ └── darwin
├── jre
│ ├── bin
│ └── lib
│ ├── applet
│ ├── cmm
│ ├── ext
│ ├── fonts
│ ├── images
│ │ └── cursors
│ ├── jfr
│ ├── jli
│ ├── management
│ ├── security
│ └── server
...
However, if you try to use JDK with R, there is a small surprise behind the corner. To get Java running in R, you call javareconf as follows
sudo R CMD javareconf
But, there is a small but. javareconf assumes that whenever you JAVA_HOME matches “/Home$” using grep, it points to system based Java installation – instead of Oracle’s one. This makes things weird. Half of your settings point to JDK and other half to JVM installed by Apple. Things can go wrong, and will.
To overcome this issue, you need to create fake Java home directory – without “/Home” in the name.
cd /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents
ln -s Home RHome
then, you need to export it as JAVA_HOME
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/RHome
and run javareconf with settings allowing to compile the code
sudo R CMD javareconf \
JAVA_HOME=${JAVA_HOME} \
JAVA=${JAVA_HOME}/bin/java \
JAVAC=${JAVA_HOME}/bin/javac \
JAVAH=${JAVA_HOME}/bin/javah \
JAR=${JAVA_HOME}/bin/jar \
JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/server \
JAVA_CPPFLAGS="-I${JAVA_HOME}/include -I${JAVA_HOME}/include/darwin"
Now, you are able to test whether rJava compiles fine. Download rJava package and try to configure it manually (outside R) from sources.
rJava can be downloaded here: https://cran.r-project.org/web/packages/rJava/index.html
curl https://cran.r-project.org/src/contrib/rJava_0.9-8.tar.gz -o rJava_0.9-8.tar.gz
tar zxf rJava_0.9-8.tar.gz
cd rJava
./configure
if it passes, start R and install package from sources
> install.packages("rJava", type="source")
Note that with messy Java installations (Apple’s JREs, Oracle’s JDKs) you might need to play with setting up Java env properly in a first place. It might be needed to update file
`/usr/libexec/java_home`/../Info.plist
if you want to edit the file, make sure to use some editor you prefer (e.g. VI)
sudo vi `/usr/libexec/java_home`/../Info.plist
and make sure your JVM can handle JNI. You need to change:
<key>JVMCapabilities</key>
<array>
<string>CommandLine</string>
</array>
to
<key>JVMCapabilities</key>
<array>
<string>CommandLine</string>
<string>JNI</string>
</array>
Note!
rJava requires XQuartz for compilation, make sure to install it as well: https://www.xquartz.org
Take a look here, for more details: recipe № 029
Recently, I had this issue with JNI code calling Python script using shared libraries provided by Python itself.
The schema is quite easy to follow, and you can easily apply it inside C based code.
// Initialize Python (can be done once - for a whole life of JVM)
Py_Initialize();
// Remember to reserve PyGILState
// It is not thread safe
gstate = PyGILState_Ensure();
// c_str will contain content of java.lang.String passed
// from Java
const char *c_str;
PyRun_SimpleString( c_str );
// once finished remember to release GIL
PyGILState_Release(gstate);
Py_Finalize();
You can find fully working sample (Linux/macOS) here – python via JNA. In one of our projects we had to go via JNI for some post processing computations done by Python, otherwise, we had huge performance drawbacks. In fact, we have noticed ~100 times faster executions of the code while using Python with JNI together with some additional caching – reference. Note that we were running code using 256 CPUs, where both Java and Python were running on all CPUs. As for passing java.lang.String from Java to C, in case you are not familiar with it, take a look here – passing java.lang.String value from Java to C.
Usually, this approach works perfectly fine. Unless something is not quite OK. Last week I got really strange error coming from very simple Python script with numpy installed. And the error was:
…
File “../python2.7/site-packages/numpy/core/__init__.py”, line 24, in
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you’re working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
which is not quite promising. It looks like something is really broken. And it looks really serious. Now, let’s check what exactly is happening there. Let’s take a look at __init__.pyt inside core.
try:
from . import multiarray
except ImportError:
msg = """
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
"""
raise ImportError(msg)
What we can see here is that multiarray is one to blame. We need to find out what multiarray is.
> find . -name "multiarray*"
./numpy/core/multiarray_tests.so
./numpy/core/multiarray.so
In fact, it’s a shared library that is loaded from Python. We need to know why does it fail inside JVM. Do you remember the execution schema?
JVM -> SomeJavaClass -> JNI -> Python library (libpython2.7) -> numpy
Let’s see what happens if we try to load this library inside JVM. We can use supper simple code for that
public class TestMulti {
static {
// System.loadLibrary takes name of library as argument
// System.load takes full path of file
System.load("../numpy/core/multiarray.so");
}
public static void main(String[] args) {}
}
And now you are talking! It looks like multiarray.so is not able to find some symbols.
> java -Djava.library.path=... -cp . TestMulti
Exception in thread "main" java.lang.UnsatisfiedLinkError: .../multiarray.so: undefined symbol: _Py_ZeroStruct
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at python.PythonJustLoad.(PythonJustLoad.java:14)
Symbol _Py_ZeroStruct comes from libpython2.7. You can check it quite easily
> nm .../libpython2.7.so | grep _Py_Zero
00000000003c6cc0 D _Py_ZeroStruct
It looks like location of libpython2.7 is specified on LD_LIBRARY_PATH.
> env | grep LD_LIBRARY_PATH | tr ':' '\n' | grep python
.../python/2.7
Digging through StackOverflow lead me to people who had similar issues: Java: load shared librariees with dependencies. And I know that JNA solves the issue. However, I prefered to stick to JNI. So, we need to make sure that symbols from libpython2.7 are loaded properly while still using JNI. We have two options here
> export LD_PRELOAD=.../libpython2.7.so
This approach will enforce loading library priori to any other library loaded by process
> man ld.so
...
...
LD_PRELOAD
A list of additional, user-specified, ELF shared libraries to be loaded before all others.
The items of the list can be separated by spaces or colons. This can be used to
selectively override functions in other shared libraries. The libraries are searched for
using the rules given under DESCRIPTION. For set-user-ID/set-group-ID ELF binaries, preload
pathnames containing slashes are ignored, and libraries in the standard search directories
are loaded only if the set-user-ID permission bit is enabled on the library file.
...
...
However, this is not quite what we want. This way, we will load Python library all over the place. Each and every process will load it. Not exactly what we are looking for. What we can do, instead, is to enforce loading of Python library directly within JNI code
void loadLibrary() {
if(handle == NULL) {
handle = dlopen("libpython2.7.so", RTLD_LAZY | RTLD_GLOBAL);
if (!handle) {
fprintf(stderr, "%s\n", dlerror());
exit(EXIT_FAILURE);
}
}
}
...
...
loadLibrary();
// Initialize Python (can be done once for whole live of JVM)
Py_Initialize();
// Remember to reserve PyGILState
// It is not thread safe
gstate = PyGILState_Ensure();
// c_str will contain content of java.lang.String passed
// from Java
const char *c_str;
PyRun_SimpleString( c_str );
// once finished remember to release GIL
PyGILState_Release(gstate);
Py_Finalize();
Two things here. First, we enforce RTLD_GLOBAL and make symbols from libpython2.7 visible for multiarray.so (note that this is not default behavior of JNI – JVM loads libs using dlopen(filename, RTLD_LAZY). Second, we are inside JVM itself and we enforce loading of library only in case the code is actually executed. And that’s what solves the issue.
Have fun with JNI!
References:
1. OpenJDK – loading libraries
2. Java: load shared librariees with dependencies
3. man ld.so
4. Embedding Python code
cat MyClass.java | grep ^import | sort | uniq
In case you are looking for sample code related to JVM and accessing it from C (like this one):
take a look here: recipe № 027.
If you are getting following errors while trying to install xlsx package in R (macOS)
> library(xlsx)
Loading required package: rJava
Error : .onLoad failed in loadNamespace() for 'rJava', details:
> library(xlsx)
JavaVM: requested Java version ((null)) not available. Using Java at "" instead.
JavaVM: Failed to load JVM: /bundle/Libraries/libserver.dylib
JavaVM FATAL: Failed to load the jvm library.
Error : .onLoad failed in loadNamespace() for 'xlsx', details:
call: .jinit()
error: JNI_GetCreatedJavaVMs returned -1
Error: package or namespace load failed for ‘xlsx’
try following
> sudo R CMD javareconf -n
> R
> install.packages('rJava', type='source')
> install.packages('xlsx')
> library(xlsx)
defaults write org.R-project.R force.LANG en_US.UTF-8