R, rJava, Java and macOS is a mess. In fact, Java itself is a mess when it comes to macOS. First of all, you can have different Java installations: Oracle based or system based.
# this is the place where Java Framework resides
/System/Library/Frameworks/JavaVM.framework
# this is the place where Oracle based Java will be installed
/Library/Java/JavaVirtualMachines
Apple’s layout is quite different comparing to what you expect from typical JDK installation.
tree -d
.
├── Frameworks -> Versions/Current/Frameworks
├── Resources -> Versions/Current/Resources
└── Versions
├── A
│ ├── Commands
│ ├── Frameworks
│ │ ├── JavaNativeFoundation.framework
│ │ │ ├── Resources -> Versions/Current/Resources
│ │ │ └── Versions
│ │ │ ├── A
│ │ │ │ ├── Resources
│ │ │ │ │ └── English.lproj
│ │ │ │ └── _CodeSignature
│ │ │ └── Current -> A
│ │ └── JavaRuntimeSupport.framework
│ │ ├── Resources -> Versions/Current/Resources
│ │ └── Versions
│ │ ├── A
│ │ │ ├── Resources
│ │ │ │ └── English.lproj
│ │ │ └── _CodeSignature
│ │ └── Current -> A
│ ├── JavaPluginCocoa.bundle
│ │ └── Contents
│ │ ├── MacOS
│ │ ├── Resources
...
Anyway, if you want to configure R such way it works properly with JVM, you have to make some tweaking. First of all, you have to play with JAVA_HOME. In macOS you have this special command: java_home
> /usr/libexec/java_home -V
Matching Java Virtual Machines (2):
1.8.0_11, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
1.8.0_05, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home
/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
# if you run it without any switches it will show currently used Java
/usr/libexec/java_home
/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
so far, so good. If you tree this one, you will see that structure is quite similar to other systems (e.g. Linux, Windows).
tree -d `/usr/libexec/java_home`
/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home
├── bin
├── db
│ ├── bin
│ └── lib
├── include
│ └── darwin
├── jre
│ ├── bin
│ └── lib
│ ├── applet
│ ├── cmm
│ ├── ext
│ ├── fonts
│ ├── images
│ │ └── cursors
│ ├── jfr
│ ├── jli
│ ├── management
│ ├── security
│ └── server
...
However, if you try to use JDK with R, there is a small surprise behind the corner. To get Java running in R, you call javareconf as follows
sudo R CMD javareconf
But, there is a small but. javareconf assumes that whenever you JAVA_HOME matches “/Home$” using grep, it points to system based Java installation – instead of Oracle’s one. This makes things weird. Half of your settings point to JDK and other half to JVM installed by Apple. Things can go wrong, and will.
To overcome this issue, you need to create fake Java home directory – without “/Home” in the name.
cd /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents
ln -s Home RHome
then, you need to export it as JAVA_HOME
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/RHome
and run javareconf with settings allowing to compile the code
sudo R CMD javareconf \
JAVA_HOME=${JAVA_HOME} \
JAVA=${JAVA_HOME}/bin/java \
JAVAC=${JAVA_HOME}/bin/javac \
JAVAH=${JAVA_HOME}/bin/javah \
JAR=${JAVA_HOME}/bin/jar \
JAVA_LD_LIBRARY_PATH=${JAVA_HOME}/jre/lib/server \
JAVA_CPPFLAGS="-I${JAVA_HOME}/include -I${JAVA_HOME}/include/darwin"
Now, you are able to test whether rJava compiles fine. Download rJava package and try to configure it manually (outside R) from sources.
rJava can be downloaded here: https://cran.r-project.org/web/packages/rJava/index.html
curl https://cran.r-project.org/src/contrib/rJava_0.9-8.tar.gz -o rJava_0.9-8.tar.gz
tar zxf rJava_0.9-8.tar.gz
cd rJava
./configure
if it passes, start R and install package from sources
> install.packages("rJava", type="source")
Note that with messy Java installations (Apple’s JREs, Oracle’s JDKs) you might need to play with setting up Java env properly in a first place. It might be needed to update file
`/usr/libexec/java_home`/../Info.plist
if you want to edit the file, make sure to use some editor you prefer (e.g. VI)
sudo vi `/usr/libexec/java_home`/../Info.plist
and make sure your JVM can handle JNI. You need to change:
<key>JVMCapabilities</key>
<array>
<string>CommandLine</string>
</array>
to
<key>JVMCapabilities</key>
<array>
<string>CommandLine</string>
<string>JNI</string>
</array>
Note!
rJava requires XQuartz for compilation, make sure to install it as well: https://www.xquartz.org
Take a look here, for more details: recipe № 029
Recently, I had this issue with JNI code calling Python script using shared libraries provided by Python itself.
The schema is quite easy to follow, and you can easily apply it inside C based code.
// Initialize Python (can be done once - for a whole life of JVM)
Py_Initialize();
// Remember to reserve PyGILState
// It is not thread safe
gstate = PyGILState_Ensure();
// c_str will contain content of java.lang.String passed
// from Java
const char *c_str;
PyRun_SimpleString( c_str );
// once finished remember to release GIL
PyGILState_Release(gstate);
Py_Finalize();
You can find fully working sample (Linux/macOS) here – python via JNA. In one of our projects we had to go via JNI for some post processing computations done by Python, otherwise, we had huge performance drawbacks. In fact, we have noticed ~100 times faster executions of the code while using Python with JNI together with some additional caching – reference. Note that we were running code using 256 CPUs, where both Java and Python were running on all CPUs. As for passing java.lang.String from Java to C, in case you are not familiar with it, take a look here – passing java.lang.String value from Java to C.
Usually, this approach works perfectly fine. Unless something is not quite OK. Last week I got really strange error coming from very simple Python script with numpy installed. And the error was:
…
File “../python2.7/site-packages/numpy/core/__init__.py”, line 24, in
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you’re working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
which is not quite promising. It looks like something is really broken. And it looks really serious. Now, let’s check what exactly is happening there. Let’s take a look at __init__.pyt inside core.
try:
from . import multiarray
except ImportError:
msg = """
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control). Otherwise reinstall numpy.
"""
raise ImportError(msg)
What we can see here is that multiarray is one to blame. We need to find out what multiarray is.
> find . -name "multiarray*"
./numpy/core/multiarray_tests.so
./numpy/core/multiarray.so
In fact, it’s a shared library that is loaded from Python. We need to know why does it fail inside JVM. Do you remember the execution schema?
JVM -> SomeJavaClass -> JNI -> Python library (libpython2.7) -> numpy
Let’s see what happens if we try to load this library inside JVM. We can use supper simple code for that
public class TestMulti {
static {
// System.loadLibrary takes name of library as argument
// System.load takes full path of file
System.load("../numpy/core/multiarray.so");
}
public static void main(String[] args) {}
}
And now you are talking! It looks like multiarray.so is not able to find some symbols.
> java -Djava.library.path=... -cp . TestMulti
Exception in thread "main" java.lang.UnsatisfiedLinkError: .../multiarray.so: undefined symbol: _Py_ZeroStruct
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at python.PythonJustLoad.(PythonJustLoad.java:14)
Symbol _Py_ZeroStruct comes from libpython2.7. You can check it quite easily
> nm .../libpython2.7.so | grep _Py_Zero
00000000003c6cc0 D _Py_ZeroStruct
It looks like location of libpython2.7 is specified on LD_LIBRARY_PATH.
> env | grep LD_LIBRARY_PATH | tr ':' '\n' | grep python
.../python/2.7
Digging through StackOverflow lead me to people who had similar issues: Java: load shared librariees with dependencies. And I know that JNA solves the issue. However, I prefered to stick to JNI. So, we need to make sure that symbols from libpython2.7 are loaded properly while still using JNI. We have two options here
> export LD_PRELOAD=.../libpython2.7.so
This approach will enforce loading library priori to any other library loaded by process
> man ld.so
...
...
LD_PRELOAD
A list of additional, user-specified, ELF shared libraries to be loaded before all others.
The items of the list can be separated by spaces or colons. This can be used to
selectively override functions in other shared libraries. The libraries are searched for
using the rules given under DESCRIPTION. For set-user-ID/set-group-ID ELF binaries, preload
pathnames containing slashes are ignored, and libraries in the standard search directories
are loaded only if the set-user-ID permission bit is enabled on the library file.
...
...
However, this is not quite what we want. This way, we will load Python library all over the place. Each and every process will load it. Not exactly what we are looking for. What we can do, instead, is to enforce loading of Python library directly within JNI code
void loadLibrary() {
if(handle == NULL) {
handle = dlopen("libpython2.7.so", RTLD_LAZY | RTLD_GLOBAL);
if (!handle) {
fprintf(stderr, "%s\n", dlerror());
exit(EXIT_FAILURE);
}
}
}
...
...
loadLibrary();
// Initialize Python (can be done once for whole live of JVM)
Py_Initialize();
// Remember to reserve PyGILState
// It is not thread safe
gstate = PyGILState_Ensure();
// c_str will contain content of java.lang.String passed
// from Java
const char *c_str;
PyRun_SimpleString( c_str );
// once finished remember to release GIL
PyGILState_Release(gstate);
Py_Finalize();
Two things here. First, we enforce RTLD_GLOBAL and make symbols from libpython2.7 visible for multiarray.so (note that this is not default behavior of JNI – JVM loads libs using dlopen(filename, RTLD_LAZY). Second, we are inside JVM itself and we enforce loading of library only in case the code is actually executed. And that’s what solves the issue.
Have fun with JNI!
References:
1. OpenJDK – loading libraries
2. Java: load shared librariees with dependencies
3. man ld.so
4. Embedding Python code
cat MyClass.java | grep ^import | sort | uniq
In case you are looking for sample code related to JVM and accessing it from C (like this one):
take a look here: recipe № 027.
If you are getting following errors while trying to install xlsx package in R (macOS)
> library(xlsx)
Loading required package: rJava
Error : .onLoad failed in loadNamespace() for 'rJava', details:
> library(xlsx)
JavaVM: requested Java version ((null)) not available. Using Java at "" instead.
JavaVM: Failed to load JVM: /bundle/Libraries/libserver.dylib
JavaVM FATAL: Failed to load the jvm library.
Error : .onLoad failed in loadNamespace() for 'xlsx', details:
call: .jinit()
error: JNI_GetCreatedJavaVMs returned -1
Error: package or namespace load failed for ‘xlsx’
try following
> sudo R CMD javareconf -n
> R
> install.packages('rJava', type='source')
> install.packages('xlsx')
> library(xlsx)
defaults write org.R-project.R force.LANG en_US.UTF-8
Well, it’s possible. Let’s say we have directory with source code: main.cpp
tree
.
├── Makefile
└── main.cpp
Source code is supper simple.
int main() { return 0; }
What we want to do is to run Makefile and make sure that all occurrences of main will be changed to directory name (e.g. myCode).
All we have to do is to get location of Makefile, update it, store it inside other file and call that other file. In the meantime we can perform some source code manipulation as well.
makefilelocation = $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
makefiletarget = $(makefilelocation).target
BASENAME := $(notdir $(CURDIR))
all: call
$(warning "I will call myself")
call: copy
$(MAKE) -f $(makefiletarget) exe
exe: main.cpp
gcc -o main main.cpp
copy:
@cp main.cpp $(BASENAME).cpp
@sed -e "s/main/$(BASENAME)/g" $(makefilelocation) > $(makefiletarget)
clean:
-rm $(BASENAME).cpp $(BASENAME) $(makefiletarget)
After execution of this Makefile we will have something like this
tree
.
├── Makefile
├── Makefile.target
├── main.cpp
├── myCode
└── myCode.cpp
OK, I am not sure whether anybody really needs something like this. But, in case you want to change your executable name to the same name as your directory name, try this
BASENAME := $(notdir $(CURDIR))
all: copy exe
copy:
@cp main.cpp $(BASENAME).cpp
exe: copy $(BASENAME).cpp
gcc -o $(BASENAME).o $(BASENAME).cpp
If you get this one, remember about missing new line after shell call that spans over multiple lines that are split with \. You will get this error in case you don’t add explicit new line and you want to silence the command using @ symbol at the same time.
LIST=a b c d
all:
@echo "Starting!"
@echo "Loop over values"
@for target in `echo ${LIST}`; do \
echo "Value: $$target"; \
done; \
@echo "Done!"
Below, you have new line after last \ – this one will work without any problems.
LIST=a b c d
all:
@echo "Starting!"
@echo "Loop over values"
@for target in `echo ${LIST}`; do \
echo "Value: $$target"; \
done; \
@echo "Done!"
Alternatively, you can make last echo to be part of the multiline command. Here, you will get the expected result as well.
LIST=a b c d
all:
@echo "Starting!"
@echo "Loop over values"
@for target in `echo ${LIST}`; do \
echo "Value: $$target"; \
done; \
echo "Done!"
Whether to use this approach or not (I mean, to call shell from Makefile) is the topic for a completely different story.
More about commands modifiers: here
I am, generally, against all these automation stuff (brew, etc.) at macOS. I prefer to install everything from sources (as long as it is possible).
So, here are few steps to get Open MPI up and running at macOS. Please make sure to download source package from: https://www.open-mpi.org/software/ompi/v2.0/downloads/openmpi-2.0.2.tar.gz.
# put this file somewhere inside your $HOME
# and untar it
> tar xf openmpi-2.0.2.tar
> cd openmpi-2.0.2/
# Make sure you have XCode and command line tools
# installed - all of these if free of charge
# If you want to have Fortran support, make sure to install Fortran from here
#
# https://gcc.gnu.org/wiki/GFortran
#
> ./configure --prefix=$HOME/opt/usr/local
> make all
> make install
# After installation is done, you can
# verify it
> $HOME/usr/local/bin/mpirun --version
mpirun (Open MPI) 2.0.2
Report bugs to http://www.open-mpi.org/community/help/
Now, it’s time for Hello world! code.
/* Put this text inside hello.c file */
#include
#include
int main(int argc, char** argv) {
int rank;
int world;
MPI_Init(NULL, NULL);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &world);
printf("Hello: rank %d, world: %d\n",rank, world);
MPI_Finalize();
}
And, eventually, we can compile and run it
$HOME/opt/usr/local/bin/mpicc -o hello ./hello.c
$HOME/opt/usr/local/bin/mpirun -np 2 ./hello
Hello: rank 0, world: 2
Hello: rank 1, world: 2
As for the Fortran code, you can compile it as well
! Put this text inside hello_f.f90 file
program main
include 'mpif.h'
integer ( kind = 4 ) error
integer ( kind = 4 ) id
integer ( kind = 4 ) p
call MPI_Init ( error )
call MPI_Comm_size ( MPI_COMM_WORLD, p, error )
call MPI_Comm_rank ( MPI_COMM_WORLD, id, error )
write (*,*) 'Hello: ', id, '/', p
call MPI_Finalize ( error )
stop
end
and you can compile and run it
$HOME/opt/usr/local/bin/mpif90 -o hello_f ./hello_f.f90
$HOME/opt/usr/local/bin/mpirun -np 2 ./hello_f
Hello: 0 / 2
Hello: 1 / 2
That’s all.
Simple one liner that helps to find all config locations inside your HOME directory
ls -a1 $HOME | grep '^\.'
iTunes -> Windows -> Equaliser
# alternatively: Option + Command + E
1. Open Key Chain Access and unlock all key chains
2. use security utility to set timeout for keychain
security -v set-keychain-settings -t 72000 login.keychain
You can read more about security using man
man security
Summary: So far, the best Scratch Coding resource I have found :)
“Old school” way of learning (one that industrial era developed) is based on reading. To know something, you have to read something. Well, that’s not always the best idea. And let me remind you, that an old philosopher, long, long time ago in a land far, far away, said:
You know, Phaedrus, writing shares a strange feature with painting. The offsprings of painting stand there as if they are alive, but if anyone asks them anything, they remain most solemnly silent. The same is true of written words. You’d think they were speaking as if they had some understanding, but if you question anything that has been said because you want to learn more, it continues to signify just that very same thing forever
– Socrates
And that’s exactly what it is whet it comes to learning from books. You are always left with the content as it is, and you can hardly discuss anything with it.
Let’s be honest, reading is really boring nowadays. You have so many, various, sources of knowledge: youtube, online courses, tutorials, podcasts, screencasts, etc.
Even more, sometimes you don’t need them at all. All you need is just an inspiration. Idea of what can be done, and how can you achieve it. You don’t need all the steps, described one by one. You don’t need full recipe. General idea is more than enough. After you know what you are looking for, all you need to do is to experiment and try to achieve it.
This is exactly the way you can learn using Scratch Coding Cards. By far, this is the most engaging and most powerful tool to teach Scratch I was dealing with. I have some experience with teaching Scratch and I know that finding good way of engaging kids into Scratch is quite challenging. There are few things I done in past. Like, fighting very bad and powerful Knight – you can find him at youtube. Or, some ideas quickly sketched on a piece of paper: here (my beloved Moleskine notebooks ;) ). However, they require either following the instructions, strictly, when you watch video or they require someone extra comment from teacher – like in case of Scratch UML.
When it comes to Scratch Coding Cards, it’s all different. All you need to do, is to give one of the sets to kids, and they will solve it. That’s the beauty of this Box full of cards. Kids are simply shuffling cards, looking at them to check what can be done and which blocks are important, and few minutes later solution is there.
What you get inside the Box (which is firm and solid) are 75 cards.
Cards are grouped into activities (challenges you are supposed to code). Each challenge can be solved by following steps presented on cards. Together with my kids, we have found that strict following of the order is not required.
Each card describes some activity and show you how to achieve that. It’s supper simple. Note that my kids are non english speakers and they were still able to follow cards by simply looking and images. This was really cool!
And, one more comment. There was no cheating at all. Kids were given cards and I slowly shaded away :)
If you are looking for some ideas and new, fresh way of teaching Scratch, make sure to check this one. Really worth it.
Have fun!!
Printed: Scratch Coding Cards
Sometimes, simple solutions are the best
> iptables -A PREROUTING -t nat -p tcp --dport 80 -j REDIRECT --to-port 8888
> iptables -t nat -I OUTPUT -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8888
# in case we want to list what we have
> iptables -t nat -nvL --line-numbers
# if we decide to remove the rule - ## stands for rule No.
> iptables -D PREROUTING ## -t nat
> iptables -D OUTPUT ## -t nat
Most coders think debugging software is about fixing a mistake, but that’s bullshit.
Debugging’s actually all about finding the bug.
Coding apprentice:
– Hi there. Well, you know, there are lots of TODOs inside code.
What does it mean, really? Is it: This One Datelessly On-site?