diff --git a/Readme.md b/Readme.md index b4ece73..bc14512 100644 --- a/Readme.md +++ b/Readme.md @@ -11,56 +11,41 @@ https://drive.google.com/open?id=1TsjfLCuIKoE_Q3kDFtwoCkuLZ3mr2KpyPO-t-qeYDyU ## Requirements - - Linux (definitely won't work on Windows, may work one day on OSX) + - Linux, only tested on CentOS, Fedora, Arch - GCC, for building the native library - Java, ideally Oracle JDK 1.8 - - Gnuplot, script currently hard coded for gnuplot-wx but could use any Gnuplot front end + - Gnuplot, for displaying a plot of memory use - Jemalloc, built with profiling enabled - - Not sure if your jemalloc has profiling enabled? Run `jemalloc-config --config` and - look for `--enable-prof` in the output. If it's not there see the - [Jemalloc Runtime Error Messages](#jemalloc-runtime-error-messages) section of this readme - -## Building - -Build using `gradle assemble` which should generate an archive containing the distributable code in build/distributions - - *Note*: I've got something wrong with my gradle sub-project build ordering setup that I haven't figured out how to fix. - But if you want to build this project yourself, run assemble **twice** otherwise it will be missing the native library - in the distribution. - -### Changing QuestionableJniLib - - - Change to the JNI projects Java directory - - `cd ./QuestionableJniLib/src/main/java` - - Generate header file - - `javah -jni -d ./../c uk.co.palmr.offheapleakexample.jni.QuestionableJniLib` + - Not sure if your Jemalloc has profiling enabled? Run `jemalloc-config --config` and + look for `--enable-prof` in the output. If it's not there see the [Troubleshooting](#troubleshooting) section of + this readme ## Running - - Make sure you're running on Oracle JDK 1.8 - - `export JAVA_HOME=/path/to/oracle/jdk/on/your/machine` - Unzip distribution and go into it's folder - `unzip OffHeapLeakExample-1.0-SNAPSHOT.zip` - `cd OffHeapLeakExample-1.0-SNAPSHOT` + - Make sure you're running on Oracle JDK 1.8 + - `export JAVA_HOME=/path/to/oracle/jdk/on/your/machine` - Run the application from the root folder - `./bin/OffHeapLeakExample` - While it's paused for 20 seconds before using memory - Create a Native Memory Tracking baseline - Start logging memory to a file for plotting - `./bin/baseline_NMT_and_log_memory.sh` - - Wait some time, got to let it gobble up memory... + - Wait some time, got to let the application gobble up memory... - Look at memory use - `./bin/plot_memory.gnuplot` - Looking higher than expected? - See what the JVM has to say about the heap - `jmap -heap ` - - Investigate the heap and maybe buffers with visualvm - - `jvisualvm` + - Investigate the heap and maybe buffers with VisualVM + - `jvisualvm` or `visualvm` - Check out diff between the Native Memory Tracking baseline taken and now: - Summary diff (usually good enough): `jcmd VM.native_memory summary.diff` - Detail diff (usually too much detail): `jcmd VM.native_memory detail.diff` - Nothing particularly obvious there to explain endless memory use? Can kill the application now - - Generate out the jemalloc profile diagrams + - Generate memory allocation diagrams from the Jemalloc profiles that were recorded - `./bin/jeprof_diagrams.sh` - Hopefully by now you've spotted where the memory is going! @@ -98,37 +83,58 @@ Calls the `jeprof` command with input of all the jeprof.*.heap files and just th The graphviz files are then converted to image (PNG) files for easy viewing. -### Jemalloc Runtime Error Messages -If you see lines in the stdout from the application like: +## Building - > \: Invalid conf pair: prof:true +Build using `gradle assemble` which should generate an archive containing the distributable code in build/distributions -Then the jemalloc on your machine was not compiled with profiling enabled. + *Note*: I've got something wrong with my gradle sub-project build ordering setup that I haven't figured out how to fix. + But if you want to build this project yourself, run assemble **twice** otherwise it will be missing the native library + in the distribution. -The following is a bash script to do this if on a distro using dnf (i.e. Fedora, CentOS) +### Changing QuestionableJniLib -```bash -#!/bin/bash + - Change to the JNI projects Java directory + - `cd ./QuestionableJniLib/src/main/java` + - Generate header file + - `javah -jni -d ./../c uk.co.palmr.offheapleakexample.jni.QuestionableJniLib` -# download source rpm to this folder -sudo dnf download --source jemalloc -# install source folder, will go to user home/rpmbuilds -rpm -ivh jemalloc*.src.rpm +## Troubleshooting -# go to specs -cd ~/rpmbuild/SPECS +Below are some example error messages you might get when trying to run this project and commands. -# get deps -sudo dnf builddep jemalloc.spec +[If you use Arch Linux, these instructions should help.](./arch_linux_instructions.md) -# add --enable-prof to configure argument -sed -i '/^%configure/ s/$/ --enable-prof/' ~/rpmbuild/SPECS/jemalloc.spec +[If you use a Redhat based distribution, like Fedora or CentOS, these instructions should help.](./redhat_linux_instructions.md) -# build package -rpmbuild -bb jemalloc.spec +### Jemalloc + +If you see lines in the stdout from the application like: + + > \: Invalid conf pair: prof:true + +Then the jemalloc on your machine was not compiled with profiling enabled. + +### JMAP + +If you run `jmap -histo` and get an error like this: -# install resultant rpm -sudo rpm -ivh `ls -1B ~/rpmbuild/RPMS/x86_64/jemalloc-*.rpm | grep -v 'dev\|debug'` ``` +Exception from jmap -heap: +Exception in thread "main" java.lang.reflect.InvocationTargetException + at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) + at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) + at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) + at java.lang.reflect.Method.invoke(Method.java:606) + at sun.tools.jmap.JMap.runTool(JMap.java:197) + at sun.tools.jmap.JMap.main(JMap.java:128) +Caused by: java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap + at sun.jvm.hotspot.tools.HeapSummary.run(HeapSummary.java:146) + at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) + at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:40) + ... 6 more +``` + +Your JVM is missing debug symbols. You can check this by running `file /usr/lib/jvm/java-8-openjdk/bin/java` (or +whatever the path is to the Java runtime being used) and checking if the output ends in `stripped`. diff --git a/arch_linux_instructions.md b/arch_linux_instructions.md new file mode 100644 index 0000000..348c6a5 --- /dev/null +++ b/arch_linux_instructions.md @@ -0,0 +1,65 @@ +# Arch Linux - Instructions + +This project was originally developed from an issue on CentOS, but at home I use Arch Linux, which required more effort +than expected to get it working. + +## Java + +Arch Linux packages are stripped of debug symbols by default. This only causes issues with `jmap -histo` so the +following instructions aren't necessary, but I've recorded my steps here for future me, if nobody else. + +To start, this has only been tested with Open JDK 8, which at the time of writing was the +[java8-openjdk](https://www.archlinux.org/packages/extra/x86_64/jdk8-openjdk/) package. + +To get an installed JDK with debug symbols on Arch you have you use the +[Arch Build System](https://wiki.archlinux.org/index.php/Arch_Build_System). + +```bash +mkdir ~/abs +cd ~/abs + +svn checkout --depth=empty svn://svn.archlinux.org/packages +svn update java8-openjdk + +cd trunk + +## Edit PKGBUILD and add: +# groups=('modified') +# options=('!strip') + +# Make and install +makepkg -sic + +# Confirm it worked, should see 'not stripped' in the output of the following command +file file /usr/lib/jvm/java-8-openjdk/jre/bin/java +``` + +Note: If you haven't already, you should add `IgnoreGroup = modified` to `/etc/pacman.conf` to stop pacman updating this +package. Instead you should svn update the abs packages you have manually. + +## JVisualVM + +JVisualVM is not part of the Java package. You need to install it separately and the +[visualvm](https://www.archlinux.org/packages/extra/x86_64/visualvm/) package can be installed and run using `visualvm` + +## Jemalloc + +I went with the [AUR git version of the Jemalloc](https://aur.archlinux.org/packages/jemalloc-git/) package to make sure +I had the latest version, but the following could probably be done to the ABS version too. + +```bash +mkdir ~/aur +cd ~/aur +git clone https://aur.archlinux.org/jemalloc-git.git + +cd jemalloc-git + +## Edit PKGBUILD and add the following to the autogen.sh command in the build function: +# --enable-prof + +# Make and install +makepkg -sic + +# Verify it worked, should see '--enable-prof' in the output of the following command +jemalloc-config --config +``` diff --git a/redhat_linux_instructions.md b/redhat_linux_instructions.md new file mode 100644 index 0000000..ae254e6 --- /dev/null +++ b/redhat_linux_instructions.md @@ -0,0 +1,34 @@ +# Redhat/CentOS/Fedora Linux - Instructions + +This project was originally developed from an issue on CentOS, and then tested on Fedora. +Java on these distributions works fine with all the commands needed. But the standard Jemalloc package does not have +profiling enabled. + +## Jemalloc + +The following is a bash script to do this if on a Linux distribution using dnf: + +```bash +#!/bin/bash + +# download source rpm to this folder +sudo dnf download --source jemalloc + +# install source folder, will go to user home/rpmbuilds +rpm -ivh jemalloc*.src.rpm + +# go to specs +cd ~/rpmbuild/SPECS + +# get deps +sudo dnf builddep jemalloc.spec + +# add --enable-prof to configure argument +sed -i '/^%configure/ s/$/ --enable-prof/' ~/rpmbuild/SPECS/jemalloc.spec + +# build package +rpmbuild -bb jemalloc.spec + +# install resultant rpm +sudo rpm -ivh `ls -1B ~/rpmbuild/RPMS/x86_64/jemalloc-*.rpm | grep -v 'dev\|debug'` +```