Skip to content

Commit

Permalink
Updated instructions
Browse files Browse the repository at this point in the history
  • Loading branch information
@Palmr
Palmr committed Jan 11, 2020
1 parent bd52643 commit 2afedd7
Show file tree
Hide file tree
Showing 3 changed files with 152 additions and 47 deletions.
100 changes: 53 additions & 47 deletions Readme.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,56 +11,41 @@ https://drive.google.com/open?id=1TsjfLCuIKoE_Q3kDFtwoCkuLZ3mr2KpyPO-t-qeYDyU

## Requirements

- Linux (definitely won't work on Windows, may work one day on OSX)
- Linux, only tested on CentOS, Fedora, Arch
- GCC, for building the native library
- Java, ideally Oracle JDK 1.8
- Gnuplot, script currently hard coded for gnuplot-wx but could use any Gnuplot front end
- Gnuplot, for displaying a plot of memory use
- Jemalloc, built with profiling enabled
- Not sure if your jemalloc has profiling enabled? Run `jemalloc-config --config` and
look for `--enable-prof` in the output. If it's not there see the
[Jemalloc Runtime Error Messages](#jemalloc-runtime-error-messages) section of this readme

## Building

Build using `gradle assemble` which should generate an archive containing the distributable code in build/distributions

*Note*: I've got something wrong with my gradle sub-project build ordering setup that I haven't figured out how to fix.
But if you want to build this project yourself, run assemble **twice** otherwise it will be missing the native library
in the distribution.

### Changing QuestionableJniLib

- Change to the JNI projects Java directory
- `cd ./QuestionableJniLib/src/main/java`
- Generate header file
- `javah -jni -d ./../c uk.co.palmr.offheapleakexample.jni.QuestionableJniLib`
- Not sure if your Jemalloc has profiling enabled? Run `jemalloc-config --config` and
look for `--enable-prof` in the output. If it's not there see the [Troubleshooting](#troubleshooting) section of
this readme

## Running

- Make sure you're running on Oracle JDK 1.8
- `export JAVA_HOME=/path/to/oracle/jdk/on/your/machine`
- Unzip distribution and go into it's folder
- `unzip OffHeapLeakExample-1.0-SNAPSHOT.zip`
- `cd OffHeapLeakExample-1.0-SNAPSHOT`
- Make sure you're running on Oracle JDK 1.8
- `export JAVA_HOME=/path/to/oracle/jdk/on/your/machine`
- Run the application from the root folder
- `./bin/OffHeapLeakExample`
- While it's paused for 20 seconds before using memory
- Create a Native Memory Tracking baseline
- Start logging memory to a file for plotting
- `./bin/baseline_NMT_and_log_memory.sh`
- Wait some time, got to let it gobble up memory...
- Wait some time, got to let the application gobble up memory...
- Look at memory use
- `./bin/plot_memory.gnuplot`
- Looking higher than expected?
- See what the JVM has to say about the heap
- `jmap -heap <pid>`
- Investigate the heap and maybe buffers with visualvm
- `jvisualvm`
- Investigate the heap and maybe buffers with VisualVM
- `jvisualvm` or `visualvm`
- Check out diff between the Native Memory Tracking baseline taken and now:
- Summary diff (usually good enough): `jcmd <pid> VM.native_memory summary.diff`
- Detail diff (usually too much detail): `jcmd <pid> VM.native_memory detail.diff`
- Nothing particularly obvious there to explain endless memory use? Can kill the application now
- Generate out the jemalloc profile diagrams
- Generate memory allocation diagrams from the Jemalloc profiles that were recorded
- `./bin/jeprof_diagrams.sh`
- Hopefully by now you've spotted where the memory is going!

Expand Down Expand Up @@ -98,37 +83,58 @@ Calls the `jeprof` command with input of all the jeprof.*.heap files and just th

The graphviz files are then converted to image (PNG) files for easy viewing.

### Jemalloc Runtime Error Messages

If you see lines in the stdout from the application like:
## Building

> \<jemalloc>: Invalid conf pair: prof:true
Build using `gradle assemble` which should generate an archive containing the distributable code in build/distributions

Then the jemalloc on your machine was not compiled with profiling enabled.
*Note*: I've got something wrong with my gradle sub-project build ordering setup that I haven't figured out how to fix.
But if you want to build this project yourself, run assemble **twice** otherwise it will be missing the native library
in the distribution.

The following is a bash script to do this if on a distro using dnf (i.e. Fedora, CentOS)
### Changing QuestionableJniLib

```bash
#!/bin/bash
- Change to the JNI projects Java directory
- `cd ./QuestionableJniLib/src/main/java`
- Generate header file
- `javah -jni -d ./../c uk.co.palmr.offheapleakexample.jni.QuestionableJniLib`

# download source rpm to this folder
sudo dnf download --source jemalloc

# install source folder, will go to user home/rpmbuilds
rpm -ivh jemalloc*.src.rpm
## Troubleshooting

# go to specs
cd ~/rpmbuild/SPECS
Below are some example error messages you might get when trying to run this project and commands.

# get deps
sudo dnf builddep jemalloc.spec
[If you use Arch Linux, these instructions should help.](./arch_linux_instructions.md)

# add --enable-prof to configure argument
sed -i '/^%configure/ s/$/ --enable-prof/' ~/rpmbuild/SPECS/jemalloc.spec
[If you use a Redhat based distribution, like Fedora or CentOS, these instructions should help.](./redhat_linux_instructions.md)

# build package
rpmbuild -bb jemalloc.spec
### Jemalloc

If you see lines in the stdout from the application like:

> \<jemalloc>: Invalid conf pair: prof:true
Then the jemalloc on your machine was not compiled with profiling enabled.

### JMAP

If you run `jmap -histo` and get an error like this:

# install resultant rpm
sudo rpm -ivh `ls -1B ~/rpmbuild/RPMS/x86_64/jemalloc-*.rpm | grep -v 'dev\|debug'`
```
Exception from jmap -heap:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.tools.jmap.JMap.runTool(JMap.java:197)
at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: java.lang.RuntimeException: unknown CollectedHeap type : class sun.jvm.hotspot.gc_interface.CollectedHeap
at sun.jvm.hotspot.tools.HeapSummary.run(HeapSummary.java:146)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
at sun.jvm.hotspot.tools.HeapSummary.main(HeapSummary.java:40)
... 6 more
```

Your JVM is missing debug symbols. You can check this by running `file /usr/lib/jvm/java-8-openjdk/bin/java` (or
whatever the path is to the Java runtime being used) and checking if the output ends in `stripped`.
65 changes: 65 additions & 0 deletions arch_linux_instructions.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Arch Linux - Instructions

This project was originally developed from an issue on CentOS, but at home I use Arch Linux, which required more effort
than expected to get it working.

## Java

Arch Linux packages are stripped of debug symbols by default. This only causes issues with `jmap -histo` so the
following instructions aren't necessary, but I've recorded my steps here for future me, if nobody else.

To start, this has only been tested with Open JDK 8, which at the time of writing was the
[java8-openjdk](https://www.archlinux.org/packages/extra/x86_64/jdk8-openjdk/) package.

To get an installed JDK with debug symbols on Arch you have you use the
[Arch Build System](https://wiki.archlinux.org/index.php/Arch_Build_System).

```bash
mkdir ~/abs
cd ~/abs

svn checkout --depth=empty svn://svn.archlinux.org/packages
svn update java8-openjdk

cd trunk

## Edit PKGBUILD and add:
# groups=('modified')
# options=('!strip')

# Make and install
makepkg -sic

# Confirm it worked, should see 'not stripped' in the output of the following command
file file /usr/lib/jvm/java-8-openjdk/jre/bin/java
```

Note: If you haven't already, you should add `IgnoreGroup = modified` to `/etc/pacman.conf` to stop pacman updating this
package. Instead you should svn update the abs packages you have manually.

## JVisualVM

JVisualVM is not part of the Java package. You need to install it separately and the
[visualvm](https://www.archlinux.org/packages/extra/x86_64/visualvm/) package can be installed and run using `visualvm`

## Jemalloc

I went with the [AUR git version of the Jemalloc](https://aur.archlinux.org/packages/jemalloc-git/) package to make sure
I had the latest version, but the following could probably be done to the ABS version too.

```bash
mkdir ~/aur
cd ~/aur
git clone https://aur.archlinux.org/jemalloc-git.git

cd jemalloc-git

## Edit PKGBUILD and add the following to the autogen.sh command in the build function:
# --enable-prof

# Make and install
makepkg -sic

# Verify it worked, should see '--enable-prof' in the output of the following command
jemalloc-config --config
```
34 changes: 34 additions & 0 deletions redhat_linux_instructions.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Redhat/CentOS/Fedora Linux - Instructions

This project was originally developed from an issue on CentOS, and then tested on Fedora.
Java on these distributions works fine with all the commands needed. But the standard Jemalloc package does not have
profiling enabled.

## Jemalloc

The following is a bash script to do this if on a Linux distribution using dnf:

```bash
#!/bin/bash

# download source rpm to this folder
sudo dnf download --source jemalloc

# install source folder, will go to user home/rpmbuilds
rpm -ivh jemalloc*.src.rpm

# go to specs
cd ~/rpmbuild/SPECS

# get deps
sudo dnf builddep jemalloc.spec

# add --enable-prof to configure argument
sed -i '/^%configure/ s/$/ --enable-prof/' ~/rpmbuild/SPECS/jemalloc.spec

# build package
rpmbuild -bb jemalloc.spec

# install resultant rpm
sudo rpm -ivh `ls -1B ~/rpmbuild/RPMS/x86_64/jemalloc-*.rpm | grep -v 'dev\|debug'`
```

0 comments on commit 2afedd7

Please sign in to comment.