XXHash module - Hash string

[stychu]

How can I use your xxhash module to do hashing on a string? I can't manage to do so. Seems like this module is the only one currently available on the internet that contains xxHash128 bit version and I would like to use it to hash some strings.
Could you advise me on how to do so?

This doesn't work for me

package com.company;

import org.lwjgl.system.*;
import org.lwjgl.util.xxhash.*;

import java.io.*;
import java.nio.*;
import java.nio.channels.*;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.util.*;

import static org.lwjgl.system.MemoryStack.*;
import static org.lwjgl.system.MemoryUtil.*;
import static org.lwjgl.util.xxhash.XXHash.*;
//import static org.testng.Assert.*;

final class Main {

    private Main() {
    }

    public static void main(String[] args) {
        String babel = "key";
        System.out.println(babel);
        //Convert string to ByteBuffer:
        ByteBuffer babb = Charset.forName("UTF-8").encode(babel);

        ByteBuffer buffer = StandardCharsets.UTF_8.encode("abrakadabra");
        int SEED = 1928732;

        XXH128Hash hash128 = XXH128(babb, SEED, XXH128Hash.create());

        System.out.format("test %d-bit hash = 0x%X%X\n", 128, hash128.high64(), hash128.low64());
    }

}

[octylFractal]

You didn't specify why it did not work, but I assume your byte buffers are allocated incorrectly, see https://blog.lwjgl.org/memory-management-in-lwjgl-3/.

[httpdigest]

Instead of:
ByteBuffer babb = Charset.forName("UTF-8").encode(babel);
use:
ByteBuffer babb = org.lwjgl.system.MemoryUtil.memUTF8(babel);

The former allocates an on-heap (non-direct) ByteBuffer, which no native API exposed by LWJGL 3 supports.

[stychu]

Yeah that was the problem, it worked. Thanks

[stychu]

Although it works, I thought it will produce the same hash string as the nodejs implementations but it doesn't :(

const addon = require('xxhash-addon');

const SEED = 1999999;

const hash128seed = new addon.XXHash128(SEED);

function logHash(hasher, str) {
  const buffer = Buffer.from(str);
  const hash = hasher.hash(buffer).toString('hex');

  console.log(hash);
}

const key = 'key';

logHash(hash128seed, key);

package com.company;
/*
 * Copyright LWJGL. All rights reserved.
 * License terms: https://www.lwjgl.org/license
 */

import org.lwjgl.util.xxhash.*;

import java.nio.*;

import static org.lwjgl.util.xxhash.XXHash.*;

final class Main {

    private Main() {
    }

    public static void main(String[] args) {
        int SEED = 1999999;

        String key = "key";
        System.out.println(key);

        //Convert string to ByteBuffer:
        ByteBuffer buffer = org.lwjgl.system.MemoryUtil.memUTF8(key);

        XXH128Hash hash128 = XXH128(buffer, SEED, XXH128Hash.create());

        System.out.format("%x%x\n", hash128.high64(), hash128.low64());
    }

}

nodejs xxHash -> 5fd4b169bdfeb9a282d751c8cac401f2
lwjgl xxHash -> 6716cb96e6a8a9a74fe952461e2f0c09

Am I doing something wrong here or implementation is a bit different and won't get the same results? Any ideas?

[Spasi]

From the xxhash header:

The XXH3 algorithm is still in development.
The results it produces may still change in future versions.

Results produced by v0.7.x are not comparable with results from v0.7.y.
However, the API is completely stable, and it can safely be used for
ephemeral data (local sessions).

Avoid storing values in long-term storage until the algorithm is finalized.
XXH3's return values will be officially finalized upon reaching v0.8.0.

After which, return values of XXH3 and XXH128 will no longer change in
future versions.

In practice, XXH32 and XXH64 have been stable since first introduced to LWJGL and should be compatible with the nodejs implementation. XXH128 output on the other hand has changed two times. The xxhash bindings in LWJGL are on 0.8.0 now, so no more changes are expected. The implementation in nodejs is very likely an older version.

[stychu]

Awesome I will look into the nodejs implementations then! Thanks for the heads up!

[stychu]

Yeah I've checked the nodejs package that I use and it uses v0.7.3 and seems all hashes are different no matter if it's 32/64/128.
That's a bummer.
Maybe in the future, they will update it to v0.8.0 and then I would be able to use LWJGL. For now, I'm forced to use
https://github.com/lz4/lz4-java as I get the same hashes between java and nodejs.

Thanks for your inputs guys! Keep the good work

[Spasi]

@stychu See what @Sylvyrfysh says below, it's very likely true. Make sure you aren't hashing the string + the \0 character. I'm fairly sure that the 32&64-bit hashes haven't changed since 0.7.3.

[Sylvyrfysh]

Is it processing a null byte as the last entry for Java and no null byte in JavaScript? I believe memUTF8 takes a second argument which says whether or not it should add a null byte to the ByteBuffer, which could least to different results since it defaults to true.

…

On Mon, Mar 8, 2021, 07:39 stychu ***@***.***> wrote: Yeah I've checked the nodejs package that I use and it uses v0.7.3 and seems all hashes are different no matter if it's 32/64/128. That's a bummer. Maybe in the future, they will update it to v0.8.0 and then I would be able to use LWJGL. For now, I'm forced to use https://github.com/lz4/lz4-java as I get the same hashes between java and nodejs. Thanks for your inputs guys! Keep the good work — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#625 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTJ6QFY6MZAVYNZZBKVUIDTCTHRXANCNFSM4YVSQHHA> .

[stychu]

Oh, man! This actually worked:
ByteBuffer buffer = org.lwjgl.system.MemoryUtil.memUTF8(key, false);

I got the same hash as in nodejs <3
Thank you!!