`make build` problem

[Unik-lif]

I'm sorry that I've never known rust and wasm before, but honestly speaking I love your paper concerning Spectre, Wasm and some interesting protection methods, so I can't wait to implement these methods and get some knowledge of Wasm-Spectre Poc on my machine. I've read the manual of rustup and cargo but still can't figure out how can I fix this problem.
Basically I've setup an environment of Ubuntu 20.04.2 LTS in a fresh VM and machine as mentioned in README.md, and I ran the command in README.md. However I got the following problem when I was running "make build".

In Makefile I notice we use cargo +rust-cet build to lanuch the project:

out/rust_build/bin/rustc:
	mkdir -p out/rust_build
	cd ./rustc-cet && ./x.py build && ./x.py install
	rustup toolchain link rust-cet ./out/rust_build

build_rustc:
	# force rebuild
	mkdir -p out/rust_build
	cd ./rustc-cet && ./x.py build && ./x.py install
	rustup toolchain link rust-cet ./out/rust_build

build_lucet_nocet: rust_libloading_aslr
	cd lucet-spectre && cargo build
	cd lucet-spectre && cargo build --release

build_lucet: out/rust_build/bin/rustc build_lucet_nocet
	cd lucet-spectre && \
		CFLAGS="-fcf-protection=full" \
		CXXFLAGS="-fcf-protection=full" \
		CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER="$(CURR_DIR)/rustc-cet/rust_cet_linker" \
		CARGO_TARGET_DIR="${CURR_DIR}/lucet-spectre/target-cet" \
		cargo +rust-cet build
	cd lucet-spectre && \
		CFLAGS="-fcf-protection=full" \
		CXXFLAGS="-fcf-protection=full" \
		CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER="$(CURR_DIR)/rustc-cet/rust_cet_linker" \
		CARGO_TARGET_DIR="${CURR_DIR}/lucet-spectre/target-cet" \
		cargo +rust-cet build --release

the result is shown below:

error: 'cargo' is not installed for the toolchain 'rust-cet'
note: this is a custom toolchain, which cannot use `rustup component add`
help: if you built this toolchain from source, and used `rustup toolchain link`, then you may be able to build the component with `x.py`

I've checked the rustup show command and find that the toolchain of "rust-cet" has already been installed:

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/link/.rustup

installed toolchains
--------------------

nightly-2020-05-15-x86_64-unknown-linux-gnu (default)
rust-cet
1.43.1-x86_64-unknown-linux-gnu

installed targets for active toolchain
--------------------------------------

wasm32-wasi
x86_64-unknown-linux-gnu

active toolchain
----------------

nightly-2020-05-15-x86_64-unknown-linux-gnu (default)
rustc 1.45.0-nightly (a74d1862d 2020-05-14)

Based on some manuals about "cargo build" , I've tried simply delete "+rust-cet" in "cargo +rust-cet build" since rust-cet seemed to be installed beforehand, but I got new fault shown below:

error: could not compile `cranelift-codegen`.

Caused by:
  process didn't exit successfully:
.......(signal: 9, SIGKILL: kill)
make: *** [Makefile:307: build_lucet] Error 101

I'm looking forward to your reply,

[shravanrn]

As a starting point, I would suggest building and running the Swivel-SFI tools and benchmarks and see if you get that running. It is possible that the build script I wrote isn't perfect and assumes that this is built already. If you get the Swivel-SFI running, this may help diagnose your issue

[Unik-lif]

Thank you for your reply! btw in Cal it might be mid-night, please mind WLB, Sir.

I added "./x.py install cargo" in "build_rustc:" and now it was able to continue to run, buf it failed when we moved to "build_macro_benchmark" process in "make build", basically beacause of the some rust problems. I guess that some of the dependencies in cargo.toml may have changed a lot so it was not easy for me to continue. The error when running "build_macro_benchmark" is basically like what rust-lang/rust#49146 mentioned and simply "rustup update" couldn't fix this.

As for Swivel-SFI tools and some benchmarks, I found a lot of files missing, and I guess that I couldn't fix them. For other tests, I encountered errors like

thread '<unnamed>' panicked at 'Failed to get module "html_template_stock"', src/wasm_module.rs::71

As rust gets update so quickly it is definitely very hard for me to setup whole environments and I guess it might be nasty and profitless for you to reproduce this project.

Btw I also tried to run "make build_repros" while disabling protections of Swivel, that way maybe I could try a taste of Wasm-Spectre Poc. But the compiler told me that I didn't get special wasi-sdk in this projects installed.

unik@ubuntu:~/swivel$ make build_repros
.......
make[3]: /home/unik/swivel/safeside/build-lucet/../../wasi-sdk/build/install/opt/wasi-sdk/bin/clang++: Command not found

Maybe it was due to my failure at make build, so I wonder if you can help me by adding the special wasi-sdk in this repro.
(I've tested wasi-sdk-10.0 and I think it is different from what make build_repros needs)
best wishes.

[shravanrn]

The error when running "build_macro_benchmark" is basically like what rust-lang/rust#49146 mentioned and simply "rustup update" couldn't fix this.

It is possible that rust and cargo have changed the dependencies in a way that might need some work to get all this building.
It definitely shouldn't be too difficult as this code is only some months old. Unfortunately, I don't have the time for the next month or so to help debug this, so you may have to give this a shot and see what happens.

thread '' panicked at 'Failed to get module "html_template_stock"', src/wasm_module.rs::71

Do you see a folder called sfispectre_webserver in your source. If not, you need to run make get_source
If the folder is present, you can try

cd ./spectresfi_webserver/modules && make clean
cd ./spectresfi_webserver/modules && make -j8

and try runing the test again

I wonder if you can help me by adding the special wasi-sdk in this repro.

make bootstrap should already pull in the right wasi-sdk and rust version to your machine

[Unik-lif]

Thank you for your advice!!
I've tried some methods but unfortunately this didn't work very well.

First of all, as I ran make -j8, the result is shown below:

./tensorflow/lite/schema/schema_generated.h:21:10: fatal error: 'flatbuffers/flatbuffers.h' file not found
#include "flatbuffers/flatbuffers.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from tensorflow/lite/micro/simple_memory_allocator.cc:22:
In file included from ./tensorflow/lite/micro/memory_helpers.h:23:
./tensorflow/lite/schema/schema_generated.h:21:10: fatal error: 'flatbuffers/flatbuffers.h' file not found
#include "flatbuffers/flatbuffers.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

Secondly, I surely ran the make bootstrap command, but I couldn't find wasi-sdk under swivel folder:

unik@ubuntu:~/swivel$ make build_repros
.......
make[3]: /home/unik/swivel/safeside/build-lucet/../../wasi-sdk/build/install/opt/wasi-sdk/bin/clang++: Command not found

It seems like the wasi-sdk should be in the directory in the swivel, but I can't find this wasi-sdk. I can find wasi-sdk only in the /opt directory, and It looks like not the wasi-sdk mentioned below:

The '/home/unik/swivel/safeside/build-lucet/../../wasi-sdk/build/install/opt/wasi-sdk/bin/clang++' didn't match with the path of wasi-sdk.

I've checked the make bootstrap, and maybe what bootstrap does is not putting wasi-sdk in the swivel directory.

In fact, what my swivel looks like was basically like below: ( In make build procedure, before make build_macro_benchmark, Everything runs right with no errors, that the point where I stuck ).

unik@ubuntu:~/swivel$ ls
aligned_patch.diff  bootstrap                 install_deps         Makefile        out                rustc-cet             sfi-spectre-testing         swivel-btb-exploit  wrk
analysis            btbflush-module           lucet-spectre        Makefile.small  package-lock.json  rust_libloading_aslr  spec2017_patch_may28.patch  tmp
benchmarks          cp_spec_data_into_tmp.sh  lucet-spectre-repro  node_modules    README.md          safeside              spectresfi_webserver        wabt-1.0.19-ubuntu

Maybe that is because I failed to run make build all, so the very wasi-sdk didn't show up?

best wishes.

[shravanrn]

I think I finally have some time. I will try this on my machine over the next few days and see if I can fix any blockers.

[shravanrn]

@Unik-lif Fixed a bunch of issues in the build script to do with

• cargo for the custom rust
• custom wasi toolchain we use

Tested on a clean ubuntu lts 20.04.04 LTS and everything seems to build
I would recommend starting from a clean VM as the build dependencies may cause issues if you start with an existing build

Closing the issue for now. Please re-open if you run into any additional issues

[Unik-lif]

Great thanks for your reply and urgent fix ! !

I've been busy recently, sorry for my late response...
The custom rust work well! And a lot of bugs are fixed. However maybe there might still be some problems when compiling cranelift-codegen.

I'm testing this in a fresh Ubuntu 20.04.04 LTS in VMware, but in make build process, it seems that the cranelift-codegen can't be compiled.

link@ubuntu:~/Desktop/swivel-master$ make build
cd lucet-spectre && cargo build
   Compiling cranelift-codegen v0.62.0 (/home/link/Desktop/swivel-master/lucet-spectre/wasmtime/cranelift/codegen)
   Compiling cranelift-module v0.62.0 (/home/link/Desktop/swivel-master/lucet-spectre/wasmtime/cranelift/module)
   Compiling cranelift-frontend v0.62.0 (/home/link/Desktop/swivel-master/lucet-spectre/wasmtime/cranelift/frontend)
   Compiling cranelift-native v0.62.0 (/home/link/Desktop/swivel-master/lucet-spectre/wasmtime/cranelift/native)
   Compiling cranelift-object v0.62.0 (/home/link/Desktop/swivel-master/lucet-spectre/wasmtime/cranelift/object)
   Compiling cranelift-wasm v0.62.0 (/home/link/Desktop/swivel-master/lucet-spectre/wasmtime/cranelift/wasm)
   Compiling lucetc v0.7.0-dev (/home/link/Desktop/swivel-master/lucet-spectre/lucetc)
   Compiling lucet-wasi-sdk v0.7.0-dev (/home/link/Desktop/swivel-master/lucet-spectre/lucet-wasi-sdk)
   Compiling lucet-spectest v0.7.0-dev (/home/link/Desktop/swivel-master/lucet-spectre/lucet-spectest)
   Compiling lucet-runtime-tests v0.7.0-dev (/home/link/Desktop/swivel-master/lucet-spectre/lucet-runtime/lucet-runtime-tests)
   Compiling lucet-benchmarks v0.7.0-dev (/home/link/Desktop/swivel-master/lucet-spectre/benchmarks/lucet-benchmarks)
   Compiling lucet-concurrency-tests v0.7.0-dev (/home/link/Desktop/swivel-master/lucet-spectre/lucet-concurrency-tests)

error: could not compile `cranelift-codegen`.

Caused by:
  process didn't exit successfully: `rustc --crate-name cranelift_codegen --edition=2018 wasmtime/cranelift/codegen/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="gimli"' --cfg 'feature="std"' --cfg 'feature="unwind"' -C metadata=44e3e644d402deb9 -C extra-filename=-44e3e644d402deb9 --out-dir /home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps -C incremental=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/incremental -L dependency=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps --extern byteorder=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libbyteorder-0128a5c530927307.rmeta --extern cranelift_bforest=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libcranelift_bforest-5ed0e5eaadb18a90.rmeta --extern cranelift_codegen_shared=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libcranelift_codegen_shared-71ea624b05acd322.rmeta --extern cranelift_entity=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libcranelift_entity-27f8bf2d818993bd.rmeta --extern cranelift_spectre=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libcranelift_spectre-49fc43bd00b69dbe.rmeta --extern gimli=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libgimli-d6488458a9010d11.rmeta --extern log=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/liblog-6de3b44926e29071.rmeta --extern regalloc=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libregalloc-23e40b1e2f610840.rmeta --extern rs_graph=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/librs_graph-b16cc850a1b9738d.rmeta --extern smallvec=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libsmallvec-f4a6710f093e9397.rmeta --extern target_lexicon=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libtarget_lexicon-76446c10f9c6aca4.rmeta --extern thiserror=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/deps/libthiserror-9c2eb3ebd3107365.rmeta -C link-args=-rdynamic --cfg 'feature="x86"' -L native=/home/link/Desktop/swivel-master/lucet-spectre/target/debug/build/cranelift-spectre-d8cae6efe4495c01/out` (signal: 9, SIGKILL: kill)
make: *** [Makefile:310: build_lucet_nocet] Error 101

The 'Caused by' message is lengthy, and I hope that you are not be annoyed....
Could you please tell me what can I do to fix this kinda problem?
Best wishes!

[shravanrn]

Hmm... this is surprising. I definitely didn't run into this issue.

Basic sanity test

• Do you have enough space in your VM storage?

One of the things you can also try to do is to run the failing command alone in the terminal and see if the error shows up again

cd lucet-spectre && cargo build

If it happens again, we'll can see how to debug this.

The 'Caused by' message is lengthy

If you had to cut off part of the log to post the message, you can probably post the full log in pastebin and send the link here

I hope that you are not be annoyed....

Nah, all good :)

[Unik-lif]

Thank you for your reply！
It works well on my machine by seperating make build process step by step according to Makefile.
It's better to allocate more than 100 GB for VM disk, though after make build basically only 50 GB will be occupied.
Best wishes! close this issue here.