Unable to boot using the simple pxe server ( isci-dhcp-server and tftpd-hpa )

[PartialVolume]

Hello,
I’m currently working on a solution to erase data and obtain certificates for hundreds of disks. This project is also part of my final school project, so I’m aiming to create the solution myself rather than using a finished version. I do have one question about the EFI file in /srv/tftp — I’m not sure if I need to create it, or if my configuration is incorrect.

Thank you for taking the time to read and assist!

Originally posted by @titielgozo in #148 (comment)

[PartialVolume]

@titielgozo I've added your video here

This shows the .efi load successfully followed by a successfull load of the shredos image (O.S) file.

Then it shows a blank screen.

So as @jamesaepp says, your network configuration looks fine. The blank screen is consistent with a video issue (although not always).

The first step is to use the nomodeset version of ShredOS, which disables DRM video drivers and switches to a simple framebuffer mode. This often resolves issues caused by incompatible DRM drivers with your video hardware. You can do this by editing the srv/tftp/EFI/BOOT/grub.cfg file and adding nomodeset to the end of the kernel command line.

[PartialVolume]

@titielgozo Can you also post the model number of the HP server you are using as well as some technical details. I believe you also said Ubuntu won't boot on it either. Is that correct?

Ideally it would be nice to see the output of lspci -k but if you'll need to boot some Linux distro or ShredOS to obtain that.

Have you tried letting it boot to the blank screen then following the headless login to see if ShredOS is running in the background which will prove it's a graphics issue but will also allow you to get the output of lspci -k and to the dmesg output?

If it is a graphics issue, maybe you could try plugging a different graphics card into one of the PCIe slots. If you do this, check the bios for graphics settings, as sometimes there is some setting that allows you to select your default graphics card such as is the case with the Dell systems I use.

[titielgozo]

Informations:
The dell server is a poweredge r430 in UEFI boot.
The hp server is a dl380 gen9 in UEFI boot.

USB stick result :
for hp : Kernel doesn't support 64-bit CPUs
for dell : shredos is starting despite the HBA mode for my raid controller i can see only the usb key in storage.

PXE boot :
for hp : no video output
for dell : Cannot boot because the name file was not good ( adding some wrong charactères for the booting file name for example i had bootx64.efiy ( added a "y" for no specific reason, when i check the dhcp conf all is good )

In progress :
I'm going to install a ubuntu in the servers for give you the details about the PCI bus.
I'm going to try to use a nomodeset during the day.

[titielgozo]

Informations:
I try to boot with usb stick and i have activated nomodeset in EFI/BOOT/grub.cfg + boot/grub/grub.cfg

USB stick result in UEFI :
for hp : Kernel doesn't support 64-bit CPUs
for dell : Kernel doesn't support 64-bit CPUs

USB stick result in legacy BIOS :
for hp : shredos is starting despite the HBA mode for my raid controller i can see only the usb key in storage.
for dell : shredos is starting despite the HBA mode for my raid controller i can see only the usb key in storage.

In progress :
I'm going to ckeck all the configurations of my servers for see if i can switch CPU to 32 bits

[titielgozo]

PCI details for the DELL poweredge R430:

PCI details for the HP dl380 gen9:

[PartialVolume]

Can you confirm your usb stick is the x86_64 version of ShredOS?

When you get it to boot in legacy what's the output of uname -a?

[PartialVolume]

Those lspci -k details are unfortunately not complete. You need to pipe them to a file, then copy the file to the usb stick, plug it into you windows system and copy and paste the text here. I don't know what your level of experience with the command line is so if you don't know how to mount and copy files to the usb stick from ShredOS let me know.

[titielgozo]

I made a mistake because I took v0.38 in 32 bits so I took the version and redid the tests, now via usb I have access to my 2 servers but I don't have the disks back up. I'll send you a pci detail later today!

[titielgozo]

My 2 file who show the lspci command :

DELL-poweredge-R430

HP-DL380-GEN9

With all tests and your help, I start to view the end, now I access to shreds and I find the error with PXE boot (corrupted isc-DHCP configuration file).

But for now I can't congrats to see the disks. All my raid controllers as set up in HBA mode.

[PartialVolume]

Starting with the Dell lspci, it shows that SATA controller and a secondary sSATA controller and they both have the AHCI driver loaded. In theory you should be seeing the discs if the controller is in AHCI or HBA mode which it seems to be as shown in the lspci output C610/X99 series chipset 6-Port SATA Controller [AHCI mode]. The C610/X99 chipset certainly supports AHCI. This assumes your disks are connected to these controllers and not some other controller not shown in lspci. Can you confirm your discs are attached to SATA ports on the motherboard. What are the options you have for configuring the discs in the BIOS? Are the discs showing up in the BIOS?

00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
	Subsystem: Dell Device 0639
	Kernel driver in use: ahci

00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
	Subsystem: Dell Device 0639
	Kernel driver in use: ahci

The HP is interesting. There are no SATA controllers but there is a fibre channel controller. However as can be seen there is no driver in use. That's because ShredOS doesn't include any fibre channel drivers as default.

Just to confirm, on the HP the discs are connected via a fibre channel controller?

05:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
	Subsystem: QLogic Corp. QLE2562 PCI Express to 8Gb FC Dual Channel

05:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
	Subsystem: QLogic Corp. QLE2562 PCI Express to 8Gb FC Dual Channel

If you are connecting discs via fibre channel I could always built a special version that includes the fibre channel drivers and let you have a .img for testing or alternatively you could build ShredOS from source and add them from within make linux-menuconfig yourself.

[titielgozo]

Hello, sorry for the wait I had too much work. I don't use FC on my infra. Now I know where is the problem, with some troubleshooting I saw the raid configuration of the disk doesn't match with the HBA controller, I try to automatised the passage of the disks raid to hba.

[titielgozo]

@PartialVolume Thanks for all the help, i got my first certificat but i had one question : how to add parameters on the image of shredOS because i try to put in default the organization details? After that all was good I really appreciate you're help!

[PartialVolume]

No problem, on the drive selection screen press the c key for config.

[titielgozo]

I mean I want to put these organization settings in default on the config like using grub to add the tftp server. It's for winning time and automatize with less input to do

[PartialVolume]

The following procedure will allow ShredOS to read and save the nwipe.conf and customers files to and from your tftpd_hpa server.

• First you need to make sure your tftpd_hpa server is writable. It's important to know that tftp uses a subset of the ftp commands. It does not support authentication or changing directories which is relevant when it comes to configuring the username and password fields in the examples below. It also uses port 69 as standard unlike ftp which uses 20 & 21. The tftp server needs to support write access and creation of files by the client. In the case of tftpd_hpa that means adding -c to TFTP_OPTIONS, i.e TFTP_OPTIONS="--secure -v -c" in the config file /etc/default/tftpd-hpa on your tftp server.

The next thing you need to do is to append the kernel command line in the ShredOS grub files on your tftpd-hpa server with the following:

 shredos_config="protcol:IP_address:path:username:password:debug"

So assuming your tftpd-hpa server has an IP of 192.168.0.20 the command might look like this

 shredos_config="192.168.0.20:/:no user:no password:d"

The path is simply /
The username and password fields are must contain no user and no password as tftp doesn't support authentication.
The 'd' means verbose debug mode information is written to the file /transfer.log

	# Search /proc/cmdline for a command such as shredos_output="ftp:192.168.0.2:/home/joe/ftpdata/:jo:488993d:d"
	# Format:
	# shredos_output="protcol:IP_address:path:username:password:debug"
	# where:
	# protocol = ftp, tftp
	# path     =
	# username = username to access remote server, enter nothing between the colon delimiter and shredos will
	#            interatively ask for a username at boot up. Enter 'no user' if the server does not require a
	#            username.
	# password = password to access remote server, enter nothing between the colon delimiter and shredos will
	#            interatively ask for a password at boot up if using a protocol that supports authentication
	#            such as ftp, stfp (but not tftp). Enter 'no password' if the server does not require a
	#            password.
	# debug    = enter 'd' to enable debug mode in some protocols. In ftp mode this enables a detailed log of
	#            communication between ShredOS and the server. Only enable if you are trying to diagnose a
	#            communication error. View the contents of /transfer.log for a transcript of communications.
	#
	# Example:
	# A command to access a ftp server that has no authentication, writes to the default directory and is in debug
	# mode and does not require a interative request for username/password would look like this.
	# shredos_output="ftp:192.168.0.2::no user:no password:d"
	#
	# Adding a trailing :d tells the comms program to log in debug mode.

In addition to shredos_config that reads and writes the nwipe.conf and customers info files to the tftpd-hpa server you can also add the command shredos_output using the same syntax. shredos_output will write the nwipe_report PDF files, dmesg and nwipe_log files to the tftp server.

So you could also add, using the example Ip above,

 shredos_output="192.168.0.20:/:no user:no password:d"

so grub.cfg would look something like this;

set default="0"
set timeout="0"

menuentry "shredos" {
	linux /boot/shredos console=tty3 loglevel=3 shredos_config="192.168.0.20:/:no user:no password:d" shredos_output="192.168.0.20:/:no user:no password:d"
}

[PartialVolume]

@titielgozo I've updated the notes on saving the config and PDFs to a tftpd-hpa server, so makesure you read the updated notes rather than the github email.

[PartialVolume]

After you have ShredOS communicating with your tftpd_hpa server using the shredos_config and shredos_output commands you can then edit the organisation and customer information using the c key from the drive selection screen of nwipe and the organisation and customer information will be saved to the tftpd_hpa server when you exit nwipe at the end of a wipe or if you abort with Control C.

Then when your boot ShredOS from your tftpd_hpa server nwipe will start up with your saved organisation and customer list data.

[PartialVolume]

The only problem you might have is with the path, I don't remember if simply `/' is adequate or whether you have to enter the absolute path, so if your tftp server path is /srv/tftpd/ then you use that as the path.

Try / first then if it doesn't work, try the absolute path to where ShredOS is located on your tftpd_hpa server.

[titielgozo]

I'm sorry I didn't explain myself properly, the tftp fact is just an example because I could see in the other ticket how to do it. It's more how I could add the organization on the pdf certificates by default (the fields already prepared) in the image or in configuration via grub?

[PartialVolume]

You can't do that via grub. If you are booting via USB stick, whatever you enter for organisation via the 'c' key is saved to /etc/nwipe/nwipe.conf on the USB stick as default, so next time you boot via that particular USB stick your organisation will appear on the PDF certificates just as you originally set it.

Also you can set a organisation preview so before the disk selection screen appears it shows you your currently set organisation details, pressing 'a' to accept then displays the drive selection screen.

So in summary your organisation details are saved to the USB stick and restored automatically every time you reboot.

[PartialVolume]

Once you have edited these details they are automatically saved to /etc/nwipe/nwipe.conf so that next time you reboot ShredOS or restart nwipe (on other distros) the organisation details you entered will be restored. i.e they become the default.

Preview screen enabled

Configuration Screen

Edit Organisation

[titielgozo]

@PartialVolume oh okay thaks for all! This week i cannot advance on my project but i prevent you the next week if all working as well!

[titielgozo]

Hello, sorry for the time to give an answer. I am little bit lost because i start the securisation part in the same time. You have totally reason I have to use the totale parameters: "shredos_config="192.168.0.20:/:no user:no password:d" shredos_output="192.168.0.20:/:no user:no password:d"".
My error is from the lease of the dhcp server i don't understand why but i cannot use an exisisting lease ( see the pictures when i start the script /usr/bin/shredos_net.sh ). I don't congrats to have any connection when I'm on the Shred. So for the test i decided to add a static configuration and it's working I congrats the ping to DHCP server.

[titielgozo]

I do an another test :

With one of the server I congrats the automatisation of the DHCP lease into shredOS. But now when I start ShredOS I have during 30 seconds a black screen with a ping test to tftp server but for me the test was not good. After the test I try to ping the tftp server and it's working! So why during the starting of shred the ping not working? Does it use only one interface?

[PartialVolume]

Can you post the contents of the file called 'transfer.log'. Thanks.

[titielgozo]

The file "transfer.log" said he can't ping /. So i understand the case, I have to add the protocole "tftp" in the start of shedos_output and shredos_config so the good construction is shredos_config=“tftp:192.168.0.20:/:no user:no password:d” shredos_output=“tftp:192.168.0.20:/certificat:no user:no password:d”.

[titielgozo]

I had a bit of trouble because your example didn't contain the protocol, but the documentation did. I recontact you on the day for let you know if all is working after my test. Thanks for all an another time you are really very precious dev.

[PartialVolume]

Thank you. Sorry about the missing protocol in the example, I'll correct that just in case somebody else comes across it in the future.

[titielgozo]

So it's working but only for the shredos_config, but about the pdf certificat it's not working as well. Do i have to add something like "mput report*pdf" on the grub?
See my grub :

set default="0"
set timeout="5"
menuentry "shredos" {
	linux /boot/shredos console=tty3 loglevel=3 nomodeset shredos_config"tftp:172.16.8.1:/:no user:no password:d" shredos_output"tftp:172.16.8.1:/:no user:no password:d" loadkeys=fr nwipe_options="--autopoweroff --nousb"
}

I'm getting an error like : "exFAT/FAT32/FAT16 USB drive not found"

[titielgozo]

During my troubleshooting I find the error in dmesg.txt I missed a "=" in the start of shredos_output... I learned too much by you, I find --autonuke and --method for the automatisation of my solution. Tommorow I'm going to end my test and said to you if all was good! Thanks 🥇

[titielgozo]

So now all automatisation was good, I recover the pdf certificat on tftp server so I'm good. I have just a little question, can i add something on the grub to exclude .txt of the output? If I cannot no worry I'm going to do a script.

[PartialVolume]

No, there's no way to exclude the .txt files, other than modify the nwipe_launcher script so .txt files don't get transferred, only pdfs and then build you own version.

[titielgozo]

@PartialVolume Okay thanks for all, so I'm going to modify nwipe_launcher. I added my documentation about errors I encountered during the projet ( sorry i translate from my native language for winning times ). You can close the ticket, I really appreciated you're help!

Problems encountered

Corrupted image

When I try to install ubuntu I encounter a problem, my server boots on the grub and during the installation here is the error:

Impossible to mount /dev/loop1 when installing ubuntu because the image was actually corrupted. I was able to make a new image which worked perfectly.

Problem with isc-dhcp-server conf file

I've tried to get the various services to work, and there's no problem with tftp, but the isc-dhpc-server doesn't work, as shown in the following error message:

The problem came from a syntax error in the configuration file!

BIOS server configuration error

When I try to boot on the server that doesn't have an OS, here's my error message. This is due to the fact that my vm is in legacy mode and not UEFI.

TFTP listening problem

Now we know that the PXE server is able to retrieve the grub, we just need to make it start on the ShredOS iso

This was simply due to the fact that the tftp server wasn't listening to any ip address (conf file not set up) and also that I hadn't authorized port 69.

Boot file impossible to recover

I've come across a new problem that's quite easy to solve because in reality the path should be /EFI/BOOT/bootx64.efi to avoid any problems because the dhcp gives the ip of the tftp server so you have to go through the network location and not the system location.

To solve this problem I simply needed to go to /etc/dhcp/dhcpd.conf and update the path.

DHCP listening configuration problem

pxe-e18 server response timeout :
After creating the problem, I realize that the tftp server is only half-functional because I have transfers and TRANSFER TIMED OUT.
I used tshark to see the different requests. I notice that when I use a client to retrieve or drop a file in tftp, everything works, but when I try to boot via pxe, nothing happens on port 69 (I had set up my interfaces incorrectly).

OS version problem

The current problem is that the pxe server is working, the download is done but I get a black screen. So I decided to test with a usb key that does the same thing (I suspect I've misconfigured my bios).

According to the creator, the problem is with the graphics. He told me to add the nomodeset to the grub.cfg (/srv/tftp/boot/grub/grub.cfg).

Once the nomodeset is set up, the problem is just a little different: instead of no longer displaying video, it blocks the download at 100%.

I've defined the files in tftp, but wouldn't that cause a problem so that the PC can retrieve them if it doesn't have the rights? I'd have to try copying the shredOS file with all rights intact.

For the special boot result on the Dell server, I've found a little workaround: since a character is added at the end of the name of my bootloader, just add a space so that the character isn't included in the name. It will be retrieved by the server, but will not be part of the path because the space delimits the end. This creates an error, but the server manages to retrieve the file.

Error: The kernel does not support 64-bit CPUs.

The error appeared when I configured my server in UEFI, in legacy the problem didn't appear. After a quick analysis of my system, I noticed that I had used a 32-bit version and not a 64-bit one.

Error adding a character

Here's the error I'm getting on 2 different servers with the same configuration (secure boot disabled + uefi mode)

I noticed while editing the configuration file that if I add a space at the end of the bootloader name the y on the dell server is also added after a space so the file is corrupted.

Ping error

When I try to send the certificate via the nwipe automation, it doesn't work despite the fact that I've respected the configuration, because it passes via a ping to check that the server is working correctly, but when we do an ufw deny the ping is blocked.

To unblock it, we can't enter a port number, so we'll have to go to the pre-send configuration file: sudo vi /etc/ufw/before.rules.

Once in the file, just find the lines below and put ACCEPT at the end:

# Allow ping / ok icmp codes output
-A ufw-before-output -p icmp --icmp-type destination-unreachable -j ACCEPT
-A ufw-before-output -p icmp --icmp-type source-quench -j ACCEPT
-A ufw-before-output -p icmp --icmp-type time-exceeded -j ACCEPT
-A ufw-before-output -p icmp --icmp-type parameter-problem -j ACCEPT
-A ufw-before-output -p icmp --icmp-type echo-request -j ACCEPT

Despite solving the problem, the ping still fails.

Ping error - Interface

After 2 or 3 checks, I tried to ping with my shredos by pressing alt + f2 to open a tty, but that's when I realized that the dhcp ip was only coming up during the pxe boot.

So I decided to do some additional tests with my personal computer, which I connected to the network. I was able to see that DHCP was working and sending securely to the network because I was able to ping.

So the problem comes from the transition between the PXE boot and ShredOS as if it didn't ask for the ip again. A solution that I could probably use is to use 2 interfaces because the first is functional with the PXE boot and the second would allow communication for dhcp and certificates but this solution would make troobleshooting more complicated.

First of all, I'm going to check with an ip on my various interfaces, I'm going to compare the mac addresses between the bios and ubuntu to see if the interface where dhcp passes through isn't missing.

However, after checking, the MAC address goes back to the bone because on both sides I find the MAC address 94:18:82:0A:1D:E8.

I'm going to use this MAC address to statically mount the interface to check that it's working properly in hardware. If it's working, the problem is either firmware or a logic problem that means the server can't retrieve the ip from the boot pxe for shredos.

The ping is working, so I'll have to look into the problem.

I've done a cross-test with a second server, which is the same (dl 380 gen 9), and it retrieves the IP address correctly. The problem is either with the firmware or the BIOS configuration.

PDF recovery error

At the end of my project, I noticed that the pdf wasn't being sent to the tftp server, whereas in the other direction, the ShredOS configuration files were being retrieved at each startup and sent back at the end of each wipe.

After a while, I managed to find transfer.log, which is used only for transfer logs (usb or ftp) and in my case I found this error message:

[2024/07/12 13:19:26] archive_log.sh: No exFAT/FAT32 drive found, unable to archive nwipe log files to USB

With a little more analysis I notice that the shredos_output and shredos_config functions add log lines to show the parameters used and that's when I realize that there's no shredos_output.

So I decided to check the GRUB.cfg file and I found the problem, there was a missing “=” character after shredos_output so I couldn't get the parameters.