Evaluation of direct integration of Seagate openSeaChest

[fthobe]

Hey @vonericsen, as discussed it's my pleasure to introduce to you nwipe, historically forked from DBAN. nwipe is one of the widest, if not the widest, used open source application to wipe drives including a digitally signed PDF proof, sharing the open source spirit of OpenSeaChest in many ways (but I think @martijnvanbrummelen and @PartialVolume are better suited to explain their projects). ShredOS (maintained by PartialVolume) bakes nwipe into an easy to use bootable ISO providing probably the most effective way to use and apply nwipe.

@martijnvanbrummelen @PartialVolume ,
I would like to grab the chance to introduce you to @vonericsen , he's the maintainer of OpenSeaChest, as you probably know the official command line application for managing, among other features, the sanitisation features of Seagate drives. @vonericsen helped me a lot while writing the documentation related to #597 , I think in all of this process I haven't found anybody more straightforward than him. He's contributing in official Seagate capacity since 2017 to SeaChest, practically the only manufacturer made open source tool predating the NVME-Standard.

As discussed several issues exist for the reliable wipe of SSDs using pre-nvme standards and the reliability / compliance of the drives with standard sas / sata commands to wipe them. @vonericsen was kind enough not only to help me to get the public seagate documentation updated, but also to extensively answer many questions.

I was hoping we could use the chance to tackle at least for one manufacturer the most important points, given that @vonericsen agreed to spare some time:

1. would it make sense to implement open sea chest features into nwipe to have higher security on wipes?
2. can we in some way obtain a list of drive model numbers from Seagate that reliably respond to sata / sas / NVME commands to wipe drives;
3. just chat to get to know each other, as that's always nice, and figure out other issues that might have come up (surely all of you have some in mind).

In a perfect world nwipe / shredOS could benefit from manufacturer tool backed assured sanitisation.

[vonericsen]

Hello!

A quick bit more about myself. I have been working at Seagate since 2012. I've been developing customer tools for OEMs and Datacenters as well as end users since that time. One of the first things I was tasked to take over and maintain was SeaTools for Windows.
One of the first features I was tasked to add to SeaTools was expanding the data erasure support to add Sanitize, ATA security Erase, and SCT write same. I also added the "Erase" button and decision making to choose an erase within Seatools back then.
SeaChest was started around 2014 (before it got named SeaChest) and I've been working on it since then as well as the work to create openSeaChest.

I'm still involved in a lot of discussions around erasing/sanitizing data and am participating in discussions with the Circular Drive Initiative about how they can leverage openSeaChest to do data erasure and health evaluations as well as using other opensource tools.

can we in some way obtain a list of drive model numbers from Seagate that reliably respond to sata / sas / NVME commands to wipe drives;

I do not have a way to get a list of model numbers for what drives support what features. The way we write SeaTools and openSeaChest today is to query the drives for what features they support. This is what I have been recommended to do by the representatives for the T10 (SAS), T13 (ATA), and nvmexpress committees. The reason for this is so that it works with any drive without needing a massive model number table to keep up to date.
This has been very successful for us not only with Seagate products, but with non-Seagate drives we've tested as well. We don't get a lot of non-Seagate drives to test (other than a drive included with a new test system or maybe some engineer has one in a home system or a customer reported something to us), but many of our customers have requested that our tools work on non-Seagate drives as well, so using the ability to query capabilities has been the best way to meet this requirement.
openSeaChest will take issues reported from running and having errors with non-Seagate drives, we just don't always have the ability to get that specific drive to debug it. Also our debugging may be limited compared to Seagate products where we can just ask the firmware teams for help as needed.

To help with this we have openSeaChest_Erase -d <handle> --showEraseSupport to figure out what a device supports. Currently this outputs a list from fastest to slowest, but it also lists whether a command is a clear or purge.
We are currently working on some additional logic to add to this that checks what OS and adapters may block some capabilities preventing them from running (for example Windows 8 and later block sanitize...but NVMe has had some support added in Windows 10). These additional lookups will be part of the output from this option in the future. Linux generally doesn't block commands but there are other known issues with USB adapters we plan to take into account going forward.

I do have some knowledge about what Seagate firmware includes these days and some past history that may be helpful.
All SATA drives include support for ATA security and thus ATA security erase including the enhanced security erase. I'm sure if you go back far enough some really old drives may not

I'm also familiar with other tools like hdparm, sg3utils, smartmontools, and camcontrol since we do have customers that want to utilize those as well and we try to include the same or similar capabilities to these when we can within openSeaChest.

I will do my best to help answer your questions and also will consider any feedback or requested changes that may help with this!

[PartialVolume]

@vonericsen Pleased to meet you, I think openSeaChest will be a useful addition to ShredOS. OpenSeaChest doesn't appear to be supplied in buildroot as standard so I'll need to configure the ShredOS build process to download and compile it from https://github.com/Seagate/openSeaChest so that the latest version of OpenSeaChest is built into ShredOS.

In regards to utilising OpenSeaChest into nwipe, I'll need to get familiar with OpenSeaChest first and see if I can utilise the utilities or code into nwipe for ATA secure erase specifically. I've got plenty of Seagate drives to test on from the last twenty five years!

[fthobe]

Some background for @PartialVolume @martijnvanbrummelen
Back then we had a discussion regarding the reliability of feature availability responses:

Written by me:

That's immensely helpful. IEEE-2883r2022 is in large part built on NIST 800-88, NIST 800-88 also asks for the method (overwrite / block erase / crypto erase / other physical destruction options), is there any chance that information could find it's way in --showerasesupport as well?

Written by Tyler

I can look into what else we can put to describe each erase. It's been a while since I last reviewed NIST 800-88.
Would it be helpful just to make "This is an overwrite", "This is a cryptographic erase", etc?
With Sanitize this is easy, but ATA security is complicated due to vendor unique behavior on enhanced secure erase.
NVMe's format secure erase may also get complicated if the drive supports crypto erase since a request to crypto erase will do crypto erase and a request for user data erase is left to the vendor to decide if it also does crypto erase or something else.

I am most worried about reliability on older drives, I understand that newer drives return relatively reliable responses on feature polling, it's more S(ATA) drives I am worried about. Back then I tried to write also to T10 & T13 and it was complicated to get any reply.

[vonericsen]

@PartialVolume

In regards to utilising OpenSeaChest into nwipe, I'll need to get familiar with OpenSeaChest first and see if I can utilise the utilities or code into nwipe for ATA secure erase specifically.

We have our code broken into a few parts/sections that may be helpful for you to understand as you look into it:

• opensea-common this library contains common types, macros, functions to help us work on different OS's. It doesn't really get into any drive specific code here...more general purpose.
• opensea-transport this library is where we do all the issuing of passthrough commands for ATA, SCSI, and NVMe over different interfaces (SAS/SATA adapters, USB adapters, etc). It has sections for ATA commands, SCSI commands, and NVMe commands as well as various helper functions, definitions, and macros for working with different types of drives.
• opensea-operations this library contains more test routine type of stuff. While opensea-transport may have the ATA security erase commands themselves defined there, all they do is issue the command. In this library it uses those to create a full routine to assess capabilities and current ATA security state, then run the erase. Or with something like sanitize it can issue the command the poll for progress until completion.

The top openSeaChest project is just the CLI level to talk to these other libraries.
We set it up this way for reuse without needing to script to and from a utility. SeaTools of today uses these libraries as do some other internal test/development tools that other teams inside Seagate write.
The library separation is probably not perfect....it was originally all one library in the very early days, but this is how we've tried to separate it for many years now.

@fthobe

I am most worried about reliability on older drives, I understand that newer drives return relatively reliable responses on feature polling, it's more S(ATA) drives I am worried about. Back then I tried to write also to T10 & T13 and it was complicated to get any reply.

How old are you concerned about going back to?
I have worked with old drives on my own time to understand how ATA has evolved over time and preserve compatibility as much as I can.
The oldest drive I have tested is from 1991 and doesn't even list compliance with any standard and I was able to get it to work. Well as much as it would respond anyways since it only has CHS mode and no DMA support...this also predates ATA security, so it was issuing writes to each sector to erase it.
I've also tested some old drives some engineers had sitting on their desk like a Quantum Big Foot...that old 5.25" drive was fun and helped me identify a few other quirks to work around.
My experience with old drives and reviewing history of early drives is that as long as they report compliance to at least ATA-3 usually things work as the standards define.
I've also tested some old Maxtor and Samsung PATA drives I have at home and got out of old PC's from family and friends and they all work as I would expect them to for the features they report (ATA security erase worked on all of them when I sanitized them for the people who gave them to me).
Today when I run into quirks it's usually with USB adapters, but I have some improvements on a branch to include more retries that I think will mitigate that impact a little bit more when I finish testing it (hopefully within the next couple weeks it will be merged).

[PartialVolume]

@vonericsen Thanks for taking the time to explain the overall architecture, that's very useful and points me in the right direction. Much appreciated.

[vonericsen]

@PartialVolume,

No problem! Please feel free to ask questions, suggest improvements, report bugs, request improved documentation etc. We are happy to work on making things better where we can!

[fthobe]

Hey, tons of question from my side :)

@vonericsen

License
Is there anything special that needs to be considered regarding the partial or full use of open seachest? I saw you people have very permissible licenses <3
@PartialVolume if you want to use seachest in the ISOs we also need to provide instructions on how to get to their repo in the license file and extend the current license files with pointers to other applicable licenses.

seagate vs other T10 / T13 / NVME Drives*

You wrote:

I'm also familiar with other tools like hdparm, sg3utils, smartmontools, and camcontrol since we do have customers that want to utilize those as well and we try to include the same or similar capabilities to these when we can within openSeaChest.

How much does how you sanitize / wipe your own drives currently differ from let's say purging or clearing a 3rd party drive with hdparm or sg3utils?

I am less concerned about spinning disks as they always had a very high entry level barrier regarding the technology required to manufacture them. With SSDs we struggle a lot: while WD, SanDisk or Samsung seem to be very responsive, what drives me crazy frequently are lots of notebooks arriving with internal IT having "upgraded" to Chinese after sales disks with inconsistent firmware behaviour (like identical "unique" identifiers across multiple drives) and big blue after their storage exit stopped providing any documentation what so ever. My attempts to get hand on any of the manufacturer Linux tools was not fruitful either (I have to admit that Samsung took big steps there from having nothing to having a decent Linux cli, even if not open source). Do you have some tips here (especially regarding non major brand stuff).

[vonericsen]

@fthobe

Is there anything special that needs to be considered regarding the partial or full use of open seachest? I saw you people have very permissible licenses <3

openSeaChest is distributed under the Mozilla Public License Version 2.0
I am an engineer, not a legal person so you may want to consult with someone with a legal background, but what I understand is the MPL2 was chosen to allow customers to use these tools freely and the code more freely than under GPL (some datacenters did not want GPL code from us so MPL2 was chosen instead).
My understanding of MPL 2.0 is you can use it however you want and do not need to redistribute source code UNLESS you make changes. Changes must be published.

MPL 2.0 is also GPL compatible which seems like this would not be an issue for nwipe looking at its license that is checked in.

One note is I have a branch in opensea-common that will be pulling in some BSD 3-clause code that I modified to add some functionality. The modifications will fall under MPL 2.0 as the rest of our code does.
My understanding is this will still be GPL compatible as well.

If you were to use closed source SeaChest from seagate.com that has a different license, which is Seagate's end user license agreement....I believe it is the multi-user variant of the license, but I would need to verify that.

How much does how you sanitize / wipe your own drives currently differ from let's say purging or clearing a 3rd party drive with hdparm or sg3utils?

I use openSeaChest on basically everything personally. haha
I've reviewed the code in these other opensource tools, and they all issue the same ATA/SCSI/NVMe command from the standards as I would expect them too so they should be functionally the same since they use the same low-level OS passthrough to issue the commands that openSeaChest uses.
It's been a while since I've ran these tools for erasure rather than other features, but they are doing the correct thing from what I have reviewed when it comes to ATA security, sanitize, format unit commands.
On ATA/SATA and NVMe some of the feature detection is a bit easier than with SAS/SCSI. This has improved overtime, but

what drives me crazy frequently are lots of notebooks arriving with internal IT having "upgraded" to Chinese after sales disks with inconsistent firmware behaviour (like identical "unique" identifiers across multiple drives) and big blue after their storage exit stopped providing any documentation what so ever. My attempts to get hand on any of the manufacturer Linux tools was not fruitful either (I have to admit that Samsung took big steps there from having nothing to having a decent Linux cli, even if not open source). Do you have some tips here (especially regarding non major brand stuff).

I have not run into these myself, but I have seen similar things happen with the way USB adapters report (they almost never report the drive's unique identifiers as I would expect them to, so you have to use SAT passthrough to get that information).
I have also seen the fake capacity drives that report something much larger than they really are (which is so very wrong to deceive people on its own but when it's also data that is important that makes it so much worse)....I have ideas on how to test this, but I don't want to send these people any money of my own to write this test.

All of the T10, T13, and SATA-IO standards can easily be purchased and be used by anyone. I know there are also lots of draft versions around the web too, so anyone can easily find them and use them to implement a new device.
If the worst thing happening is a bad unique ID, that is not great, but probably won't affect other functionality much.
I know that all the big names participate in these committees for the most part, but occasionally there is not a representative for a manufacturer. What I have heard from our committee representative is they want participation from others on these to work out issues and create better definitions.
One thing to keep in mind is when a drive is reporting capabilities this is meant to instruct not just the operating system but other software what the device can do and what commands it can process so that you avoid sending a command that causes an error return.
I think it's reasonable to expect these bits to report correctly in terms of command acceptance....at least until proven otherwise.
We implement workarounds as necessary in openSeaChest and openSeaChest is not alone. The linux kernel's libata has workarounds for known drive problems too and I've seen workarounds inside USB adapter firmware as well.
Another thing from my experience that might help is knowing that each additional feature costs more money to implement. Whoever is implementing these drive's firmwares is likely to skip anything that is going to take more time or money. Implementing the minimum on a cheap SSD is probably going to amount to: report some identifying data, report some SMART attributes that are common, capable of read/write/verify to all LBAs, accepting the TRIM command, maybe ATA security since that is baked into a lot of BIOS/UEFI firmware for a password...but no guarantees on the enhanced erase capabilities, but the basic one will likely be there.
There are different ways you could filter on misbehaving drives. Most common would probably be Model number + FW revision.
You could do some vendor detection as well if it's just one vendor reporting bad data if it came down to that.

openSeaChest has one place to filter capabilities but it's largely focused around USB adapters and one PCIe adapter that was blocking a command. We have another place to do some vendor lookups for the brands Seagate has acquired in the past (for example Samsung HDDs and Maxtor and Quatum since Maxtor bought them, etc) but that doesn't filter out capabilities for commands right now since we haven't had reports about misbehaving devices around these things.
Generally when I run into "hey, this didn't work on this drive" I try to look at what happened on the low-level interface/data to figure out what went wrong or if a capability bit was interpreted incorrectly. Often times I find it's a feature interaction issue (which are more difficult to figure out since it's a nitty-gritty detail somewhere in the spec). I was reported an error last week on a drive that was bizarre, but I found retrying with a different method to accomplish the same behavior worked....so I added a retry so it would work with any drive that does the same thing again. I find a retry to be better than a filter, but it depends on the issue.

If you have any more specific examples for these drives and sanitizing data, I am happy to help look into it or try making adjustments in openSeaChest.
When it comes to data-erasure you may even want a "waterfall" approach which is similar to what I did in the old SeaTools for Windows. In that case it collected all capabilities, then chose the best & fastest one to run.
A modified version of that would be to start by collecting all available options and starting with a purge. If it fails, try a different purge, etc until the old classic overwrite with a bunch of writes across the entire drive.
If the drive supports something like DCO/HPA/AMAC then try to restore that first, but if that fails you can throw an error to tell the user it didn't work if you suspect that a drive may not be at its full capacity.
FYI in these features they all have a freezelock which might be a "try using a USB adapter or a different system" since that command can be issued by the BIOS/UEFI depending on how it's designed (I have run into some laptops and desktops that issue these as it's past the bios and before booting the OS).

If the issue is trusting that these devices did an erasure then the next best thing is to run a verification that it is erased.
NIST800-88 and IEEE2883 have some recommended verification procedures in them depending on the erase performed.
Seagate has implemented this; however it is not in openSeaChest at this time (only closed source SeaChest).
One note is IEEE2883 recommends verification after each data erasure...so if you were to combine multiple techniques it is recommended to verify after each one that is performed.

Let me know if this helps or I missed something. If you have the ability to test some of these drives and find an issue, let me know and I can help look into it a bit more. The -v 4 is more helpful in this type of debug since it spits out the raw responses from the drive which would be more helpful to figure out the issue.

[Knogle]

This is excellent content, I think it might contribute to the project in many ways! unfortunately, regarding the license. MPL 2.0 is not compatible with GPLv2-only because the MPL requires modified MPL-licensed files to remain under the MPL, while GPLv2-only requires all combined work to be under the GPLv2. These requirements conflict, making direct integration impossible. However, they can coexist in the same project if the code remains in separate files, as the licenses would apply independently. For integration, dual-licensing or moving to GPLv2-or-later/GPLv3 would resolve the conflict, as MPL 2.0 is compatible with GPLv3.

[PartialVolume]

<------ Over at ShredOS, Commit PartialVolume/shredos.x86_64@e05541a

Added Seagate's openSeaChest tools, it actually took less space and time than I was expecting, about 5MB in size so only needed to increase the image size to 310MB leaving about 25MB free in the partition for PDF certificates/logs etc.

I've not tried playing with any of the tools yet so if anybody wants to try it out, it's now committed to the master. I only need to fix a network issue for Dell USB ethernet adapters and then I'll publish a official release with .iso's and .img's.

I'm aiming for a release by the end of the week but we'll see how it goes!

[fthobe]

I've not tried playing with any of the tools yet so if anybody wants to try it out, it's now committed to the master. I only need to fix a network issue for Dell USB ethernet adapters and then I'll publish an official release with .iso's and .img's.

What's the problem here?

[fthobe]

@vonericsen

If you have any more specific examples for these drives and sanitizing data, I am happy to help look into it or try making adjustments in openSeaChest.

If you want I can send you some.

[vonericsen]

@fthobe

If you want I can send you some.

Before doing that, can you give me some examples of brands or something I can search and look into buying/finding within Seagate?