Skip to content

README_Dart3.md

Latest commit

 

History

History
298 lines (189 loc) · 18.7 KB

README_Dart3.md

File metadata and controls

298 lines (189 loc) · 18.7 KB

DART 3

DART 3 is currently in BETA mode. We encourage you to use it for testing and to report bugs and feature requests at https://github.com/APTrust/dart/issues.

Be cautious when using the beta version for essential workflows! Wait for a stable release build if you want to use this in production.

Getting Started

  1. Dowloading the App
Operating System Version Download Link SHA-256 Checksum
Windows (Intel 64-bit) 3.0 https://s3.amazonaws.com/aptrust.public.download/dart3/v3.0/windows-amd64/dart.exe a6e587694a39495ed00337bb8c816146161caf54d3c3c72031496acecb29ed28
Mac (Any CPU) 3.0 https://s3.amazonaws.com/aptrust.public.download/dart3/v3.0/mac-universal/DART.dmg 33f5f634df80b0860bd50c42f58737bb874ebd21179e5acd99b78886df31abd7
Linux (Intel 64-bit) 3.0 https://s3.amazonaws.com/aptrust.public.download/dart3/v3.0/linux-amd64/dart 79ad553782f063f07aca7044cd57d10d587ab38a869fdccade8a7c4f478b1717
  1. Follow the instructions below to start the app on your operating system.

Note that the SHA256 checksum for the Mac app is the checksum of the executable at DART/DART.app/Contents/MacOS/DART. The MacOS version is digitally signed and notarized, so you should not receive any warnings when trying to run it.

Starting DART 3 on Windows

  1. Double click on dart3.

  2. You'll see a message saying Windows protected your computer by refusing to run the app.

Windows protection message

  1. Click the More info link. You'll see the following message:

Windows protection message

  1. Click Run Anyway. You should see a new browser tab showing DART 3.

Installing & Starting DART 3 on MacOS

To install DART:

  1. Double-click on the DART.dmg file you downloaded from the URL above.
  2. Drag the DART app into your Applications directory.

If you want to install DART somewhere else, you can drag the app into any directory you like.

To start DART, simply double click the DART app icon.

Starting DART 3 on Linux

  1. First, make DART 3 executable. If you downloaded DART into your Downloads folder, the open a terminal (aka terminal.app) and type the following:
cd ~/Downloads
chmod 0755 dart3
  1. Double-click on dart3 in your file browser, or simply run ./dart3 in your terminal, and you'll see a new browser tab open with DART 3 running.

Known Issues

Major features are generally known to work in the current alpha build. However, the build has some known issues, including:

  • Trying to package in any format other than BagIt may cause errors.

Notable UI Changes

For the most part, DART 3 looks like DART 2. The one notable change is that DART 2 let you drag in files from the desktop to be bagged. In DART 3, the file browser is inside the application, to the left of the drop zone, as in the image below.

DART job files screen

Click any folder to open it. Drag a file into the drop zone to include it in the bag. Drag a folder into the drop zone to include the whole folder in the bag.

Feature Comparison

DART 3 supports all DART 2 features, plus:

  • gzipped bags
  • unserialized bags
  • local preservation of artifacts such as tag files and manifests

Here's a more complete comparison. We will update this list as necessary with each new alpha release of DART 3.

Last Updated: March 12, 2024

Feature DART 2 DART 3 Notes
Dashboard - Show Recent Jobs Yes Yes
Dashboard - Show Recent APTrust Work Items Yes Yes
View/Edit Application Settings Yes Yes
Import BagIt Profiles Yes Yes
Export BagIt Profiles Yes Yes
Edit/Customize BagIt Profiles Yes Yes
Remote Repositories APTrust only APTrust only
S3 Storage Services Yes Yes
SFTP Storage Services with password Yes Yes
SFTP Storage Services with SSH key Yes Yes
Export Settings Yes Yes
Import Settings Yes Yes
Run Bagging Jobs Yes Yes
Create tarred bags Yes Yes
Create loose bags No Yes
Create gzipped bags No Yes Currently only supports .tar.gz bags.
Run Validation-only Jobs Yes Yes
Validate tarred bags Yes Yes
Validate loose bags No Yes
Validate gzipped bats No Yes
Run upload-only jobs No Yes
Save artifacts after job runs No Yes Includes job description, job result, tag files and manifests.
Create Workflows Yes Yes
Run one-off workflow jobs Yes Yes
Run workflow batch jobs Yes Yes
Help Link Yes Yes
Context-sensitive help links No Yes
View log files Yes Yes Log rotation caused problems with log viewing in DART 2. Fixed in DART 3.
Choose how to deal with illegal characters No Yes Choose how DART should deal with illegal filename characters when bagging and validating.

Platform Rationale

DART 3 is the successor to DART 2. DART 2 is an Electron app that whose maintenance has been time consuming and difficult. We chose to write DART 3 in Go as a locally-running web app with a Wails front end for a number of reasons, including:

  • The tasks DART has to perform can be written much more clearly in Go than in JavaScript. This substantially eases our maintenance burden and makes it easier to add new features. Node's default async model is particularly ill-suited to some of DART's core tasks. (E.g. writing tar files, which MUST be done synchronously.)
  • The Go ecosystem is more stable than the Electron/Node ecosystem. We know this from years of maintaining code on both platforms.
  • Node and Electron often introduce breaking features in new releases, forcing us to abandon and rewrite working code. The Go language and the major browsers rarely do this.
  • The rest of the APTrust ecosystem is written in Go, which allows us to reuse proven code for bagging, validation and file transport. This substantially reduces the burden of having to maintain complex code in two different languages (Go and JavaScript) with identical functionality and behavior.
  • Electron apps like DART use substantial resources. Running the DART test suite consumed about 1.5 GB of RAM. DART 3 uses about 20 MB of RAM and considerably less CPU.
  • Electron builds did not always behave the same way as Electron in the development environment. Spending days of developer time to debug these issues was a poor use of developer time.
  • DART 3 now uses Dart Runner to package, validate and upload bags. This ensures consistent behavior between the two apps.
  • The use of Go in DART 3 will eventually allow us to share core code between preservation services, DART, Dart Runner and the apt-cmd suite of command-line tools.

Separation of Policy and Implementation

At this point, the DART UI is essentially a policy editor. Its job is to help the user construct a valid description of a job: what is to be packaged, what BagIt profile to use, what metadata to add, and where to send it. Once that description is created, DART serializes it to JSON and passes it off to DART Runner to do the work. DART Runner is the worker that implements the policy.

DART Runner has been in use for several years. We created it to run unattended jobs on servers that had no graphics or windowing system, only a command line. The traditional use case was that a user would create a workflow interactively in DART, then export that workflow to JSON. They would then create a CSV file listing directories to bag up, and they'd tell DART Runner to run everything in the CSV file through the workflow described by the JSON file.

This has worked well in practice for the few people using it. Some users have kicked off bag-and-upload jobs of dozens of terabytes that take weeks to run. It all runs unattended leaving the user is free to work on other things.

While we were maintaining DART 2, we effectively had one policy editor (the DART GUI) and two full implementations of the worker. The DART 2 implementation was written in JavaScript and DART Runner was written in Go. The two worked equally well for bags up to a few gigabytes, but DART Runner worked better for larger bags, particularly in the upload phase, where DART 2 would often stall.

DART 3 brings us down to one policy editor and one implementor, all written in the same language.

Security Notes

Because DART 3 exposes the local file system in the browser, it listens only on 127.0.0.1:9797, which means it will not accept outside connections.

DART 3 on the Server

In future, we may offer a "server mode" for DART 3, allowing organizations to run the app on a shared server accessible to selected staff. In that case, it will be up to the host organization to secure the DART 3 instance.

Server-hosted DART could have a number of advantages. If hosted on a server attached to shared drives, any DART user could bag and upload items from any of the shares. For long-running jobs, such as bagging and uploading multi-terabyte packages, users could kick off jobs that won't tie up their own workstations. Large jobs and workflows could run in the background for days while users attend to other tasks.

Prerequisites for Development

  • Go > 1.23
  • Docker (to run Minio and SFTP containers)

Notes for Developers

Testing: ./scripts/run.sh tests

Building for realease: ./scripts/build_dart.rb

Running in dev mode: ./scripts/run.sh dart

Note that running in dev mode also starts a local SFTP server and a Minio server, both in docker containers. DART will print the URLs and credentials for these local services in the console so you can look into them if necessary.

Also, if you want to run DART 3 through the VS Code debugger and have it upload to local S3 and SFTP servers, use this:

./scripts/run.sh services

Then start DART 3 in debug mode in VS Code. From there, you can use DART through http://localhost:8444.

Also, not that a number of tests call util.ProjectRoot(), which actually comes from Dart Runner's util code and points to Dart Runner's project root. If you have installed the Dart Runner dependency (which is listed in go.mod and should) be installed automatically by Go, then util.ProjectRoot() will point to the root directory of your local copy of Dart Runner.

Building for Release

Before cutting a new release, be sure you have done the following:

  1. Installed Wails
  2. Installed Xcode, if you're running on Mac: xcode-select --install
  3. Installed npm: brew install npm
  4. Added AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KE to your environment.
  5. Set up your app-specific password on the Apple Developer site for Apple code signing.
  6. Run wails doctor to ensure that your system has all of the Wails requirements.
  7. Tag the release using git tag -a <version> -m "Version <number>", where version is something like "v3.4" (lower-case v) and number is something like "3.4". You can run git tag -l to see a list of existing tags.
  8. In ReleaseNotes.md, put today's date next to the heading for version. Example: # v3.4 - August 24, 2026
  9. If you're code-signing on Mac, be sure you have a signing certificate and that you have created a local file called codesign.env that is based on codesign_example.env. Note that that file includes instructions on how to set up Mac code signing. If you have trouble running the code signing process on Mac, run ./scripts/diagnose_cert_file.sh for diagnostics.

Now you can do a test run of the the release script by running ./scripts/release.sh --precheck <version>.

If the test run reports no errors, run the full release script: ./scripts/release.sh <version>

The release script does the following:

  1. Builds DART
  2. Signs and notarizes DART on a Mac
  3. Creates S3 folders for the release
  4. Uploads the new binaries to the S3 folders
  5. Prints out a list of release links and checksums
  6. Reminds the developer--you--to update the DART download links in a number of locations. The script prints the locations to the screen and also prints the new URLs and checksums, so you can cut and paste them.

Note that when you run release.sh on a Mac, it builds, signs and uploads only the MacOS version. You will need to build the Wails app on each platform and architecture separately. This means we build Windows amd64 on a Windows amd64 machine, Linux amd64 on a Linux amd64 box, etc. On those architectures, simple run ./scripts/release.sh <version> and the script will do what it needs to do.

Once we have CI/CD set up in GitLab, we should be able to do cross-platform builds using a method like the one described here: https://wails.io/docs/guides/crossplatform-build/

For more on Wails build options, see https://wails.io/docs/guides/manual-builds/. Also note that the build system (and many other things) will be changing in Wails 3. See https://v3alpha.wails.io/whats-new/ for details.

We are currently not code-signing Windows builds because the process of obtaining a Windows signing certificate is prohibitive.

Note on Wails

Wails build is currently working in dart3-wails branch, but Wails drag-and-drop is broken, so we're currently using our own drag-and-drop implementation.

See these open Wails issues:

  • Drag and drop from the desktop did not work through most of the DART 3 development period, which is why we use drag and drop from inside the app window. See Wails issue wailsapp/wails#3563.
  • See also: wailsapp/wails#3949

Live Local Development

To run in live development mode, run wails dev in the project directory. This will run a Vite development server that will provide very fast hot reload of your frontend changes. If you want to develop in a browser and have access to your Go methods, there is also a dev server that runs on http://localhost:34115. Connect to this in your browser, and you can call your Go code from devtools.

Note - When using VS Code, choose the Golang HTML Template formatter instead of the basic HTML formatter for the HTML files in this project. The basic formatter breaks the double-braces that Go templates use to wrap variables and function calls.

Debugging

To debug with VS Code:

  1. Click the Debug icon in VS Code's left nav.
  2. Select "Launch Package" from the list in the upper left corner beside the words Run and Debug.
  3. Click the green Start Debugging arrow.
  4. Go to http://localhost:8444 (a browser window should open automatically with that URL).

Now you can set breakpoints in the code.

Also, use ./scripts/run.sh services to run local SFTP and S3 services during DART debugging.

Working with Dart Runner

DART 3 relies on Dart Runner to package and ship bags. Often, when fixing items in DART, you'll actually be fixing items in Dart Runner. The easiest way to do this is to add the following to go.mod, so it points to your local Dart Runner source code repo. This prevents you having to push code to Dart Runner and then pull it back down into DART.

# In go.mod

// Uncomment this to use local version of dart-runner instead of
// GitHub version. This makes development much quicker, as you
// can instantly pick up changes to dart runner. It's also less
// messy that git submodules.

replace github.com/APTrust/dart-runner => ../dart-runner

require (
	~~github.com/APTrust/dart-runner v0.0.0-20251011144618-6e22996d0b69~~
	github.com/gin-gonic/gin v1.10.0
	github.com/google/uuid v1.6.0
	github.com/stretchr/testify v1.10.0
	github.com/wailsapp/wails/v2 v2.10.2
)

This code replaces Dart Runner from github.com with Dart Runner on the local file system, assuming that Dart Runner is one level up the file tree from DART.

Note that when you build a release, you should get rid of the replace and point to a specific build of Dart Runner.

If you want to change the version of Dart Runner in a dev or release build, use the following command from the command line:

go get github.com/dart-runner@version

Where version is the commit id you want to build with. For example:

go get github.com/dart-runner@87709f079baaefd79d76873b711e2c6b64bec688

That will update go.mod and go.sum as necessary.

Testing

./scripts/run tests

Code Signing for Mac OS

You will need an Apple developer certificate to sign and notarize DART builds so they can run on Mac OS without a lot of hassles and scary warnings. For info on how to get the required certificate and app-specific password, see the codesign_example.env file.

Once you've set that up, you can build the app using the Building instructions above. After the app is built, if your customized codesign.env file is ready, then just run this script from the project's top-level directory:

./scripts/mac_sign_and_notarize.sh

It should only prompt you for the password for your .p12 certificate file. Note that the notarization step of the signing process often appears to hang for 30 seconds - 10 minutes. That's part of Apple's process.

When the app is fully signed and notarized, it should appear here in the top-level directory with the name DART-signed.dmg. You should rename it to DART.dmg before uploading it to S3 for public distribution.

NOTE: You may want to run the signing script from a terminal window instead of from VS Code's built-in terminal. Running from within VS Code will cause Mac to prompt you to elevate VS Code's permissions, which may be dangerous.

Eventually, we will do code signing at the end of the CI/CD pipeline. For instructions on that, see https://wails.io/docs/guides/signing/.