New secret links design and huge speed ups

A brand new and much better design for secret links. Previous secret links were pure capabilities in the URL. This meant revoking them required rotating all the keys which is expensive. They are also very long, e.g. https://peergos.net/#%7B%22secretLink%22:true%2c%22link%22:%22#6MDZhRRPT4ugkJuUfcWtaZodN5QYzkZKJtHpDHomFJrVhNSZysiFYimpgtcA2F/6MDZhRRPT4ugkJuUfcRzRbPpFimcBNJx2N9TJDnL4W3ETYhwdsWdvgCkXkwipF/FCYSFhpQ1xD2cydr6CFQ6UwFkgB82pWReAUzKVDxe4KA/5Pf7SvCG1mMtui2aPd9F3SH2wdwsPro1GxTa7VfxkWrj9XQGAUB%22%2c%22open%22:true%2c%22path%22:%22/demo/%22%7D

The new secret links require the assistance of your home server (or a mirror). The capability that was in the URL before is now encrypted and stored in a new champ on the identity writer data. There is a new server API to lookup a secret link and return the ciphertext. This is then decrypted locally using the hash in the URL and optionally an additional password supplied by the user.

This means the URLs can be much shorter (4x), e.g. https://your-domain/secret/z59vuwzfFDoy1Mya69m7j1nZHL7hH4yLpAN7qKQusEUREveNVJifs9R/1368969#cWNWXq9jPahK

The new champ includes the mirror bat in its root block so the champ itself is not public, meaning the number of secret links you have is not revealed to anyone but your server and mirrors.

Because we are using the server, we can get it to enforce expiry times or n-use links. Revoking these links is instant because you just need to delete the mapping from the champ. We also allow writable secret links, with a suitable warning that holders of the link could fill your space quota (until we implement subquotas). Links can also had additional password protection.

If you migrate to another server, your links come with you and all pre-existing secret links will continue to work. There is no limit to the number of links you can create, because they are just normal blocks that contribute to your space usage.

The password based encryption on each link is designed to take 100 years to brute force if you have 1,000,000 GPUs, each of which can do 1M scrypt hashes per second.

Uploading small files has been optimised by uploading cbor blocks and raw blocks concurrently, rather than in 2 steps. Also by grouping small raw blocks and writing them directly to Peergos server, rather than S3 in direct S3 case.

The reason this is so beneficial is direct S3 uploads from browser are constrained by http1.1. This means we can only upload 5 blocks concurrently over the 5 connections provided by the browser.

Features:

• new secret links design with:
  • expiring secret links
  • n-use secret links
  • password protected secret links
  • writable secret links
  • instant revocation
  • supported when migrating server
• mkdir in shell CLI
• fullscreen support in html browser
• support large file streaming in apps
• double click to open in any installed app
• open json files in text editor
• move built-in tasks app to an installable app
• ebook reader app
• support downloading all your data in a single zip file
• ease installation of recommended apps supporting a given file type

Optimisations:

• Local S3 small block buffer (50x faster small writes)
• 10x faster signing and signature verification (use webcrypto Ed25519 when present)
• 2-3x faster bulk small file/folder upload
• 20% faster large file upload
• cache profile pictures in newsfeed

Fixes:

• support wasm mimetype
• fix failed uploads in a different writing space
• fix an edge case in revoking write access
• Support general mimetypes in the gateway for public websites

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/3ef5e383e9fe01ee2710b13ab15b7e733bae5e52

Local peer discovery

Peergos can now discover peers on the local network (LAN) using mdns. This means two peers on the same LAN can friend each other and share files etc entirely offline.

Features:

• local peer discovery using mdns
• launch recommended-app for secret links to supported formats (docx, clsx etc.)
• make server configurable in writable-link generator cli

Optimisations:

• cache pointer targets
• improve newsfeed performance

Fixes:

• bulk mirror of raw blocks
• offline login
• fix stay logged in with MFA enabled
• remove any cached login data if MFA is enabled

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/f6788b5a3366148e21dcf70d56a5a735e4c6243b

60x faster mirroring, chat API

The initial pki sync is now 60x faster (down from 10min to 10s). The pki sync also happens in the background now. There is a new chat api enabling fully fledged 3rd party chat apps.

Features:

• Extended chat API for apps
• Background pki sync
• Show profile picture in drive header
• Improved newsfeed design
• Thumbnails displayed in picker dialogue
• Directly launch specific apps from search results
• Additional app permission to retrieve profile photos

Optimisations:

• Mirroring data is about 60X faster now

Fixes:

• Use direct S3 blockstore when available
• Fix bulk chat invites
• Improve S3 error handling

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/04354df8a322502916e792a496b1e7604c84a9a0

Server identity rotation and internationalisation

Peergos v0.16.0

Servers can now rotate their identity keypair independently of the users on that server. Clients verify the rotation using signed IPNS records if they update their host, which are periodically republished. Internationalisation support is included with simplified Chinese supported already.

Features

• remove 2 GB limit for size of zipped folder that can be downloaded
• server identity rotation (and CLI)
• select all/none

Optimisations

• make GC about 100x more scalable by storing reachability and block list on disk instead of ram

Fixes

• fix native image builds
• handle OPFS returning empty blocks
• remove excess space at bottom of login page
• fix race condition with repeated updateSocial calls during login
• fix multi-select in UI
• fix occasional failures to resolve a peerid in dht
• miscellaneous web ui improvements
• context menu dark mode

Mirrored at https://peergos.net/public/peergos/releases

Release commit:
web-ui/512532b3a2e280f5d15e0b89bc8a050f050b76f4

Latex support, 15x less bandwidth

Peergos v0.15.0

Large performance improvements after switch to Nabu. 15x reduction in baseline bandwidth usage. Now ~12KB/s is typical. Also 60% less CPU usage. The size of peergos.js has also been halved by a bug fix in jpack!

Features:

• Latex support in notes app
• Latex support in markdown app
• Support UTF-8 in text mimetype detection
• Usage CLI for server operators
• Authed usage call

Optimisations:

• Don't query blockstore when calculating usage for users that haven't changed
• Add local cbor block cache for S3 to make champ gets 10x faster
• 2x less RAM and bandwidth usage in Kademlia
• Huge reduction to kademlia routing table size by not adding non-server peers
• Don't call identify or kademlia on incoming connections for peers we already know
• Reduce baseline bandwidth usage from 1.5Mb/s to 0.1Mb/s by switching to a new bitswap protocol ID
• Fix Jpack to halve size of compiled peergos.js (2.1 MB to 1.1 MB)

Fixes:

• Handle S3 internal errors as rate limits
• Correct usage update during migration to new node
• Improve information for GC errors
• Make GC tolerant to non existent blocks
• Handle .apk mimetype
• Treat http connection refused as a rate limit
• Darkmode fix for sub pages in md viewer
• Fix opening depth 2 sub-pages in md viewer

Published at https://peergos.net/public/peergos/releases

Release commit:
web-ui/ac4bba146e5faad4f98ab9ca1b5aebd330823667

Notes app + bug fixes

A new app and a few bug fixes from the previous major rewrite in v0.14.0.

New apps:

• notes

Features:

• ipfs metrics for S3 calls, bitswap and kademlia bandwidth
• decimal storage sizes

Optimisations:

• cache peer id in AuthedStorage
• inline block request authoriser (no separate allow listener)
• block aggressive bitswap peers
• don't retrieve blocks to auth a bitswap request for them

Fixes:

• timezone for block auth generation
• use bare peer id in bat auth
• always persist generated ipfs identity
• make sure block metadata db is populted for file blockstores also
• handle empty block files
• filter blocks of wrong type in listing blockstore

Release commit:
web-ui/163f59fcbbff6e3cf2814de985b7bba5c0585e97

Java IPFS implementation (Nabu), block metadata db, versioned S3 buckets.

This is an exciting new release that means we finally control our whole stack as we switch to a Java implementation of IPFS (that we wrote). We introduce a blockmetadata store that stores for each cid the size, links and BATs. This massively speeds up GC, space usage updates, and read authorizations.

Features:

• Switch from Golang based ipfs-nucleus to Java based Nabu
• Support versioned S3 buckets
• Introduce block metadata db to massively speed up many operations
• infini filter (infinitely expandable bloom filter)
• only use authWrites for raw blocks, and send the BATs for these
• store BATs in block metadata store

Privacy improvements

• don't broadcast block retrievals to DHT or bitswap
• don't provide blocks to DHT unless we are running a mirror (also reduces CPU and bandwidth)

Optimisations:

• 10x less bandwidth usage when idle
• faster peer dials to get blocks
• less CPU and RAM usage
• inline p2p block get authorization
• block repeated block get attempts with invalid auth
• huge speed up to cacheless delete (e.g. in incognito mode)
• use owner to lookup peer to retrieve block from in the common case
• improve sign up oplog from 10 pointer updates and 29 blocks to, 2 pointer updates and 10 blocks
• upgrade HTTP client in JavaPoster

Fixes:

• timezone issue in BAT verification
• fix handling of writable secret links
• Bulk delete on B2 (which is not S3 compliant)
• CAS exception making social post
• support Ed25519 PKI peer id
• event loading in calendar

Release commit:
web-ui/59c4fa5820dd7964a70d992c7bec97ff252e98ea

v0.13.0

Two factor auth (2FA), Multi-file-select, new apps

Features:

• 2FA including TOTP and webauthn
• select multiple files/folders to {delete, download, cut, copy, paste}
• enable quic, noise
• require token to signup if not only listening on localhost
• support HEIC and JPEGXL image mimetypes
• require Java 17
• build docker image
• add delete chat and setTitle to chat API
• human readable file sizes
• new and improved gallery with media controls and prefetch
• app grid on launcher page
• file picker api for apps
• export to PDF in paintz, ck-editor and tui-markdown editor apps
• app to convert html to docx
• new draw.io app

Optimisations:

• download chunks in parallel (much faster downloads)
• make bloom adds async
• bulk block writes for small blocks (speed up deletes, uploads)
• parallelise block puts to S3 in bulk write
• bulk delete
• add OPFS (Origin Private File System) cache for blocks in browser

Fixes:

• fix svg mimetype case
• allow wildcard listen addresses (fixes docker)
• fix immediate login after changing password in browser
• many miscellaneous fixes

App-store (recommended apps), full-text search

Peergos v0.12.0

App-store (recommended apps), full-text search

Features:

• full text search of files
• implement a bulk upload progress bar, rather than one per file
• add "app store" for easy installation of recommended apps
• allow secret links to open apps
• enable noise security protocol for future compatibility
• support available apps in the newsfeed directly
• show available apps for opening a file in context menu

Optimisations:

• make GC 60x faster using block metadata store
• make usage calculation 60x faster using block metadata store
• increase buffer size when streaming video in browser
• improve offline champ lookups

Fixes:

• miscellaneous reliability fixes
• mirroring to S3 based node
• legacy directory modification
• granting write access
• better offline detection
• timezone handling in recurring calendar events
• better concurrent file modification handling

Download binaries from https://peergos.net/public/peergos/releases

Offline login and viewing of files

Peergos v0.11.0

Offline login and viewing of files.

Features:

• include retrying mutable pointers, and extra feedback during CLI upload
• add chat api for peergos apps
• support local app dev via a proxy
• support http HEAD and OPTIONS in applications
• add offline block and pointer cache in browser (using indexeddb)
• add ability for apps to ask user to save a file
• peergos is now a Progressive Web App (installable from browser on sensible OSes)
• 100% offline login and viewing of recent files

Optimisations:

• reduce S3 read auth bandwidth by 80X to 10,000X
• parallelise usage calculation
• constrain CPU usage of ipfs-nucleus
• big speedups across the board by always using a buffered network access
• speed up news feed building and retrieving shared files by hoisting pointer gets
• halve S3 read latency from browser (no CORS preflight)
• make bulk import of calendars 30x faster
• use bloom filter in S3 case to reduce request rate by 1000X

Fixes:

• auto open for public link to a single file
• paste file bugs
• uploading huge videos through Java CLI
• include https support in native images
• support dash encoded mp4 videos
• fix space usage growth - now uploading and deleting a file returns you to prior usage
• add a transaction store GC
• fix build on windows
• allow empty summary in calendar import
• make calendar import idempotent
• fix duplicate progress bar bug during multiple file upload
• handle S3 rate limits during signup
• delay updating progress bars until blocks have been flushed from buffered storage
• handle video thumbnail error causing upload hangs
• htmlviewer deep links

Download binaries from: https://peergos.net/public/peergos/releases