SPX x Boost Pre-release Testing

[Reiers]

Hello SPX Fellows

We need some feedback on Boost - please fill out the reply card below. If you are not using Boost yet - you will find all the information you need in this post.

🚀 <-------! This post if you are participating in this testing round.
If you aren't able to join for reasons, please reach out to @jennijuju or me on Slack!!!

Overview

The purpose of Boost is to replace the current version of Lotus Markets. It is currently being developed as a standalone repository to allow the team to iterate quickly on a redesign of the system, prove the merit of the new design early and often through internal and community facing demos, and to be releasable as a standalone binary that can run alongside the existing Lotus process.

Alpha Testing

The alpha testing for boost has already been completed with positive results. We have integrated the feedback from the initial testing back into the boost. You can check the results of alpha testing filecoin-project/boost#403.

Pre-release Testing

After the successful initial testing, Boost team is looking to test the product with a wider range of storage providers. We are inviting all the interested storage providers from the community to participate in this testing before the main launch. The objective of this program is to test boost's robustness and performance with different sized storage providers, deal sources and configuration.

Expectations

• Improve the storage deal flow
• Make replay-ability of deal stages much easier when the process experiences a recoverable error, such as "out of funds".
• Reduce state complexity and debuggability through reduction based state transitions
• Provide more flexibility of each deal stage to enable future extensibility/customization
• Improve visibility for Storage Providers into their systems
• Add a simple webui to leverage the new Markets APIs
• Better visibility of fund allocation, deal states including data transfer, and sealing pipelines
• Extensible data transfer
• Online deals support fetching a CAR file over HTTP
• Improve team velocity
• Reduce complexity in deal stage management

Instructions

- No Complete Rollback
❗ Please note that Boost uses a sqllite DB for the deal metadata and logs. Once boost has been enabled, the new deals cannot be rolled back to the lotus market. If you decide to roll back after the pre-release testing, you will lose all the metadata for the deal made with boost. However, this will have no impact on the sealed data itself.

• lotus and lotus-miner need to be on v1.15.2 or higher

• All instructions on using boost can be found on the boost website under migrating from lotus to boost section.
  (For any issues with the documentation, please also include that directly in your feedback.)

Please fill out this report reply card with your results, as well as any qualitative feedback that you may have.

"your minerID here"	YES	NO
Changing deal ask config from UI is working
Adding fund to market actor is working through UI
Deal transfer is working in UI for all deals
Publishing deals from UI is working
All the deal parameters and logs are available in deal detail page
All the legacy deals are available in the UI
Sealing Pipeline displays the correct stage for deal sealing

Metrics	-/+%
Success rate and number of boost deals made (Estuary, Bidbot, boost-client, etc)
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)
Hardware spec of Boost node
Average speed of transfers experienced previously with legacy markets (GraphSync)
Average speed of transfers on Boost
Network bandwidth
Number of deals initially migrated

⚠️ Reminders and Risk Mitigations

- Please backup everything in the repositories before making any changes.
Roll back to legacy markets

- Reporting Issues
Please fill out this template

- Questions
Start here Slack #boost and #boost-help
@LexLuthr will be monitoring the relevant slack channels and Github

- User tutorials
We would appreciate help with user tutorials.
Example tutorial on how to store files with boost

[SBudo]

f0773157	YES	NO
Changing deal ask config from UI is working	X
Adding fund to market actor is working through UI	X
Deal transfer is working in UI for all deals	X
Publishing deals from UI is working	X
All the deal parameters and logs are available in deal detail page	X
All the legacy deals are available in the UI	X
Sealing Pipeline displays the correct stage for deal sealing	X

Metrics	-/+%
Success rate and number of boost deals made (Estuary, Bidbot, boost-client, etc)	95% (103 deals total)
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	87% (360 deals total)
Hardware spec of Boost node	16 vCores, 128GB RAM, 10TB HDD + 128GB SSD (Virtual Server)
Average speed of transfers experienced previously with legacy markets (GraphSync)	10-20Mbps
Average speed of transfers on Boost	150-170Mbps
Network bandwidth	200Mbps
Number of deals initially migrated	13,461

[SBudo]

Currently having a data transfer problem with Estuary where the connection to the peer is unsuccessful:
https://filecoinproject.slack.com/archives/C03CKDLEWG1/p1654062988967799

2022-06-01 16:16:12.4503mshttp-transport: terminating http transfer: exhausted max attempts
err: failed to send http req: Get "libp2p://12D3KooWLV128pddyvoG6NBvoZw7sSrgpMTPtjnpu3mSmENqhtL7": failed to dial 12D3KooWLV128pddyvoG6NBvoZw7sSrgpMTPtjnpu3mSmENqhtL7: * [/ip4/172.31.32.185/tcp/6745] dial tcp4 0.0.0.0:26101->172.31.32.185:6745: i/o timeout
maxAttempts: 15

[SBudo]

This has been fixed by the Estuary team. A few of the nodes were publishing their private IP address as opposed to their NAT'd public IPs

[namoo1]

Meest NAM

f01806491	YES	NO
Changing deal ask config from UI is working	x
Adding fund to market actor is working through UI		x
Deal transfer is working in UI for all deals		x
Publishing deals from UI is working	x
All the deal parameters and logs are available in deal detail page	x
All the legacy deals are available in the UI	x
Sealing Pipeline displays the correct stage for deal sealing	x

Metrics
Success rate and number of boost deals made (Estuary, Bidbot, boost-client, etc)	Estuary 10%, Bidbot 90%, evergreen 90%, Filswan 90%(only error comP mismatch) (TOTAL 1,602)
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	Estuary 10%, Bidbot 90%, evergreen 90% Filswan 90%(Only error comP mismatch)
Hardware spec of Boost node	TR3970x RAM256GiB 2TiB NVME*2 RAID0(dagstore, fstmp) retrieve store 4TiB NVME
Average speed of transfers experienced previously with legacy markets (GraphSync)	Average 5MiB/sec (Retrieve and run "GraphSync" at the same time)
Average speed of transfers on Boost	ToTal 3TiB measured 48hr = Average 17MiB/sec (Only GraphSync run)
Network bandwidth	1Gbps
Number of deals initially migrated	about 150

1. "given data does not match expected commP" Errors occur frequently, but when you "import" again, 90% is successful
2. Warnings like this continue to appear after migration
   WARN dagstore [email protected]/dagstore_gc.go:65 failed to persist shard {"shard": "baga6ea4seaqp2hm24ooa4ddlpf4zdhmxeeppeotb4jp5ror4jn2o4pj4lq772oy", "error": "failed to put shard state: leveldb/table: corruption on data-block (pos=577317): checksum mismatch, want=0x9242b249 got=0xd48a48b7 [file=334947.ldb]"}
3. When "Retrieve" is performed, the following cache file is generated in the ./lotus to take up capacity

1653796588228729182.car
1653796588228729183.car

4. ERROR dt-impl impl/receiver.go:135 received error message on data transfer: stream reset
   It happens often
5. DEBUG rcmgr [email protected]/scope.go:242 blocked memory reservation {"scope": "conn-114360", "size": 786432, "priority": 152, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":0,"NumConnsInbound":0,"NumConnsOutbound":1,"NumFD":0,"Memory":0}, "error": "resource limit exceeded"}
   Happens frequently

[dirkmc]

@namoo1 could you please open separate issues against the boost repo for each of these 5 problems you're seeing so that we can help fix them. Please include repro steps and all relevant logs.

[tmyuu]

filecoin-project/boost#522 (comment)

[benjaminh83]

f02576	YES	NO
Changing deal ask config from UI is working	x
Adding fund to market actor is working through UI		x
Deal transfer is working in UI for all deals		x
Publishing deals from UI is working	x
All the deal parameters and logs are available in deal detail page	x
All the legacy deals are available in the UI	x
Sealing Pipeline displays the correct stage for deal sealing	x

Metrics
Success rate and number of boost deals made (Estuary, Bidbot, boost-client, etc)	Estuary 10%, Bidbot 90%, evergreen 90%, Filswan 90%(only error comP mismatch) (TOTAL 1,602)
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	Estuary 10%, Bidbot 90%, evergreen 90% Filswan 90%(Only error comP mismatch)
Hardware spec of Boost node	EPYC 7282 RAM 512GiB 4TiB NVME*2 RAID0(dagstore, fstmp) retrieve store 8TiB NVME
Average speed of transfers experienced previously with legacy markets (GraphSync)	mostly slow
Average speed of transfers on Boost	Too fast - I do not notice them after migration to boostd
Network bandwidth	1Gbps
Number of deals initially migrated	5569

I can move funds to escrow in UI. I find this option quite useless, as escrow is always getting filled and needs to be withdrawn from time to time. Also I have no idea what the purpose of adding the new PUBLISH_STORAGE_DEALS_WALLET and
PLEDGE_COLLAT_WALLET was, as part of migrating to boostd. I have 10 FIL sitting in each of them, and they are doing nothing.

Deal transfers are not easy to see in UI as they tempt to be fast for storage deals. It would be much more relevant if also having retrieval in there. This is where transfers are hanging.

I almost get no deals from Estuary anymore, so really hard for me to collect data on transfers. Deals from @LexLuthr went right through.

[cryptowhizzard]

f01608291	YES	NO
Changing deal ask config from UI is working	x
Adding fund to market actor is working through UI
Deal transfer is working in UI for all deals
Publishing deals from UI is working	x
All the deal parameters and logs are available in deal detail page	x
All the legacy deals are available in the UI	x
Sealing Pipeline displays the correct stage for deal sealing	x

[jamerduhgamer]

f01851683	YES	NO
Changing deal ask config from UI is working	X
Adding fund to market actor is working through UI	X
Deal transfer is working in UI for all deals	X
Publishing deals from UI is working	X
All the deal parameters and logs are available in deal detail page	X
All the legacy deals are available in the UI	X
Sealing Pipeline displays the correct stage for deal sealing	X

Metrics	-/+%
Success rate and number of boost deals made (Estuary, Bidbot, boost-client, etc)	100% - 269
Success rate and number of legacy deals made (Estuary, Bidbot, lotus-light, etc)	100% - 1
Hardware spec of Boost node	1 TB RAM, 64C AMD EPYC CPU, 3 x 3.8 TB NVMe drives RAID0
Average speed of transfers experienced previously with legacy markets (GraphSync)	Didn't work
Average speed of transfers on Boost	23 transferring deals = ~20-25 Mbps
Network bandwidth	100G
Number of deals initially migrated	0 - brand new boost node