Split MPP by maximizing expected delivered amount #2792
Conversation
As suggested by @renepickhardt in #2785
There was a problem hiding this comment.
Concept ACK (can't give a proper ACK in github on this repo)
As noted in my comment on the code I checked that finding the max should be equivalent to 1/x - 1/(c_1 - x) - ... - 1/(c_n - x). With my poor scala skills I belive the code computes that formular. (Actually https://www.codeconvert.ai/scala-to-python-converter suggested to me that my understanding of the scala code for private def optimizeExpectedValue seemed correctly in which case I think your code does what it is supposed to do.
That being said: I find it extremely elegant to do the binary search in this way to find the maximum of the expected value curve. Kudos to this! I made this less elegant in my eval framework as my code (with evaluating on a grid of 100 points) works with any probabilistic prior and I wanted to check also non uniform priors.
If I understand correctly you also didn't include your local knowledge / belief about remote channel's liquidity into the max expected value computation but only used the already allocated inflight htlcs. (I think substracting the used capacity is correctly)
ps: I am not sure if you get notifications on closed PR's but I commented with two diagrams that splitting of the liquidity which one has certainty about still makes sense.
All that being said: As mentioned on my original PR I would be very delighted and happy if you can share results from the mainnet A/B test of your node and give me permission to utilize them in the publication / paper that I expect to finalize soon. In particular I am curious if the method works as intendet.
Thank you for this patch!
| val mid = (high + low) / 2 | ||
| val d = route.drop(1).foldLeft(1.0 / mid) { case (x, edge) => | ||
| val availableCapacity = edge.capacity - usedCapacity.getOrElse(edge.desc.shortChannelId, 0 msat) | ||
| x - 1.0 / (availableCapacity.toLong - mid) |
There was a problem hiding this comment.
For a uniform distribution between 0 and cap there is indeed only one max in the expected value curve and thus the derrivative on that domain has indeed only one 0.
For two channels I computed that setting the derrivative of the expected value to 0 is equivalent to
1/x = 1/(c_1-x) + 1/(c_2-x)
assuming this generalizes to several channels (because of the product rule of derrivatives for the probability of the path this very much looks like this (I think I computed that before)). Thus I think it is indeed reasonable to look that one only has to check the sign of the term 1/x - 1/(c_1 - x) - ... - 1/(c_n - x) In deed I compared it to my local python code and the results matched also for more than 2 channels
There was a problem hiding this comment.
If we consider the polynomial P(x) = x(c0 - x)(c1 - x)(c2 - x)..., the full derivative is actually P(x) / x - P(x) / (c0 - x) - P(x) / (c1 - x) - P(x) / (c2 - x) - ... = P(x) (1 / x - 1 / (c0 - x) - 1 / (c1 - x) - 1 / (c2 - x) - ...). We know that P(x) is positive on (0, min(c0, c1, c2, ...)) so the sign is the same as 1 / x - 1 / (c0 - x) - 1 / (c1 - x) - 1 / (c2 - x) - ....
There was a problem hiding this comment.
Thanks for laying out the proof. It's funny as I remember trying to find a formular during last summer to solve this analytically and I recognizedc the same structure. But I couldn't non numerically solve for x which is why I stopped there.
However I still wonder how does the computation change if for example the channel c_2 has a non zero low estimate? Evaluating the full polynomial n times to find the max works easily when using conditional probabilities. I have a travel day tomorrow and might sit down and try your approach for the conditional probabilities. I am sure a similar formula can be found.
| @@ -394,6 +394,7 @@ eclair { | |||
| mpp { | |||
| min-amount-satoshis = 15000 // minimum amount sent via partial HTLCs | |||
| max-parts = 5 // maximum number of HTLCs sent per payment: increasing this value will impact performance | |||
| splitting-strategy = "randomize" // Can be either "full-capacity", "randomize" or "max-expected-amount" | |||
There was a problem hiding this comment.
I thought that using "randomize" by default wouldn't change the current behavior as the default sets "randomize-route-selection = true", however the code flips this "randomize" flag in several places. I need to think more about it to not change the default behavior, any ideas?
There was a problem hiding this comment.
That's true, when the default configuration fails, we fall back to a retry using the randomize flag set to true. IIRC we only override this to set it to true, is that correct or am I missing some cases? As long as the first attempts use the configured values, I think it's ok if eclair has a fallback on failure that uses the algorithm we think as being the best option.
|
That PR was unfortunately forgotten as we had a lot on our plate around that time, do you think it makes sense to work on it now and integrate it before the next release @thomash-acinq? I'll spend time reviewing it if you think it's useful to have. |
As suggested by @renepickhardt in #2785