Pangolin performance via Newt

[steuerlexi]

Hey everyone,

First of all, thanks for the great work on Pangolin – I really like how clean and straightforward it is to set up.

I’m currently testing Pangolin in a fairly performant setup:
• Home connection: 1 Gbit fiber
• VPS: 2.5 Gbit throughput
• Service behind Pangolin: Pingvin Share (self-hosted file sharing tool)

The problem I’m running into is related to throughput performance. When I upload a file to Pingvin through Pangolin, I only get about 4 MB/s (≈32 Mbit/s), and the speed occasionally drops even further. The same happens when I try to download the file back – it’s consistently low and doesn’t reflect the bandwidth I actually have on either side.

So now I’m wondering:
Where’s the bottleneck? Is it:
• A known limitation or default setting in Pangolin?
• Related to WireGuard performance under load?
• Possibly something in the configuration (like MTU size, buffer sizes, etc.)?

I’d really appreciate if someone could share similar experiences, tuning tips, or diagnostic steps I might try.

Thanks in advance!
Alex

[steuerlexi]

Is nobody else experiencing this problem ?

[AstralDestiny]

Can't replicate it..
For me pangolin is on a dedicated host colo, Not the fastest speeds it's only 1Gbps/300mbps, But cheap for a remote system,
at the test network with Newt, Newt runs in a ipvlan so it skips all of the host's iptables and other routing and talks directly with the host's nic so it can go line speeds uninterrupted (ipvlan in l2), Goes out to the switch then hops over to the host running Pingvin Share, Can upload at 50Mbps which is the max for the network I was testing on from a friend's host and uploaded a 938MB file which uploaded consistently at 50Mbps. The Pingvin host took the connection at the nic level and bound it right to the container skipping the docker host's network stack entirely and Pingvin happily got the file at the max speed of the uploading test host. The host running Pingvin is using ipvlan in l3 instead of l2 but same still occurs it bypasses the docker host's network stack and well it's entire stack so it's just 2 containers talking directly to eachother across hosts so no limiting factors past kernel processing.

Can you tell me how far your vps is from your newt? and how your internal network is structured and if using hairpinning by chance? Where the connection comes from newt and instead of talking directly to say pingvin you put the host ip so it's doing containerA(newt) > Host(Same host) > ContainerB

Anyways let me know.

[Razdnut]

I'm having the same issue. Maybe the problem is with Traefik( or gerbil ??)since the issue persists even locally. Some people have fixed it by switching VPS. Temporarily, I solved it by enabling the Cloudflare Proxy, which tripled my upload speed.

[Razdnut]

I switched to Hetzner CAX11 (arm64), and in terms of performance, it seems to be one of the best. The speed has improved significantly. I can also play 4K HDR videos with a bitrate of 78000 kbps without interruptions and without making changes to buffering, with crowdsec added. Location: Nuremberg. VPS with a declared speed of 500 mbps (info here) tested with iperf3 at 430 Mbps. At home, I have a 2.5 Gbps fiber connection.
Suggestion: avoid installing newt with Docker. Install it directly. I've found that Docker tends to slow down the connection (there are many articles on this topic and on the effects of overlay2), especially in bridge mode. Personally, I created a newt.service that starts on server boot, and I believe this has improved both upload and download speeds.
If we consider that there are two layers in between, one with Docker and one from the tunnel, it’s not surprising that the speed decreases. Additionally, Traefik doesn’t excel in this aspect either. I am currently testing the new fastproxy, but it only supports HTTP 1.

[johanngrobe]

I don't know what happened, but suddenly my performance issues are gone. For my part it had nothing to do with pangolin nor newt. I had the same performance issues with a direct wireguard connection and tailscale. Hetzner Nürnberg works flawlessly as well. It seems to be my internet provider that had issues.

[steuerlexi]

I don't know what happened, but suddenly my performance issues are gone. For my part it had nothing to do with pangolin nor newt. I had the same performance issues with a direct wireguard connection and tailscale. Hetzner Nürnberg works flawlessly as well. It seems to be my internet provider that had issues.

very good to hear this. So Hetzner seems to be the best choice. What are you guys paying for the service at the moment?

[synologyy]

~5€

[steuerlexi]

I decided to explore more VPS services and revisited Ionos out of curiosity.

Ionos, in combination with my ISP "Deutsche Glasfaser," did not connect directly to the Ionos data center in Berlin but instead routed through Amsterdam, which significantly increased latency and reduced throughput. I also evaluated 1Blu and netcup. Here is my top 4 list:

1. Hetzner - 10ms
2. 1Blu - 10ms
3. netcup - 13ms
4. Ionos - 40-90ms

I decided to go with netcup as they hat the best offer with 2 euros

[alexandrescieux]

I’m also experiencing a severe performance issue with my Pangolin/Newt/Gerbil setup, and I can’t understand why.

It works flawlessly for serving low-bandwidth workloads (websites) — it's great, and I love it.
However, it struggles when it comes to streaming (4K video / FLAC audio) or downloading files.
I only get around 8–10 Mb/s (~1 MB/s), which causes long buffering times and very slow file downloads.

---

My setup is as follows:

• Home server

  • Ryzen 9 9955HX
  • 10 Gbit SFP+ port
  • Proxmox VE 8.4
  • Newt 1.3.1 in a dedicated VM, installed via Docker Compose following the documentation

• Router

  • GL.iNet Flint 2 with 2.5 Gb LAN ports for WAN and LAN
  • Running OpenWRT

• VPS (OVH VPS S - 2 vCPU / 2 GB RAM - Gravelines datacenter)

  • Pangolin 1.6.2 + Gerbil 1.0.0 + Traefik 3.4.4, installed via Docker Compose as per the documentation

---

What I’ve tried to identify the root cause:

Suspected CPU bottleneck (WireGuard overhead)

I initially suspected that Newt and/or Gerbil/Traefik were CPU-bound due to WireGuard overhead.
However, running docker stats on the VPS and checking the Proxmox dashboard ruled that out:
Pangolin/Gerbil/Traefik/Newt combined consistently stay under 10% CPU and 10% RAM usage, even when launching 5 streams or downloads in parallel.

---

Suspected bandwidth bottleneck

I ran speedtest-cli in different environments:

• On Proxmox (Debian): ~500 Mb/s up/down (symmetric sub-gigabit fiber from my ISP)
• Inside the Newt VM: ~500 Mb/s up/down (same as the hypervisor)
• On the VPS: ~450 Mb/s up/down (slightly below OVH’s advertised rate)

Everything seems to have a decent internet connection.

---

Suspected network throughput between components

I deployed iperf3 in server mode via Docker on the VPS (same conditions as Pangolin/Gerbil/Traefik), then:

• Ran iperf3 in client mode from Proxmox: 450 Mb/s up/down
• Ran iperf3 in client mode from the Newt VM directly (not via Docker): 450 Mb/s up/down
• Ran iperf3 in client mode from the Newt VM via Docker: 450 Mb/s up/down

So everything seems to have proper link speed internally despite the abstraction layers (VMs, Docker, VPS).

---

When monitoring netin/netout in the Newt VM via the Proxmox dashboard, I can clearly see the 1 MB/s cap on the graphs (not necessarily blaming Newt, but it aligns with the observed limit).

---

Given the above, it doesn’t appear to be a CPU, RAM, or bandwidth issue — which leaves us with either the Newt/Gerbil/Traefik codebase or their configuration.

Let me know if you need more details or logs — I’d be happy to help debug this further.

[alexandrescieux]

Happy to have contributed to the analysis!

Unfortunately, I'm not experienced enough in Go to make such a deep change to the networking codebase.
I'll be looking forward to seeing it included in a future release.
Bought a limited supporter key, keep up the good work !

For now, the current speed brute-forcing packets one by one from a Hetzner CAX11 is sufficient for a single-user Pangolin instance like mine. However, it could become a bottleneck in an enterprise setup.

[Zedifuu]

I've also been facing the same issue as the above I thought I was going mad! and ended up switching between a number of VPS providers to see if that was the issue. All facing similar results with iperf and transfer speeds.

I have been trying to get a basic wireguard setup between pangolin and Opnsense, in the meantime with not much luck, Basic Wireguard connections dont seem documented 🥲

Should this discussion above maybe be translated into an issue on the Newt repository so it's easier to find and not forgotten about?

[steuerlexi]

@oschwartz10612 @johanngrobe @synologyy @alexandrescieux

Following the ongoing discussions about performance bottlenecks within the Pangolin and Newt setup, it has become clear that a key limitation is the sequential processing of packets, which significantly restricts throughput speeds especially for high-bandwidth use cases like 4K video streaming and large file transfers.

While the core team has acknowledged that parallel processing is both desirable and feasible, it has not yet been implemented, primarily due to resource constraints and the initial focus on delivering a functional and efficient baseline.

Therefore, we are reaching out to the community:

Would any developer be willing to contribute by creating a pull request to implement parallel processing in the Newt/Gerbil/Pangolin codebase?

Your contribution would be invaluable in overcoming this current software limitation and could dramatically improve performance and user experience. For those interested, we can provide guidance on the project structure and current bottlenecks to help get started.

In the meantime, some workarounds such as avoiding Docker for Newt installation, choosing high-performance VPS providers like Hetzner CAX11, and minimizing proxy layers (e.g., Traefik) can help improve performance temporarily. However, the real solution lies in parallelizing packet handling.

If you are interested or have any questions about how to proceed with this contribution, please do not hesitate to reach out. Your participation would make a significant impact!

Thank you for considering this request.

[oschwartz10612]

Change would be in https://github.com/fosrl/newt/blob/main/proxy/manager.go

If someone wants this would be cool to add.

If not I DO want to do this soon and I am sorry for the slow speed. SO much we are doing these days lol.

I also think there is an issue with too many UDP sockets being created that needs looking at...

[steuerlexi]

Thank you so much @oschwartz10612 I can see there’s so much happening, and we all truly appreciate your efforts. This is an amazing product that really has no real competitors! :-)

Interesting fact around the UDP sockets!

[steuerlexi]

btw. i would like to share my workaround until we have this. It´s simple. Basically let every service/ressource have it´s own newt tunnel/sites.

[MisterGoodcat]

I've been pulling my hair out over this until I found this discussion. I had no problems at all for several months, but only with low-bandwidth services. Recently I added my photo library to the setup, and then realized I'm not able to stream any videos reliably.

The behavior I see is consistently the following: with larger files, the initial transfer speed is high for a few seconds (meaning a few megabytes), and then it rapidly drops to unacceptable levels (around 10 kb/s!). After a quite long time (1-2 minutes) it equally rapidly starts to pick up speed again and then keeps saturating my line until the transfer is complete (several mb/s). If I interrupt the transfer and restart, it starts all over again with the same behavior.

During the time it's only dripping, the load on the VPS is non-existent. Once it picks up speed again I see Traefik at ~25% CPU. I did extensive testing to check for bandwidth and speeds all along the setup, and I'm able to saturate my public line to and from the VPS (Strato Germany) no matter what (raw transfer through SCP, with or without Docker etc.). I'm also able to max out the local GBit network when I access the services only locally.

It doesn't sound like the exact symptoms you describe, but similar in that I cannot find any issues at any stage of the setup, but when I add Newt to the equation it reproducible shows this behavior.

[aszurnasirpal]

Just want to upvote to raise attention to @oschwartz10612. I face the same. My Pangolin is located on an Oracle Free Tier instance (so 480 Mbps network cap), but in fact, while traffic is routed via Pangolin, the real speed seems to be limited to 2-3MB/s, instead of 60 that I should have)

[Karsto-sys]

+1

[AstralDestiny]

@MisterGoodcat Mind bugging me in the pangolin discord by chance?, When running did you use the iperf within the namespace of the gerbil container downstream to newt then off to a service without hairpinning locally? and kept it running then started to shunt data through and see if it remains stable or not?

[lamixer]

A google search brought me here. I am running Nextcloud behind Pangolin on a VPS for local resources only (no tunnels / Gerbil / Newt at all). Max out around 3 megabit/s when should be at least 200. I guess not a Newt problem, or not exclusively Newt anyway.

[MisterGoodcat]

@MisterGoodcat Mind bugging me in the pangolin discord by chance?, When running did you use the iperf within the namespace of the gerbil container downstream to newt then off to a service without hairpinning locally? and kept it running then started to shunt data through and see if it remains stable or not?

Shortly after that weekend of analysis I moved away from Pangolin to a different setup for my requirements. Apologies that I don't remember the specifics because I tried so many combinations back then (and it's been a few months), and that at this time I don't have a repro environment anymore.

I really like the project and still watch its progress and this discussion, to maybe reconsider it in the future. All the best!

[alexandrescieux]

I migrated my Pangolin instance from the default SQLite database (pangolin:1.12.2) to a PostgreSQL backend (pangolin:postgresql-1.12.2).
I’m seeing a noticeable improvement in UI responsiveness and in the loading speed of static web pages (which was already reasonable). However, I’m still experiencing the same issue when streaming or downloading large files, even with PostgreSQL.

It was definitely worth trying, and the performance boost for the UI and static pages is appreciated.
My guess is that SQLite is simply too slow on VPS setups that rely on shared spinning-rust HDD backends.

[aszurnasirpal]

Just out of curiosity. How did you migrate to PostgreSQL? using this script or differently?

[alexandrescieux]

I use Pangolin as a hobby/self-hosting setup, so I was able to afford some downtime. I deployed the new image on a fresh database, redeployed Newt with the new secrets, and reapplied my blueprints that contain all my configuration as code.
So unfortunately, I didn’t use any migration script and can’t provide any feedback on that part.

[thutex]

using this script

i've just looked at it, and this script is extremely outdated and does not have all the correct tables/table structures,
so it cannot be used in its present form.
it's probably not an enormous amount of work to get it fixed up though.

[aszurnasirpal]

Ohh, thank you very much! You saved me a lot of time that I would have spent trying to figure out why it isn't working.

[steuerlexi]

This discussion seems to be one of the most active ones, yet I haven’t seen this issue being addressed. I used to really like Pangolin, but it’s time for me to move on from the project.

Thank you very much for everything so far. I wish the project and everyone involved all the best — and as they say, paths always cross again.

[Karsto-sys]

Look at https://github.com/wiredoor/wiredoor
Its similar but whitout the speed issue I get nearly 100% bandwith on the same vps I used Pangolin, really happy with it

[gaizkaeu]

Same here. I couldn't get more than 6 MBps, used Wiredoor and now +20 MBps...

[keonramses]

Unfortunately issue still persists in the latest newt version. (IPV6 only)

local site:


newt site:


wireguard site:

[AstralDestiny]

What's your upload speed? at the newt location ? and can you do an iperf3 test?

[keonramses]

Newt upload speed is above. Here is the iperf3 test result.

[joshuadow]

After struggling with inconsistent throughput over Newt, I switched to a Basic WireGuard Site. Setup was pretty painless and immediately fixed the “bursty stall/buffering” behavior for long-lived streams.

Problem

Persistent connections would stall in bursts over Newt tunnels (CPU wasn’t pegged). Shorter/lower-bitrate flows were ok which strongly pointed to PMTU/fragmentation (TCP bursts + long stalls).

Fix

Use a Basic WireGuard Site and route traffic over kernel WireGuard. Key points:

• Point resources to the WG peer IP, not your LAN IP.
• Pin MTU (1280) + add TCP MSS clamping (otherwise wg-quick can revert MTU to 1420 after restart and the buffering comes back).

---

Solution

1) Create a Basic WireGuard Site in Pangolin

• Create the site and bring up WireGuard on a LAN gateway LXC/VM: wg-quick@wg0
• Confirm handshake: wg show

Note: the server-side WG interface may live inside the gerbil container (not on the pangolin host):

docker exec -it gerbil ip -br addr   # should show wg0 with e.g. 100.89.x.x/24

2) Point the resource upstream to the WG peer IP (NOT the LAN IP)

In Pangolin, set the resource Target/Upstream to:
https://{WG_PEER_IP}:{PORT}
Do not target 192.168.x.x directly for Basic WG sites unless you’re forwarding/NATing on the peer.

3) On the WG peer gateway: DNAT only what you need to {TARGET_IP}:{PORT}

Put the following in /etc/wireguard/wg0.conf on the WG peer (replace {TARGET_IP}, {PORT}, and your LAN interface name if it’s not eth0):

[Interface]
Address = <WG_PEER_IP_CIDR>
PrivateKey = ...
MTU = 1280

# Forward <PORT> from WG -> target
PostUp   = sysctl -w net.ipv4.ip_forward=1
PostUp   = iptables -t nat -A PREROUTING -i %i -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_IP>:<PORT>
PostUp   = iptables -t nat -A POSTROUTING -o eth0 -p tcp -d <TARGET_IP> --dport <PORT> -j MASQUERADE
PostUp   = iptables -A FORWARD -i %i -o eth0 -p tcp -d <TARGET_IP> --dport <PORT> -j ACCEPT
PostUp   = iptables -A FORWARD -i eth0 -o %i -p tcp -s <TARGET_IP> --sport <PORT> -j ACCEPT

# MSS clamp (prevents PMTU blackholes / burst-stall behavior)
PostUp   = iptables -t mangle -A FORWARD -i %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
PostUp   = iptables -t mangle -A FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

PostDown = iptables -t nat -D PREROUTING -i %i -p tcp --dport <PORT> -j DNAT --to-destination <TARGET_IP>:<PORT>
PostDown = iptables -t nat -D POSTROUTING -o eth0 -p tcp -d <TARGET_IP> --dport <PORT> -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -o eth0 -p tcp -d <TARGET_IP> --dport <PORT> -j ACCEPT
PostDown = iptables -D FORWARD -i eth0 -o %i -p tcp -s <TARGET_IP> --sport <PORT> -j ACCEPT
PostDown = iptables -t mangle -D FORWARD -i %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
PostDown = iptables -t mangle -D FORWARD -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

4) Persist across reboots

systemctl enable --now wg-quick@wg0

Quick sanity checks
ip -d link show wg0 | grep mtu # note that 1420 caused buffering to return
iptables -t nat -S | grep
iptables -t mangle -S | grep TCPMSS

[van-geaux]

After struggling with inconsistent throughput over Newt, I switched to a Basic WireGuard Site. Setup was pretty painless and immediately fixed the “bursty stall/buffering” behavior for long-lived streams.

I've confirmed that this workaround works for me. In mine I'm using docker for the wireguard and I had to bound the apps to the wireguard network, example:

services:
  wireguard:
    image: lscr.io/linuxserver/wireguard:latest
    container_name: wireguard
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    environment:
      - PUID=1027
      - PGID=100
      - TZ=Asia/Jakarta
    volumes:
      - /mnt/user/docker/00000-wireguard:/config
      - /lib/modules:/lib/modules:ro
    ports:
      - 51820:51820/udp
      - 3000:3000                                           # the app's port
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
    restart: unless-stopped

  whoami:
    image: ghcr.io/traefik/whoami:latest
    container_name: whoami
    # network_mode: host
    network_mode: "service:wireguard"       # in network mode bound it to wireguard service
    # ports:
    #   - 3000:3000
    depends_on:
      - wireguard
    restart: unless-stopped

edit:
Forgot to mention that I need to put the wireguard config from pangolin in a wg0.conf inside wireguard's /config dir

[essys]

I wonder if there are any clues as to what might be causing this.
Has there been any official feedback from the maintainers/developers on this issue?

[memesalot]

Same issue here.

[BSZ7]

Yeah, I’ve been running into the same issue with large file streams. I also tried replacing Newt with a plain WireGuard connection, but unfortunately the speeds still dropped quite a bit. For now, I’ve had to switch to a different tool for the affected applications.

Really hoping this gets improved at some point. Everything else works great, so it would be amazing to see this resolved.

[kylepyke]

Same issue here. I understand there's likely no easy fix, bu it seems like a pretty critical issue.

[garysan-uk]

I've encountered same issue. Using a 2vCPU VPS with Pangolin installed and running on it. Newt, as a service on my Win11P box. Have nice 'elegant' solution that I can point to (plex.mydomain.com, overseer.mydomain.com, etc.) but Plex streaming is limited to 720p/4mbps through the Newt tunnel... I've had to switch out to Tailscale which isn't quite as elegant, as it requires something installed on my Plex users' machines.

I too would really like to see this resolved as Pangolin is a super-neat solution otherwise.

[Boscovitz]

Same here. Streaming 4K over newt is impossible. Working ok on standard Wireguard.

[balgerion]

Racknerd 2vcpu streaming 4k was working but it took good minute to start switched to std WireGuard and its almost instant

[vadim0872]

In your setup did you happen to have CrowdSec protection? There's an adacent issue where CrowdSec can't handle large file uploads, but I'm curious if they're related somehow

NO, NO NO , it not problem in crowdsec, I turn off it, and it is no changes. When big is uploading in htop I see CPU load on server.js process, that seems that is pangolin core is bottlenecking.

[essys]

I agree, this has nothing to do with Crowdsec, as I did not use Crowdsec in my tests, but I did experience serious performance issues with Newt.
I have not yet tested the direct Wireguard connection yet.

[Razdnut]

It should be added that Traefik is slower than other reverse proxies in circulation.
A test with Traefik and a tunnel wireguard should be done to test performance.

[vadim0872]

Upload/Download - no problem and max speed with traefik, without pangolin

[kylepyke]

I was referring to this issue, which I’m also having:

crowdsecurity/crowdsec#3837

Apologies, was just polling to see if the issues were linked by having Crowdsec installed, but clearly not.

[garysan-uk]

I'd love to see a straightforward set of instructions for having Pangolin on a VPS, and setting up a WG client at the Windows 11 end... I can't be the only one with this setup (currently with Newt, and struggling with the speed for streaming media). I'm using Tailscale now which does work but I'd like to make use of the VPS I'm paying for.

Everyone says setting a WG tunnel up is the answer but I can't find anywhere that instructs exactly how to do it - I mean, like I'm an idiot and have no experience of WG. They either have Windows at both ends or a Linux distro where my Win 11 box is... Please?

[GlenNicholls]

This isn’t really relevant to this discussion so you’d get the best help in a new discussion or issue with specific details regarding your setup, what you’ve already done, and a clear outline of what is not working.

With that said, the quick install guide in the docs should get you up and running on your VPS. Then the only steps once you have pangolin installed is to open the required ports on your VPS, install Newt on your windows system, point newt to your VPS static IP, and create a newt site in pangolin. You could use wireguard instead of newt but the config is slightly more complicated.

The easiest way to set up newt would be via docker or docker compose on your windows system. While I haven’t deployed on Windows, docker should make the OS mostly irrelevant.

EDIT: sorry, I misread your question. It sounds like you got things working with newt but experienced speed/bandwidth issues and would like to get this working with WG. It’d be helpful if you posted your WG config and pangolin WG site config with sensitive info obfuscated.

[garysan-uk]

Not sure if anyone else has noticed a difference but since I updated Newt to v1.10.0 and Pangolin (self hosted on a VPS) to v.15.4 - things have markedly improved and I'm able to now stream directly from my media server at 1080p/12mbps (previously I couldn't achieve anything above 720p/4mbps, as many above have also mentioned).

Perhaps someone else can join these dots as to which is having the desired effect, but I installed Newt using the newt_windows_installer.exe as opposed to the previous version I had, which was just the Newt.exe - (always been installed/set-up as a service on Win 11 Pro). This time around, I noticed the installer also installed wintun.dll, along with Newt.exe. Now, I do remember some mention of perhaps needing wintun.dll in the Pangolin/Newt docs but as Newt 'appeared' to be working ok, I didn't think any more about it at the time.

Maybe it's the new version of Newt or maybe it's the inclusion of the wintun.dll into the Newt directory this time around but so far, it's working really well for me now and just what I wanted it to be. Thanks 🙏🏻

[LaurenceJJones]

Hey all, I wanted to jump in as one of the newest members of the Pangolin team to keep communication clear on this issue.

I know this has been open for a while, but a lack of active replies does not mean this discussion has been deprioritized. @oschwartz10612 has an active task to investigate and improve performance, and I've also opened follow-up tasks to improve Go-side performance over time.

The performance work merged into dev so far includes:
fosrl/newt#204
fosrl/newt#205

In Newt, we currently use two WireGuard netstack implementations: netstack and netstack2. The main tunnel to Gerbil is still using netstack, while child/downstream tunnels use netstack2, which contains the newer performance improvements.

A full switch is not just a simple toggle. Even though the implementations are compatible, we still need proper test coverage before promoting this broadly, to avoid inconsistencies or regressions.

[memesalot]

Awesome thanks!

[v1rusnl]

@LaurenceJJones Just to clarify, this did not make it into 1.10.2 or 1.10.3, right?

[LaurenceJJones]

Nope, as I just put in comment in my testing even with netstack2, the perf improvement was very minimal, I found more optimizing the kernal options gave better results (10mb/s on larger packets -> 140mb/s). However, be wary about changing these, maybe we can write a guide on what to tinker with.

[seanjones80]

Aye would be good to see a cheeky guide on this :)

[v1rusnl]

Nope, as I just put in comment in my testing even with netstack2, the perf improvement was very minimal, I found more optimizing the kernal options gave better results (10mb/s on larger packets -> 140mb/s). However, be wary about changing these, maybe we can write a guide on what to tinker with.

You mean stuff like net.core.rmem_max, net.core.wmem_max, net.core.rmem_default, net.core.wmem_default etc.?

Changing the default way to small values e.g. for rmem and wmem made a big change for e.g. loading speed of e.g. images etc. Not just via Newt, but also my local Pangolin Instance running on the Newt Client.

[TheD3vilsAdvocate]

I tried a lot of things including Newt inside Docker, outside docker, a direct connection (ie DNS to open port without pangolin) and ultimately Pangolin + Wireguard Basic connecting to a Unifi Router.

What's pretty clear is that VPS performance and also Host performance play a minor role.

When using immich, I was getting top speeds with a direct connection and with wireguard basic. It goes up to 25MBsec and beyond, but it still starts pretty slow, i.e 600KBsec-3MBsec. That's apparently a TCP limitation and after about 20 seconds or so it should go through the roof.

Wireguard basic gives me much better speeds than newt overall. So I'm happy it works and that I can keep using Pangolin that way.

Interestingly, downloading a large file via Nextcloud will work with 15MBsec using both Newt and WG-Basic, but WG has less fluctuations. Here the difference is somehow minor compared to immich. I suspect Nextcloud Overhead but who knows...

[twocoolbeans]

Food for thought...speed while using Pangolin in some situations may have little to do with Pangolin or newt. Some CGNAT's don't play well with UDP—especially when it comes to video streaming. WireGuard is always over UDP. In order to test if UDP behind a CGNAT is impacting speed, try accessing a resource behind an SSH reverse tunnel (which is always over TCP) and compare the speed to accessing the resource behind Pangolin.

For example, Jellyfin over a reverse SSH tunnel:
ssh -R 8096:localhost:8096 user@domain

[LaurenceJJones]

Yes as well as there a few kernal optimizations you can do with UDP packets that can improve speed but from my testing so far, even with netstack2 it didnt really improve speeds. However, changing the kernal changed it from 10 mb/s to 140 mb/s. So in short, it all depends on the infrastructure and tcp/udp.

[szszl0]

Hey,
I recently moved from free Pangolin Cloud to selfhosted Pangolin on my VPS. I have exactly same public resources setup as i had before. Newt site is also installed exactly the same way as it was when i was using Pangolin Cloud - in an LXC container as systemd service inside my Proxmox home server. My home is behind cgnat without ipv6 but it has not been an issue with Pangolin Cloud. My VPS is both ipv4 and v6 capable.
Network speed and latency are flawless on both VPS and home server. Resources "work" in a way that they are loading eventually but it takes from 10 to about 60 seconds... Exposed services arent a problem - they work fast when i connect to them using local network or tailscale vpn.
So long story short - exactly same setup worked great on Pangolin Cloud but sucks on my VPS. Any ideas why? VPS's CPU and memory are fine during resource loading.

Edit:
P. S. Pangolin dashboard itself loads super fast, in less than a second. Only the resources are slow. For example Home Assistant dashboard is loading 10-15s, Immich loads in 10s but uploading one photo takes few minutes. Gotify and Termix load much longer than 60s and sometimes dont load at all (it loads the service ui background and sometimes some ui elements but doesnt load data at all)

[TheD3vilsAdvocate]

I moved my whole install from a Docker for Windows Setup to a Ubuntu VM on Proxmox Setup. Very comparable hardware, but now I'm getting much faster speeds with newt than with wireguard. It makes almost zero sense.

I can now start downloading a file with 8MBsec and it goes up to 25MBsec and beyond quickly instead of starting with 600KBsec via Newt previously or 900KB via WG previously and taking 30secs before reaching better speeds.

So NOW the experience is much better because the buffering is almost nothing at all.

I suppose the TCP backend on linux machines is much better than on Windows.

[szszl0]

well, i never used windows for that. Both homelab and vps are running linux

[Hielyr]

I'd just like to add my voice to this discussion as I've just discovered it.

I wrote a decently detailed report in issue #2905 on when I started experiencing this issue and what I tried doing. Hopefully it helps even a tiny bit.

#2905 (comment)

I'm still having the issue unfortunately, and my experiences mirrors that of many here - decent initial burst of speed, but then almost immediately slows to a trickle - but I have learned from this discussion thread that I should try setting the site up as a wireguard connection instead of using Newt. I'll give that a try next.

[alexandrescieux]

I am more and more convinced that we need Newt and Gerbil to use a native, kernel-spaced wireguard interface for us to have decent link speed.

I think the bottleneck we are seeing is linked to the Docker + wireguard-go user-spaced implementation introducing too much context-switching.

As mentioned in 2356, Newt has the USE_NATIVE_INTERFACE variable, but Gerbil does not. I haven't seen any improvement after activating USE_NATIVE_INTERFACE on Newt, so maybe the bottleneck is on Gerbil side ?

[TheD3vilsAdvocate]

It's really a bizarre issue...
I can max out my VPS' max transfer speed of 500Mbit sec (55MBsec). Newt connects to my Ubuntu-based VM on Proxmox.

Happy to help out if anybody wants to know anything.

[autoclave73]

I was having this issue on truenas, I converted newt to a custom app to run as root with USE_NATIVE_INTERFACE and got a 10x bandwidth improvement, from 2MB/s to 20MB/s. Truenas is using an old little intel C2750 cpu. I tried network_mode: host as well but that didn't help.

[v1rusnl]

I was having this issue on truenas, I converted newt to a custom app to run as root with USE_NATIVE_INTERFACE and got a 10x bandwidth improvement, from 2MB/s to 20MB/s. Truenas is using an old little intel C2750 cpu. I tried network_mode: host as well but that didn't help.

Did you set up custom if up down rules? When using USE_NATIVE_INTERFACE=true, my private resources stop working.

Edit:
Needed to set up the following rules to make it work with USE_NATIVE_INTERFACE=true:

# 1) Forwarding for Newt-Tunnel in DOCKER-USER
sudo iptables -I DOCKER-USER -i newt -o YOUR-LAN-IF -j ACCEPT
sudo iptables -I DOCKER-USER -i YOUR-LAN-IF -o newt -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

# 2) NAT for LAN postrouting
sudo iptables -t nat -A POSTROUTING -s 100.90.128.0/24 -o YOUR-LAN-IF -j MASQUERADE

YOUR-LAN-IF= Interface Name of your LAN Interface, e.g. eth0 or enp2s0

[v1rusnl]

In meanwhile I've done it all, sysctl optimizations, ethtool scripts, USE_NATIVE_INTERFACE=true with Newt.

Finally I've come to the following (perhaps obvious) conclusion:

1. Gerbil must get a Kernel Space option somehow in the future like newt does (USE_NATIVE_INTERFACE) to use full potential also of a small VPS like Hetzner CX23 (2vCPU, 4GB).
2. The recommended specs for running Pangolin when serving e.g. media streaming like Jellyfin have to be more than the ones recommended on the Website (I would go at least with 4 vCPUs). Without 1, this is even more important.

[oschwartz10612]

Yeah USE_NATIVE_INTERFACE is not really a supported option right now. It only makes a interface on the host so you can address the peer by its peer IP instead of the resources but it breaks everything else. We leave it undocumented because it is not a full feature quite yet. I agree I think we need to bump the specs on the website - will do.

[v1rusnl]

I agree I think we need to bump the specs on the website - will do.

Bad times looking at the price increases from Hetzner 😄

[Karsto-sys]

Or you can just use another tunnel solution. I get full bandwidth with one V core and 2GB RAM.

[v1rusnl]

Gotcha, plain wg already running in Kernel Space as a sidecar. But that's not the point of discussion. The point is making Pangolin better.

[v1rusnl]

Yeah USE_NATIVE_INTERFACE is not really a supported option right now. It only makes a interface on the host so you can address the peer by its peer IP instead of the resources but it breaks everything else. We leave it undocumented because it is not a full feature quite yet. I agree I think we need to bump the specs on the website - will do.

@oschwartz10612 Are there any disadvantages when the user set up the routing by himself on the newt host like in https://github.com/orgs/fosrl/discussions/512#discussioncomment-17356826 ?