Virtual Tuners: Problem with high-bitrate DVB-Stream

[neo7530]

Hi.

I've tried to tune into a high-bitrate DVB stream, supplied via udp-multicast ( 3 TV-Channels and 3 Radio channels). Ts-Reader, streamer and childpipe - all have the same behavior. The Channels will be found in SAT>IP Client, but the video is unwatchable. Have uploaded a test-ts of this stream to: https://ddns.dynbox.net/owncloud/index.php/s/6FaYTk68zBHWsJm

my mapping looks like this:

#EXTM3U

# To use Translation you need to set in the web interface
# - 'Advertise as' to exp. 'DVB-S2'
# - 'Transformation enable' should be checked
# - and a valid m3u file like this

#EXTINF:-1 satip-freq="12344", Translation to: ChildPIPE exec="wget -qO- HTTP-Stream from TSDUCK streaming"
rtp://@#1/?msys=childpipe&exec="wget%20-qO-%20http:%2F%2F10.24.10.51:7777%2Fudp%2F224.10.10.10:1234"&pcrtimer=0

#EXTINF:-1 satip-freq="12345", Translation to: ChildPIPE exec="wget -qO- HTTP-Stream from TSDUCK streaming"
rtp://@#1/?msys=file&uri="multiprog.ts"&pcrtimer1

#EXTINF:-1 satip-freq="12363", Translation to: Stream 1 Multicast UDP input
rtsp://@#1/?msys=streamer&uri="[email protected]:1234"

Playing the UDP-Multicast via VLC works, playing the translated (via udpxy) stream direct works with VLC also. Playing via SATPI (running on the same machine) via VLC doesn't work (getting programs but sluttery video/audio)

Could you download my testfile for testing, please and tell me, what is wrong with the Stream?

Greetings, Marco.

[Barracuda09]

Hi Marco,

Thanks for your interesting use case.

I have downloaded the file, I will test this probably later this weekend.
But first I have some questions you could answer in the meantime.

• What machine do you use, I expect a PC?
• Did you try pcrtimer=0 (or leave it out) for every input (FILE and childpipe)? I would expect a higher value than 0, is only necessary for reading from files.
• How did you set these value Internal Software Pid Filtering and Filter PCR for timing in the frontend tab of the WebIF?

Kind regards,

Marc

[neo7530]

Hi Marc. I tested all variations with pcrtimer without any success... it is a pc (i7 32GB ram) sat-ip client is connected via ethernet. Filtering etc was tested too without success...

[neo7530]

Hi Marc.

The issue seems to be the stuffing pid (0x1fff). After remuxing the file without pid 1FFF the file is playing back fine. Can you build in a filter, that filters out the stuffing pid? It is always 0x1FFF.

Kind regards
Marco.

[Barracuda09]

Hi Marco,

Could you upload this remuxed file. You can send the link to my e-mail if you like.

i have done some testing with the stuffed file. I added the drop of 0x1FFF (null packets) but then it still reads the file to slow. So this does not seem to have the desired outcome.

Why is the file stuffed with 0x1FFF packets, to make it constant bit-rate? or was this a mistake?

Kind regards,
marc

[neo7530]

This is to match the muxrate for my dektec modulator. I need a cbr for transmitting.

[Barracuda09]

Can you build SatPI yourself? and edit some files?

[neo7530]

Yep i can do.

[Barracuda09]

Which virtual-tuner are you planing to use?
childpipe or streamer (Do not use File as it does not work and should be removed)

[neo7530]

I use childpipe, because i read from udp port with socat.

[Barracuda09]

Ok then you could try to decrease the 150 ms to 20 ms.
And turn off Filter PCR for timing in webUI and no pcrtimer=xx in the request :

SATPI/src/input/childpipe/TSReader.cpp

Lines 103 to 114 in 91dabfb

	const long interval = pcrDelta - std::chrono::duration_cast<std::chrono::microseconds>(_t1 - _t2).count();
	if (interval > 0) {
	std::this_thread::sleep_for(std::chrono::microseconds(interval));
	}
	_t1 = std::chrono::steady_clock::now();
	_deviceData.getFilter().getPCRData()->clearPCRDelta();
	} else {
	std::this_thread::sleep_for(std::chrono::microseconds(150 + pcrTimer));
	}
	return true;
	}

[neo7530]

Ok. Will test tomorrow. 👍

[lars18th]

Hi @Barracuda09 ,

After a lot of time without writing here, I need to comment this because is related with the problem that suffers @neo7530 :

• Months ago I've stopped to update the SATPI in my environment. The reason is the changes that you've introduced after my patches around the software pid filtering for the childpipe input. After this point (the changes that you have introduced), the problem of glitches in the stream output are inherent to the SATPI and they are NOT fixable. Whats the problem? The problem is that you've enforced to read when the input buffer is full (to remember: my original function is readFullTSPacket() and you've changed it with the finalCall parameter). As we have discussed time ago, my approach to to read from the device at every loop. But your approach is different. I agree with you that it's a good idea to "protect" of a overloading from the input stream. However, this "control" requires to leave to the user the option to disabled it when he "trusts" the source. And that's is true if the stream is comming from an external tuner with a CBR over a UDP socket.

In resume, after this commit 9870875 the input troubles appear. I recommend to @neo7530 to compile one version before this commit and check the results.

In my case, I'm freezed with these ancient versions (with my original patches) because I'm using SATPI only for UDP+FILE+HDHomeRun inputs with the REMAPPING functionality.

I hope this helps.
Regards.

[Barracuda09]

Hi @lars18th

Nice to hear from you again, hope your doing well.

I have good results (no freezes) with lowering this timer in function bool TSReader::isDataAvailable() in combination with the provided test file here #199 (comment).

We have never lowed this value, it is now more or less the same result as your approuce with the do while loop (I would think)

Kind regards,
Marc

[lars18th]

Hi @Barracuda09 ,

I'm going not really well. However, I'm glad to participate here.

Regarding the last changes, I'll check them in the next weeks. However, I want to note two key points (from my point of view):

1. It will be useful to leave the option of configuring the waiting timer. The reason is that you don't know the performance of the device where the SATPI runs. Therefore, using a low power device with a hardware pid filtering, it will be preferable to use a higher value. But with a performance server with software pid filtering, the idea is to reduce it to the minimun. My suggestion is to use a number from a CONSTANT (that you can modify at compile time), and add a command line option to change the default value. You agree?
2. The second question is if you want to trust or not the source. If you trust that it doesn't produce over a maximun threshold, then it will be preferable to read in the loop until "all the incoming buffer is full or no more data is pending". This has sense when you're using the software pid filtering. Because all packets to discard will be removed in the same loop. Without waiting for the next read. Take note that if the source is "trusted", then it sends TS packets of the same PES packet in one chunk. The problem with the current implementation is that you can insert some wating delay inside a full PES packet. And that's the reason of the glitches. However, with the approach to read "at full speed" (in the sense of fulfill the buffer if data is pending to read) then the problem goes out even the waiting timer is higher. Remember my initial implementation. On it, I'm reading and discarding packets (pid filtering) until the buffer is full or no more packets are pending. That's the difference with the current implementation.

I hope this helps you to improve the implementation.
Regards.

[lars18th]

Hi @Barracuda09 ,

Please, revert back this commit: ad810dd

When you want to receive the FULL TRANSPORT STREAM, then you want all pids, including the NULL stuff. Because you want to maintain the CBR of the full stream. This code breaks this. So I suggest to change it to:

// Check is this the beginning of the TS and no Transport error indicator and not a NULL packet when filtering
if (ptr[0] != 0x47 || (ptr[1] & 0x80) == 0x80 || (ptr[1] == 0x1F && ptr[2] == 0xFF && !_pidTable.isAllPID())) {

Regards.

[Barracuda09]

Hi @lars18th

Said to here that. And I appreciated your input and suggestions.

• The code does not discard the NULL packets with this change, please see code snippet:
  

  SATPI/src/mpegts/Filter.cpp

  Lines 142 to 155 in ad810dd

  	for (std::size_t i = begin; i < size; ++i) {
  	const unsigned char *ptr = buffer.getTSPacketPtr(i);
  	// Check is this the beginning of the TS and no Transport error indicator and not a NULL packet
  	if (ptr[0] != 0x47 || (ptr[1] & 0x80) == 0x80 || (ptr[1] == 0x1F && ptr[2] == 0xFF)) {
  	if (filter && !_pidTable.isAllPID()) {
  	buffer.markTSForPurging(i);
  	}
  	continue;
  	}
  	// get PID and CC from TS
  	const uint16_t pid = ((ptr[1] & 0x1f) << 8) | ptr[2];
  	// If pid was not opened, skip this one (and perhaps purge it)
  	if (!_pidTable.isPIDOpened(pid)) {
  	if (filter && !_pidTable.isAllPID()) {

  
  It will only mark the packet for purging, when filtering is on and not pids all is "selected"
  So this will only continue and not try to parse it further thru the code. So in other words, it is a small optimization to not parse it further.
  So please rethink you opinion, or confinse me other wish ;-)

• About the time based reading the time is adjustable by adding &pcrtimer=xx to the request. Please view https://github.com/Barracuda09/SATPI/wiki/Virtual-tuners-explained again.

I have good results with these changes and the test file provided and running this setup on a Pi3. So please give these changes a try. Please note, I do run a regulair build of SatPI so no debug.

Kind regards,
Marc

[lars18th]

Hi @Barracuda09 ,

Thank you for commenting this. Now, I think you're right. However, perhaps a simple #define WAIT_TIMER 15 in the code will be a good option. Do you think so?

Futhermore, let me to check it in the next weeks. Perhaps you're right (and I hope it) and the problems are gone.

Regards.

[Barracuda09]

Hi @neo7530

Ok. Will test tomorrow. 👍

Did you manage to test this?