[webkit-dev] [MSE] Range ends inclusion is deleting wanted MediaSample's

[Alicia Boya García]

On 14/11/17 22:55, Jer Noble wrote:
> 
> Yeah, the spec is not the best here. 

I agree. While WebKit replicates the specified steps faithfully, it's
not clear they are good steps in the first place.

I spent quite a bit of time trying to wrap my head around the three (!)
frame removal cases, all of them apparently doing the same but with
slightly different ranges and being used in slightly different situations...

Even after all the complexity I found too some cases that are unhandled
by the spec.

> If you have, e.g. a sample with a
> PTS of 0 and a duration of 100, and then insert a sample with a PTS of
> 50 and a duration of 100, you’d expect that to cause the first sample to
> be removed. But a strict reading of the spec says that sample stays.

Or maybe cropped (that would be of questionable utility in practice
though)... But yup, the spec says that (supposing they are consecutive)
no check at all is performed because PTS < highestEndTimestamp.

>  Now you have two overlapping samples.  It can get even weirder if you
> insert a sample with a PTS of 25 and a duration of 50.  Now, strictly
> implementing the spec, you have a sample overlaps on both ends of
> another sample.  What does that even mean for a decoder?  It’s almost
> guaranteed to generate a decode error, unless both of the overlapping
> samples are I-frames.

It would be implementation specific. In the case of GStreamer, for
instance, durations are not needed for decoders to work, only DTS. Same
can be said for the presentation, that only needs to know the total
length of the movie (or at the very least know whether it should
continue playing) and the PTS of each frame. So the first frame would be
shown for 25 ms, then the second one for 75 ms. No corruption should
occur as the frames have been decoded in the correct order.

More serious problems can arise from manipulating the array of frames as
many assumptions would be broken: frames would be played for less than
their declared durations, the relation "time to frame" would no longer
have 0..1 multiplicity, the sum of durations would be longer than the
last frame end even though they are thought to be consecutive and the
first frame has PTS=0 and so on... Definitively undesirable consequences
independently of the platform.

> 
> I think the intent of the spec is clear: if any part of a previous
> sample overlaps the new one, it has to be removed, and all samples that
> depend on the removed samples must be removed as well.  And in order to
> do that, you have to take the entire duration of the sample into account.
> 

I wish that part of the MSE spec was rewritten in a simpler yet more
comprehensive way... because as of today, that's not what the spec says.
Some complexity may need to appear nevertheless because of the float
conversions caused by JS interaction.

-- Alicia.