[webkit-dev] HTML5 tokenizer landing soon

[David Hyatt]

On Jun 14, 2010, at 3:48 PM, Adam Barth wrote:

> 
> We ended up using the same algorithm as the old tokenizer to manage
> insertion points, however, we moved all the work into a separate
> InputStream data structure:
> 
> http://trac.webkit.org/browser/trunk/WebCore/html/HTML5DocumentParser.h#L75
> 
> The old code was actually pretty clever once I figured out what it was
> doing.  We're considering moving InputStream into its own file instead
> of keeping it as an inner class of the document parser.
> 

If you're talking about the segmented string stuff, I added that to the existing tokenizer. :)

dave