What is an HTML tokenizer and what does it do?

[JAB Creations]

What is an HTML tokenizer? What does it do and how would that be useful?

[zcorpan]

An HTML tokenizer is part of an HTML parser. To parse HTML, you first tokenize the input stream (a sequence of bytes or characters) into a sequence of tokens, where a token is text, start tag, end tag, doctype, or comment. Then the HTML tree builder looks at the sequence of tokens and builds a DOM tree.

[JAB Creations]

So text to DOM tree essentially, thanks Z!