Improve Markdown code block tokens (#17591)

Browse Source

* Improve Markdown code block tokens

Code blocks without a language weren't tokenized.
Code blocks didn't have their ending ``` punctuation tokenized.
Code blocks used to only have one token. Now each block has the following tokens available for syntax highlighters:
- Starting and ending ``` punctuations
- Code block's language setting
- Code snippet

* Variable whitespace for MD code block ``` token

Allow for variable amount of whitespacing before ``` code blocks

* Reorder raw blocks

Raw blocks were preventing tokenizing as languaged blocks. Putting them on bottom resolves this.

* Fix MD block detection when following paragraph

Used to require a new line inbetween ``` code blocks and preceding paragraph text.

* Prevent broken language grammar leaks in MD fences

Prevents leaks in MD code fences while also capturing the closing fence punctuations.

* Update Markdown tokenizer test file

...

This commit is contained in:

Josh Peng

2017-01-03 10:03:16 -08:00

committed by Matt Bierner

parent 7b663a459a

commit b9a362a185

2 changed files with 1638 additions and 287 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

1843

extensions/markdown/syntaxes/markdown.tmLanguage

File diff suppressed because it is too large Load Diff