@yozora/tokenizer-text
Any characters not given an interpretation by the above rules will be parsed as plain textual content.
- See github flavor markdown spec for details.
- See Live Examples for an intuitive impression.
Install
- npm
- Yarn
- pnpm
npm install --save @yozora/tokenizer-text
yarn add @yozora/tokenizer-text
pnpm add @yozora/tokenizer-text
Usage
@yozora/tokenizer-text has been integrated into @yozora/parser / @yozora/parser-gfm-ex / @yozora/parser-gfm,
so you can use YozoraParser
/ GfmExParser
/ GfmParser
directly.
- Basic Usage
- YozoraParser
- GfmParser
- GfmExParser
@yozora/tokenizer-text cannot be used alone, it needs to be registered in Parser as a plugin-in before it can be used.
import { DefaultParser } from '@yozora/core-parser'
import ParagraphTokenizer from '@yozora/tokenizer-paragraph'
import TextTokenizer from '@yozora/tokenizer-text'
import TextTokenizer from '@yozora/tokenizer-text'
const parser = new DefaultParser()
.useFallbackTokenizer(new ParagraphTokenizer())
.useFallbackTokenizer(new TextTokenizer())
.useTokenizer(new TextTokenizer())
// parse source markdown content
parser.parse("hello $.;'there")
import YozoraParser from '@yozora/parser'
const parser = new YozoraParser()
// parse source markdown content
parser.parse("hello $.;'there")
import GfmParser from '@yozora/parser-gfm'
const parser = new GfmParser()
// parse source markdown content
parser.parse("hello $.;'there")
import GfmExParser from '@yozora/parser-gfm-ex'
const parser = new GfmExParser()
// parse source markdown content
parser.parse("hello $.;'there")
Options
Name | Type | Required | Default |
---|---|---|---|
name | string | false | "@yozora/tokenizer-text" |
priority | number | false | TokenizerPriority.FALLBACK |
-
name
: The unique name of the tokenizer, used to bind the token it generates, to determine the tokenizer that should be called in each life cycle of the token in the entire matching / parsing phase. -
priority
: Priority of the tokenizer, determine the order of processing, high priority priority execution. interruptable. In addition, in thematch-block
stage, a high-priority tokenizer can interrupt the matching process of a low-priority tokenizer.Exception: Delimiters of type
full
are always processed before other type delimiters.
Types
@yozora/tokenizer-text produce Text type nodes. See @yozora/ast for full base types.
import type { Literal } from '@yozora/ast'
export const TextType = 'text'
export type TextType = typeof TextType
/**
* Text represents everything that is just text.
* @see https://github.com/syntax-tree/mdast#text
* @see https://github.github.com/gfm/#textual-content
*/
export type Text = Literal<TextType>
Live Examples
-
Basic.
-
Internal spaces are preserved verbatim.