@yozora/core-parser
@yozora/core-parser provides a plug-in system that can load tokenizer for collaborative processing in the lifecycles.
Install
- npm
- Yarn
- pnpm
npm install --save @yozora/core-parser
yarn add @yozora/core-parser
pnpm add @yozora/core-parser
Usage
@yozora/core-parser provides a default Parser
implementation
DefaultParser.
import { DefaultParser } from '@yozora/core-parser'
import EmphasisTokenizer from '@yozora/tokenizer-emphasis'
import ParagraphTokenizer from '@yozora/tokenizer-paragraph'
import TextTokenizer from '@yozora/tokenizer-text'
const parser = new DefaultParser({ shouldReservePosition: true })
.useFallbackTokenizer(new ParagraphTokenizer())
.useFallbackTokenizer(new TextTokenizer())
// add custom tokenizers through `useTokenizer()`
.useTokenizer(new EmphasisTokenizer())
// Parse literal string
parser.parse('source content such as **emphasis**')
// Parse literal string lists
parser.parse(['source content', 'additional *content*'])
// Parse iterable string
// The Parser will processing at every time a line is read.
function* genContents () {
yield 'source content1\n'
yield 'source content2\n'
return 'source content3\n'
}
parser.parse(genContents())
// Reserve the positions
parser.parse('source content', { shouldReservePosition: true })
DefaultParser Options
-
Constructor Options
Name Required Default Type blockFallbackTokenizer
false
null
BlockFallbackTokenizer | null
inlineFallbackTokenizer
false
null
InlineFallbackTokenizer | null
defaultParseOptions
false
See below Parse Options -
blockFallbackTokenizer
: Fallback tokenizer on processing block structure phase. -
inlineFallbackTokenizer
: Fallback tokenizer on processing inline structure phase. -
defaultParseOptions
: Default options forparse()
. Default:{
presetDefinitions: [],
presetFootnoteDefinitions: [],
shouldReservePosition: false
}
-
-
Parse Options
Name Required Default Type presetDefinitions
false
defaultParserOptions.presetDefinitions
[Association][src-Association] list. presetFootnoteDefinitions
false
defaultParserOptions.presetFootnoteDefinitions
boolean
shouldReservePosition
false
defaultParserOptions.shouldReservePosition
boolean
presetDefinitions
: Preset definition meta data list.presetFootnoteDefinitions
: Preset footnote definition meta data list.shouldReservePosition
: Whether it is necessary to reserve the position of the Node parsed.
Lifecycle
math-block
-
API: IMatchBlockPhaseApi
Name Description extractPhrasingLines
Extract phrasing content lines from block token. rollbackPhrasingLines
Re-match token from phrasing content lines. registerDefinitionIdentifier
Register a definition identifier. registerFootnoteDefinitionIdentifier
Register a footnote definition identifier. -
Hook: IMatchBlockHook
Name Required Description isContainingBlock
true
Whether if it is a container block. eatOpener
true
Try to match new block data. eatAndInterruptPreviousSibling
false
Try to interrupt the eatContinuationText action of the last sibling node. eatContinuationText
false
Try to eat the Continuation Text, and check if it is still satisfied to current opening MatchToken eatLazyContinuationText
false
Try to eat the Laziness Continuation Text, and check if it is still satisfied to current opening MatchToken onClose
false
Called when the token is saturated. -
Example tokenizers:
- @yozora/tokenizer-admonition
- @yozora/tokenizer-blockquote
- @yozora/tokenizer-fenced-code
- @yozora/tokenizer-heading
- @yozora/tokenizer-html-block
- @yozora/tokenizer-indented-code
- @yozora/tokenizer-definition
- @yozora/tokenizer-list
- @yozora/tokenizer-math
- @yozora/tokenizer-paragraph
- @yozora/tokenizer-setext-heading
- @yozora/tokenizer-table
- @yozora/tokenizer-thematic-break
parse-block
-
API: IParseBlockPhaseApi
Name Description shouldReservePosition
Whether it is necessary to reserve the position in the Node produced. processInlines
Process node points into inline nodes. parseBlockTokens
Parse block tokens to Yozora AST nodes. -
Hook: IParseBlockHook
Name Required Description parse
true
Processing a specified token list to Node list. -
Example tokenizers:
- @yozora/tokenizer-admonition
- @yozora/tokenizer-blockquote
- @yozora/tokenizer-fenced-code
- @yozora/tokenizer-heading
- @yozora/tokenizer-html-block
- @yozora/tokenizer-indented-code
- @yozora/tokenizer-definition
- @yozora/tokenizer-list
- @yozora/tokenizer-math
- @yozora/tokenizer-paragraph
- @yozora/tokenizer-setext-heading
- @yozora/tokenizer-table
- @yozora/tokenizer-thematic-break
match-inline
-
API: IMatchInlinePhaseApi
Name Description hasDefinition
Check if there is exists a definition with the given identifier. hasFootnoteDefinition
Check if there is exists a footnote definition with the given identifier. getNodePoints
Get the node points. getBlockStartIndex
Start index of current block token. getBlockEndIndex
End index of current block token. resolveFallbackTokens
Resolve fallback inline tokens. resolveInternalTokens
Resolve raw contents with the fallback inline tokenizer. -
Hook: IMatchInlineHook
Name Required Description findDelimiter
true
Find an inline token delimiter. isDelimiterPair
false
Check if the given two delimiters can be combined into a pair. processDelimiterPair
false
Process a pair of delimiters. processSingleDelimiter
false
Process a single delimiter (its type should be one of 'both' and 'full'). -
Example tokenizers:
- @yozora/tokenizer-autolink
- @yozora/tokenizer-autolink-extension
- @yozora/tokenizer-break
- @yozora/tokenizer-delete
- @yozora/tokenizer-emphasis
- @yozora/tokenizer-html-inline
- @yozora/tokenizer-image
- @yozora/tokenizer-image-reference
- @yozora/tokenizer-inline-code
- @yozora/tokenizer-inline-math
- @yozora/tokenizer-link
- @yozora/tokenizer-link-reference
- @yozora/tokenizer-text
parse-inline
-
API: IParseInlinePhaseApi
Name Description shouldReservePosition
Whether it is necessary to reserve the position in the Node produced. getNodePoints
Get the node points. calcPosition
Calculate position of token. hasDefinition
Check if there is exists a definition with the given identifier. hasFootnoteDefinition
Check if there is exists a footnote definition with the given identifier. parseInlineTokens
Parse inline tokens to Yozora AST nodes. -
Hook: IParseInlineHook
Name Required Description parse
true
Processing token list to Node list. -
Example tokenizers:
- @yozora/tokenizer-autolink
- @yozora/tokenizer-autolink-extension
- @yozora/tokenizer-break
- @yozora/tokenizer-delete
- @yozora/tokenizer-emphasis
- @yozora/tokenizer-html-inline
- @yozora/tokenizer-image
- @yozora/tokenizer-image-reference
- @yozora/tokenizer-inline-code
- @yozora/tokenizer-inline-math
- @yozora/tokenizer-link
- @yozora/tokenizer-link-reference
- @yozora/tokenizer-text