@yozora/tokenizer-indented-code

github flavor markdown spec

An indented code block is composed of one or more indented chunks separated by blank lines. An indented chunk is a sequence of non-blank lines, each indented four or more spaces. The contents of the code block are the literal contents of the lines, including trailing line endings, minus four spaces of indentation. An indented code block has no info string.

• See github flavor markdown spec for details.
• See Live Examples for an intuitive impression.

Install

npm

npm install --save @yozora/tokenizer-indented-code

Yarn

yarn add @yozora/tokenizer-indented-code

pnpm

pnpm add @yozora/tokenizer-indented-code

Usage

tip

@yozora/tokenizer-indented-code has been integrated into @yozora/parser / @yozora/parser-gfm-ex / @yozora/parser-gfm, so you can use YozoraParser / GfmExParser / GfmParser directly.

Basic Usage

@yozora/tokenizer-indented-code cannot be used alone, it needs to be registered in YastParser as a plugin-in before it can be used.

import { DefaultYastParser } from '@yozora/core-parser'
import ParagraphTokenizer from '@yozora/tokenizer-paragraph'
import TextTokenizer from '@yozora/tokenizer-text'
import IndentedCodeTokenizer from '@yozora/tokenizer-indented-code'

const parser = new DefaultYastParser()
  .useBlockFallbackTokenizer(new ParagraphTokenizer())
  .useInlineFallbackTokenizer(new TextTokenizer())
  .useTokenizer(new IndentedCodeTokenizer())

// parse source markdown content
parser.parse(`
    <a/>
    *hi*
  
    - one
`)

YozoraParser

import YozoraParser from '@yozora/parser'

const parser = new YozoraParser()

// parse source markdown content
parser.parse(`
    <a/>
    *hi*
  
    - one
`)

GfmParser

import GfmParser from '@yozora/parser-gfm'

const parser = new GfmParser()

// parse source markdown content
parser.parse(`
    <a/>
    *hi*
  
    - one
`)

GfmExParser

import GfmExParser from '@yozora/parser-gfm-ex'

const parser = new GfmExParser()

// parse source markdown content
parser.parse(`
    <a/>
    *hi*
  
    - one
`)

Options

Name	Type	Required	Default
name	string	false	"@yozora/tokenizer-indented-code"
priority	number	false	TokenizerPriority.ATOMIC

• name: The unique name of the tokenizer, used to bind the token it generates, to determine the tokenizer that should be called in each life cycle of the token in the entire matching / parsing phase.

• priority: Priority of the tokenizer, determine the order of processing, high priority priority execution. interruptable. In addition, in the match-block stage, a high-priority tokenizer can interrupt the matching process of a low-priority tokenizer.

Types

@yozora/tokenizer-indented-code produce Code type nodes. See @yozora/ast for full base types.

import type { YastLiteral } from '@yozora/ast'

export const CodeType = 'code'
export type CodeType = typeof CodeType

/**
 * Code represents a block of preformatted text, such as ASCII art or computer
 * code.
 * @see https://github.com/syntax-tree/mdast#code
 * @see https://github.github.com/gfm/#code-fence
 */
export interface Code extends YastLiteral<CodeType> {
  /**
   * Language of the codes
   */
  lang?: string
  /**
   * Meta info string
   */
  meta?: string
}

Live Examples

• Basic.

  #77

  yozora

    

  pretty-json

    

  a simple indented code block

  1
  2
      a simple
        indented code block

• If there is any ambiguity between an interpretation of indentation as a code block and as indicating that material belongs to a list item, the list item interpretation takes precedence.

  #78

  yozora

    

  pretty-json

    

  - foo bar

  1
  2
  3
    - foo
  
      bar

• The contents of a code block are literal text, and do not get parsed as Markdown.

  #80

  yozora

    

  pretty-json

    

  <a/> *hi* - one

  1
  2
  3
  4
      <a/>
      *hi*
  
      - one

• Here we have three chunks separated by blank lines.

  #81

  yozora

    

  pretty-json

    

  chunk1 chunk2 chunk3

  1
  2
  3
  4
  5
  6
  7
      chunk1
  
      chunk2
    
   
   
      chunk3

• Any initial spaces beyond four will be included in the content, even in interior blank lines.

  #82

  yozora

    

  pretty-json

    

  chunk1 chunk2

  1
  2
  3
      chunk1
        
        chunk2

• An indented code block cannot interrupt a paragraph. (This allows hanging indents and the like.)

  #83

  yozora

    

  pretty-json

    

  Foo bar

  1
  2
  Foo
      bar

• However, any non-blank line with fewer than four leading spaces ends the code block immediately. So a paragraph may occur immediately after indented code.

  #84

  yozora

    

  pretty-json

    

  foo bar

  1
  2
      foo
  bar

• And indented code can occur immediately before and after other kinds of blocks.

  #85

  yozora

    

  pretty-json

    

  # Heading foo Heading ------ foo ----

  1
  2
  3
  4
  5
  6
  # Heading
      foo
  Heading
  ------
      foo
  ----

• The first line can be indented more than four spaces.

  #86

  yozora

    

  pretty-json

    

  foo bar

  1
  2
          foo
      bar

• Blank lines preceding or following an indented code block are not included in it.

  #87

  yozora

    

  pretty-json

    

  foo

  1
  2
  3
  4
  
      
      foo
      

• Trailing spaces are included in the code block’s content.

  #88

  yozora

    

  pretty-json

    

  foo

  1
      foo  

Related

• @yozora/ast
• @yozora/parser
• @yozora/parser-gfm
• @yozora/parser-gfm-ex
• @yozora/tokenizer-fenced-code
• @yozora/tokenizer-inline-code
• @yozora/react-code
• @yozora/react-markdown
• Sourcecode
• Code| Yozora AST
• Indented code blocks | Github Flavor Markdown Spec
• Code | Mdast