@yozora/tokenizer-list

github flavor markdown spec

A list is a sequence of one or more list items of the same type. The list items may be separated by any number of blank lines.

Two list items are of the same type if they begin with a list marker of the same type. Two list markers are of the same type if

• (a) they are bullet list markers using the same character (-, +, or *) or

• (b) they are ordered list numbers with the same delimiter (either . or )).

A list is an ordered list if its constituent list items begin with ordered list markers, and a bullet list if its constituent list items begin with bullet list markers.

The start number of an ordered list is determined by the list number of its initial list item. The numbers of subsequent list items are disregarded.

A list is loose if any of its constituent list items are separated by blank lines, or if any of its constituent list items directly contain two block-level elements with a blank line between them. Otherwise a list is tight. (The difference in HTML output is that paragraphs in a loose list are wrapped in <p> tags, while paragraphs in a tight list are not.)

• See github flavor markdown spec for details.
• See Live Examples for an intuitive impression.

---

A list marker is a bullet list marker or an ordered list marker.

A bullet list marker is a -, +, or * character.

An ordered list markeris a sequence of $1-9$ arabic digits (0-9), followed by either a . character or a ) character. (The reason for the length limit is that with 10 digits we start seeing integer overflows in some browsers.)

The following rules define list items:

1. Basic case. If a sequence of lines $L_s$ constitute a sequence of blocks $B_s$ starting with a non-whitespace character, and $M$ is a list marker of width $W$ followed by $1 \leqslant N \leqslant 4$ spaces, then the result of prepending $M$ and the following spaces to the first line of $L_s$, and indenting subsequent lines of $L_s$ by $W + N$ spaces, is a list item with Bs as its contents. The type of the list item (bullet or ordered) is determined by the type of its list marker. If the list item is ordered, then it is also assigned a start number, based on the ordered list marker.

Exceptions:

1. When the first list item in a list interrupts a paragraph—that is, when it starts on a line that would otherwise count as paragraph continuation text—then

   • (a) the lines Ls must not begin with a blank line, and
   • (b) if the list item is ordered, the start number must be $1$.

2. If any line is a thematic break then that line is not a list item.

3. Item starting with indented code. If a sequence of lines $L_s$ constitute a sequence of blocks $B_s$ starting with an indented code block, and $M$ is a list marker of width $W$ followed by one space, then the result of prepending $M$ and the following space to the first line of $L_s$, and indenting subsequent lines of $L_s$ by $W + 1$ spaces, is a list item with $B_s$ as its contents. If a line is empty, then it need not be indented. The type of the list item (bullet or ordered) is determined by the type of its list marker. If the list item is ordered, then it is also assigned a start number, based on the ordered list marker.

4. Item starting with a blank line. If a sequence of lines $L_s$ starting with a single blank line constitute a (possibly empty) sequence of blocks $B_s$, not separated from each other by more than one blank line, and $M$ is a list marker of width $W$, then the result of prepending $M$ to the first line of $L_s$, and indenting subsequent lines of $L_s$ by $W + 1$ spaces, is a list item with $B_s$ as its contents. If a line is empty, then it need not be indented. The type of the list item (bullet or ordered) is determined by the type of its list marker. If the list item is ordered, then it is also assigned a start number, based on the ordered list marker.

5. Indentation. If a sequence of lines $L_s$ constitutes a list item according to rule #1, #2, or #3, then the result of indenting each line of $L_s$ by $1-3$ spaces (the same for each line) also constitutes a list item with the same contents and attributes. If a line is empty, then it need not be indented.

6. Laziness. If a string of lines $L_s$ constitute a list item with contents $B_s$, then the result of deleting some or all of the indentation from one or more lines in which the next non-whitespace character after the indentation is paragraph continuation text is a list item with the same contents and attributes. The unindented lines are called lazy continuation lines.

• See github flavor markdown spec for details.
• See Live Examples for an intuitive impression.

Install

npm

npm install --save @yozora/tokenizer-list

Yarn

yarn add @yozora/tokenizer-list

pnpm

pnpm add @yozora/tokenizer-list

Usage

tip

@yozora/tokenizer-list has been integrated into @yozora/parser / @yozora/parser-gfm-ex / @yozora/parser-gfm, so you can use YozoraParser / GfmExParser / GfmParser directly.

Basic Usage

@yozora/tokenizer-list cannot be used alone, it needs to be registered in Parser as a plugin-in before it can be used.

import { DefaultParser } from '@yozora/core-parser'
import ParagraphTokenizer from '@yozora/tokenizer-paragraph'
import TextTokenizer from '@yozora/tokenizer-text'
import ListTokenizer from '@yozora/tokenizer-list'

const parser = new DefaultParser()
  .useFallbackTokenizer(new ParagraphTokenizer())
  .useFallbackTokenizer(new TextTokenizer())
  .useTokenizer(new ListTokenizer({
    emptyItemCouldNotInterruptedTypes: [ParagraphType],
    enableTaskListItem: false,
  }))

// parse source markdown content
parser.parse(`
- a
- b
  - c
  - d
  - e
- f
- g

---

- [ ] This is a TODO item.
- [-] This is a processing TODO item.
- [x] This is a finished TODO item.

---

1. This is an ordered list item

a. This is an another type of ordered list item
`)

YozoraParser

import YozoraParser from '@yozora/parser'

const parser = new YozoraParser()

// parse source markdown content
parser.parse(`
- a
- b
  - c
  - d
  - e
- f
- g

---

- [ ] This is a TODO item.
- [-] This is a processing TODO item.
- [x] This is a finished TODO item.

---

1. This is an ordered list item

a. This is an another type of ordered list item
`)

GfmParser

import GfmParser from '@yozora/parser-gfm'

const parser = new GfmParser()

// parse source markdown content
parser.parse(`
- a
- b
  - c
  - d
  - e
- f
- g

---

- [ ] This is a TODO item.
- [-] This is a processing TODO item.
- [x] This is a finished TODO item.

---

1. This is an ordered list item

a. This is an another type of ordered list item
`)

GfmExParser

import GfmExParser from '@yozora/parser-gfm-ex'

const parser = new GfmExParser()

// parse source markdown content
parser.parse(`
- a
- b
  - c
  - d
  - e
- f
- g

---

- [ ] This is a TODO item.
- [-] This is a processing TODO item.
- [x] This is a finished TODO item.

---

1. This is an ordered list item

a. This is an another type of ordered list item
`)

Options

Name	Type	Required	Default
name	string	false	"@yozora/tokenizer-list"
priority	number	false	TokenizerPriority.CONTAINING_BLOCK
emptyItemCouldNotInterruptedTypes	string[]	false	[ParagraphType, PhrasingContentType]
enableTaskListItem	boolean	false	false

• name: The unique name of the tokenizer, used to bind the token it generates, to determine the tokenizer that should be called in each life cycle of the token in the entire matching / parsing phase.

• priority: Priority of the tokenizer, determine the order of processing, high priority priority execution. interruptable. In addition, in the match-block stage, a high-priority tokenizer can interrupt the matching process of a low-priority tokenizer.

• emptyItemCouldNotInterruptedTypes: Specify an array of Node types that could not be interrupted by this Tokenizer if the current list-item is empty.

  See https://github.github.com/gfm/#example-263.

• enableTaskListItem: Should enable task list item (extension).

Types

@yozora/tokenizer-list produce List and ListItem type nodes. See @yozora/ast for full base types.

import type { ListItem, Parent } from '@yozora/ast'

export const ListType = 'list'
export type ListType = typeof ListType

/**
 * List represents a list of items.
 * @see https://github.com/syntax-tree/mdast#list
 * @see https://github.github.com/gfm/#list
 */
export interface List extends Parent<ListType> {
  /**
   * Whether it is an ordered lit.
   */
  ordered: boolean
  /**
   * The starting number of a ordered list-item.
   */
  start?: number
  /**
   * Marker of a unordered list-item, or delimiter of an ordered list-item.
   */
  marker: number
  /**
   * Whether if the list is loose.
   * @see https://github.github.com/gfm/#loose
   */
  spread: boolean
  /**
   * Lists are container block.
   */
  children: ListItem[]
}

Live Examples

• Basic.

  #231

  yozora

    

  pretty-json

    

  A paragraph with two lines. indented code > A block quote.

  1
  2
  3
  4
  5
  6
  A paragraph
  with two lines.
  
      indented code
  
  > A block quote.

• Item Starting with indented code.

  #248

  yozora

    

  pretty-json

    

  - foo bar

  1
  2
  3
  - foo
  
        bar

• Item Starting with a blank line.

  #256

  yozora

    

  pretty-json

    

  - foo - ``` bar ``` - baz

  1
  2
  3
  4
  5
  6
  7
  8
  -
    foo
  -
    ```
    bar
    ```
  -
        baz

• Indentation.

  #264

  yozora

    

  pretty-json

    

  1. A paragraph with two lines. indented code > A block quote.

  1
  2
  3
  4
  5
  6
   1.  A paragraph
       with two lines.
  
           indented code
  
       > A block quote.

• Lazyniess.

  #268

  yozora

    

  pretty-json

    

  1. A paragraph with two lines. indented code > A block quote.

  1
  2
  3
  4
  5
  6
    1.  A paragraph
  with two lines.
  
            indented code
  
        > A block quote.

• Indentation in sublist.

  #272

  yozora

    

  pretty-json

    

  - foo - bar - baz - boo

  1
  2
  3
  4
  - foo
    - bar
      - baz
        - boo

• Task list item (extension).

  #279

  gfm-ex

    

  pretty-json

    

  - [ ] foo - [x] bar

  1
  2
  - [ ] foo
  - [x] bar

• In order to solve of unwanted lists in paragraphs with hard-wrapped numerals, we allow only lists starting with $1$ to interrupt paragraphs.

  #284

  yozora

    

  pretty-json

    

  The number of windows in my house is 14. The number of doors is 6.

  1
  2
  The number of windows in my house is
  14.  The number of doors is 6.

• There can be any number of blank lines between items.

  #286

  yozora

    

  pretty-json

    

  - foo - bar - baz

  1
  2
  3
  4
  5
  6
  - foo
  
  - bar
  
  
  - baz

• To separate consecutive lists of the same type, or to separate a list from an indented code block that would otherwise be parsed as a subparagraph of the final list item, you can insert a blank HTML comment.

  #288

  yozora

    

  pretty-json

    

  - foo - bar <!-- --> - baz - bim

  1
  2
  3
  4
  5
  6
  7
  - foo
  - bar
  
  <!-- -->
  
  - baz
  - bim

• List items need not be indented to the same level. The following list items will be treated as items at the same list level, since none is indented enough to belong to the previous list item.

  #290

  yozora

    

  pretty-json

    

  - a - b - c - d - e - f - g

  1
  2
  3
  4
  5
  6
  7
  - a
   - b
    - c
     - d
    - e
   - f
  - g

• Note, however, that list items may not be indented more than three spaces. Here - e is treated as a paragraph continuation line, because it is indented more than three spaces.

  #292

  yozora

    

  pretty-json

    

  - a - b - c - d - e

  1
  2
  3
  4
  5
  - a
   - b
    - c
     - d
      - e

• And here, 3. c is treated as in indented code block, because it is indented four spaces and preceded by a blank line.

  #293

  yozora

    

  pretty-json

    

  1. a 2. b 3. c

  1
  2
  3
  4
  5
  1. a
  
    2. b
  
      3. c

• This is a loose list, because there is a blank line between two of the list items.

  #294

  yozora

    

  pretty-json

    

  - a - b - c

  1
  2
  3
  4
  - a
  - b
  
  - c

• So is this, with a empty second item.

  #295

  yozora

    

  pretty-json

    

  * a * * c

  1
  2
  3
  4
  * a
  *
  
  * c

• These are loose lists, even though there is no space between the items, because one of the items directly contains two block-level elements with a blank line between them.

  #296

  yozora

    

  pretty-json

    

  - a - b c - d

  1
  2
  3
  4
  5
  - a
  - b
  
    c
  - d

• This is a tight list, because the blank lines are in a code block.

  #298

  yozora

    

  pretty-json

    

  - a - ``` b ``` - c

  1
  2
  3
  4
  5
  6
  7
  - a
  - ```
    b
  
  
    ```
  - c

• This is a tight list, because the blank line is between two paragraphs of a sublist. So the sublist is loose while the outer list is tight.

  #299

  yozora

    

  pretty-json

    

  - a - b c - d

  1
  2
  3
  4
  5
  - a
    - b
  
      c
  - d

• This is a tight list, because the blank line is inside the block quote.

  #300

  yozora

    

  pretty-json

    

  * a > b > * c

  1
  2
  3
  4
  * a
    > b
    >
  * c

• This list is tight, because the consecutive block elements are not separated by blank lines.

  #301

  yozora

    

  pretty-json

    

  - a > b ``` c ``` - d

  1
  2
  3
  4
  5
  6
  - a
    > b
    ```
    c
    ```
  - d

• A single-paragraph list is tight.

  #302

  yozora

    

  pretty-json

    

  - a

  1
  - a

• This list is loose, because of the blank line between the two block elements in the list item.

  #304

  yozora

    

  pretty-json

    

  1. ``` foo ``` bar

  1
  2
  3
  4
  5
  1. ```
     foo
     ```
  
     bar

• Here the outer list is loose, the inner list tight.

  #305

  yozora

    

  pretty-json

    

  * foo * bar baz

  1
  2
  3
  4
  * foo
    * bar
  
    baz

Related

• @yozora/ast
• @yozora/parser
• @yozora/parser-gfm
• @yozora/parser-gfm-ex
• @yozora/react-list
• @yozora/react-list-item
• @yozora/react-markdown
• Sourcecode
• List | Yozora AST
• ListItem | Yozora AST
• Lists | Github Flavor Markdown Spec
• List items | Github Flavor Markdown Spec
• Task list items (extension) | Github Flavor Markdown Spec
• List | Mdast