Accessibility/MathML

Summary

This page describes a draft of changes to be made to add MathML accessibility to Gecko (WIP).

Background

Scope

For now, this will cover the translation of Presentation MathML to an accessible tree. Content MathML support can be added at a later time (display is currently unsupported by Gecko). An interface will be provided to allow ATs to traverse the MathML tree.

Element types

The following includes descriptions for the various types of MathML elements for the purpose of this document. Some examples are pulled from MDN.

Tokens

These are the smallest units in MathML carrying semantic meaning. They are often represented by a single symbol or number. These include the following:

• mi - identifier
• mn - number
• mo - operator
• mtext - text
• mspace - spacing
• ms - string literal

Valid token values include single characters, numbers, and many MathML-specific constants and symbols, e.g. α for the Greek lower alpha symbol. mtext is used for arbitrary text with no particular semantic meaning to the equation itself. ms is used for arbitrary string literals meant to be interpreted by a programming language or other system. A special case is mglyph (image), which is used inside other token elements when a Unicode character is unavailable. Example markup is listed below.

<math> 
  <mi>y</mi>  
  <mn>10</mn>
  <mo>+</mo>
  <mtext>Arbitrary text</mtext>
  <mspace width="1em"/>
  <ms>InterpretedStringLiteral</ms>
  <mi>
    <mglyph src="an-image" alt="an-image-description"/>
  </mi>
</math>

Layout

These have been split into three groups: layout elements that describe semantic meaning, layout elements only for formatting and styling, and layout elements for describing elementary math.

Semantic meaning

These carry semantic meaning in the presentation. Nested inside them are either more layout elements or tokens. These include the following:

• mrow - row, containing any number of elements
• mfrac - fraction, containing a numerator and denominator
• msqrt - square root, containing a base
• mroot - root, containing a base and index
• mfenced - fenced block, containing any number of elements
• menclose - enclosed block, containing any number of elements

mrow is used for grouping a logical sub-expression in an equation. mfenced is used for enclosing an expression in parentheses with separators. Custom values can be used (defaults to '(', ')', and ','). menclose is used to enclose an expression in a specified notation. Example markup is listed below.

<math>
  <mrow>
    <mfenced>
      <mrow>
        <mi>x</mi>
        <mo>+</mo>
        <mi>y</mi>
      </mrow>
    </mfenced>
    <mo>+</mo>
    <mrow>
      <mn>2</mn>
    </mrow>
  </mrow>
  <mfrac>
    <mi>x</mi>
    <mn>2</mn>
  </mfrac>
  <msqrt>
    <mn>2</mn>
  </msqrt>
  <mroot>
    <mi>x</mi>
    <mn>5</mn>
  </mroot>
  <mfenced>
    <mi>x</mi>
    <mi>y</mi>
    <mi>z</mi>
  </mfenced>
  <menclose notation="circle">
    <mi>a</mi>
    <mo>+</mo>
    <mi>b</mi>
  </menclose>
</math>

The above describes (x+y) + 2, x/2 as a fraction, the square root of 2, the 5th root of x, a 3D vector (x, y, z), and (a + b) circled.

Formatting

These add formatting information when displaying content. These include the following:

• mstyle - changes styling for child elements, containing any number of elements
• mpadded - adds padding for child elements, containing any number of elements
• mphantom - renders child elements invisibly, containing any number of elements

Example markup is listed below.

<math>
  <mstyle dir="rtl" mathcolor="blue">
    <mpadded height="100px" width="200px">
      <mi>x</mi>
      <mphantom>
        <mo>+</mo>
        <mi>y</mi>
      </mphantom>
    </mpadded>
  </mstyle>
</math>

The above describes the expression "x + y" rendered in blue from right to left, with extra padding, and with the "+ y" not displayed.

Elementary math

These describe elements for formatting elementary math problems. These include the following:

• mstack - rows of numbers aligned on digits, containing any number of elements
• mlongdiv - a long division container, containing a divisor, the result, then any number of elements
• msgroup - groups rows together for horizontal alignment, containing any number of elements
• msrow - a row in the stack, containing any number of elements
• mscarries - carries, borrows, and crossouts for the following row, containing any number of elements
• mscarry - a single carry, borrow, or crossout for a column, containing any number of elements
• msline - a drawn line in a stack

After the first two elements inside mlongdiv, it behaves like an mstack. The <none/> element is used to insert spacing, e.g. in msrow to offset an operator, or mscarries to denote no carrying needs to be done. mscarries contains mscarry elements or <none/>. Example markup is listed below.

<math>
  <mstack>
    <mscarries>
      <none/>
      <mscarry crossout="updiagonalstrike">
        <mn>1</mn>
      </mscarry>
      <mscarry location="w">
        <mn>1</mn>
      </mscarry>
    </mscarries>
    <mn>523</mn>
    <msrow>
      <mo>-</mo>
      <none/>
      <mn>15</mn>
    </msrow>
    <msline/>
    <mn>508</mn>
  </mstack>
  <mlongdiv>
    <mn>5</mn>
    <mn>1</mn>
    <mn>5</mn>
  </mlongdiv>
</math>

The above describes the subtraction of 15 from 523, with a borrow from the ten's column to the one's column, as well as the division of 5 by 5 resulting in 1.

Scripts and limits

These elements describe various scripts and limits. These include the following:

• msub - subscript, containing a base and a subscript
• msup - superscript, containing a base and a superscript
• msubsup - subscript and superscript, containing a base, a subscript, and a superscript
• munder - underscript, containing a base and an underscript
• mover - overscript, containing a base and an overscript
• munderover - underscript and overscript, containing a base, an underscript, and an overscript
• mmultiscripts - multiple scripts, with children described below

mmultiscripts uses a special syntax. The first child is the base, then a possible subscript and superscript. If prescripts are desired, a <mprescripts/> tag is included, then a possible subscript and superscript. <none/> can be substituted for scripts if no script is desired for each of these cases. Example markup is listed below.

<math>
  <msubsup>
    <mi>b</mi>
    <mn>1</mn>
    <mn>2</mn>
  </msubsup>
  <munderover>
    <mo>∑</mo>
    <mi>5</mi>
    <mrow>
      <mi>n</mi>
      <mo>=</mo>
      <mn>1</mn>
    </mrow>
  </munderover>
  <mi>n</mi>
  <mo>=</mo>
  <mn>15</mn>
  <mmultiscripts>
    <mi>x</mi>
    <none/>
    <mi>a</mi>
    <mprescripts/>
    <mi>b</mi>
    <none/>
  </mmultiscripts>
</math>

The above describes the base b with superscript 1 and subscript 2, the summation of n from 1 to 5 being 15, and x with post-superscript a and pre-subscript b.

Interface

nsIAccessibleMathML

Inherit from nsISupports.

readonly attribute AString tokenValue;
// Get token value for token MathML elements (mi, mo, mn, mtext, mspace, ms,
// mglyph).  Returns empty string if not a token element.

readonly attribute boolean isToken;
// Check if this element is a token MathML element.

readonly attribute nsIPersistentProperties attributes;
// Get attributes specific to the MathML element.

Two options for describing relations between MathML elements:

Option 1: Explicit readonly attributes (accessors) for going from parent elements to children and vice versa. e.g.

readonly attribute nsIAccessibleMathML radicand;
readonly attribute nsIAccessibleMathML rootIndex;
readonly attribute nsIAccessibleMathML under;
...
readonly attribute nsIAccessibleMathML parent;

Returns nullptr if the related accessible doesn't exist.

Option 2: Relations linking accessibles based on role, so we won't have attributes/functions in the interface. An example would be for a fraction structure, as follows.

<math>
  <mfrac>
    <mi>a</mi>
    <mn>2</mn>
  </mfrac>
</math>

mfrac would have RELATION_NUMERATOR_IS -> mi and RELATION_DENOMINATOR_IS -> mn or similar. mi would have RELATION_NUMERATOR_FOR -> mfrac and mn would have RELATION_DENOMINATOR_FOR -> mfrac.

MathMLAccessible

Inherit from AccessibleWrap, and implements nsIAccessibleMathML.

• NativeAttributes - Override Accessible, and append MathML specific attributes if the element accepts them.
• GetTokenValue - Sets token value string if applicable.
• GetIsToken - Returns true if element is a token element.

Roles

One role for each of the following constructs (41 of them):

Tokens

• mi
• mn
• mo
• mtext
• mspace
• ms
• mglyph

Layout (semantic meaning)

• mrow
• mfrac
• msqrt
• mroot
• mfenced
• menclose

Layout (formatting)

• mstyle
• mpadded
• mphantom

Layout (elementary math)

• mstack
• mlongdiv
• msgroup
• msrow
• mscarries
• mscarry
• msline

Script/limits

• msub
• msup
• msubsup
• munder
• mover
• munderover
• mmultiscripts

Tables/matrices

• mtable
• mlabeledtr
• mtr
• mtd

Table formatting

• maligngroup
• malignmark

Semantics annotation

• semantics
• annotation
• annotation-xml

Other

• maction
• merror

Sample Gecko roles

• MATHML_IDENTIFIER
• MATHML_NUMBER
• MATHML_OPERATOR
• MATHML_TEXT
• MATHML_SPACE
• MATHML_STRING_LITERAL
• MATHML_GLYPH
• MATHML_ROW
• MATHML_FRACTION
• MATHML_SQUARE_ROOT
• MATHML_ROOT
• MATHML_FENCED
• MATHML_ENCLOSED
• MATHML_STYLE
• MATHML_PADDED
• MATHML_PHANTOM
• MATHML_SUB
• MATHML_SUP
• MATHML_SUB_SUP
• MATHML_UNDER
• MATHML_OVER
• MATHML_UNDER_OVER
• MATHML_MULTISCRIPTS
• MATHML_TABLE
• MATHML_LABELED_ROW
• MATHML_TABLE_ROW
• MATHML_CELL
• MATHML_ALIGNMENT_GROUP
• MATHML_ALIGNMENT_MARK
• MATHML_ACTION
• MATHML_ERROR
• MATHML_SEMANTICS
• MATHML_ANNOTATION
• MATHML_XML_ANNOTATION
• MATHML_STACK
• MATHML_LONG_DIVISION
• MATHML_STACK_GROUP
• MATHML_STACK_ROW
• MATHML_STACK_CARRIES
• MATHML_STACK_CARRY
• MATHML_STACK_LINE