Accessibility/MathML
Summary
This page describes a draft of changes to be made to add MathML accessibility to Gecko (WIP).
Background
Scope
For now, this will cover the translation of Presentation MathML to an accessible tree. Content MathML support can be added at a later time (display is currently unsupported by Gecko). An interface will be provided to allow ATs to traverse the MathML tree.
Element types
The following includes descriptions and examples for the various types of MathML elements for the purpose of this document. Some examples are pulled from MDN.
Tokens
These are the smallest units in MathML carrying semantic meaning. They are often represented by a single symbol or number. These include the following:
- mi - identifier
- mn - number
- mo - operator
- mtext - text
- mspace - spacing
- ms - string literal
Valid token values include single characters, numbers, and many MathML-specific constants and symbols, e.g. α for the Greek lower alpha symbol. mtext is used for arbitrary text with no particular semantic meaning to the equation itself. ms is used for arbitrary string literals meant to be interpreted by a programming language or other system. A special case is mglyph (image), which is used inside other token elements when a Unicode character is unavailable. Another special case is the malignmark element, which will be described below in the table and matrix section. Example markup is listed below.
<math> <mi>y</mi> <mo>+</mo> <mn>10</mn> <mspace width="1em"/> <mtext>Arbitrary text</mtext> <mspace width="1em"/> <ms>InterpretedStringLiteral</ms> <mspace width="1em"/> <mi> <mglyph src="x.png" alt="x"/> </mi> </math>
Layout
These have been split into three groups: layout elements that describe semantic meaning, layout elements only for formatting and styling, and layout elements for describing elementary math.
Semantic meaning
These carry semantic meaning in the presentation. Nested inside them are either more layout elements or tokens. These include the following:
- mrow - row, containing any number of elements
- mfrac - fraction, containing a numerator and denominator
- msqrt - square root, containing a base
- mroot - root, containing a base and index
- mfenced - fenced block, containing any number of elements
- menclose - enclosed block, containing any number of elements
mrow is used for grouping a logical sub-expression in an equation. mfenced is used for enclosing an expression in parentheses with separators. Custom values can be used (defaults to '(', ')', and ','). menclose is used to enclose an expression in a specified notation. Example markup is listed below.
<math> <mrow> <mfenced> <mrow> <mi>x</mi> <mo>+</mo> <mi>y</mi> </mrow> </mfenced> <mo>+</mo> <mrow> <mn>2</mn> </mrow> </mrow> <mspace width="1em"/> <mfrac> <mi>x</mi> <mn>2</mn> </mfrac> <mspace width="1em"/> <msqrt> <mn>2</mn> </msqrt> <mspace width="1em"/> <mroot> <mi>x</mi> <mn>5</mn> </mroot> <mspace width="1em"/> <mfenced> <mi>x</mi> <mi>y</mi> <mi>z</mi> </mfenced> <mspace width="1em"/> <menclose notation="circle"> <mi>a</mi> <mo>+</mo> <mi>b</mi> </menclose> </math>
Formatting
These add formatting information when displaying content. These include the following:
- mstyle - changes styling for child elements, containing any number of elements
- mpadded - adds padding for child elements, containing any number of elements
- mphantom - renders child elements invisibly, containing any number of elements
Example markup is listed below.
<math> <mstyle dir="rtl" mathcolor="blue"> <mpadded height="100px" width="200px"> <mi>x</mi> <mphantom> <mo>+</mo> <mi>y</mi> </mphantom> </mpadded> </mstyle> </math>
Elementary math
These describe elements for formatting elementary math problems. These include the following:
- mstack - rows of numbers aligned on digits, containing any number of elements
- mlongdiv - a long division container, containing a divisor, the result, then any number of elements
- msgroup - groups rows together for horizontal alignment, containing any number of elements
- msrow - a row in the stack, containing any number of elements
- mscarries - carries, borrows, and crossouts for the following row, containing any number of elements
- mscarry - a single carry, borrow, or crossout for a column, containing any number of elements
- msline - a drawn line in a stack
After the first two elements inside mlongdiv, it behaves like an mstack. The <none/> element is used to insert spacing, e.g. in msrow to offset an operator, or mscarries to denote no carrying needs to be done. mscarries contains mscarry elements or <none/>. Example markup is listed below.
<math> <mstack> <mscarries> <none/> <mscarry crossout="updiagonalstrike"> <mn>1</mn> </mscarry> <mscarry location="w"> <mn>1</mn> </mscarry> </mscarries> <mn>523</mn> <msrow> <mo>-</mo> <none/> <mn>15</mn> </msrow> <msline/> <mn>508</mn> </mstack> <mspace width="1em"/> <mlongdiv> <mn>5</mn> <mn>1</mn> <mn>5</mn> </mlongdiv> </math>
Scripts and limits
These elements describe various scripts and limits. These include the following:
- msub - subscript, containing a base and a subscript
- msup - superscript, containing a base and a superscript
- msubsup - subscript and superscript, containing a base, a subscript, and a superscript
- munder - underscript, containing a base and an underscript
- mover - overscript, containing a base and an overscript
- munderover - underscript and overscript, containing a base, an underscript, and an overscript
- mmultiscripts - multiple scripts, with children described below
mmultiscripts uses a special syntax. The first child is the base, then any number of pairs of post-subscripts and post-superscripts. If prescripts are desired, a <mprescripts/> tag is included, then any number of pairs of pre-subscripts and pre-superscripts. <none/> can be substituted for scripts if no script is desired in a particular pair. Example markup is listed below.
<math display="block"> <msubsup> <mi>b</mi> <mn>1</mn> <mn>2</mn> </msubsup> <mspace width="1em"/> <munderover> <mo>∑</mo> <mrow> <mi>n</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>5</mi> </munderover> <mi>n</mi> <mo>=</mo> <mn>15</mn> <mspace width="1em"/> <mmultiscripts> <mi>x</mi> <none/> <mi>a</mi> <mprescripts/> <mi>b</mi> <none/> </mmultiscripts> </math>
Tables/matrices
These elements describe a table or matrix. This is also used for layout. These include the following:
- mtable - the container for a table, contains any number of mtr or mlabeledtr elements
- mtr - a row, containing any number of mtd elements
- mlabeledtr - a labelled row, having a caption on the side, containing any number of mtd elements
- mtd - a table cell, containing any number of elements
For mlabeledtr, the first mtd contained inside becomes the caption.
In addition, there are two elements used for table formatting:
- maligngroup - denotes a column in a table, used for alignment
- malignmark - denotes an alignment point inside an alignment group
malignmark is used inside a token element to set the alignment point. Example markup is listed below.
<math> <mrow> <mo>[</mo> <mtable> <mtr> <mtd> <mn>1</mn> </mtd> <mtd> <mn>0</mn> </mtd> </mtr> <mtr> <mtd> <mn>0</mn> </mtd> <mtd> <mn>1</mn> </mtd> </mtr> </mtable> <mo>]</mo> </mrow> <mspace width="1em"/> <mtable> <mlabeledtr> <mtd> <mtext>Volume of a sphere</mtext> </mtd> <mtd> <mi>V</mi> <mo>=</mo> <mfrac> <mn>4</mn> <mn>3</mn> </mfrac> <mo></mo> <mi>π</mi> <mo></mo> <msup> <mi>r</mi> <mn>3</mn> </msup> </mtd> </mlabeledtr> </mtable> <mspace width="1em"/> <mtable> <mtr> <mrow> <mn>3</mn> <mo></mo> <maligngroup/> <mi>x</mi> <maligngroup/> <mo>+</mo> <maligngroup/> <mi>y</mi> <maligngroup/> <mo>=</mo> <maligngroup/> <mn>7</mn> </mrow> </mtr> <mtr> <mrow> <mphantom> <mn>0</mn> <mo></mo> <maligngroup/> <mi>x</mi> <maligngroup/> <mo>+</mo> </mphantom> <maligngroup/> <mi>y</mi> <maligngroup/> <mo>=</mo> <maligngroup/> <mn>1</mn> </mrow> </mtr> </mtable> </math>
Semantics annotation
These elements provide explicit annotation for MathML. These include the following:
- semantics - starts a semantic annotation block, containing MathML and annotations
- annotation - annotation information in non-XML format
- annotation-xml - annotation information in XML format
The first child of semantics is the content being annotated, with the following annotation elements annotating the content.
Example markup is listed below.
<math> <semantics> <!-- Presentation MathML --> <mrow> <msup> <mi>x</mi> <mn>2</mn> </msup> <mo>+</mo> <mi>y</mi> </mrow> <!-- Content MathML --> <annotation-xml encoding="MathML-Content"> <apply> <plus/> <apply> <power/> <ci>x</ci> <cn type="integer">2</cn> </apply> <ci>y</ci> </apply> </annotation-xml> <!-- annotate TeX --> <annotation encoding="application/x-tex"> x^{2} + y </annotation> </semantics> </math>
Other
Other elements not fitting into the above categories.
- maction - binds actions to expressions
- merror - displays content as an error message
maction elements allow for interactivity with MathML content. Example markup is listed below.
<math> <merror> <mrow> <mtext>Division by zero: </mtext> <mfrac> <mn>1</mn> <mn>0</mn> </mfrac> </mrow> </merror> </math>
<math> <maction actiontype="toggle"> <mfrac> <mn>6</mn> <mn>8</mn> </mfrac> <mfrac> <mrow> <mn>3</mn> <mo>⋅</mo> <mn>2</mn> </mrow> <mrow> <mn>4</mn> <mo>⋅</mo> <mn>2</mn> </mrow> </mfrac> <mfrac> <mn>3</mn> <mn>4</mn> </mfrac> </maction> </math>
Interface
nsIAccessibleMathML
Inherit from nsISupports.
readonly attribute AString tokenValue; // Get token value for token MathML elements (mi, mo, mn, mtext, mspace, ms, // mglyph). Returns empty string if not a token element.
There's several different options for describing relations between MathML elements:
Option 1: Relations linking accessibles based on role, so we won't have attributes/functions in the interface. An example would be for a fraction structure, as follows.
<math> <mfrac> <mi>a</mi> <mn>2</mn> </mfrac> </math>
mfrac would have RELATION_NUMERATOR_IS -> mi and RELATION_DENOMINATOR_IS -> mn or similar. mi would have RELATION_NUMERATOR_FOR -> mfrac and mn would have RELATION_DENOMINATOR_FOR -> mfrac.
A list of potential relations is as follows:
- MATHML_NUMERATOR_FOR
- MATHML_NUMERATOR_IS
- MATHML_DENOMINATOR_FOR
- MATHML_DENOMINATOR_IS
- MATHML_RADICAND_FOR
- MATHML_RADICAND_IS
- MATHML_ROOT_INDEX_FOR
- MATHML_ROOT_INDEX_IS
- MATHML_UNDER_FOR
- MATHML_UNDER_IS
- MATHML_OVER_FOR
- MATHML_OVER_IS
- MATHML_BASE_FOR
- MATHML_BASE_IS
- MATHML_SUBSCRIPT_FOR
- MATHML_SUBSCRIPT_IS
- MATHML_SUPERSCRIPT_FOR
- MATHML_SUPERSCRIPT_IS
- MATHML_PRE_SUBSCRIPT_FOR
- MATHML_PRE_SUBSCRIPT_IS
- MATHML_PRE_SUPERSCRIPT_FOR
- MATHML_PRE_SUPERSCRIPT_IS
- MATHML_POST_SUBSCRIPT_FOR
- MATHML_POST_SUBSCRIPT_IS
- MATHML_POST_SUPERSCRIPT_FOR
- MATHML_POST_SUPERSCRIPT_IS
- MATHML_ROW_FOR
- MATHML_IN_ROW
Option 2: Explicit readonly attributes (accessors) for going from parent elements to children and vice versa. e.g.
readonly attribute nsIAccessibleMathML radicand; readonly attribute nsIAccessibleMathML rootIndex; readonly attribute nsIAccessibleMathML under; ... readonly attribute nsIAccessibleMathML parent;
Returns nullptr if the related accessible doesn't exist. Note that specialised elements such as mrow or mmultiscripts would return arrays of elements.
nsIAccessibleMathMLTable
Inherit from nsISupports.
/** * Return columns count in the table. */ readonly attribute long columnCount; /** * Return rows count in the table. */ readonly attribute long rowCount; /** * Return the MathML accessible object at the given row and column in the * table. * * @param rowIndex [in] the row index to retrieve the cell at * @param columnIndex [in] the column index to retrieve the cell at */ nsIAccessibleMathML getCellAt(in long rowIndex, in long columnIndex); /** * Return the number of columns occupied by the MathML accessible cell at * the specified row and column in the table. The result differs from 1 if * the specified cell spans multiple columns. * * @param row [in] row index of the cell to return the column extent for * @param column [in] column index of the cell to return the column extent * for */ long getColumnExtentAt(in long row, in long column); /** * Return the number of rows occupied by the MathML accessible cell at the * specified row and column in the table. The result differs from 1 if the * specified cell spans multiple rows. * * @param row [in] row index of the cell to return the column extent for * @param column [in] column index of the cell to return the column extent * for */ long getRowExtentAt(in long row, in long column); /** * Return the label MathML accessible of the specified row in the table if * available (mlabeledtr), or nullptr otherwise. * * @param rowIndex [in] the row index to retrieve label mtd for */ nsIAccessibleMathML getRowLabel(in long rowIndex);
nsIAccessibleMathMLTableCell
Inherit from nsISupports.
/** * Return container MathML table accessible. */ readonly attribute nsIAccessibleMathMLTable table; /** * Return column index of this cell. */ readonly attribute long columnIndex; /** * Return row index of this cell. */ readonly attribute long rowIndex; /** * Return the number of columns occupied by this cell. The result differs * from 1 if the specified cell spans multiple columns. */ readonly attribute long columnExtent; /** * Return the number of rows occupied by this accessible cell. The result * differs from 1 if the specified cell spans multiple rows. */ readonly attribute long rowExtent;
MathMLAccessible
Inherit from AccessibleWrap, and implements nsIAccessibleMathML.
- NativeAttributes - Override Accessible, and append MathML specific attributes if the element accepts them.
- RelationByType - Override Accessible, process MathML specific relations if we go that route.
- GetTokenValue - Sets token value string if applicable.
- GetIsToken - Returns true if element is a token element.
Roles
One role for each of the following constructs (41 of them):
Tokens
- mi
- mn
- mo
- mtext
- mspace
- ms
- mglyph
Layout (semantic meaning)
- mrow
- mfrac
- msqrt
- mroot
- mfenced
- menclose
Layout (formatting)
- mstyle
- mpadded
- mphantom
Layout (elementary math)
- mstack
- mlongdiv
- msgroup
- msrow
- mscarries
- mscarry
- msline
Script/limits
- msub
- msup
- msubsup
- munder
- mover
- munderover
- mmultiscripts
Tables/matrices
- mtable
- mlabeledtr
- mtr
- mtd
Table formatting
- maligngroup
- malignmark
Semantics annotation
- semantics
- annotation
- annotation-xml
Other
- maction
- merror
Sample Gecko roles
- MATHML_IDENTIFIER
- MATHML_NUMBER
- MATHML_OPERATOR
- MATHML_TEXT
- MATHML_SPACE
- MATHML_STRING_LITERAL
- MATHML_GLYPH
- MATHML_ROW
- MATHML_FRACTION
- MATHML_SQUARE_ROOT
- MATHML_ROOT
- MATHML_FENCED
- MATHML_ENCLOSED
- MATHML_STYLE
- MATHML_PADDED
- MATHML_PHANTOM
- MATHML_SUB
- MATHML_SUP
- MATHML_SUB_SUP
- MATHML_UNDER
- MATHML_OVER
- MATHML_UNDER_OVER
- MATHML_MULTISCRIPTS
- MATHML_TABLE
- MATHML_LABELED_ROW
- MATHML_TABLE_ROW
- MATHML_CELL
- MATHML_ALIGNMENT_GROUP
- MATHML_ALIGNMENT_MARK
- MATHML_ACTION
- MATHML_ERROR
- MATHML_SEMANTICS
- MATHML_ANNOTATION
- MATHML_XML_ANNOTATION
- MATHML_STACK
- MATHML_LONG_DIVISION
- MATHML_STACK_GROUP
- MATHML_STACK_ROW
- MATHML_STACK_CARRIES
- MATHML_STACK_CARRY
- MATHML_STACK_LINE