Metadata-Version: 2.4
Name: Mathics_Scanner
Version: 2.0.0
Summary: Character Tables and Tokenizer for Mathics and the Wolfram Language.
Maintainer: Mathics3 Group
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://mathics.org/
Keywords: Mathematica,Wolfram,Interpreter,Shell,Math,CAS
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Topic :: Scientific/Engineering :: Physics
Classifier: Topic :: Software Development :: Interpreters
Requires-Python: >=3.10
Description-Content-Type: text/x-rst
License-File: COPYING.txt
License-File: AUTHORS.txt
Requires-Dist: PyYAML
Requires-Dist: chardet
Requires-Dist: click
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Provides-Extra: full
Requires-Dist: ujson; extra == "full"
Dynamic: license-file

|Workflows| |Pypi Installs| |Latest Version| |Supported Python Versions|

|Packaging status|

Mathics Character Tables and Scanner
=====================================

This repository really contains two things:

* extensive tables describing WL symbols, operators, and their properties
* a tokenizer or scanner portion for the Wolfram Language.

Concerning the first item, there are two commented YAML files that contain a
full set of translations between:

* Wolfram Language named characters,
* Unicode/ASCII equivalents and Unicode and WL code-points,
* AMSLaTeX equivalent names,
* Operator name (if symbol is an operator),
* Operator arity (if an operator)
* Operator precedence (if an operator)
* Operator associativity (if an operator)
* Keyboard escape sequences for the symbol

Uses
----

The scanner and character tables are used inside `Mathics3 <https://mathics.org>`_. However information can
also be used by other programs for tokenizing and formatting Wolfram Language code.

For example, tables are used in `mathics-pygments <https://pypi.org/project/Mathics-Scanner/>`_, a Pygments-based
lexer and highlighter for Mathematica/Wolfram Language source code.

This library may be useful if you need to work with the Wolfram Language
named character and convert it to various formats.

Usage
-----

- For tokenizing and scanning Wolfram Language code, use the
  ``mathics_scanner.tokenizer.Tokenizer`` class.
- To convert between Wolfram Language named characters and Unicode/ASCII, use
  the ``mathics_scanner.characters.replace_wl_with_plain_text`` and
  ``mathics_scanner.characters.replace_unicode_with_wl`` functions.
- To convert between qualified names of named characters (such ``FormalA`` for
  ``\[FormalA]``) and Wolfram's internal representation use the
  ``m̀athics_scanner.characters.named_characters`` dictionary.

To regenerate JSON-format tables, run:

::

   $ mathics3-generate-json-table
   $ mathics3-generate-operator-json-table

Without options, ``mathics3-generate-json-table`` produces the maximum set of correspondences.

In most applications, though, you may need just a few of these. The
``--field`` option can be used to narrow the list of entries to output in JSON. Run
``mathics3-generate-json-table --help`` for a full list of fields.


Utility for showing token parsing
---------------------------------

There is a command-line utility, ``mathics3-tokens``, for showing how
an input stream is tokenized. The ``--CodeTokenize`` or ``-C`` option
will try to show the token as it would appear using CodeTokenize. type
``mathics3-tokens --help`` information on command-line options.

Implementation
--------------

For notes on the implementation of the packages or details on the conversion
scheme please read `Implementation <https://mathics-scanner.readthedocs.io/en/latest/implementation.html>`_.

Contributing
------------

Please feel encouraged to contribute to this package or Mathics3! Create your
own fork, make the desired changes, commit, and make a pull request.

License
-------

Mathics3 is released under the GNU General Public License Version 3 (GPL3).

.. |Workflows| image:: https://github.com/Mathics3/mathics-scanner/actions/workflows/ubuntu.yml/badge.svg
.. |Packaging status| image:: https://repology.org/badge/vertical-allrepos/mathics-scanner.svg
			    :target: https://repology.org/project/mathics-scanner/versions
.. |Latest Version| image:: https://badge.fury.io/py/Mathics-Scanner.svg
		 :target: https://badge.fury.io/py/Mathics-Scanner
.. |Pypi Installs| image:: https://pepy.tech/badge/Mathics-Scanner
.. |Supported Python Versions| image:: https://img.shields.io/pypi/pyversions/Mathics-Scanner.svg
