From 67e57d74929200757ae4946f9c2279ba9bf583e9 Mon Sep 17 00:00:00 2001 From: "Elf M. Sternberg" Date: Fri, 2 Sep 2016 21:12:04 -0700 Subject: [PATCH] ELI5-ification underway\! --- README.rst | 106 +++++++------- docs/_sources/details.txt | 111 ++++++++++++++ docs/_sources/eli5.txt | 37 +++++ docs/_sources/index.txt | 5 +- docs/_sources/installation.txt | 14 +- docs/_sources/readme.txt | 2 +- docs/_sources/usage.txt | 19 ++- docs/contributing.html | 10 +- docs/details.html | 223 +++++++++++++++++++++++++++++ docs/doctrees/details.doctree | Bin 0 -> 29142 bytes docs/doctrees/eli5.doctree | Bin 0 -> 6701 bytes docs/doctrees/environment.pickle | Bin 9983 -> 18168 bytes docs/doctrees/index.doctree | Bin 6402 -> 5674 bytes docs/doctrees/installation.doctree | Bin 7732 -> 7567 bytes docs/doctrees/readme.doctree | Bin 2771 -> 15404 bytes docs/doctrees/usage.doctree | Bin 2939 -> 4735 bytes docs/eli5.html | 126 ++++++++++++++++ docs/index.html | 24 ++-- docs/installation.html | 20 +-- docs/objects.inv | 4 +- docs/readme.html | 76 +++++++++- docs/searchindex.js | 2 +- docs/src/details.rst | 111 ++++++++++++++ docs/src/eli5.rst | 37 +++++ docs/src/index.rst | 5 +- docs/src/installation.rst | 14 +- docs/src/readme.rst | 2 +- docs/src/usage.rst | 19 ++- docs/usage.html | 24 +++- 29 files changed, 873 insertions(+), 118 deletions(-) create mode 100644 docs/_sources/details.txt create mode 100644 docs/_sources/eli5.txt create mode 100644 docs/details.html create mode 100644 docs/doctrees/details.doctree create mode 100644 docs/doctrees/eli5.doctree create mode 100644 docs/eli5.html create mode 100644 docs/src/details.rst create mode 100644 docs/src/eli5.rst diff --git a/README.rst b/README.rst index 981cc49..c38fb60 100644 --- a/README.rst +++ b/README.rst @@ -1,67 +1,65 @@ Synopsis -------- -**Polyloader** is a python module that enables the discovery and loading -of heterogenous source code packages. This discovery and loading is -critical to the functioning of other programming languages that use the -Python AST and Python interpreter, languages such as Hy, Doge, and -Mochi. - -Problem Statement ------------------ - -The Python module loader system is hard-coded to prevent the discovery -of heterogenous source code packages. From Python 2.6 through the -current (as of this writing) Python 3.5, the import mechanism allows for -the creation of file finders and importers that would transform Python's -import syntax into a *path,* assert whether or not that path could be -made to correspond to a *syntax object*, and then attempt to *load* that -syntax object as a Python module. Python *packages*, however, are -assumed to be uniformly made up of Python syntax objects, be they -**.py** source files, **.pyc/.pyo** bytecode, or **.so/.dll** files with -an exposed Python-to-C API. In Python 2 these suffixes are hard-coded -into the source in the **imp** builtin module; in Python 3 these -suffixes are constants defined in a private section of **importlib**; in -either case, they are unavailable for modification. This lack of access -to the extensions list prevents the *discovery* of heterogenous source +**Polyloader** is a python module that extends the Python `import` +statement to enable the discovery and loading of heterogenous source code packages. -The discovery mechanism is outlined in Python's **pkgutil** module; -features such as **pkgutil.iter_modules** do not work with heterogenous -source code, which in turn means that one cannot write, for one -important example, Django management commands in an alternative syntax. +Say What? +--------- -**polyloader** is a Python module that intercepts calls to the default -finder, loader, and package module iterator, and if the path resolves to -an alternative syntax, provide the appropriate finder, loader and -iterator. **polyloader** is different from traditional importlib shims -in that it directly affects the default source file loader, and thus -allows for the discovery and importation of suffixes not listed in -Python's defaults. +The ``import`` statement is how the Python interpreter finds a module +written in Python and loads it, turning it into variables, executable +functions, constructable classes, and other Python objects, and then +exposes those objects to the currently running program. -To use: -------- +The ``import`` statement has long been extensible so that things other +than Python code could be imported, but this feature has always had two +limitations: -Import polyloader in your python script's launcher or library, as well -as the syntax compiler(s) you plan to use. For example, if you have -`Mochi `__ and -`Hy `__ installed, and you wanted to -write a Django app, edit manage.py and add the following lines at the -top: +1. It's annoyingly hard to write an importer. (Believe me. Polyloader + *is* one!) +2. For filesystem-based modules (which is 99% of them) Python's importer + only understands one loader type per directory. It's not possible to + store code or data written in something other than Python in the same + directory with Python module code and load it via ``import``. -:: +The former requires a certain degree of abstraction and thought. For +the latter, most people ignore the problem and load module configuration +files written in JSON or YAML or whatever directly. This is fine, +except when you want to write in one of Python's extended languages like +Hy or Coconut in a framework like Django, Flask or Glitch. - from mochi.main import compile_file as mochi_compile - from hy.importer import ast_compile as hy_compile - from polyloader import polyimport - polyimport(mochi_compile, ['.mochi']) - polyimport(hy_compile, ['.hy'])} +**Polyloader** eliminates these limitations. -Now your views can be written in Hy and your models in Mochi, and -everything will just work. +What's the real problem? +------------------------ -Dependencies ------------- +The real problem is that Python's traditional extensions, ``.py``, +``.pyc/.pyo``, and ``.so/.dll`` files, are hard-coded in Python. In +Python 2, they're in the ``imp`` builtin; In Python 3, they're defined +in a private section of `importlib`. Either way, they're not accessible +for modification and extension. + +This problem is made harder by the ``pkglib`` module, which is part of +Python's standard library. This module uses ``inspect.getmoduleinfo``, +which again only recognizes the usual extensions. Which means you can't +list multilingual modules either; this hampers the development of Django +management commands in a syntax other than Python. + +What the solution? +------------------ + +At its heart, the Python import system runs two different internal +mechanisms to figure out what the *import string* (the dotted terms +after the word "import") "means." Each mechanism has one or more +*finders*, and the first finder to report "I have a *loader* that knows +what that import string means" wins. + +The very last finder is for the filesystem. The solution is to get in +front of that finder with one that can handle all the other syntax +loaders *and* knows how to fall back on the last one for those files the +last one handles. + +That's what ``polyloader`` does. -polymorph is self-contained. It has no dependencies other than Python -itself and your choice of language. diff --git a/docs/_sources/details.txt b/docs/_sources/details.txt new file mode 100644 index 0000000..fc64dd2 --- /dev/null +++ b/docs/_sources/details.txt @@ -0,0 +1,111 @@ +Details on Import and Polyloader +================================ + +Welcome to the Python Import ELI5! +********************************** + +What is `import`? +----------------- + +``import`` is a Python statement that finds a *module* that is accessible +by the currently running process, loads it, and makes its contents +available to the scope in which the statement was made. A statement +like + +``import django.db.models`` + +is looking for a module named ``django.db.models``. If the statement is +successful, ``django.db.models`` will be a variable name in the scope, +it will be a Python *object* (of ``class Module``, but that's not +important), and it will have functions, class constructors, variables +and constants. These Python objects will be accessible through the dot +operator. + +An alternative way of writing import statements is + +``from django.utils import encoding`` + +And then the variable will just be ``encoding``. The ``encoding`` +module has a function for handling unicode-to-web translation. +Accessing it through the dot operator, it looks like this: + +``ready_str = encoding.smart_str(unready_str)`` + +We call the parts of the import statement that describe the module the +*import string*. + +``sys.path_hooks``: How does Python know where to look? +------------------------------------------------------- + +That's what's funny. Python has two independent ways of making sense of +of the import string. The old system is based on the assumption that +everything is a filesystem, with folders and filenames. This is called +the ``sys.path_hooks`` system. + +In the old system, the parts of the import string would be split up, and +then a collection of directories would be scanned to see if the first +name in the import string could be matched with a subdirectory. If it +could, that directory would be scanned until the last name on the import +string. If that name was a *filename*, it would be loaded as a module. +If that name was a *directory* and that directory had a file named +``__init__.py``, then that file would be loaded as the module. + +The ``sys.path_hooks`` array has a list of different methods for trying to +scan a filesystem for the parts of the import string. A ``path_hook`` is +a function that takes a path to a directory; if it can handle the +contents of the directory, it returns a **Finder**, an object whose job +is to figure out how to load the module; if it can't, it returns an +ImportError exception. The object that loads the module is called, +naturally, a **Loader**. + +* To read more about **Finders**, see :ref:`eli5-finders` +* To read more about **Loaders**, see :ref:`eli5-loaders` +* To read more about **Path Hooks**, see :ref:`eli5-pathhooks` + +``sys.path``: What directories are searched? +-------------------------------------------- + +The list of directories is stored in an array, ``sys.path``. This path is +initialized by Python when it starts up, but programs can modify it at +run-time to point to extra directories if they want. + + +``sys.meta_path``: What is the new system? +------------------------------------------ + +The new system is called ``sys.meta_path``, and it's an array of +**Finders**, objects that have one method, ``find_module(fullname)``. +It's an anything-goes API that gives developers the freedom to import +modules from anywhere: databases, archives, remote web resources, even +code written on-the-fly internally. The new system can apply any +meaning at all to the import string. + +In Python, the import string is offered to each object in +``sys.meta_path`` before being offered to each ``sys.path_hook``. The +filesystem is typically the last finder tried. + +To read more about **Meta Paths**, see :ref:`eli5-metapaths` + +Is it different between Python 2 and Python 3? +--------------------------------------------- + +Python 3 moves almost everything about this process into python's +library, leaving only a bare minimum of functionality inside the Python +executable to load this library and run it. When the Python developers +did that, they added a lot of functionality to make it easier to write +new import modules. The old way still works, but there are now *Module +Specifications*, which are metadata about a module, and the old +``path_hooks`` system is now just a ``meta_path`` handler added to the +new system as the last resort. + +To read more about **Module Specifications**, see :ref:`eli5-specs` + +Does the old system still matter? +--------------------------------- + +Yes, for one reason: *iteration*. Iteration is the ability to take a +path where you believe Python modules can be found, and list through +them. This facility is useful for large frameworks where a user wants +to add new commands, or new objects, or new operations; Django uses this +facility a lot! The ``pkgutil`` library depends upon Finders being able +to iterate their contents. diff --git a/docs/_sources/eli5.txt b/docs/_sources/eli5.txt new file mode 100644 index 0000000..667777f --- /dev/null +++ b/docs/_sources/eli5.txt @@ -0,0 +1,37 @@ +ELI5: Explain Like I'm Five! +============================ + +.. _eli5-finders: + +Finders +======= + +TODO + +.. _eli5-loaders: + +Loaders +======= + +TODO + +.. _eli5-pathhooks: + +Path Hooks +========== + +TODO + +.. _eli5-metapaths: + +Meta Paths +========== + +TODO + +.. _eli5-specs: + +Module Specifications +===================== + +TODO diff --git a/docs/_sources/index.txt b/docs/_sources/index.txt index 71cd28f..fb05f4b 100644 --- a/docs/_sources/index.txt +++ b/docs/_sources/index.txt @@ -3,8 +3,8 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Welcome to py-polymorphic-loader's documentation! -====================================== +Welcome to polyloader +===================== Contents: @@ -14,6 +14,7 @@ Contents: readme installation usage + details contributing authors history diff --git a/docs/_sources/installation.txt b/docs/_sources/installation.txt index 5eca886..578e385 100644 --- a/docs/_sources/installation.txt +++ b/docs/_sources/installation.txt @@ -8,11 +8,11 @@ Installation Stable release -------------- -To install py-polymorphic-loader, run this command in your terminal: +To install polyloader, run this command in your terminal: .. code-block:: console - $ pip install py-polymorphic-loader + $ pip install polyloader If you don't have `pip`_ installed, this `Python installation guide`_ can guide you through the process. @@ -24,19 +24,19 @@ you through the process. From sources ------------ -The sources for py-polymorphic-loader can be downloaded from the `Github repo`_. +The sources for polyloader can be downloaded from the `Github repo`_. You can either clone the public repository: .. code-block:: console - $ git clone git://github.com/elfsternberg/py-polymorphic-loader + $ git clone git://github.com/elfsternberg/polyloader Or download the `tarball`_: .. code-block:: console - $ curl -OL https://github.com/elfsternberg/py-polymorphic-loader/tarball/master + $ curl -OL https://github.com/elfsternberg/polyloader/tarball/master Once you have a copy of the source, you can install it with: @@ -45,5 +45,5 @@ Once you have a copy of the source, you can install it with: $ python setup.py install -.. _Github repo: https://github.com/elfsternberg/py-polymorphic-loader -.. _tarball: https://github.com/elfsternberg/py-polymorphic-loader/tarball/master +.. _Github repo: https://github.com/elfsternberg/polyloader +.. _tarball: https://github.com/elfsternberg/polyloader/tarball/master diff --git a/docs/_sources/readme.txt b/docs/_sources/readme.txt index 72a3355..a6210d3 100644 --- a/docs/_sources/readme.txt +++ b/docs/_sources/readme.txt @@ -1 +1 @@ -.. include:: ../README.rst +.. include:: ../../README.rst diff --git a/docs/_sources/usage.txt b/docs/_sources/usage.txt index 587eb68..5f01efd 100644 --- a/docs/_sources/usage.txt +++ b/docs/_sources/usage.txt @@ -2,6 +2,21 @@ Usage ===== -To use py-polymorphic-loader in a project:: +Import polyloader in your python script's launcher or library, as well +as the syntax compiler(s) you plan to use. For example, if you have +`Mochi `__ and `Hy +`__ installed, and you wanted to +write a Django app, edit manage.py and add the following lines at the +top: + +:: + + from mochi.main import compile_file as mochi_compile + from hy.importer import ast_compile as hy_compile + import polyloader + polyloader.install(mochi_compile, ['.mochi']) + polyloader.install(hy_compile, ['.hy'])} + +Now your views can be written in Hy and your models in Mochi, and +everything will just work. - import py-polymorphic-loader diff --git a/docs/contributing.html b/docs/contributing.html index 7565fbc..758ed39 100644 --- a/docs/contributing.html +++ b/docs/contributing.html @@ -25,7 +25,7 @@ - +