elf
/
polyloader
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Polyloader for Python 2 (2.6 and 2.7) is GREEN. Woot! 

Unit tests included to ensure that Polyloader doesn't interfere with
standard Python import behavior, while still allowing for arbitrary
compilers to be associated with new file suffixes on the fly.

WOOT!

I wrote unit tests!

The syntax of the compilers has been change; I chose to trust Python's
source file loaders, so now we get a default stream in whatever format
the current version of Python is most comfortable with.

Made a note in my TODO about a Context Manager for making Polyloaders
"live" and then removing them automatically when a procedure goes out
of scope.

I have this weird sensation that this might actually work...
This commit is contained in:
Elf M. Sternberg 2016-06-30 22:11:10 -07:00
parent d6ea4d2dcf
commit 0e03ef38e0
18 changed files with 666 additions and 604 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -9,3 +9,4 @@ bower_components
build/*
.cache/*
.tox/*
dist/*

4
Makefile
View File

@ -75,11 +75,11 @@ servedocs: docs
release: clean
python setup.py sdist upload
python setup.py bdist_wheel upload
python setup.py bdist upload
dist: clean
python setup.py sdist
python setup.py bdist_wheel
python setup.py bdist
ls -l dist
install: clean

1
docs/index.rst
View File

@ -17,6 +17,7 @@ Contents:
contributing
authors
history
todo
Indices and tables
==================

51
docs/notes.html
View File

@ -1,51 +0,0 @@
<p>Python IMPORT</p>
<p>What is the <em>exact</em> syntax of the python <code>import</code> command?</p>
<p>Clarifying terminology</p>
<p>The language used for describing the import mechanism is confusing, often horribly so. Let's go with some clarification first.</p>
<p>When the 'import <fullname>' command is called, a procedure is triggered. That procedure then:</p>
<ul>
<li>attempts to <em>find</em> a corresponding python <em>module</em></li>
<li>attempts to <em>load</em> that corresponding module into <em>bytecode</em></li>
<li>Associates the bytecode with the name via sys.modules[fullname]</li>
<li>Exposes the bytecode to the calling scope.</li>
</ul>
<p>Only the first three matter for our purposes.</p>
<h2 id="finding">FINDING</h2>
<p><em>Finding</em> is the act of identifying the a resource that can be compiled into a meaningful Python module. This resource could a file on a filesystem, a cell in a database, a remote web object, a stream of bytes in an object store, some content object in a compressed archive, or anything that can meaningfully be said described as an array of bytes. It could even be dynamically generated in some way.</p>
<p><em>Finding</em> typically involves scanning a collection of <em>resources</em> against a collection of <em>finders</em>. <em>Finding</em> ends when &quot;<em>finder <strong>A</strong></em>, given <em>fullname <strong>B</strong></em>, reports that a corresponding module can be found in <em>resource <strong>C</strong></em>, and that the resource can be loaded with <em>loader <strong>D</strong></em>.&quot;</p>
<h3 id="metafinders">METAFINDERS</h3>
<p><em>Finders</em> come first, and <em>MetaFinders</em> come before all other kinds of finders.</p>
<p><em>Most finding is done in the context of <code>sys.path</code></em>; that is, Python's primary means of organizing Python modules is to have them somewhere on the local filesystem. Sometimes, however, you want to get in front of that scan. That's what you do with a MetaFinder: A MetaFinder may have its own take on what to do with <code>sys.path</code>; it may choose to ignore <code>sys.path</code> entirely and do something with the import <em>fullname</em> that has nothing to do with the local filesystem.</p>
<p>A Finder is any object with the following function: [Loader|None] find_module([self|cls], fullname:string, path:[string|None])</p>
<p>If find_module returns None if it cannot find a loader resource for fullname &amp; path.</p>
<p>MetaFinders are placed into the list <code>sys.meta_path</code> by whatever code needs a MetaFinder, and persist for the duration of the runtime provided they're not removed or replaced. Being a list, the search is ordered and first one wins. MetaFinders may be instantiated in any way the developer desires.</p>
<h3 id="path_hook">PATH_HOOK</h3>
<p><em>PathHooks</em> are how <code>sys.path</code> is scanned to determine the which Finder should be associated with a given directory path.</p>
<p>A <em>PathHook</em> is a function: [Finder|None] <anonymous function>(path:string)</p>
<p>A <em>PathHook</em> is a function that takes a given directory path and, if the PathHook can identify a corresponding Finder for the modules in that directory path, returns the Finder, otherwise it returns None.</p>
<p>If no <code>sys.meta_path</code> finder returns a loader, the full array of <code>sys.paths ⨯ sys.path_hooks</code> is compared until a path_hook says it can handle the path and the corresponding finder says it can handle the fullname. If no match happens, Python's default import behavior is triggered.</p>
<p>PathHooks are placed into the list <code>sys.path_hooks</code>; like <code>sys.meta_path</code>, the list is ordered and first one wins.</p>
<h3 id="loader">LOADER</h3>
<p><em>Loaders</em> are returned by <em>Finders</em>, and are constructed by Finders with whatever resources the developer specifies the Finder has and can provide.</p>
<p>a collection of <em>finders</em> the <em>fullname</em> (the dot-separated string passed to the <code>import</code> function).</p>
<p>to find a corresponding python module, which is then compiled into Python bytecode and incorporated into the python runtime, where it will be accessible to the importing function or modules</p>
<p>MetaFinder: A python object with a single method:</p>
<pre><code>(Loader|None) find_module(self, fullname:string, path:(string|None))</code></pre>
<p>Python 2.7</p>
<p>iter_modules (iter_importers) -&gt; calls iter_importer_modules for each importer in iter_importers</p>
<p>iter_importers (meta_path, get_importer) -&gt; returns every importer in sys.meta_path + map(get_importer, sys.path)</p>
<p>get_importer(path):</p>
<pre><code>returns a filtered list of sys.path_hooks for importers that can
handle this path; if there is no match, returns ImpImporter(),
which supplies a module iterator (ImpImporter.iter_modules) that
relies on getmodulename.
* A path_hook is a function of (path -&gt; Maybe importer)</code></pre>
<p>iter_modules(path, get_importer, prefix) -&gt; calls iter_importer_modules for each importer returned by path.map(get_importer)</p>
<p>iter_importer_modules(importer) -&gt; returns list of (filename, ispkg) for each module understood by the importer * The method called depends on the class of the importer * The default is a generic call for &quot;no specific importer&quot; * For FILES, iter_import_modules returns a list of files whose extensions match those in imp.get_suffixes(), which is hard- coded into the interpreter. * MEANING: Unless your importer can handle heterogenous module suffixes, SourceFiles.iter_importer_modules can only find homogeonous modules.</p>
<p>This relationship issue holds for Python 2.6 as well.</p>
<p>Python 3.3</p>
<pre><code>The same issue holds, although now most of the extensions have been
moved to importlib._bootstrap.</code></pre>
<p>It is the relationship between importlib.machinery.FileFinder and <em>iter</em>file_finder_modules</p>
<p>That's killing us.</p>

236
docs/notes.md
View File

@ -1,236 +0,0 @@
# Python IMPORT
What is the *exact* syntax of the python `import` command? What does it
mean when you write:
```
import alpha.beta.gamma
from alpha import beta
from alpha.beta import gamma
from .delta import epsilon
```
In each case, python is attempting to resolve the collection of dotted
names into a *module object*.
**module object**: A resource that is or can be compiled into a
meaningful Python *module*. This resource could a file on a filesystem,
a cell in a database, a remote web object, a stream of bytes in an
object store, some content object in a compressed archive, or anything
that can meaningfully be said described as an array of bytes. It could
even be dynamically generated!
**module**: The organizational unit of Python code. A namespace
containing Python objects, including classes, functions, submodules, and
immediately invoked code. Modules themselves may be collected into
*packages*.
**package**: A python module which contains submodules or even
subpackages. The most common packaging scheme is a directory folder; in
this case the folder is a module if it contains an `__init__.py` file,
and it is a *package* if it contains other modules. The name of the
package is the folder name; the name of a submodule would be
`foldername.submodule`. This is called *regular packaging*. An
alternative method is known as *namespace packaging*.
Python has a baroque but generally flexible mechanism for defining how
the dotted name is turned into a *module object*, which it calls *module
finding*, and for how that *module object* is turned into a *code
object* within the current Python session, called *module loading*.
Python also has a means for *listing* modules. *Listing* is usually
done on a list of paths, using an appropriate means for accessing the
contents at the end of a path.
The technical definition of a *package* is a module with a `__path__`, a
list of paths that contain submodules for the package. Subpackages get
their own` __path__`. A package can therefore accomodate `.` and `..`
prefixes in submodules, indicating relative paths to sibling modules. A
package can also and to its own `__path__` collection to enable access
to submodules elsewhere.
# Clarifying terminology
The language used for describing the import mechanism is confusing,
often horribly so. Let's go with some clarification first.
When the 'import <fullname>' command is called, a procedure is
triggered. That procedure then:
* attempts to *find* a corresponding python *module*
* attempts to *load* that corresponding module into *bytecode*
* Associates the bytecode with the name via sys.modules[fullname]
* Exposes the bytecode to the calling scope.
Only the first three matter for our purposes.
## FINDING
*Finding* is the act of identifying a resource that corresponds to the
import string and can be compiled into a meaningful Python module. The
import string is typically called the *fullname*.
*Finding* typically involves scanning a collection of *resources*
against a collection of *finders*. *Finding* ends when *finder `A`*,
given *fullname `B`*, reports that a corresponding module can be found
in *resource `C`*, and that the resource can be loaded with *loader
`D`*."
### METAFINDERS
*Finders* come first, and *MetaFinders* come before all other kinds of
finders.
_Most finding is done in the context of `sys.path`_; that is, Python's
primary means of organizing Python modules is to have them somewhere on
the local filesystem, which makes sense. Sometimes, however, you want
to get in front of that scan. That's what you do with a MetaFinder: A
MetaFinder may have its own take on what to do with `sys.path`; it may
choose to ignore `sys.path` entirely and do something with the import
*fullname* that has nothing to do with the local filesystem.
A Finder is any object with the following function:
[Loader|None] find_module([self|cls], fullname:string, path:[string|None])
If find_module returns None if it cannot find a loader resource for
fullname & path.
A MetaFinder is placed into the list `sys.meta_path` by whatever code
needs the MetaFinder, and it persists for the duration of the runtime,
unless it is later removed or replaced. Being a list, the search is
ordered; first match wins. MetaFinders may be instantiated in any way
the developer desires before being added into `sys.meta_path`.
### PATH_HOOK
*PathHooks* are how `sys.path` is scanned to determine the which Finder
should be associated with a given directory path.
A *PathHook* is a function:
[Finder|None] <anonymous function>(path:string)
A *PathHook* is a function that takes a given directory path and, if the
PathHook can identify a corresponding Finder for the modules in that
directory path, returns the Finder, otherwise it returns None.
If no `sys.meta_path` finder returns a loader, the full array of
`sys.paths ⨯ sys.path_hooks` is compared until a PathHook says it can
handle the path and the corresponding finder says it can handle the
fullname. If no match happens, Python's default import behavior is
triggered.
PathHooks are placed into the list `sys.path_hooks`; like
`sys.meta_path`, the list is ordered and first one wins.
### LOADER
*Loaders* are returned by *Finders*, and are constructed by Finders with
whatever resources the developer specifies the Finder has and can
provide. The Loader is responsible for pulling the content of the
*module object* into Python's memory and processing it into a *module*,
whether by calling Python's `eval()/compile()` functions on standard
Python code, or by some other means.
a collection of *finders* the *fullname* (the dot-separated string passed to the `import`
function).
to find a
corresponding python module, which is then compiled into Python bytecode
and incorporated into the python runtime, where it will be accessible to
the importing function or modules
MetaFinder: A python object with a single method:
(Loader|None) find_module(self, fullname:string, path:(string|None))
Python 2.7
iter_modules (iter_importers) ->
calls iter_importer_modules for each importer in iter_importers
iter_importers (meta_path, get_importer) ->
returns every importer in sys.meta_path + map(get_importer, sys.path)
get_importer(path):
returns a filtered list of sys.path_hooks for importers that can
handle this path; if there is no match, returns ImpImporter(),
which supplies a module iterator (ImpImporter.iter_modules) that
relies on getmodulename.
* A path_hook is a function of (path -> Maybe importer)
iter_modules(path, get_importer, prefix) ->
calls iter_importer_modules for each importer returned by path.map(get_importer)
iter_importer_modules(importer) ->
returns list of (filename, ispkg) for each module understood by the importer
* The method called depends on the class of the importer
* The default is a generic call for "no specific importer"
* For FILES, iter_import_modules returns a list of files whose
extensions match those in imp.get_suffixes(), which is hard-
coded into the interpreter.
* MEANING: Unless your importer can handle heterogenous module
suffixes, SourceFiles.iter_importer_modules can only find
homogeonous modules.
This relationship issue holds for Python 2.6 as well.
Python 3.3
The same issue holds, although now most of the extensions have been
moved to importlib._bootstrap.
It is the relationship between
importlib.machinery.FileFinder
and
_iter_file_finder_modules
That's killing us.
---
So the ONLY thing I have to do, according to Python, is assert that
there's a dir/__init__.suff and attempt to load it! If I do that, I can
make it work?
No: The search for __init__.suff is only the first
---
test_import: test_with_extension "py" and "my"
test_execute_bit_not_set (on Posix system, .pyc files got their
executable bit set if the .py file had it set; it looks as if Python
just copied the permissions, if it had permission to do so. We should
follow the example of 2.7 & 3.4, and NOT set +x if we can help it).
test_rewrite_pyc_with_read_only_source (on Posix systems, if the .py
file had read-only set, the .pyc file would too, making updates
problematic).
test_import_name_binding
test_bug7732 (attempt to import a '.my' file that's not a file)
These are more Hy-related:
test_module_with_large_stack (see python example)
test_failing_import_sticks
test_failing_reload

152
docs/notes.txt
View File

@ -1,152 +0,0 @@
Clarifying terminology
The language used for describing the import mechanism is confusing,
often horribly so. Let's go with some clarification first.
When the 'import <fullname>' command is called, a procedure is
triggered. That procedure then:
* attempts to *find* a corresponding python *module*
* attempts to *load* that corresponding module into *bytecode*
* Associates the bytecode with the name via sys.modules[fullname]
* Exposes the bytecode to the calling scope.
Only the first three matter for our purposes.
## FINDING
*Finding* is the act of identifying the a resource that can be compiled
into a meaningful Python module. This resource could a file on a
filesystem, a cell in a database, a remote web object, a stream of bytes
in an object store, some content object in a compressed archive, or
anything that can meaningfully be said described as an array of bytes.
It could even be dynamically generated in some way.
*Finding* typically involves scanning a collection of *resources*
against a collection of *finders*. *Finding* ends when "*finder __A__*,
given *fullname __B__*, reports that a corresponding module can be found
in *resource __C__*, and that the resource can be loaded with *loader
__D__*."
### METAFINDERS
*Finders* come first, and *MetaFinders* come before all other kinds of
finders.
_Most finding is done in the context of `sys.path`_; that is, Python's
primary means of organizing Python modules is to have them somewhere on
the local filesystem. Sometimes, however, you want to get in front of
that scan. That's what you do with a MetaFinder: A MetaFinder may have
its own take on what to do with `sys.path`; it may choose to ignore
`sys.path` entirely and do something with the import *fullname* that has
nothing to do with the local filesystem.
A Finder is any object with the following function:
[Loader|None] find_module([self|cls], fullname:string, path:[string|None])
If find_module returns None if it cannot find a loader resource for
fullname & path.
MetaFinders are placed into the list `sys.meta_path` by whatever code
needs a MetaFinder, and persist for the duration of the runtime provided
they're not removed or replaced. Being a list, the search is ordered
and first one wins. MetaFinders may be instantiated in any way the
developer desires.
### PATH_HOOK
*PathHooks* are how `sys.path` is scanned to determine the
which Finder should be associated with a given directory path.
A *PathHook* is a function:
[Finder|None] <anonymous function>(path:string)
A *PathHook* is a function that takes a given directory path and, if the
PathHook can identify a corresponding Finder for the modules in that
directory path, returns the Finder, otherwise it returns None.
If no `sys.meta_path` finder returns a loader, the full array of
`sys.paths ⨯ sys.path_hooks` is compared until a path_hook says it can
handle the path and the corresponding finder says it can handle the
fullname. If no match happens, Python's default import behavior is
triggered.
PathHooks are placed into the list `sys.path_hooks`; like
`sys.meta_path`, the list is ordered and first one wins.
### LOADER
*Loaders* are returned by *Finders*, and are constructed by Finders with
whatever resources the developer specifies the Finder has and can
provide.
a collection of *finders* the *fullname* (the dot-separated string passed to the `import`
function).
to find a
corresponding python module, which is then compiled into Python bytecode
and incorporated into the python runtime, where it will be accessible to
the importing function or modules
MetaFinder: A python object with a single method:
(Loader|None) find_module(self, fullname:string, path:(string|None))
Python 2.7
iter_modules (iter_importers) ->
calls iter_importer_modules for each importer in iter_importers
iter_importers (meta_path, get_importer) ->
returns every importer in sys.meta_path + map(get_importer, sys.path)
get_importer(path):
returns a filtered list of sys.path_hooks for importers that can
handle this path; if there is no match, returns ImpImporter(),
which supplies a module iterator (ImpImporter.iter_modules) that
relies on getmodulename.
* A path_hook is a function of (path -> Maybe importer)
iter_modules(path, get_importer, prefix) ->
calls iter_importer_modules for each importer returned by path.map(get_importer)
iter_importer_modules(importer) ->
returns list of (filename, ispkg) for each module understood by the importer
* The method called depends on the class of the importer
* The default is a generic call for "no specific importer"
* For FILES, iter_import_modules returns a list of files whose
extensions match those in imp.get_suffixes(), which is hard-
coded into the interpreter.
* MEANING: Unless your importer can handle heterogenous module
suffixes, SourceFiles.iter_importer_modules can only find
homogeonous modules.
This relationship issue holds for Python 2.6 as well.
Python 3.3
The same issue holds, although now most of the extensions have been
moved to importlib._bootstrap.
It is the relationship between
importlib.machinery.FileFinder
and
_iter_file_finder_modules
That's killing us.

10
docs/todo.rst Normal file
View File

@ -0,0 +1,10 @@
Currently assumes that Polyloaders last for the lifetime of the Python
instance. There is no standardized mechanism for removing a
compiler/suffix set from a running instance.
1. Create a standardized mechanism for removing a compiler/suffix from
the running instance.
2. Create a 'with' context manager that creates a scope in which a
Polyloader compiler/suffix pair is active, and automatically removes
that pair upon leaving the scope.

6
polyloader/__init__.py
View File

@ -6,9 +6,9 @@ __email__ = 'elf.sternberg@gmail.com'
__version__ = '0.1.0'
if sys.version_info[0:2] >= (2, 6):
from ._python2 import install
from ._python2 import install, reset
if sys.version_info[0] >= 3:
from ._python3 import install
from ._python3 import install, reset
__all__ = ['install']
__all__ = ['install', 'reset']

70
polyloader/_python2.py
View File

@ -1,9 +1,9 @@
import io
import os
import os.path
import stat
import sys
import imp
import types
import pkgutil
@ -25,11 +25,10 @@ class PolyLoader():
if overlap:
raise RuntimeError("Override of native Python extensions is not permitted.")
overlap = suffixes.intersection(
set([suffix for (compiler, suffix) in cls._loader_handlers]))
set([suffix for (_, suffix) in cls._loader_handlers]))
if overlap:
raise RuntimeWarning(
"Insertion of %s overrides already installed compiler." %
', '.join(list(overlap)))
# Fail silently
return
cls._loader_handlers += [(compiler, suf) for suf in suffixes]
def load_module(self, fullname):
@ -42,26 +41,29 @@ class PolyLoader():
matches = [(compiler, suffix) for (compiler, suffix) in self._loader_handlers
if self.path.endswith(suffix)]
if matches.length == 0:
if len(matches) == 0:
raise ImportError("%s is not a recognized module?" % fullname)
if matches.length > 1:
if len(matches) > 1:
raise ImportError("Multiple possible resolutions for %s: %s" % (
fullname, ', '.join([suffix for (compiler, suffix) in matches])))
compiler = matches[0]
compiler = matches[0][0]
with io.FileIO(self.path, 'r') as file:
source_text = file.read()
module = compiler(source_text, fullname, self.path)
code = compiler(source_text, self.path, fullname)
module = types.ModuleType(fullname)
module.__file__ = self.path
module.__name__ = self.fullname
module.__name__ = fullname
module.__package__ = '.'.join(fullname.split('.')[:-1])
if self.is_package:
module.__path__ = [os.path.dirname(self.path)]
module.__path__ = [os.path.dirname(module.__file__)]
module.__package__ = fullname
exec(code, module.__dict__)
sys.modules[fullname] = module
return module
@ -76,29 +78,29 @@ class PolyLoader():
class PolyFinder(object):
def __init__(self, path=None):
self.path = path or '.'
def _pl_find_on_path(self, fullname, path=None):
splitname = fullname.split(".")
if self.path is None and splitname[-1] != fullname:
return None
dirpath = "/".join(splitname)
path = [os.path.realpath(self.path)]
def _pl_find_on_path(self, fullname, path=None):
subname = fullname.split(".")[-1]
if self.path is None and subname != fullname:
return None
path = os.path.realpath(self.path)
fls = [("%s/__init__.%s", True), ("%s.%s", False)]
for (fp, ispkg) in fls:
for (compiler, suffix) in PolyLoader._loader_handlers:
composed_path = fp % ("%s/%s" % (path, dirpath), suffix)
composed_path = fp % ("%s/%s" % (path, subname), suffix)
if os.path.isdir(composed_path):
raise IOError("Invalid: Directory name ends in recognized suffix")
if os.path.isfile(composed_path):
return PolyLoader(fullname, composed_path, ispkg)
# Fall back onto Python's own methods.
try:
file, filename, etc = imp.find_module(fullname, path)
except ImportError:
file, filename, etc = imp.find_module(subname, [path])
except ImportError as e:
return None
return pkgutil.ImpLoader(fullname, file, filename, etc)
def find_module(self, fullname, path=None):
return self._pl_find_on_path(fullname)
@ -106,12 +108,13 @@ class PolyFinder(object):
def getmodulename(path):
filename = os.path.basename(path)
suffixes = ([(-len(suf[0]), suf[0]) for suf in imp.get_suffixes()] +
[(-len(suf[1]), suf[1]) for suf in PolyLoader.loader_handlers])
[(-len(suf[1]), suf[1]) for suf in PolyLoader._loader_handlers])
suffixes.sort()
for neglen, suffix in suffixes:
if filename[neglen:] == suffix:
return (filename[:neglen], suffix)
return filename[:neglen]
return None
def iter_modules(self, prefix=''):
if self.path is None or not os.path.isdir(self.path):
return
@ -126,7 +129,7 @@ class PolyFinder(object):
filenames.sort()
for fn in filenames:
modname = self.getmodulename(fn)
if modname=='__init__' or modname in yielded:
if modname == '__init__' or modname in yielded:
continue
path = os.path.join(self.path, fn)
@ -141,7 +144,7 @@ class PolyFinder(object):
dircontents = []
for fn in dircontents:
subname = self.getmodulename(fn)
if subname=='__init__':
if subname == '__init__':
ispkg = True
break
else:
@ -152,16 +155,19 @@ class PolyFinder(object):
yield prefix + modname, ispkg
def _polyloader_pathhook(path):
if not os.path.isdir(path):
raise ImportError('Only directories are supported', path = path)
raise ImportError('Only directories are supported', path=path)
return PolyFinder(path)
def install(compiler, suffixes):
if not PolyLoader._installed:
sys.path_hooks.append(_polyloader_pathhook)
PolyLoader._installed = True
PolyLoader._install(compiler, suffixes)
def reset():
PolyLoader._loader_handlers = []
PolyLoader._installed = False

125
polyloader/_python3.py
View File

@ -1,29 +1,3 @@
# polyloader.py
#
# Copyright (c) 2016 Elf M. Sternberg <elf.sternberg@gmail.com>
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.
#
""" Utilities for initializing extended path-hooks into the Python runtime """
__all__ = [] # Exports nothing; this module is called for its side-effects
import os
import sys
@ -33,11 +7,6 @@ from pkgutil import iter_importer_modules
import sys
__author__ = 'Elf M. Sternberg'
__version__ = '2016.05.29'
__contact__ = 'elf.sternberg@gmail.com'
if sys.version_info[0:2] in [(3,3), (3,4)]:
from importlib._bootstrap import _get_supported_file_loaders
sourcefile_recognizer = 'importlib.SourceFileLoader'
@ -54,7 +23,7 @@ def _call_with_frames_removed(f, *args, **kwds):
return f(*args, **kwds)
class ExtendedSourceFileLoader(machinery.SourceFileLoader):
class PolySourceFileLoader(machinery.SourceFileLoader):
"""Override the get_code method. Falls back on the SourceFileLoader
if it's a Python file, which will generate pyc files as needed,
or works its way into the Extended version. This method does
@ -91,8 +60,14 @@ class ExtendedSourceFileLoader(machinery.SourceFileLoader):
# Provide a working namespace for our new FileFinder.
class ExtendedFileFinder(machinery.FileFinder):
class PolySourceFileFinder(machinery.FileFinder):
# Taken from inspect.py and modified to support alternate suffixes.
@staticmethod
def getmodulename(path):
@ -148,28 +123,27 @@ class ExtendedFileFinder(machinery.FileFinder):
yield prefix + modname, ispkg
pass
# Monkeypatch both path_hooks and iter_importer_modules to make our
# modules recognizable to the module iterator functions. This is
# probably horribly fragile, but there doesn't seem to be a more
# robust way of doing it at the moment, and these names are stable
# from python 2.7 up.
def install(compiler, suffixes):
""" Install a compiler and suffix(es) into Python's sys.path_hooks, so
that modules ending with thoses suffixes will be parsed into
python executable modules automatically.
"""Install a specialized version of FileFinder that will search
through alternative extensions first for syntax files and, upon
encountering one, will return a specialized version of
SourceFileLoader for that syntax. By replacing this into
path_hook this makes both import and iter_modules work as
expected.
"""
if sys.version[0] >= 3:
filefinder = [(f, i) for i, f in enumerate(sys.path_hooks)
if repr(f).find('path_hook_for_FileFinder') != -1]
if not filefinder:
return
filefinder, fpos = filefinder[0]
ExtendedSourceFileLoader._source_handlers = (ExtendedSourceFileLoader._source_handlers +
[(compiler, tuple(suffixes))])
filefinder = [(f, i) for i, f in enumerate(sys.path_hooks)
if repr(f).find('.path_hook_for_FileFinder') != -1]
if not filefinder:
return
filefinder, fpos = filefinder[0]
supported_loaders = _get_supported_file_loaders()
print([repr(i) for i in supported_loaders])
@ -186,52 +160,9 @@ def install(compiler, suffixes):
if sys.path[0] != "":
sys.path.insert(0, "")
if sys.version[0:2] == (2, 7):
def loader(path):
class Loader(object):
def __init__(self, path):
self.path = path
def is_package(self, fullname):
dirpath = "/".join(fullname.split("."))
for pth in sys.path:
pth = os.path.abspath(pth)
composed_path = "%s/%s/__init__.%s" % (pth, dirpath, suffix)
if os.path.exists(composed_path):
return True
return False
def load_module(self, name):
if name in sys.modules:
return sys.modules[name]
module = compiler(fullname, path)
module.__file__ = path
sys.modules[name] = module
if '.' in name:
parent_name, child_name = name.rsplit('.', 1)
setattr(sys.modules[parent_name], child_name, module)
sys.modules[name] = module
return module
return Loader()
class MetaImporter(object):
def find_module(self, fullname, path=None):
if fullname == 'numpy' or fullname.startswith('numpy.'):
_mapper.PerpetrateNumpyFixes()
if fullname in ('_hashlib', 'ctypes'):
raise ImportError('%s is not available in ironclad yet' % fullname)
lastname = fullname.rsplit('.', 1)[-1]
for d in (path or sys.path):
pyd = os.path.join(d, lastname + '.pyd')
if os.path.exists(pyd):
return loader(pyd)
return None
sys.meta_path.insert(0, MetaImporter())
iter_importer_modules.register(MetaImporter, meta_iterate_modules)
def reset():
pass

1
tests/polytestmix/test1.py
View File

@ -0,0 +1 @@
result = "Success for 1: Test One"

48
tests/ptutils.py Normal file
View File

@ -0,0 +1,48 @@
import UserDict
import os
class EnvironmentVarGuard(UserDict.DictMixin):
"""Class to help protect the environment variable properly. Can be used as
a context manager."""
def __init__(self):
self._environ = os.environ
self._changed = {}
def __getitem__(self, envvar):
return self._environ[envvar]
def __setitem__(self, envvar, value):
# Remember the initial value on the first access
if envvar not in self._changed:
self._changed[envvar] = self._environ.get(envvar)
self._environ[envvar] = value
def __delitem__(self, envvar):
# Remember the initial value on the first access
if envvar not in self._changed:
self._changed[envvar] = self._environ.get(envvar)
if envvar in self._environ:
del self._environ[envvar]
def keys(self):
return self._environ.keys()
def set(self, envvar, value):
self[envvar] = value
def unset(self, envvar):
del self[envvar]
def __enter__(self):
return self
def __exit__(self, *ignore_exc):
for (k, v) in self._changed.items():
if v is None:
if k in self._environ:
del self._environ[k]
else:
self._environ[k] = v
os.environ = self._environ

1
tests/relimport.py Normal file
View File

@ -0,0 +1 @@
from .test_import import *

273
tests/test_import.py Normal file
View File

@ -0,0 +1,273 @@
import polyloader
import pytest
import py_compile
import ptutils
import stat
import sys
import os
import imp
import random
import struct
class Compiler:
def __init__(self, pt):
self.pt = pt
def __call__(self, source_text, filename, *extra):
return compile("result='Success for %s: %s'" %
(self.pt, source_text.rstrip()), filename, "exec")
def __repr__(self):
return "Compiler %s" % (self.pt)
polyloader.install(Compiler("2"), ['2'])
TESTFN = '@test'
def clean_tmpfiles(path):
if os.path.exists(path):
os.remove(path)
if os.path.exists(path + 'c'):
os.remove(path + 'c')
if os.path.exists(path + 'o'):
os.remove(path + 'o')
def unload(name):
try:
del sys.modules[name]
except KeyError:
pass
# This exists mostly to test that the inclusion of PolyLoader doesn't
# break anything else in Python.
# Tests commented out: 2
class Test_Imports(object):
def test_case_sensitivity(self):
# Brief digression to test that import is case-sensitive: if we got
# this far, we know for sure that "random" exists.
try:
import RAnDoM
except ImportError:
pass
else:
assert(False)
def test_import(self):
sys.path.insert(0, os.curdir)
ext = 'py'
try:
# The extension is normally ".py", perhaps ".pyw".
source = TESTFN + os.extsep + ext
pyc = TESTFN + os.extsep + "pyc"
with open(source, "w") as f:
print >> f, ("# This tests Python's ability to import a", ext,
"file.")
a = random.randrange(1000)
b = random.randrange(1000)
print >> f, "a =", a
print >> f, "b =", b
if os.path.exists(source):
print "%s EXISTS..." % source
try:
print "IMPORT..."
mod = __import__(TESTFN)
print "END IMPORT..."
except ImportError, err:
print("import from %s (%s) failed: %s" % (ext, os.curdir, err))
assert(False)
else:
assert(mod.a == a)
assert(mod.b == b)
finally:
os.remove(source)
try:
if not sys.dont_write_bytecode:
imp.reload(mod)
except ImportError, err:
self.fail("import from .pyc/.pyo failed: %s" % err)
finally:
clean_tmpfiles(source)
unload(TESTFN)
finally:
del sys.path[0]
def test_execute_bit_not_copied(self):
# Issue 6070: under posix .pyc files got their execute bit set if
# the .py file had the execute bit set, but they aren't executable.
oldmask = os.umask(022)
sys.path.insert(0, os.curdir)
try:
fname = TESTFN + os.extsep + "py"
f = open(fname, 'w').close()
os.chmod(fname, (stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH |
stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH))
__import__(TESTFN)
fn = fname + 'c'
if not os.path.exists(fn):
fn = fname + 'o'
if not os.path.exists(fn):
self.fail("__import__ did not result in creation of "
"either a .pyc or .pyo file")
s = os.stat(fn)
assert(s.st_mode & (stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH))
assert(not (s.st_mode & (stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)))
finally:
os.umask(oldmask)
clean_tmpfiles(fname)
unload(TESTFN)
del sys.path[0]
def test_imp_module(self):
# Verify that the imp module can correctly load and find .py files
# XXX (ncoghlan): It would be nice to use test_support.CleanImport
# here, but that breaks because the os module registers some
# handlers in copy_reg on import. Since CleanImport doesn't
# revert that registration, the module is left in a broken
# state after reversion. Reinitialising the module contents
# and just reverting os.environ to its previous state is an OK
# workaround
orig_path = os.path
orig_getenv = os.getenv
with ptutils.EnvironmentVarGuard():
x = imp.find_module("os")
new_os = imp.load_module("os", *x)
assert(os == new_os)
assert(orig_path == new_os.path)
assert(orig_getenv != new_os.getenv)
def test_module_with_large_stack(self, module='longlist'):
# Regression test for http://bugs.python.org/issue561858.
filename = module + os.extsep + 'py'
# Create a file with a list of 65000 elements.
with open(filename, 'w+') as f:
f.write('d = [\n')
for i in range(65000):
f.write('"",\n')
f.write(']')
# Compile & remove .py file, we only need .pyc (or .pyo).
with open(filename, 'r') as f:
py_compile.compile(filename)
os.remove(filename)
# Need to be able to load from current dir.
sys.path.append('')
# This used to crash.
exec 'import ' + module
# Cleanup.
del sys.path[-1]
clean_tmpfiles(filename)
def test_failing_import_sticks(self):
source = TESTFN + os.extsep + "py"
with open(source, "w") as f:
print >> f, "a = 1 // 0"
# New in 2.4, we shouldn't be able to import that no matter how often
# we try.
sys.path.insert(0, os.curdir)
try:
for i in [1, 2, 3]:
with pytest.raises(ZeroDivisionError) as zde:
__import__(TESTFN)
assert(sys.modules.get(TESTFN) == None)
finally:
del sys.path[0]
clean_tmpfiles(source)
# def test_failing_reload(self):
# # A failing reload should leave the module object in sys.modules.
# source = TESTFN + os.extsep + "py"
# with open(source, "w") as f:
# print >> f, "a = 1"
# print >> f, "b = 2"
#
# sys.path.insert(0, os.curdir)
# try:
# mod = __import__(TESTFN)
# assert(sys.modules.get(TESTFN) != None)
# assert(mod.a == 1)
# assert(mod.b == 2)
#
# # On WinXP, just replacing the .py file wasn't enough to
# # convince reload() to reparse it. Maybe the timestamp didn't
# # move enough. We force it to get reparsed by removing the
# # compiled file too.
# clean_tmpfiles(TESTFN)
#
# # Now damage the module.
# with open(source, "w") as f:
# print >> f, "a = 10"
# print >> f, "b = 20 // 0"
#
# with pytest.raises(ZeroDivisionError) as zde:
# imp.reload(mod)
#
# # But we still expect the module to be in sys.modules.
# mod = sys.modules.get(TESTFN)
# assert(mod != None)
#
# # We should have replaced a w/ 10, but the old b value should
# # stick.
# assert(mod.a == 10)
# assert(mod.b == 2)
#
# finally:
# del sys.path[0]
# clean_tmpfiles(source)
# unload(TESTFN)
def test_infinite_reload(self):
# http://bugs.python.org/issue742342 reports that Python segfaults
# (infinite recursion in C) when faced with self-recursive reload()ing.
sys.path.insert(0, os.path.dirname(__file__))
try:
with pytest.raises(ImportError):
import infinite_reload
finally:
del sys.path[0]
def test_import_name_binding(self):
# import x.y.z binds x in the current namespace.
import test as x
import test.test_support
assert(x == test)
assert(hasattr(test.test_support, "__file__"))
# import x.y.z as w binds z as w.
import test.test_support as y
assert(y == test.test_support)
def test_import_initless_directory_warning(self):
with pytest.raises(ImportError) as ie:
__import__('site-packages')
def test_import_by_filename(self):
path = os.path.abspath(TESTFN)
with pytest.raises(ImportError) as c:
__import__(path)
assert("Import by filename is not supported." == c.value[0])
# def test_bug7732(self):
# source = TESTFN + '.2'
# os.mkdir(source)
# try:
# with pytest.raises(IOError) as iie:
# mod = imp.find_module(TESTFN, ["."])
# print("Found mod %s" % mod)
# finally:
# os.rmdir(source)

61
tests/test_paths.py Normal file
View File

@ -0,0 +1,61 @@
import polyloader
import pytest
import py_compile
import ptutils
import stat
import sys
import os
import imp
import random
import struct
class Compiler:
def __init__(self, pt):
self.pt = pt
def __call__(self, source_text, filename, *extra):
return compile("result='Success for %s: %s'" %
(self.pt, source_text.rstrip()), filename, "exec")
def __repr__(self):
return "Compiler %s" % (self.pt)
polyloader.install(Compiler("2"), ['2'])
TESTFN = '@test'
def clean_tmpfiles(path):
if os.path.exists(path):
os.remove(path)
if os.path.exists(path + 'c'):
os.remove(path + 'c')
if os.path.exists(path + 'o'):
os.remove(path + 'o')
def unload(name):
try:
del sys.modules[name]
except KeyError:
pass
class Test_Paths:
path = TESTFN
def setup_method(self, method):
os.mkdir(self.path)
self.syspath = sys.path[:]
def teardown_method(self, method):
rmtree(self.path)
sys.path[:] = self.syspath
# Regression test for http://bugs.python.org/issue1293.
def test_trailing_slash(self):
with open(os.path.join(self.path, 'test_trailing_slash.2'), 'w') as f:
f.write("testdata = 'test_trailing_slash'")
sys.path.append(self.path+'/')
mod = __import__("test_trailing_slash")
self.assertEqual(mod.testdata, 'test_trailing_slash')
unload("test_trailing_slash")

124
tests/test_polyloader.py
View File

@ -8,44 +8,110 @@ test_polyloader
Tests for `polyloader` module.
"""
import pytest
from polyloader import polyloader
import polyloader
import copy
import sys
# Note that these compilers don't actually load much out of the
# source files. That's not the point. The point is to show that the
# correct compiler has been found for a given extension.
def compiler(pt):
def _compiler(source_text, modulename):
return compile("result='Success for %s: %s'" % (pt, source_text.rstrip()), modulename, "exec")
return _compiler
class Test_Polymorph_1(object):
class ImportEnvironment(object):
def __init__(self):
pass
def __enter__(self):
polyloader.reset()
self.path = copy.copy(sys.path)
self.path_hooks = copy.copy(sys.path_hooks)
self.meta_path = copy.copy(sys.meta_path)
self.modules = copy.copy(sys.modules)
self.path_importer_cache = copy.copy(sys.path_importer_cache)
return sys
def __exit__(self, type, value, traceback):
sys.path = self.path
sys.path_hooks = self.path_hooks
sys.meta_path = self.meta_path
sys.modules = self.modules
sys.path_importer_cache = self.path_importer_cache
class Compiler:
def __init__(self, pt):
self.pt = pt
def __call__(self, source_text, filename, *extra):
return compile("result='Success for %s: %s'" %
(self.pt, source_text.rstrip()), filename, "exec")
def __repr__(self):
return "Compiler %s" % (self.pt)
def compiler(pt):
return Compiler(pt)
# Functionally, the "From" test and the "Direct" test should be
# indistinguishable. What's interesting about them, though, is that
# in the context of the caller, the "Direct" test now has test and
# test.polytestmix as objects in the calling context. They're fairly
# lightweight, but they do exist, and they do honor the __all__ and
# __path__ cases.
#
# Also:
# See http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html
# for some 'gotchas' to look forward to. Whee.
class Test_Polymorph_From(object):
def test_import1(self):
polyloader.install(compiler("2"), ['.2'])
polyloader.install(compiler("3"), ['.3'])
from .polytestmix import test2
from .polytestmix import test3
assert(test2.result == "Success for 2: Test Two")
assert(test3.result == "Success for 3: Test Three")
with ImportEnvironment() as sys:
polyloader.install(compiler("2"), ['2'])
polyloader.install(compiler("3"), ['3'])
from .polytestmix import test2
from .polytestmix import test3
from .polytestmix import test1
assert(test1.result == "Success for 1: Test One")
assert(test2.result == "Success for 2: Test Two")
assert(test3.result == "Success for 3: Test Three")
class Test_Polymorph_Direct(object):
def test_import2(self):
with ImportEnvironment() as sys:
polyloader.install(compiler("2"), ['2'])
polyloader.install(compiler("3"), ['3'])
import tests.polytestmix.test2
import tests.polytestmix.test3
import tests.polytestmix.test1
assert(tests.polytestmix.test1.result == "Success for 1: Test One")
assert(tests.polytestmix.test2.result == "Success for 2: Test Two")
assert(tests.polytestmix.test3.result == "Success for 3: Test Three")
class Test_Polymorph_Module(object):
def test_import3(self):
with ImportEnvironment() as sys:
polyloader.install(compiler("3"), ['3'])
polyloader.install(compiler("2"), ['2'])
from tests.polytestmix.test3 import result as result3
from tests.polytestmix.test2 import result as result2
from tests.polytestmix.test1 import result as result1
assert(result1 == "Success for 1: Test One")
assert(result2 == "Success for 2: Test Two")
assert(result3 == "Success for 3: Test Three")
class Test_Polymorph_Iterator(object):
''' The Django Compatibility test: Can we load arbitrary modules from a package? '''
def test_iterator(self):
import os
import pkgutil
import inspect
polyloader.install(compiler("2"), ['.2'])
polyloader.install(compiler("3"), ['.3'])
from .polytestmix import test2
from .polytestmix import test3
target_dir = os.path.join(
os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))),
'polytestmix')
modules = set([name for _, name, is_pkg in pkgutil.iter_modules([target_dir])
if not is_pkg and not name.startswith('_')])
assert(modules == set(['test1', 'test2', 'test3']))
with ImportEnvironment() as sys:
import os
import inspect
polyloader.install(compiler("2"), ['.2'])
polyloader.install(compiler("3"), ['.3'])
import pkgutil
target_dir = os.path.join(
os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))),
'polytestmix')
modules = set([name for (_, name, is_pkg) in pkgutil.iter_modules([target_dir])
if not is_pkg and not name.startswith('_')])
assert(modules == set(['test1', 'test2', 'test3']))

99
tests/test_relativeimport.py Normal file
View File

@ -0,0 +1,99 @@
import polyloader
import pytest
import py_compile
import ptutils
import stat
import sys
import os
import imp
import random
import struct
class Compiler:
def __init__(self, pt):
self.pt = pt
def __call__(self, source_text, filename, *extra):
return compile("result='Success for %s: %s'" %
(self.pt, source_text.rstrip()), filename, "exec")
def __repr__(self):
return "Compiler %s" % (self.pt)
polyloader.install(Compiler("2"), ['2'])
TESTFN = '@test'
def clean_tmpfiles(path):
if os.path.exists(path):
os.remove(path)
if os.path.exists(path + 'c'):
os.remove(path + 'c')
if os.path.exists(path + 'o'):
os.remove(path + 'o')
def unload(name):
try:
del sys.modules[name]
except KeyError:
pass
class Test_RelativeImports:
def teardown_class(cls):
unload("tests.relimport")
def setup_class(cls):
unload("tests.relimport")
def test_relimport_star(self):
# This will import * from .test_import.
from . import relimport
assert(hasattr(relimport, "Test_Imports"))
def test_issue3221(self):
# Regression test for http://bugs.python.org/issue3221.
def check_absolute():
exec "from os import path" in ns
def check_relative():
exec "from . import relimport" in ns
# Check both OK with __package__ and __name__ correct
ns = dict(__package__='tests', __name__='test.notarealmodule')
check_absolute()
check_relative()
# Check both OK with only __name__ wrong
ns = dict(__package__='tests', __name__='notarealpkg.notarealmodule')
check_absolute()
check_relative()
# Check relative fails with only __package__ wrong
ns = dict(__package__='foo', __name__='test.notarealmodule')
with pytest.warns(RuntimeWarning) as rw:
check_absolute()
with pytest.raises(SystemError) as se:
check_relative()
# Check relative fails with __package__ and __name__ wrong
ns = dict(__package__='foo', __name__='notarealpkg.notarealmodule')
with pytest.warns(RuntimeWarning) as se:
check_absolute()
with pytest.raises(SystemError) as se:
check_relative()
# Check both fail with package set to a non-string
ns = dict(__package__=object())
with pytest.raises(ValueError) as ve:
check_absolute()
with pytest.raises(ValueError) as ve:
check_relative()
def test_absolute_import_without_future(self):
# If explicit relative import syntax is used, then do not try
# to perform an absolute import in the face of failure.
# Issue #7902.
with pytest.raises(ImportError) as ie:
from .os import sep
assert(False)

7
tox.ini
View File

@ -1,5 +1,6 @@
[tox]
envlist = py26, py27, py33, py34, py35
envlist = py26, py27
; , py33, py34, py35
[testenv]
setenv =
@ -7,8 +8,10 @@ setenv =
deps =
-r{toxinidir}/requirements_dev.txt
commands =
py.test --basetemp={envtmpdir}
py.test --basetemp={envtmpdir} []
[flake8]
max-line-length = 99
; If you want to make tox run the tests with the same versions, create a
; requirements.txt with the pinned versions and uncomment the following lines: