diff --git a/.gitignore b/.gitignore index 0ecf7ee..2048d89 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,4 @@ bower_components build/* .cache/* .tox/* +dist/* diff --git a/Makefile b/Makefile index c69fbfc..64e22a0 100644 --- a/Makefile +++ b/Makefile @@ -75,11 +75,11 @@ servedocs: docs release: clean python setup.py sdist upload - python setup.py bdist_wheel upload + python setup.py bdist upload dist: clean python setup.py sdist - python setup.py bdist_wheel + python setup.py bdist ls -l dist install: clean diff --git a/docs/index.rst b/docs/index.rst index ea96316..71cd28f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -17,6 +17,7 @@ Contents: contributing authors history + todo Indices and tables ================== diff --git a/docs/notes.html b/docs/notes.html deleted file mode 100644 index eb1d6a5..0000000 --- a/docs/notes.html +++ /dev/null @@ -1,51 +0,0 @@ -

Python IMPORT

-

What is the exact syntax of the python import command?

-

Clarifying terminology

-

The language used for describing the import mechanism is confusing, often horribly so. Let's go with some clarification first.

-

When the 'import ' command is called, a procedure is triggered. That procedure then:

- -

Only the first three matter for our purposes.

-

FINDING

-

Finding is the act of identifying the a resource that can be compiled into a meaningful Python module. This resource could a file on a filesystem, a cell in a database, a remote web object, a stream of bytes in an object store, some content object in a compressed archive, or anything that can meaningfully be said described as an array of bytes. It could even be dynamically generated in some way.

-

Finding typically involves scanning a collection of resources against a collection of finders. Finding ends when "finder A, given fullname B, reports that a corresponding module can be found in resource C, and that the resource can be loaded with loader D."

-

METAFINDERS

-

Finders come first, and MetaFinders come before all other kinds of finders.

-

Most finding is done in the context of sys.path; that is, Python's primary means of organizing Python modules is to have them somewhere on the local filesystem. Sometimes, however, you want to get in front of that scan. That's what you do with a MetaFinder: A MetaFinder may have its own take on what to do with sys.path; it may choose to ignore sys.path entirely and do something with the import fullname that has nothing to do with the local filesystem.

-

A Finder is any object with the following function: [Loader|None] find_module([self|cls], fullname:string, path:[string|None])

-

If find_module returns None if it cannot find a loader resource for fullname & path.

-

MetaFinders are placed into the list sys.meta_path by whatever code needs a MetaFinder, and persist for the duration of the runtime provided they're not removed or replaced. Being a list, the search is ordered and first one wins. MetaFinders may be instantiated in any way the developer desires.

-

PATH_HOOK

-

PathHooks are how sys.path is scanned to determine the which Finder should be associated with a given directory path.

-

A PathHook is a function: [Finder|None] (path:string)

-

A PathHook is a function that takes a given directory path and, if the PathHook can identify a corresponding Finder for the modules in that directory path, returns the Finder, otherwise it returns None.

-

If no sys.meta_path finder returns a loader, the full array of sys.paths ⨯ sys.path_hooks is compared until a path_hook says it can handle the path and the corresponding finder says it can handle the fullname. If no match happens, Python's default import behavior is triggered.

-

PathHooks are placed into the list sys.path_hooks; like sys.meta_path, the list is ordered and first one wins.

-

LOADER

-

Loaders are returned by Finders, and are constructed by Finders with whatever resources the developer specifies the Finder has and can provide.

-

a collection of finders the fullname (the dot-separated string passed to the import function).

-

to find a corresponding python module, which is then compiled into Python bytecode and incorporated into the python runtime, where it will be accessible to the importing function or modules

-

MetaFinder: A python object with a single method:

-
(Loader|None) find_module(self, fullname:string, path:(string|None))
-

Python 2.7

-

iter_modules (iter_importers) -> calls iter_importer_modules for each importer in iter_importers

-

iter_importers (meta_path, get_importer) -> returns every importer in sys.meta_path + map(get_importer, sys.path)

-

get_importer(path):

-
returns a filtered list of sys.path_hooks for importers that can
-handle this path; if there is no match, returns ImpImporter(),
-which supplies a module iterator (ImpImporter.iter_modules) that
-relies on getmodulename.  
-
-* A path_hook is a function of (path -> Maybe importer)
-

iter_modules(path, get_importer, prefix) -> calls iter_importer_modules for each importer returned by path.map(get_importer)

-

iter_importer_modules(importer) -> returns list of (filename, ispkg) for each module understood by the importer * The method called depends on the class of the importer * The default is a generic call for "no specific importer" * For FILES, iter_import_modules returns a list of files whose extensions match those in imp.get_suffixes(), which is hard- coded into the interpreter. * MEANING: Unless your importer can handle heterogenous module suffixes, SourceFiles.iter_importer_modules can only find homogeonous modules.

-

This relationship issue holds for Python 2.6 as well.

-

Python 3.3

-
The same issue holds, although now most of the extensions have been
-moved to importlib._bootstrap.
-

It is the relationship between importlib.machinery.FileFinder and iterfile_finder_modules

-

That's killing us.

diff --git a/docs/notes.md b/docs/notes.md deleted file mode 100644 index 2257d5f..0000000 --- a/docs/notes.md +++ /dev/null @@ -1,236 +0,0 @@ -# Python IMPORT - -What is the *exact* syntax of the python `import` command? What does it -mean when you write: - -``` -import alpha.beta.gamma -from alpha import beta -from alpha.beta import gamma -from .delta import epsilon -``` - -In each case, python is attempting to resolve the collection of dotted -names into a *module object*. - -**module object**: A resource that is or can be compiled into a -meaningful Python *module*. This resource could a file on a filesystem, -a cell in a database, a remote web object, a stream of bytes in an -object store, some content object in a compressed archive, or anything -that can meaningfully be said described as an array of bytes. It could -even be dynamically generated! - -**module**: The organizational unit of Python code. A namespace -containing Python objects, including classes, functions, submodules, and -immediately invoked code. Modules themselves may be collected into -*packages*. - -**package**: A python module which contains submodules or even -subpackages. The most common packaging scheme is a directory folder; in -this case the folder is a module if it contains an `__init__.py` file, -and it is a *package* if it contains other modules. The name of the -package is the folder name; the name of a submodule would be -`foldername.submodule`. This is called *regular packaging*. An -alternative method is known as *namespace packaging*. - -Python has a baroque but generally flexible mechanism for defining how -the dotted name is turned into a *module object*, which it calls *module -finding*, and for how that *module object* is turned into a *code -object* within the current Python session, called *module loading*. - -Python also has a means for *listing* modules. *Listing* is usually -done on a list of paths, using an appropriate means for accessing the -contents at the end of a path. - -The technical definition of a *package* is a module with a `__path__`, a -list of paths that contain submodules for the package. Subpackages get -their own` __path__`. A package can therefore accomodate `.` and `..` -prefixes in submodules, indicating relative paths to sibling modules. A -package can also and to its own `__path__` collection to enable access -to submodules elsewhere. - -# Clarifying terminology - -The language used for describing the import mechanism is confusing, -often horribly so. Let's go with some clarification first. - -When the 'import ' command is called, a procedure is -triggered. That procedure then: - -* attempts to *find* a corresponding python *module* -* attempts to *load* that corresponding module into *bytecode* -* Associates the bytecode with the name via sys.modules[fullname] -* Exposes the bytecode to the calling scope. - -Only the first three matter for our purposes. - -## FINDING - -*Finding* is the act of identifying a resource that corresponds to the -import string and can be compiled into a meaningful Python module. The -import string is typically called the *fullname*. - -*Finding* typically involves scanning a collection of *resources* -against a collection of *finders*. *Finding* ends when *finder `A`*, -given *fullname `B`*, reports that a corresponding module can be found -in *resource `C`*, and that the resource can be loaded with *loader -`D`*." - -### METAFINDERS - -*Finders* come first, and *MetaFinders* come before all other kinds of -finders. - -_Most finding is done in the context of `sys.path`_; that is, Python's -primary means of organizing Python modules is to have them somewhere on -the local filesystem, which makes sense. Sometimes, however, you want -to get in front of that scan. That's what you do with a MetaFinder: A -MetaFinder may have its own take on what to do with `sys.path`; it may -choose to ignore `sys.path` entirely and do something with the import -*fullname* that has nothing to do with the local filesystem. - -A Finder is any object with the following function: - [Loader|None] find_module([self|cls], fullname:string, path:[string|None]) - -If find_module returns None if it cannot find a loader resource for -fullname & path. - -A MetaFinder is placed into the list `sys.meta_path` by whatever code -needs the MetaFinder, and it persists for the duration of the runtime, -unless it is later removed or replaced. Being a list, the search is -ordered; first match wins. MetaFinders may be instantiated in any way -the developer desires before being added into `sys.meta_path`. - -### PATH_HOOK - -*PathHooks* are how `sys.path` is scanned to determine the which Finder -should be associated with a given directory path. - -A *PathHook* is a function: - [Finder|None] (path:string) - -A *PathHook* is a function that takes a given directory path and, if the -PathHook can identify a corresponding Finder for the modules in that -directory path, returns the Finder, otherwise it returns None. - -If no `sys.meta_path` finder returns a loader, the full array of -`sys.paths ⨯ sys.path_hooks` is compared until a PathHook says it can -handle the path and the corresponding finder says it can handle the -fullname. If no match happens, Python's default import behavior is -triggered. - -PathHooks are placed into the list `sys.path_hooks`; like -`sys.meta_path`, the list is ordered and first one wins. - -### LOADER - -*Loaders* are returned by *Finders*, and are constructed by Finders with -whatever resources the developer specifies the Finder has and can -provide. The Loader is responsible for pulling the content of the -*module object* into Python's memory and processing it into a *module*, -whether by calling Python's `eval()/compile()` functions on standard -Python code, or by some other means. - - - -a collection of *finders* the *fullname* (the dot-separated string passed to the `import` -function). - - - -to find a -corresponding python module, which is then compiled into Python bytecode -and incorporated into the python runtime, where it will be accessible to -the importing function or modules - -MetaFinder: A python object with a single method: - - (Loader|None) find_module(self, fullname:string, path:(string|None)) - - - - - -Python 2.7 - -iter_modules (iter_importers) -> - calls iter_importer_modules for each importer in iter_importers - -iter_importers (meta_path, get_importer) -> - returns every importer in sys.meta_path + map(get_importer, sys.path) - -get_importer(path): - - returns a filtered list of sys.path_hooks for importers that can - handle this path; if there is no match, returns ImpImporter(), - which supplies a module iterator (ImpImporter.iter_modules) that - relies on getmodulename. - - * A path_hook is a function of (path -> Maybe importer) - -iter_modules(path, get_importer, prefix) -> - calls iter_importer_modules for each importer returned by path.map(get_importer) - -iter_importer_modules(importer) -> - returns list of (filename, ispkg) for each module understood by the importer - * The method called depends on the class of the importer - * The default is a generic call for "no specific importer" - * For FILES, iter_import_modules returns a list of files whose - extensions match those in imp.get_suffixes(), which is hard- - coded into the interpreter. - * MEANING: Unless your importer can handle heterogenous module - suffixes, SourceFiles.iter_importer_modules can only find - homogeonous modules. - -This relationship issue holds for Python 2.6 as well. - -Python 3.3 - - The same issue holds, although now most of the extensions have been - moved to importlib._bootstrap. - -It is the relationship between - importlib.machinery.FileFinder -and - _iter_file_finder_modules - -That's killing us. - - - ---- - -So the ONLY thing I have to do, according to Python, is assert that -there's a dir/__init__.suff and attempt to load it! If I do that, I can -make it work? - -No: The search for __init__.suff is only the first - - ---- - -test_import: test_with_extension "py" and "my" -test_execute_bit_not_set (on Posix system, .pyc files got their -executable bit set if the .py file had it set; it looks as if Python -just copied the permissions, if it had permission to do so. We should -follow the example of 2.7 & 3.4, and NOT set +x if we can help it). - -test_rewrite_pyc_with_read_only_source (on Posix systems, if the .py -file had read-only set, the .pyc file would too, making updates -problematic). - -test_import_name_binding - -test_bug7732 (attempt to import a '.my' file that's not a file) - - - -These are more Hy-related: - -test_module_with_large_stack (see python example) - -test_failing_import_sticks - -test_failing_reload - - diff --git a/docs/notes.txt b/docs/notes.txt deleted file mode 100644 index ad4c919..0000000 --- a/docs/notes.txt +++ /dev/null @@ -1,152 +0,0 @@ -Clarifying terminology - -The language used for describing the import mechanism is confusing, -often horribly so. Let's go with some clarification first. - -When the 'import ' command is called, a procedure is -triggered. That procedure then: - -* attempts to *find* a corresponding python *module* -* attempts to *load* that corresponding module into *bytecode* -* Associates the bytecode with the name via sys.modules[fullname] -* Exposes the bytecode to the calling scope. - -Only the first three matter for our purposes. - -## FINDING - -*Finding* is the act of identifying the a resource that can be compiled -into a meaningful Python module. This resource could a file on a -filesystem, a cell in a database, a remote web object, a stream of bytes -in an object store, some content object in a compressed archive, or -anything that can meaningfully be said described as an array of bytes. -It could even be dynamically generated in some way. - -*Finding* typically involves scanning a collection of *resources* -against a collection of *finders*. *Finding* ends when "*finder __A__*, -given *fullname __B__*, reports that a corresponding module can be found -in *resource __C__*, and that the resource can be loaded with *loader -__D__*." - -### METAFINDERS - -*Finders* come first, and *MetaFinders* come before all other kinds of -finders. - -_Most finding is done in the context of `sys.path`_; that is, Python's -primary means of organizing Python modules is to have them somewhere on -the local filesystem. Sometimes, however, you want to get in front of -that scan. That's what you do with a MetaFinder: A MetaFinder may have -its own take on what to do with `sys.path`; it may choose to ignore -`sys.path` entirely and do something with the import *fullname* that has -nothing to do with the local filesystem. - -A Finder is any object with the following function: - [Loader|None] find_module([self|cls], fullname:string, path:[string|None]) - -If find_module returns None if it cannot find a loader resource for -fullname & path. - -MetaFinders are placed into the list `sys.meta_path` by whatever code -needs a MetaFinder, and persist for the duration of the runtime provided -they're not removed or replaced. Being a list, the search is ordered -and first one wins. MetaFinders may be instantiated in any way the -developer desires. - -### PATH_HOOK - -*PathHooks* are how `sys.path` is scanned to determine the -which Finder should be associated with a given directory path. - -A *PathHook* is a function: - [Finder|None] (path:string) - -A *PathHook* is a function that takes a given directory path and, if the -PathHook can identify a corresponding Finder for the modules in that -directory path, returns the Finder, otherwise it returns None. - -If no `sys.meta_path` finder returns a loader, the full array of -`sys.paths ⨯ sys.path_hooks` is compared until a path_hook says it can -handle the path and the corresponding finder says it can handle the -fullname. If no match happens, Python's default import behavior is -triggered. - - -PathHooks are placed into the list `sys.path_hooks`; like -`sys.meta_path`, the list is ordered and first one wins. - -### LOADER - -*Loaders* are returned by *Finders*, and are constructed by Finders with -whatever resources the developer specifies the Finder has and can -provide. - - - -a collection of *finders* the *fullname* (the dot-separated string passed to the `import` -function). - - - -to find a -corresponding python module, which is then compiled into Python bytecode -and incorporated into the python runtime, where it will be accessible to -the importing function or modules - -MetaFinder: A python object with a single method: - - (Loader|None) find_module(self, fullname:string, path:(string|None)) - - - - - -Python 2.7 - -iter_modules (iter_importers) -> - calls iter_importer_modules for each importer in iter_importers - -iter_importers (meta_path, get_importer) -> - returns every importer in sys.meta_path + map(get_importer, sys.path) - -get_importer(path): - - returns a filtered list of sys.path_hooks for importers that can - handle this path; if there is no match, returns ImpImporter(), - which supplies a module iterator (ImpImporter.iter_modules) that - relies on getmodulename. - - * A path_hook is a function of (path -> Maybe importer) - -iter_modules(path, get_importer, prefix) -> - calls iter_importer_modules for each importer returned by path.map(get_importer) - -iter_importer_modules(importer) -> - returns list of (filename, ispkg) for each module understood by the importer - * The method called depends on the class of the importer - * The default is a generic call for "no specific importer" - * For FILES, iter_import_modules returns a list of files whose - extensions match those in imp.get_suffixes(), which is hard- - coded into the interpreter. - * MEANING: Unless your importer can handle heterogenous module - suffixes, SourceFiles.iter_importer_modules can only find - homogeonous modules. - -This relationship issue holds for Python 2.6 as well. - -Python 3.3 - - The same issue holds, although now most of the extensions have been - moved to importlib._bootstrap. - -It is the relationship between - importlib.machinery.FileFinder -and - _iter_file_finder_modules - -That's killing us. - - - - - diff --git a/docs/todo.rst b/docs/todo.rst new file mode 100644 index 0000000..6668b49 --- /dev/null +++ b/docs/todo.rst @@ -0,0 +1,10 @@ +Currently assumes that Polyloaders last for the lifetime of the Python +instance. There is no standardized mechanism for removing a +compiler/suffix set from a running instance. + +1. Create a standardized mechanism for removing a compiler/suffix from + the running instance. + +2. Create a 'with' context manager that creates a scope in which a + Polyloader compiler/suffix pair is active, and automatically removes + that pair upon leaving the scope. diff --git a/polyloader/__init__.py b/polyloader/__init__.py index 405e87c..55d8878 100644 --- a/polyloader/__init__.py +++ b/polyloader/__init__.py @@ -6,9 +6,9 @@ __email__ = 'elf.sternberg@gmail.com' __version__ = '0.1.0' if sys.version_info[0:2] >= (2, 6): - from ._python2 import install + from ._python2 import install, reset if sys.version_info[0] >= 3: - from ._python3 import install + from ._python3 import install, reset -__all__ = ['install'] +__all__ = ['install', 'reset'] diff --git a/polyloader/_python2.py b/polyloader/_python2.py index bee8879..05bc5cc 100644 --- a/polyloader/_python2.py +++ b/polyloader/_python2.py @@ -1,9 +1,9 @@ import io import os import os.path -import stat import sys import imp +import types import pkgutil @@ -25,11 +25,10 @@ class PolyLoader(): if overlap: raise RuntimeError("Override of native Python extensions is not permitted.") overlap = suffixes.intersection( - set([suffix for (compiler, suffix) in cls._loader_handlers])) + set([suffix for (_, suffix) in cls._loader_handlers])) if overlap: - raise RuntimeWarning( - "Insertion of %s overrides already installed compiler." % - ', '.join(list(overlap))) + # Fail silently + return cls._loader_handlers += [(compiler, suf) for suf in suffixes] def load_module(self, fullname): @@ -42,26 +41,29 @@ class PolyLoader(): matches = [(compiler, suffix) for (compiler, suffix) in self._loader_handlers if self.path.endswith(suffix)] - if matches.length == 0: + if len(matches) == 0: raise ImportError("%s is not a recognized module?" % fullname) - if matches.length > 1: + if len(matches) > 1: raise ImportError("Multiple possible resolutions for %s: %s" % ( fullname, ', '.join([suffix for (compiler, suffix) in matches]))) - compiler = matches[0] + compiler = matches[0][0] with io.FileIO(self.path, 'r') as file: source_text = file.read() - module = compiler(source_text, fullname, self.path) + code = compiler(source_text, self.path, fullname) + + module = types.ModuleType(fullname) module.__file__ = self.path - module.__name__ = self.fullname + module.__name__ = fullname module.__package__ = '.'.join(fullname.split('.')[:-1]) if self.is_package: - module.__path__ = [os.path.dirname(self.path)] + module.__path__ = [os.path.dirname(module.__file__)] module.__package__ = fullname + exec(code, module.__dict__) sys.modules[fullname] = module return module @@ -76,29 +78,29 @@ class PolyLoader(): class PolyFinder(object): def __init__(self, path=None): self.path = path or '.' - - def _pl_find_on_path(self, fullname, path=None): - splitname = fullname.split(".") - if self.path is None and splitname[-1] != fullname: - return None - - dirpath = "/".join(splitname) - path = [os.path.realpath(self.path)] + def _pl_find_on_path(self, fullname, path=None): + subname = fullname.split(".")[-1] + if self.path is None and subname != fullname: + return None + + path = os.path.realpath(self.path) fls = [("%s/__init__.%s", True), ("%s.%s", False)] for (fp, ispkg) in fls: for (compiler, suffix) in PolyLoader._loader_handlers: - composed_path = fp % ("%s/%s" % (path, dirpath), suffix) + composed_path = fp % ("%s/%s" % (path, subname), suffix) + if os.path.isdir(composed_path): + raise IOError("Invalid: Directory name ends in recognized suffix") if os.path.isfile(composed_path): return PolyLoader(fullname, composed_path, ispkg) # Fall back onto Python's own methods. try: - file, filename, etc = imp.find_module(fullname, path) - except ImportError: + file, filename, etc = imp.find_module(subname, [path]) + except ImportError as e: return None return pkgutil.ImpLoader(fullname, file, filename, etc) - + def find_module(self, fullname, path=None): return self._pl_find_on_path(fullname) @@ -106,12 +108,13 @@ class PolyFinder(object): def getmodulename(path): filename = os.path.basename(path) suffixes = ([(-len(suf[0]), suf[0]) for suf in imp.get_suffixes()] + - [(-len(suf[1]), suf[1]) for suf in PolyLoader.loader_handlers]) + [(-len(suf[1]), suf[1]) for suf in PolyLoader._loader_handlers]) suffixes.sort() for neglen, suffix in suffixes: if filename[neglen:] == suffix: - return (filename[:neglen], suffix) - + return filename[:neglen] + return None + def iter_modules(self, prefix=''): if self.path is None or not os.path.isdir(self.path): return @@ -126,7 +129,7 @@ class PolyFinder(object): filenames.sort() for fn in filenames: modname = self.getmodulename(fn) - if modname=='__init__' or modname in yielded: + if modname == '__init__' or modname in yielded: continue path = os.path.join(self.path, fn) @@ -141,7 +144,7 @@ class PolyFinder(object): dircontents = [] for fn in dircontents: subname = self.getmodulename(fn) - if subname=='__init__': + if subname == '__init__': ispkg = True break else: @@ -152,16 +155,19 @@ class PolyFinder(object): yield prefix + modname, ispkg - def _polyloader_pathhook(path): if not os.path.isdir(path): - raise ImportError('Only directories are supported', path = path) + raise ImportError('Only directories are supported', path=path) return PolyFinder(path) - + def install(compiler, suffixes): if not PolyLoader._installed: sys.path_hooks.append(_polyloader_pathhook) PolyLoader._installed = True PolyLoader._install(compiler, suffixes) - + + +def reset(): + PolyLoader._loader_handlers = [] + PolyLoader._installed = False diff --git a/polyloader/_python3.py b/polyloader/_python3.py index ad60c79..7a8b419 100644 --- a/polyloader/_python3.py +++ b/polyloader/_python3.py @@ -1,29 +1,3 @@ -# polyloader.py -# -# Copyright (c) 2016 Elf M. Sternberg -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER -# DEALINGS IN THE SOFTWARE. -# - -""" Utilities for initializing extended path-hooks into the Python runtime """ -__all__ = [] # Exports nothing; this module is called for its side-effects - import os import sys @@ -33,11 +7,6 @@ from pkgutil import iter_importer_modules import sys -__author__ = 'Elf M. Sternberg' -__version__ = '2016.05.29' -__contact__ = 'elf.sternberg@gmail.com' - - if sys.version_info[0:2] in [(3,3), (3,4)]: from importlib._bootstrap import _get_supported_file_loaders sourcefile_recognizer = 'importlib.SourceFileLoader' @@ -54,7 +23,7 @@ def _call_with_frames_removed(f, *args, **kwds): return f(*args, **kwds) -class ExtendedSourceFileLoader(machinery.SourceFileLoader): +class PolySourceFileLoader(machinery.SourceFileLoader): """Override the get_code method. Falls back on the SourceFileLoader if it's a Python file, which will generate pyc files as needed, or works its way into the Extended version. This method does @@ -91,8 +60,14 @@ class ExtendedSourceFileLoader(machinery.SourceFileLoader): # Provide a working namespace for our new FileFinder. -class ExtendedFileFinder(machinery.FileFinder): +class PolySourceFileFinder(machinery.FileFinder): + + + + + + # Taken from inspect.py and modified to support alternate suffixes. @staticmethod def getmodulename(path): @@ -148,28 +123,27 @@ class ExtendedFileFinder(machinery.FileFinder): yield prefix + modname, ispkg pass - -# Monkeypatch both path_hooks and iter_importer_modules to make our -# modules recognizable to the module iterator functions. This is -# probably horribly fragile, but there doesn't seem to be a more -# robust way of doing it at the moment, and these names are stable -# from python 2.7 up. + + def install(compiler, suffixes): - """ Install a compiler and suffix(es) into Python's sys.path_hooks, so - that modules ending with thoses suffixes will be parsed into - python executable modules automatically. + """Install a specialized version of FileFinder that will search + through alternative extensions first for syntax files and, upon + encountering one, will return a specialized version of + SourceFileLoader for that syntax. By replacing this into + path_hook this makes both import and iter_modules work as + expected. """ - - if sys.version[0] >= 3: - filefinder = [(f, i) for i, f in enumerate(sys.path_hooks) - if repr(f).find('path_hook_for_FileFinder') != -1] - if not filefinder: - return - filefinder, fpos = filefinder[0] - ExtendedSourceFileLoader._source_handlers = (ExtendedSourceFileLoader._source_handlers + - [(compiler, tuple(suffixes))]) + + filefinder = [(f, i) for i, f in enumerate(sys.path_hooks) + if repr(f).find('.path_hook_for_FileFinder') != -1] + if not filefinder: + return + filefinder, fpos = filefinder[0] + + + supported_loaders = _get_supported_file_loaders() print([repr(i) for i in supported_loaders]) @@ -186,52 +160,9 @@ def install(compiler, suffixes): if sys.path[0] != "": sys.path.insert(0, "") - if sys.version[0:2] == (2, 7): - def loader(path): - class Loader(object): - def __init__(self, path): - self.path = path - - def is_package(self, fullname): - dirpath = "/".join(fullname.split(".")) - for pth in sys.path: - pth = os.path.abspath(pth) - composed_path = "%s/%s/__init__.%s" % (pth, dirpath, suffix) - if os.path.exists(composed_path): - return True - return False - - def load_module(self, name): - if name in sys.modules: - return sys.modules[name] - - module = compiler(fullname, path) - module.__file__ = path - sys.modules[name] = module - if '.' in name: - parent_name, child_name = name.rsplit('.', 1) - setattr(sys.modules[parent_name], child_name, module) - sys.modules[name] = module - return module - - return Loader() - class MetaImporter(object): - def find_module(self, fullname, path=None): - if fullname == 'numpy' or fullname.startswith('numpy.'): - _mapper.PerpetrateNumpyFixes() - if fullname in ('_hashlib', 'ctypes'): - raise ImportError('%s is not available in ironclad yet' % fullname) - - lastname = fullname.rsplit('.', 1)[-1] - for d in (path or sys.path): - pyd = os.path.join(d, lastname + '.pyd') - if os.path.exists(pyd): - return loader(pyd) - - return None - sys.meta_path.insert(0, MetaImporter()) - iter_importer_modules.register(MetaImporter, meta_iterate_modules) + - +def reset(): + pass diff --git a/tests/polytestmix/test1.py b/tests/polytestmix/test1.py index e69de29..c99c3f1 100644 --- a/tests/polytestmix/test1.py +++ b/tests/polytestmix/test1.py @@ -0,0 +1 @@ +result = "Success for 1: Test One" diff --git a/tests/ptutils.py b/tests/ptutils.py new file mode 100644 index 0000000..e883e8f --- /dev/null +++ b/tests/ptutils.py @@ -0,0 +1,48 @@ +import UserDict +import os + +class EnvironmentVarGuard(UserDict.DictMixin): + + """Class to help protect the environment variable properly. Can be used as + a context manager.""" + + def __init__(self): + self._environ = os.environ + self._changed = {} + + def __getitem__(self, envvar): + return self._environ[envvar] + + def __setitem__(self, envvar, value): + # Remember the initial value on the first access + if envvar not in self._changed: + self._changed[envvar] = self._environ.get(envvar) + self._environ[envvar] = value + + def __delitem__(self, envvar): + # Remember the initial value on the first access + if envvar not in self._changed: + self._changed[envvar] = self._environ.get(envvar) + if envvar in self._environ: + del self._environ[envvar] + + def keys(self): + return self._environ.keys() + + def set(self, envvar, value): + self[envvar] = value + + def unset(self, envvar): + del self[envvar] + + def __enter__(self): + return self + + def __exit__(self, *ignore_exc): + for (k, v) in self._changed.items(): + if v is None: + if k in self._environ: + del self._environ[k] + else: + self._environ[k] = v + os.environ = self._environ diff --git a/tests/relimport.py b/tests/relimport.py new file mode 100644 index 0000000..50aa497 --- /dev/null +++ b/tests/relimport.py @@ -0,0 +1 @@ +from .test_import import * diff --git a/tests/test_import.py b/tests/test_import.py new file mode 100644 index 0000000..5d6c211 --- /dev/null +++ b/tests/test_import.py @@ -0,0 +1,273 @@ +import polyloader +import pytest +import py_compile +import ptutils +import stat +import sys +import os +import imp +import random +import struct + +class Compiler: + def __init__(self, pt): + self.pt = pt + + def __call__(self, source_text, filename, *extra): + return compile("result='Success for %s: %s'" % + (self.pt, source_text.rstrip()), filename, "exec") + + def __repr__(self): + return "Compiler %s" % (self.pt) + +polyloader.install(Compiler("2"), ['2']) + +TESTFN = '@test' + +def clean_tmpfiles(path): + if os.path.exists(path): + os.remove(path) + if os.path.exists(path + 'c'): + os.remove(path + 'c') + if os.path.exists(path + 'o'): + os.remove(path + 'o') + +def unload(name): + try: + del sys.modules[name] + except KeyError: + pass + +# This exists mostly to test that the inclusion of PolyLoader doesn't +# break anything else in Python. + +# Tests commented out: 2 + +class Test_Imports(object): + + def test_case_sensitivity(self): + # Brief digression to test that import is case-sensitive: if we got + # this far, we know for sure that "random" exists. + try: + import RAnDoM + except ImportError: + pass + else: + assert(False) + + def test_import(self): + sys.path.insert(0, os.curdir) + ext = 'py' + try: + # The extension is normally ".py", perhaps ".pyw". + source = TESTFN + os.extsep + ext + pyc = TESTFN + os.extsep + "pyc" + + with open(source, "w") as f: + print >> f, ("# This tests Python's ability to import a", ext, + "file.") + a = random.randrange(1000) + b = random.randrange(1000) + print >> f, "a =", a + print >> f, "b =", b + + if os.path.exists(source): + print "%s EXISTS..." % source + + try: + print "IMPORT..." + mod = __import__(TESTFN) + print "END IMPORT..." + except ImportError, err: + print("import from %s (%s) failed: %s" % (ext, os.curdir, err)) + assert(False) + else: + assert(mod.a == a) + assert(mod.b == b) + finally: + os.remove(source) + + try: + if not sys.dont_write_bytecode: + imp.reload(mod) + except ImportError, err: + self.fail("import from .pyc/.pyo failed: %s" % err) + finally: + clean_tmpfiles(source) + unload(TESTFN) + finally: + del sys.path[0] + + def test_execute_bit_not_copied(self): + # Issue 6070: under posix .pyc files got their execute bit set if + # the .py file had the execute bit set, but they aren't executable. + oldmask = os.umask(022) + sys.path.insert(0, os.curdir) + try: + fname = TESTFN + os.extsep + "py" + f = open(fname, 'w').close() + os.chmod(fname, (stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH | + stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)) + __import__(TESTFN) + fn = fname + 'c' + if not os.path.exists(fn): + fn = fname + 'o' + if not os.path.exists(fn): + self.fail("__import__ did not result in creation of " + "either a .pyc or .pyo file") + s = os.stat(fn) + assert(s.st_mode & (stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)) + assert(not (s.st_mode & (stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH))) + finally: + os.umask(oldmask) + clean_tmpfiles(fname) + unload(TESTFN) + del sys.path[0] + + def test_imp_module(self): + # Verify that the imp module can correctly load and find .py files + + # XXX (ncoghlan): It would be nice to use test_support.CleanImport + # here, but that breaks because the os module registers some + # handlers in copy_reg on import. Since CleanImport doesn't + # revert that registration, the module is left in a broken + # state after reversion. Reinitialising the module contents + # and just reverting os.environ to its previous state is an OK + # workaround + orig_path = os.path + orig_getenv = os.getenv + with ptutils.EnvironmentVarGuard(): + x = imp.find_module("os") + new_os = imp.load_module("os", *x) + assert(os == new_os) + assert(orig_path == new_os.path) + assert(orig_getenv != new_os.getenv) + + def test_module_with_large_stack(self, module='longlist'): + # Regression test for http://bugs.python.org/issue561858. + filename = module + os.extsep + 'py' + + # Create a file with a list of 65000 elements. + with open(filename, 'w+') as f: + f.write('d = [\n') + for i in range(65000): + f.write('"",\n') + f.write(']') + + # Compile & remove .py file, we only need .pyc (or .pyo). + with open(filename, 'r') as f: + py_compile.compile(filename) + os.remove(filename) + + # Need to be able to load from current dir. + sys.path.append('') + + # This used to crash. + exec 'import ' + module + + # Cleanup. + del sys.path[-1] + clean_tmpfiles(filename) + + + def test_failing_import_sticks(self): + source = TESTFN + os.extsep + "py" + with open(source, "w") as f: + print >> f, "a = 1 // 0" + + # New in 2.4, we shouldn't be able to import that no matter how often + # we try. + sys.path.insert(0, os.curdir) + try: + for i in [1, 2, 3]: + with pytest.raises(ZeroDivisionError) as zde: + __import__(TESTFN) + assert(sys.modules.get(TESTFN) == None) + finally: + del sys.path[0] + clean_tmpfiles(source) + +# def test_failing_reload(self): +# # A failing reload should leave the module object in sys.modules. +# source = TESTFN + os.extsep + "py" +# with open(source, "w") as f: +# print >> f, "a = 1" +# print >> f, "b = 2" +# +# sys.path.insert(0, os.curdir) +# try: +# mod = __import__(TESTFN) +# assert(sys.modules.get(TESTFN) != None) +# assert(mod.a == 1) +# assert(mod.b == 2) +# +# # On WinXP, just replacing the .py file wasn't enough to +# # convince reload() to reparse it. Maybe the timestamp didn't +# # move enough. We force it to get reparsed by removing the +# # compiled file too. +# clean_tmpfiles(TESTFN) +# +# # Now damage the module. +# with open(source, "w") as f: +# print >> f, "a = 10" +# print >> f, "b = 20 // 0" +# +# with pytest.raises(ZeroDivisionError) as zde: +# imp.reload(mod) +# +# # But we still expect the module to be in sys.modules. +# mod = sys.modules.get(TESTFN) +# assert(mod != None) +# +# # We should have replaced a w/ 10, but the old b value should +# # stick. +# assert(mod.a == 10) +# assert(mod.b == 2) +# +# finally: +# del sys.path[0] +# clean_tmpfiles(source) +# unload(TESTFN) + + def test_infinite_reload(self): + # http://bugs.python.org/issue742342 reports that Python segfaults + # (infinite recursion in C) when faced with self-recursive reload()ing. + + sys.path.insert(0, os.path.dirname(__file__)) + try: + with pytest.raises(ImportError): + import infinite_reload + finally: + del sys.path[0] + + def test_import_name_binding(self): + # import x.y.z binds x in the current namespace. + import test as x + import test.test_support + assert(x == test) + assert(hasattr(test.test_support, "__file__")) + + # import x.y.z as w binds z as w. + import test.test_support as y + assert(y == test.test_support) + + def test_import_initless_directory_warning(self): + with pytest.raises(ImportError) as ie: + __import__('site-packages') + + def test_import_by_filename(self): + path = os.path.abspath(TESTFN) + with pytest.raises(ImportError) as c: + __import__(path) + assert("Import by filename is not supported." == c.value[0]) + +# def test_bug7732(self): +# source = TESTFN + '.2' +# os.mkdir(source) +# try: +# with pytest.raises(IOError) as iie: +# mod = imp.find_module(TESTFN, ["."]) +# print("Found mod %s" % mod) +# finally: +# os.rmdir(source) + diff --git a/tests/test_paths.py b/tests/test_paths.py new file mode 100644 index 0000000..6da4180 --- /dev/null +++ b/tests/test_paths.py @@ -0,0 +1,61 @@ +import polyloader +import pytest +import py_compile +import ptutils +import stat +import sys +import os +import imp +import random +import struct + +class Compiler: + def __init__(self, pt): + self.pt = pt + + def __call__(self, source_text, filename, *extra): + return compile("result='Success for %s: %s'" % + (self.pt, source_text.rstrip()), filename, "exec") + + def __repr__(self): + return "Compiler %s" % (self.pt) + +polyloader.install(Compiler("2"), ['2']) + +TESTFN = '@test' + +def clean_tmpfiles(path): + if os.path.exists(path): + os.remove(path) + if os.path.exists(path + 'c'): + os.remove(path + 'c') + if os.path.exists(path + 'o'): + os.remove(path + 'o') + +def unload(name): + try: + del sys.modules[name] + except KeyError: + pass + + +class Test_Paths: + path = TESTFN + + def setup_method(self, method): + os.mkdir(self.path) + self.syspath = sys.path[:] + + def teardown_method(self, method): + rmtree(self.path) + sys.path[:] = self.syspath + + # Regression test for http://bugs.python.org/issue1293. + def test_trailing_slash(self): + with open(os.path.join(self.path, 'test_trailing_slash.2'), 'w') as f: + f.write("testdata = 'test_trailing_slash'") + sys.path.append(self.path+'/') + mod = __import__("test_trailing_slash") + self.assertEqual(mod.testdata, 'test_trailing_slash') + unload("test_trailing_slash") + diff --git a/tests/test_polyloader.py b/tests/test_polyloader.py index 5942db9..ea8a8b9 100644 --- a/tests/test_polyloader.py +++ b/tests/test_polyloader.py @@ -8,44 +8,110 @@ test_polyloader Tests for `polyloader` module. """ -import pytest - -from polyloader import polyloader +import polyloader +import copy +import sys # Note that these compilers don't actually load much out of the # source files. That's not the point. The point is to show that the # correct compiler has been found for a given extension. -def compiler(pt): - def _compiler(source_text, modulename): - return compile("result='Success for %s: %s'" % (pt, source_text.rstrip()), modulename, "exec") - return _compiler -class Test_Polymorph_1(object): +class ImportEnvironment(object): + def __init__(self): + pass + + def __enter__(self): + polyloader.reset() + self.path = copy.copy(sys.path) + self.path_hooks = copy.copy(sys.path_hooks) + self.meta_path = copy.copy(sys.meta_path) + self.modules = copy.copy(sys.modules) + self.path_importer_cache = copy.copy(sys.path_importer_cache) + return sys + + def __exit__(self, type, value, traceback): + sys.path = self.path + sys.path_hooks = self.path_hooks + sys.meta_path = self.meta_path + sys.modules = self.modules + sys.path_importer_cache = self.path_importer_cache + + +class Compiler: + def __init__(self, pt): + self.pt = pt + + def __call__(self, source_text, filename, *extra): + return compile("result='Success for %s: %s'" % + (self.pt, source_text.rstrip()), filename, "exec") + + def __repr__(self): + return "Compiler %s" % (self.pt) + +def compiler(pt): + return Compiler(pt) + +# Functionally, the "From" test and the "Direct" test should be +# indistinguishable. What's interesting about them, though, is that +# in the context of the caller, the "Direct" test now has test and +# test.polytestmix as objects in the calling context. They're fairly +# lightweight, but they do exist, and they do honor the __all__ and +# __path__ cases. +# +# Also: +# See http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html +# for some 'gotchas' to look forward to. Whee. + +class Test_Polymorph_From(object): def test_import1(self): - polyloader.install(compiler("2"), ['.2']) - polyloader.install(compiler("3"), ['.3']) - from .polytestmix import test2 - from .polytestmix import test3 - assert(test2.result == "Success for 2: Test Two") - assert(test3.result == "Success for 3: Test Three") + with ImportEnvironment() as sys: + polyloader.install(compiler("2"), ['2']) + polyloader.install(compiler("3"), ['3']) + from .polytestmix import test2 + from .polytestmix import test3 + from .polytestmix import test1 + assert(test1.result == "Success for 1: Test One") + assert(test2.result == "Success for 2: Test Two") + assert(test3.result == "Success for 3: Test Three") + +class Test_Polymorph_Direct(object): + def test_import2(self): + with ImportEnvironment() as sys: + polyloader.install(compiler("2"), ['2']) + polyloader.install(compiler("3"), ['3']) + import tests.polytestmix.test2 + import tests.polytestmix.test3 + import tests.polytestmix.test1 + assert(tests.polytestmix.test1.result == "Success for 1: Test One") + assert(tests.polytestmix.test2.result == "Success for 2: Test Two") + assert(tests.polytestmix.test3.result == "Success for 3: Test Three") + +class Test_Polymorph_Module(object): + def test_import3(self): + with ImportEnvironment() as sys: + polyloader.install(compiler("3"), ['3']) + polyloader.install(compiler("2"), ['2']) + from tests.polytestmix.test3 import result as result3 + from tests.polytestmix.test2 import result as result2 + from tests.polytestmix.test1 import result as result1 + assert(result1 == "Success for 1: Test One") + assert(result2 == "Success for 2: Test Two") + assert(result3 == "Success for 3: Test Three") class Test_Polymorph_Iterator(object): ''' The Django Compatibility test: Can we load arbitrary modules from a package? ''' def test_iterator(self): - import os - import pkgutil - import inspect - polyloader.install(compiler("2"), ['.2']) - polyloader.install(compiler("3"), ['.3']) - from .polytestmix import test2 - from .polytestmix import test3 - target_dir = os.path.join( - os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))), - 'polytestmix') - modules = set([name for _, name, is_pkg in pkgutil.iter_modules([target_dir]) - if not is_pkg and not name.startswith('_')]) - assert(modules == set(['test1', 'test2', 'test3'])) - - + with ImportEnvironment() as sys: + import os + import inspect + polyloader.install(compiler("2"), ['.2']) + polyloader.install(compiler("3"), ['.3']) + import pkgutil + target_dir = os.path.join( + os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))), + 'polytestmix') + modules = set([name for (_, name, is_pkg) in pkgutil.iter_modules([target_dir]) + if not is_pkg and not name.startswith('_')]) + assert(modules == set(['test1', 'test2', 'test3'])) diff --git a/tests/test_relativeimport.py b/tests/test_relativeimport.py new file mode 100644 index 0000000..ba10d4f --- /dev/null +++ b/tests/test_relativeimport.py @@ -0,0 +1,99 @@ +import polyloader +import pytest +import py_compile +import ptutils +import stat +import sys +import os +import imp +import random +import struct + +class Compiler: + def __init__(self, pt): + self.pt = pt + + def __call__(self, source_text, filename, *extra): + return compile("result='Success for %s: %s'" % + (self.pt, source_text.rstrip()), filename, "exec") + + def __repr__(self): + return "Compiler %s" % (self.pt) + +polyloader.install(Compiler("2"), ['2']) + +TESTFN = '@test' + +def clean_tmpfiles(path): + if os.path.exists(path): + os.remove(path) + if os.path.exists(path + 'c'): + os.remove(path + 'c') + if os.path.exists(path + 'o'): + os.remove(path + 'o') + +def unload(name): + try: + del sys.modules[name] + except KeyError: + pass + + +class Test_RelativeImports: + + def teardown_class(cls): + unload("tests.relimport") + + def setup_class(cls): + unload("tests.relimport") + + def test_relimport_star(self): + # This will import * from .test_import. + from . import relimport + assert(hasattr(relimport, "Test_Imports")) + + def test_issue3221(self): + # Regression test for http://bugs.python.org/issue3221. + def check_absolute(): + exec "from os import path" in ns + def check_relative(): + exec "from . import relimport" in ns + + # Check both OK with __package__ and __name__ correct + ns = dict(__package__='tests', __name__='test.notarealmodule') + check_absolute() + check_relative() + + # Check both OK with only __name__ wrong + ns = dict(__package__='tests', __name__='notarealpkg.notarealmodule') + check_absolute() + check_relative() + + # Check relative fails with only __package__ wrong + ns = dict(__package__='foo', __name__='test.notarealmodule') + with pytest.warns(RuntimeWarning) as rw: + check_absolute() + with pytest.raises(SystemError) as se: + check_relative() + + # Check relative fails with __package__ and __name__ wrong + ns = dict(__package__='foo', __name__='notarealpkg.notarealmodule') + with pytest.warns(RuntimeWarning) as se: + check_absolute() + with pytest.raises(SystemError) as se: + check_relative() + + # Check both fail with package set to a non-string + ns = dict(__package__=object()) + with pytest.raises(ValueError) as ve: + check_absolute() + with pytest.raises(ValueError) as ve: + check_relative() + + def test_absolute_import_without_future(self): + # If explicit relative import syntax is used, then do not try + # to perform an absolute import in the face of failure. + # Issue #7902. + with pytest.raises(ImportError) as ie: + from .os import sep + assert(False) diff --git a/tox.ini b/tox.ini index 0b8729e..4df6d80 100644 --- a/tox.ini +++ b/tox.ini @@ -1,5 +1,6 @@ [tox] -envlist = py26, py27, py33, py34, py35 +envlist = py26, py27 +; , py33, py34, py35 [testenv] setenv = @@ -7,8 +8,10 @@ setenv = deps = -r{toxinidir}/requirements_dev.txt commands = - py.test --basetemp={envtmpdir} + py.test --basetemp={envtmpdir} [] +[flake8] +max-line-length = 99 ; If you want to make tox run the tests with the same versions, create a ; requirements.txt with the pinned versions and uncomment the following lines: