The pidgy literate computing shell

https://ipython.readthedocs.io/en/stable/development/execution.html#execution-semantics

A powerful feature of the jupyter ecosystem is a generalized implementation of the Shell & Kernel model for interactive computing in interfaces like the terminal and notebooks. That is to say that different programming languages can use the same interfaces, jupyter supports over 100 languages now. The general ability to support different languages is possible because of configurable interfaces for the IPython.InteractiveShell and ipykernel.

import ipykernel.kernelapp, ipykernel.zmqshell, nbconvert, traitlets, pidgy, types, pluggy, IPython, jinja2
class pidgyShell(ipykernel.zmqshell.ZMQInteractiveShell):

The pidgy shell is wrapper around the existing IPython shell experience. It explicitly defines the tangle and weave conventions of literate programming to each interactive computing execution. Once the shell is configured, it can be reused as a jupyter kernel or IPython extension that supports the pidgy [Markdown]/[IPython] metalanguage and metasyntax.

    environment = traitlets.Any(nbconvert.exporters.TemplateExporter().environment)

pidgy specification

    @pidgy.specification(firstresult=True)
    def tangle(str:str)->str:

The tangle step operates on an input string that will become compiled source code. In a literate program, the source is written primarily in the documentation language and tangling converts it to the programming language. In pidgy, the tangle steps target valid IPython which is a superset of [Python], and requires further processing.

    input_transformers_post = traitlets.List([pidgy.tangle.demojize])

pidgy includes the ability the use emojis as valid python names through the existing traitlets configuration system.

    class pidgyManager(IPython.core.inputtransformer2.TransformerManager):
        def transform_cell(self, cell):
            shell = IPython.get_ipython()
            return super(type(self), self).transform_cell(
                (shell and hasattr(shell, 'manager') and shell.manager.hook.tangle)(str=cell))

    input_transformer_manager = traitlets.Instance(pidgyManager, args=tuple())

    ast_transformers = traitlets.List([pidgy.tangle.ExtraSyntax(), pidgy.testing.Definitions()])

Another feature of IPython is the ability to intercept [Abstract Syntax Tree]s and change their representation or capture metadata. After these transformations are applied, IPython compile the tree into a valid types.CodeType.

    @pidgy.specification
    def post_execute(self):
        ...


    @pidgy.specification
    def post_run_cell(self, result):

The weave step happens after execution, the tangle step happens before. Weaving only occurs if the input is computationally verified. It allows different representations of the input to be displayed. pidgy will implement templated Markdown displays of the input and formally test the contents of the input.

    def _post_run_cell(self, result):
        self.manager.hook.post_run_cell(result=result)

    def _post_exec(self):
        self.manager.hook.post_execute()


    enable_html_pager = traitlets.Bool(True)
    definitions = traitlets.List()
    manager = traitlets.Instance('pluggy.PluginManager', args=('pidgy',))
    loaders = traitlets.Dict()
    weave = traitlets.Any()

    @traitlets.default('weave')
    def _default_weave(self): return pidgy.weave.Weave(self)

pidgy mixes the standard IPython configuration system and its own pluggy specification and implementation.

Initializing the pidgy shell

    def init_pidgy(self):

Initialize pidgy specific behaviors.

        self.manager.add_hookspecs(pidgyShell)
        for object in (
            pidgy.tangle, self.weave, pidgy.testing
        ):

The tangle and weave implementations are discussed in other parts of this document. Here we register each of them as pluggy hook implementations.

            self.manager.register(object)
        self.events.register("post_run_cell", types.MethodType(pidgyShell._post_run_cell, self))
        self.events.register("post_execute", types.MethodType(pidgyShell._post_exec, self))


        if pidgy.pidgyLoader not in self.loaders:

pidgy enters a loader context allowing [Markdown] and notebook files to be used permissively as input.

            self.loaders[pidgy.pidgyLoader] = pidgy.pidgyLoader().__enter__()

It also adds a few extra features to the shell.

        self.user_ns["shell"] = self
        self.user_ns.update({k: v for k, v in vars(IPython.display).items()
            if pidgy.util.istype(v, IPython.core.display.DisplayObject)
        })

and allows json syntax as valid python input.

        pidgy.tangle.init_json()
        pidgy.magic.load_ipython_extension(self)

    def __init__(self, *args, **kwargs):

Override the initialization of the conventional IPython kernel to include the pidgy opinions.

        super().__init__(*args, **kwargs)
        self.init_pidgy()

pidgy extension.

    def load_ipython_extension(shell):

The pidgy kernel makes it easy to access the pidgy shell, but it can also be used an IPython extension.

        shell.add_traits(manager=pidgyShell.manager, loaders=pidgyShell.loaders, definitions=pidgyShell.definitions, weave=pidgyShell.weave)
        shell._post_run_cell = types.MethodType(pidgyShell._post_run_cell, shell)
        shell._post_exec = types.MethodType(pidgyShell._post_exec, shell)
        pidgyShell.init_pidgy(shell)
        shell.input_transformer_manager = pidgyShell.input_transformer_manager.default_value

    def unload_ipython_extension(self):
        self.events.unregister("post_run_cell", self._post_run_cell)
        self.events.unregister("post_run_cell", pidgy.weave.post_run_cell)
        loader = self.loaders.pop(pidgy.pidgyLoader)
        if loader is not None:
            loader.__exit__(None, None, None)


load_ipython_extension = pidgyShell.load_ipython_extension
unload_ipython_extension = pidgyShell.unload_ipython_extension