Initializing the ModuleManager

This page describes how a ModuleManager goes from empty to holding a wired call graph built from a set of plugins. Plugins themselves are described in Designing the Plugin Class. The ModuleManager is described in Designing the Module Manager.

What is ModuleManager Initialization?

Initialization is the process that:

  1. loads the modules from each plugin the user asks for, and

  2. sets default submodules between plugins, e.g., makes SCF’s driver use NUX’s Born-Oppenheimer Hamiltonian module when NUX is loaded.

The ModuleManager is then ready for the user to change it further and run it.

Previous Approaches to Initialization

Originally, the end user managed initialization by hand. There was a convention that plugins would declare/define load_modules and set_defaults functions. The end user would call them. This approach has several problems:

  • Wiring only works if every provider has been loaded before any set_defaults is called. Nothing enforces that order.

  • Every package that combines plugins has to rewrite this sequence.

  • PluginPlay cannot tell which plugin a module came from or which plugins are loaded.

To try to add some level of automation, we added a function ModuleManager::set_default<PT>. The idea was that a user could call it to set a default module for a property type. ModuleManagerPIMPL::at then quietly filled unready submodules of that type whenever a module is looked up. This feature had no users outside PluginPlay’s own tests, and was removed. The problem was that property types are not unique enough for a single default to make sense.

Initialization Considerations

  1. Loading order does not matter

    • Plugins can be loaded in any order.

  2. Wiring happens after loading

    • “Is my preferred provider loaded?” only has a reliable answer once all plugins are loaded. Wiring is therefore a separate step that runs afterwards.

  3. User choices win

    • Between loading and wiring, the user may change modules, e.g., pick submodules or add modules of their own. Wiring must not overwrite those choices.

  4. Unique names

    • Two plugins may use the same module key, e.g., "Overlap". Every module must still be addressable without ambiguity.

  5. Wiring can be run again

    • Running the wiring step again, e.g., after loading another plugin, only fills submodules that are still unset.

  6. No early errors

    • A submodule that is still unset after wiring is not an error. The user can set it at any time before running the call graph.

Initialization Strategy

Storage

The ModuleManager owns each loaded plugin object, and each plugin owns its ModuleMap (see Designing the Plugin Class). So the ModuleManager has one ModuleMap per loaded plugin, plus a ModuleMap of its own. The ModuleManager’s own map holds modules that do not come from a plugin, such as modules added directly by the user, lambda modules, and facade modules. Copies the user makes through the ModuleManager (copy_module and rename_module) also go in its own map, even when the original is one of a plugin’s modules, so a plugin’s map only holds the modules the plugin provides. The maps are not merged into one pool. A submodule may still be any module in any map, so links can cross maps.

The empty plugin name, "", is reserved for the ModuleManager’s own map.

Uniqueness and Lookup

  • ModuleMap enforces unique module keys within a plugin.

  • The ModuleManager enforces unique plugin names. Loading a second plugin with the same name throws. Like module keys, plugin names are compared case-insensitively, everywhere they are used: when loading, in lookups, and when matching preferred providers.

Together these guarantee that the pair (plugin name, module key) is unique. The ModuleManager therefore offers two ways to look up a module, and every function that takes a module key (at, change_input, change_submod, run_as, and so on) supports both:

  1. With a plugin name and a module key. This always works. The "" plugin name means the ModuleManager’s own map.

  2. With only a module key. This works as long as the key appears in exactly one map. If it appears in more than one map the call throws and the error lists the plugins that contain the key.

Both forms are spelled with a QualifiedKey, a small class holding an optional plugin name and a module key. A QualifiedKey is implicitly created from a string, which gives the key-only form, so the functions keep a single signature and existing calls keep working:

mm.at("SCF Driver");                  // key-only
mm.at({"scf", "SCF Driver"});         // plugin name and module key
mm.change_submod({"scf", "SCF Driver"}, "Hamiltonian",
                 {"nux", "Born-Oppenheimer approximation"});

In Python, a string or a (plugin name, module key) tuple can be passed wherever a QualifiedKey is expected.

Note that {"", key} is a qualified key, not a key-only one. The two forms look up different things: a key-only key searches every map, while {"", key} only looks in the ModuleManager’s own map. They find the same module only when no plugin uses key. The qualified form is the only way to reach a module in the ModuleManager’s own map when a plugin also uses its key:

mm.add_module<Rectangle>("Shared");   // own map
mm.load_plugin<ProviderA>();          // also has a "Shared" module

mm.at("Shared");                      // throws: ambiguous
mm.at({"", "Shared"});                // the own map's module
mm.at({"provider_a", "Shared"});      // ProviderA's module

The key-only form keeps current code working, since today’s module keys are effectively unique across the NWChemEx plugins.

The ModuleManager does not store which keys appear in more than one map. Instead, a key-only lookup searches every map. There are only a handful of plugins, so this is cheap, and a stored record could go out of date when the user copies or erases a module after loading.

count(key) returns the number of maps with the key, size() returns the total number of modules, and keys() returns the qualified key of every module. The ModuleManager no longer offers begin() and end(); code that needs the modules loops over plugin_names() and calls get_modules(name), which returns that map.

Caches

Memoized results are stored in module caches, which are looked up by name. Two plugins may use the same module key, so a plugin’s module uses the cache named "<plugin name>::<module key>". Modules in the ModuleManager’s own map use the module key, as before.

This naming is not collision-free: a module in the ModuleManager’s own map with the key "my_plugin::Area" shares a cache with the module "Area" of the plugin "my_plugin". Since plugin names must be identifiers, plugin names can not collide this way, only module keys containing "::" can. Rather than change the cache names of the ModuleManager’s own modules, the user documentation asks developers to avoid "::" in module keys.

A plugin may set a submodule and then erase that module’s key. The module is still used, so load_plugin also gives every module reachable through submodules the runtime and caches, naming its cache after the key it was last stored under.

Loading Plugins

pluginplay::ModuleManager mm;
mm.load_plugin<scf::SCF>();
mm.load_plugin<integrals::Integrals>();
mm.load_plugin<nux::NUX>();
mm.load_plugin<chemcache::ChemCache>();

load_plugin<T>():

  1. creates a T (T must derive from Plugin and be default constructible). T’s constructor sets the plugin’s name and preferred providers and builds its ModuleMap, including its internal wiring,

  2. throws if a plugin with the same name() is already loaded,

  3. gives each of the plugin’s modules the ModuleManager’s runtime and caches, and

  4. takes ownership of the plugin object. The plugin’s ModuleMap is now the ModuleManager’s map for that plugin; nothing is copied.

Python has an equivalent, mm.load_plugin(scf.SCF()), which takes an instance because Python has no template parameters. Because the plugin holds live modules, the same instance must not be loaded into two ModuleManager objects.

Wiring Defaults

After loading, the user may make changes, and then calls:

mm.wire_defaults();

It iterates over the loaded plugins in the order they were loaded, and for each plugin P:

  1. For each name N in P.preferred_providers(), in order:

    • If no plugin named N is loaded, skip it.

    • Otherwise call P.default_submods_from(N).

    • The result is a list of defaults, each of which is a triple (module_key, callback_key, provider_key)

  2. For each default triple:

    • If the user erased P’s module module_key, move to the next triple.

    • If P’s module module_key has a submodule with key callback_key that is already set, move to the next triple.

    • Else, if the provider does not have a module with key provider_key, log a warning and move to the next triple.

    • Otherwise set callback_key to the provider’s module provider_key.

These changes are done to P’s live map, so it includes any changes the user made and any wiring done by earlier calls in the loop. Because wiring only sets unset submodules:

  • choices the user made before wire_defaults are kept,

  • the order of preferred_providers sets priority, since the first provider to set a submodule wins, and

  • calling wire_defaults again does not change submodules that are already set.

The order in which plugins are loaded does not change the result. Each plugin only changes its own map, so the result for P depends only on its own provider list.