Nornir Proxy module

Nornir Proxy Module is a core component of Nornir Proxy Minion. However, users rarely have to interact with it directly unless they writing their own Execution or Runner or State or whatever modules for SaltStack.

What is important to understand are configuration parameters you can use with Nornir Proxy Minion, as they can help to alter default behavior or control various aspects of Nornir Proxy Minion life cycle.


Single Nornir proxy-minion can work with hundreds of devices as opposed to conventional proxy-minion that normally dedicated to managing one device/system only.

As a result, Nornir proxy-minion requires less resources to run tasks against same number of devices. During idle state only one proxy minion process is active, that significantly reduces amount of memory required to manage device endpoints.

Proxy-module required way of operating is multiprocessing set to True - default value, that way each task executed in dedicated process.


Nornir 3.x uses modular approach for plugins. As a result required plugins need to be installed separately from Nornir Core library. Main collection of plugins to install is nornir-salt. Nornir Salt repository contains many function used by Salt Nornir Proxy Minion module and is mandatory to have on the system where proxy minion process runs.

Nornir Proxy Configuration Parameters

Below parameters can be specified in Proxy Minion Pillar.

  • proxytype - string of value nornir

  • multiprocessing - boolean, True by default, multiprocessing is a recommended way to run this proxy, threading mode also works, but might be prone to memory consumption issues

  • process_count_max - int, default is -1 no limit, maximum number of processes to use to limit a number of tasks waiting to execute

  • nornir_filter_required - boolean, default is False, to indicate if Nornir filter is mandatory for tasks executed by this proxy-minion. Nornir has access to multiple devices, by default, if no filter provided, task will run for all devices, nornir_filter_required allows to change behavior to opposite, if no filter provided, task will not run at all. It is a safety measure against running task for all devices accidentally, instead, filter FB="*" can be used to run task for all devices.

  • runner - dictionary, Nornir Runner plugin parameters, default is RetryRunner

  • inventory - dictionary, Nornir Inventory plugin parameters, default is DictInventory populated with data from proxy-minion pillar, pillar data ignored by any other inventory plugins

  • child_process_max_age - int, default is 660s, seconds to wait before forcefully kill child process

  • watchdog_interval - int, default is 30s, interval in seconds between watchdog runs

  • proxy_always_alive - boolean, default is True, keep connections with devices alive or tear them down immediately after each job

  • connections_idle_timeout - int, seconds, default is 1 equivalent to proxy_always_alive set to True, if value equals 0 renders same behavior as if proxy_always_alive is False, if value above 1 - all host’s device connections torn down after it was not in use for longer then idle timeout value even if proxy_always_alive set to True

  • job_wait_timeout - int, default is 600s, seconds to wait for job return until give up

  • memory_threshold_mbyte - int, default is 300, value in MBytes above each to trigger memory_threshold_action

  • memory_threshold_action - str, default is log, action to implement if memory_threshold_mbyte exceeded, possible actions: log - send syslog message, restart - shuts down proxy minion process.

  • nornir_workers - number of Nornir instances to create, each instance has worker thread associated with it allowing to run multiple tasks against hosts, as each worker dequeue tasks from jobs queue, default is 3

  • files_base_path - str, default is /var/salt-nornir/{proxy_id}/files/, OS path to folder where to save files on a per-host basis using ToFileProcessor <>_,

  • files_max_count - int, default is 5, maximum number of file version for tf argument used by ToFileProcessor

  • nr_cli - dictionary of default arguments to use with nr.cli execution module function, default is none

  • nr_cfg - dictionary of default arguments to use with nr.cfg execution module function, default is none

  • nr_nc - dictionary of default arguments to use with execution module function, default is none

  • event_progress_all - boolean, default is False, if True emits progress events for all tasks using SaltEventProcessor <>_, per-task event_progress argument overrides event_progress_all parameter.

Nornir uses inventory to store information about devices to interact with. Inventory can contain information about hosts, groups and defaults. Nornir inventory defined in proxy-minion pillar.

Nornir proxy-minion pillar example:

  proxytype: nornir
  process_count_max: 3
  multiprocessing: True
  nornir_filter_required: True
  proxy_always_alive: True
  connections_idle_timeout: 1
  watchdog_interval: 30
  child_process_max_age: 660
  job_wait_timeout: 600
  memory_threshold_mbyte: 300
  memory_threshold_action: log
  files_base_path: "/var/salt-nornir/{proxy_id}/files/"
  files_max_count: 5
  event_progress_all: True
  nr_cli: {}
  nr_cfg: {}
  nr_nc: {}
     plugin: RetryRunner
        num_workers: 100
        num_connectors: 10
        connect_retry: 3
        connect_backoff: 1000
        connect_splay: 100
        task_retry: 3
        task_backoff: 1000
        task_splay: 100
        reconnect_on_fail: True
        task_timeout: 600
        creds_retry: ["local_creds"]
     plugin: DictInventory

    platform: ios
    location: B1
    groups: [lab]
    platform: ios
    location: B2
    groups: [lab]

    username: nornir
    password: nornir
        optional_args: {dest_file_system: "system:"}

        username: admin
        password: admin

# user_defined Nornir configuration
  any_key: any_value

To use other type of inventory or runner plugins define their settings in pillar configuration:

     plugin: threaded
         num_workers: 100
     plugin: SimpleInventory
       host_file: "/var/salt-nonir/proxy-id-1/hosts.yaml"
       group_file: "/var/salt-nonir/proxy-id-1/groups.yaml"
       defaults_file: "/var/salt-nonir/proxy-id-1/defaults.yaml"

Starting with version Salt-Nornir 0.15.0 Nornir user_defined parameters can be specified under pillar configuration key:

    foo: 123

Nornir Task plugins can access user defined configuration under task attribute.

Starting with version Salt-Nornir 0.15.0 jinja_env configuration honored and sourced using config.get execution module function, as a result jinja_env configuration can be specified in Master configuration or proxy pillar. However, to be sourced from master, pillar_opts should be set to True in master configuration.

Sample Master jinja_env configuration:

pillar_opts: true
  lstrip_blocks: true
  trim_blocks: true

Sample proxy minion pillar jinja_env configuration:

  proxytype: nornir

  lstrip_blocks: false
  trim_blocks: false

Nornir runners

Nornir Runner plugins define how to run tasks against hosts. If no runner dictionary provided in proxy-minion pillar, Nornir initialized using Nornir Salt RetryRunner plugin with these default settings:

runner = {
    "plugin": "RetryRunner",
    "options": {
        "num_workers": 100,
        "num_connectors": 10,
        "connect_retry": 3,
        "connect_backoff": 1000,
        "connect_splay": 100,
        "task_retry": 3,
        "task_backoff": 1000,
        "task_splay": 100,
        "reconnect_on_fail": True,
        "task_timeout": 600,
        "creds_retry": [{"username": "admin", "password": "admin", "port": 2022}]

Nornir Proxy Module Functions

Common CLI Arguments Summary




Function to re-instantiate Nornir object instance refreshing pillar


Function to place job in worker thread jobs queue


Retrieve Nornir Proxy Minion grains


Refresh Nornir Proxy Minion grains


Initiate Nornir Proxy-module


Un-gracefully shutdown Nornir Proxy Minion process


Produces a list of hosts’ names managed by this Proxy


To retrieve values from nornir_data Nornir Proxy Minion dictionary


To test Nornir Proxy Minion process


Utility function to manage Proxy Minion queues


Used to run Nornir Task


Gracefully shutdown Nornir Instance


Produces a dictionary of Nornir Proxy Minion statistics


Utility function to manage Proxy Minion workers


salt_nornir.proxy.nornir_proxy_module._refresh_nornir(*args, **kwargs)


salt_nornir.proxy.nornir_proxy_module.execute_job(task_fun, kwargs, identity)

Function to submit job request to Nornir Proxy minion jobs queue, wait for job to be completed and return results.

  • task_fun – (str) name of nornir task function/plugin to import and run

  • kwargs – (dict) any arguments to submit to Nornir task **kwargs

  • identity – (dict) dictionary of uuid4, jid, function_name keys

identity parameter used to identify job results in results queue and must be unique for each submitted job.



Populate grains



Does nothing, returns empty dictionary


salt_nornir.proxy.nornir_proxy_module.init(opts, loader=None, init_queues=True, wkr_stats=None)

Initiate Nornir by calling InitNornir()

  • opts – (dict) proxy minion options

  • loader – (obj) SaltStack loader context object

  • init_queues – (bool) if True, initializes multiprocessing queues, set to False if “nr.nornir refresh workers_only=True” called

  • wkr_stats – list of worker stats dictionaries to preserve them across Nornir refresh


salt_nornir.proxy.nornir_proxy_module.kill_nornir(*args, **kwargs)

This function kills Nornir process and its child process as fast as possible.


this function kills main Nornir process and does not recover it



Return a list of hosts managed by this Nornir instance


Fx – filters to filter hosts



Helper function to return values from nornir_data dictionary, used by nr.cli, nr.cfg and execution module functions to retrieve default kwargs values from respective proxy settings’ attributes.


key – (str or list) if string return value for single key, if list return a dictionary keyed by items in given key list.



Function to return a report of installed packages and their versions, useful for troubleshooting dependencies.


Return Nornir proxy status



Function to retrieve operational data for job queues


call – (str) utility to invoke - results_queue_dump

Supported calls:

  • results_queue_dump - drain items from result queue and return their content,

    put items copies back into the queue afterwards

run, loader, identity, name, nr, wkr_data, **kwargs)

Function for worker Thread to run Nornir tasks.

  • task – (obj) callable task function

  • loader – (obj) __salt__.loader object instance

  • identity – (dict) Task results queue identity for SaltEventProcessor

  • kwargs – (dict) passed to after extracting CLI arguments

  • name – (str) Nornir task name to run

  • nr – (obj) Worker instance Nornir object



This function implements this protocol to perform Nornir graceful shutdown:

  1. Signal worker and watchdog threads to stop

  2. Close all connections to devices

  3. Close jobs and results queues

  4. Kill all child processes

  5. Delete Nornir object

Proxy Minion process keeps running afterwards, but cannot do anything.


salt_nornir.proxy.nornir_proxy_module.stats(*args, **kwargs)

Function to gather and return stats about Nornir proxy process.


stat – name of stat to return, returns all by default

Returns dictionary with these parameters:

  • proxy_minion_id - if of this proxy minion

  • main_process_is_running - set to 0 if not running and to 1 otherwise

  • main_process_start_time - time.time() function to indicate process start time in epoch

  • main_process_start_date - time.ctime() function date to indicate process start time

  • main_process_uptime_seconds - int, main proxy minion process uptime

  • main_process_ram_usage_mbyte - int, RAM usage

  • main_process_pid - main process ID i.e. PID

  • main_process_host - hostname of machine where proxy minion process is running

  • jobs_started - int, overall number of jobs started

  • jobs_completed - int, overall number of jobs completed

  • jobs_failed - int, overall number of jobs failed

  • jobs_job_queue_size - int, size of jobs queue, indicating number of jobs waiting to start

  • jobs_res_queue_size - int, size of results queue, indicating number of results waiting to be collected by child process

  • tasks_completed - overall number of completed Nornir tasks (including subtasks)

  • tasks_failed - overall number of failed Nornir tasks (including subtasks)

  • hosts_count - int, number of hosts/devices managed by this proxy minion

  • hosts_connections_active - int, overall number of connection active to devices

  • hosts_tasks_failed - overall number of hosts that failed all tasks within single job

  • timestamp - time.ctime() timestamp of stats function run

  • watchdog_runs - int, overall number of watchdog thread runs

  • watchdog_child_processes_killed - int, number of stale child processes killed by watchdog

  • watchdog_dead_connections_cleaned - int, number of stale hosts’ connections cleaned by watchdog

  • child_processes_count - int, number of child processes currently running

  • main_process_fd_count - int, number of file descriptors in use by main proxy minion process

  • main_process_fd_limit - int, fd count limit imposed by Operating System for minion process



Function to retrieve operational data for Nornir Worker instances


call – (str) utility to invoke

Supported calls:

  • stats - return worker statistics keyed by worker id with these parameters:

    • is_busy - boolean, indicates if worker doing the work

    • worker_jobs_completed - counter of completed jobs

    • worker_jobs_failed - counter of completely failed jobs

    • worker_connections - hosts’ connections info

    • worker_jobs_queue - size of the worker specific jobs queue

    • worker_hosts_tasks_failed - counter of overall host failed tasks

    • worker_jobs_started - counter of started jobs