Nornir Proxy module

Nornir Proxy Module is a core component of Nornir Proxy Minion. However, users rarely have to interact with it directly unless they writing their own Execution or Runner or State or whatever modules for SaltStack.

What is important to understand are configuration parameters you can use with Nornir Proxy Minion, as they can help to alter default behavior or control various aspects of Nornir Proxy Minion life cycle.

Introduction

Single Nornir proxy-minion can work with hundreds of devices as opposed to conventional proxy-minion that normally dedicated to managing one device/system only.

As a result, Nornir proxy-minion requires less resources to run tasks against same number of devices. During idle state only one proxy minion process is active, that significantly reduces amount of memory required to manage device endpoints.

Proxy-module required way of operating is multiprocessing set to True - default value, that way each task executed in dedicated process.

Dependencies

Nornir 3.x uses modular approach for plugins. As a result required plugins need to be installed separately from Nornir Core library. Main collection of plugins to install is nornir-salt. Nornir Salt repository contains many function used by Salt Nornir Proxy Minion module and is mandatory to have on the system where proxy minion process runs.

Nornir Proxy Configuration Parameters

Below parameters can be specified in Proxy Minion Pillar.

proxytype - string of value nornir
multiprocessing - boolean, True by default, multiprocessing is a recommended way to run this proxy, threading mode also works, but might be prone to memory consumption issues
process_count_max - int, default is -1 no limit, maximum number of processes to use to limit a number of tasks waiting to execute
nornir_filter_required - boolean, default is False, to indicate if Nornir filter is mandatory for tasks executed by this proxy-minion. Nornir has access to multiple devices, by default, if no filter provided, task will run for all devices, nornir_filter_required allows to change behavior to opposite, if no filter provided, task will not run at all. It is a safety measure against running task for all devices accidentally, instead, filter FB="*" can be used to run task for all devices.
runner - dictionary, Nornir Runner plugin parameters, default is RetryRunner
inventory - dictionary, Nornir Inventory plugin parameters, default is DictInventory populated with data from proxy-minion pillar, pillar data ignored by any other inventory plugins
child_process_max_age - int, default is 660s, seconds to wait before forcefully kill child process
watchdog_interval - int, default is 30s, interval in seconds between watchdog runs
proxy_always_alive - boolean, default is True, keep connections with devices alive or tear them down immediately after each job
connections_idle_timeout - int, seconds, default is 1 equivalent to proxy_always_alive set to True, if value equals 0 renders same behavior as if proxy_always_alive is False, if value above 1 - all host’s device connections torn down after it was not in use for longer then idle timeout value even if proxy_always_alive set to True
job_wait_timeout - int, default is 600s, seconds to wait for job return until give up
memory_threshold_mbyte - int, default is 300, value in MBytes above each to trigger memory_threshold_action
memory_threshold_action - str, default is log, action to implement if memory_threshold_mbyte exceeded, possible actions: log - send syslog message, restart - shuts down proxy minion process.
nornir_workers - number of Nornir instances to create, each instance has worker thread associated with it allowing to run multiple tasks against hosts, as each worker dequeue tasks from jobs queue, default is 3
files_base_path - str, default is /var/salt-nornir/{proxy_id}/files/, OS path to folder where to save files on a per-host basis using ToFileProcessor <https://nornir-salt.readthedocs.io/en/latest/Processors/ToFileProcessor.html>_,
files_max_count - int, default is 5, maximum number of file version for tf argument used by ToFileProcessor
nr_cli - dictionary of default arguments to use with nr.cli execution module function, default is none
nr_cfg - dictionary of default arguments to use with nr.cfg execution module function, default is none
nr_nc - dictionary of default arguments to use with nr.nc execution module function, default is none
event_progress_all - boolean, default is False, if True emits progress events for all tasks using SaltEventProcessor <https://nornir-salt.readthedocs.io/en/latest/Processors/SaltEventProcessor.html>_, per-task event_progress argument overrides event_progress_all parameter.

Nornir uses inventory to store information about devices to interact with. Inventory can contain information about hosts, groups and defaults. Nornir inventory defined in proxy-minion pillar.

Nornir proxy-minion pillar example:

proxy:
  proxytype: nornir
  process_count_max: 3
  multiprocessing: True
  nornir_filter_required: True
  proxy_always_alive: True
  connections_idle_timeout: 1
  watchdog_interval: 30
  child_process_max_age: 660
  job_wait_timeout: 600
  memory_threshold_mbyte: 300
  memory_threshold_action: log
  files_base_path: "/var/salt-nornir/{proxy_id}/files/"
  files_max_count: 5
  event_progress_all: True
  nr_cli: {}
  nr_cfg: {}
  nr_nc: {}
  runner:
     plugin: RetryRunner
     options:
        num_workers: 100
        num_connectors: 10
        connect_retry: 3
        connect_backoff: 1000
        connect_splay: 100
        task_retry: 3
        task_backoff: 1000
        task_splay: 100
        reconnect_on_fail: True
        task_timeout: 600
        creds_retry: ["local_creds"]
  inventory:
     plugin: DictInventory

hosts:
  IOL1:
    hostname: 192.168.217.10
    platform: ios
    location: B1
    groups: [lab]
  IOL2:
    hostname: 192.168.217.7
    platform: ios
    location: B2
    groups: [lab]

groups:
  lab:
    username: nornir
    password: nornir
    connection_options:
      napalm:
        optional_args: {dest_file_system: "system:"}

defaults:
  data:
    credentials:
      local_creds:
        username: admin
        password: admin

# user_defined Nornir configuration
configuration:
  any_key: any_value

To use other type of inventory or runner plugins define their settings in pillar configuration:

proxy:
  runner:
     plugin: threaded
     options:
         num_workers: 100
  inventory:
     plugin: SimpleInventory
     options:
       host_file: "/var/salt-nonir/proxy-id-1/hosts.yaml"
       group_file: "/var/salt-nonir/proxy-id-1/groups.yaml"
       defaults_file: "/var/salt-nonir/proxy-id-1/defaults.yaml"

Starting with version Salt-Nornir 0.15.0 Nornir user_defined parameters can be specified under pillar configuration key:

configuration:
  params:
    foo: 123

Nornir Task plugins can access user defined configuration under task.nr.config.user_defined task attribute.

Starting with version Salt-Nornir 0.15.0 jinja_env configuration honored and sourced using config.get execution module function, as a result jinja_env configuration can be specified in Master configuration or proxy pillar. However, to be sourced from master, pillar_opts should be set to True in master configuration.

Sample Master jinja_env configuration:

pillar_opts: true
jinja_env:
  lstrip_blocks: true
  trim_blocks: true

Sample proxy minion pillar jinja_env configuration:

proxy:
  proxytype: nornir

jinja_env:
  lstrip_blocks: false
  trim_blocks: false

Nornir runners

Nornir Runner plugins define how to run tasks against hosts. If no runner dictionary provided in proxy-minion pillar, Nornir initialized using Nornir Salt RetryRunner plugin with these default settings:

runner = {
    "plugin": "RetryRunner",
    "options": {
        "num_workers": 100,
        "num_connectors": 10,
        "connect_retry": 3,
        "connect_backoff": 1000,
        "connect_splay": 100,
        "task_retry": 3,
        "task_backoff": 1000,
        "task_splay": 100,
        "reconnect_on_fail": True,
        "task_timeout": 600,
        "creds_retry": [{"username": "admin", "password": "admin", "port": 2022}]
    },
}

Nornir Proxy Module Functions

Common CLI Arguments Summary
Name	Description
refresh_nornir	Function to re-instantiate Nornir object instance refreshing pillar
execute_job	Function to place job in worker thread jobs queue
grains	Retrieve Nornir Proxy Minion grains
grains_refresh	Refresh Nornir Proxy Minion grains
init	Initiate Nornir Proxy-module
kill_nornir	Un-gracefully shutdown Nornir Proxy Minion process
list_hosts	Produces a list of hosts’ names managed by this Proxy
nr_data	To retrieve values from `nornir_data` Nornir Proxy Minion dictionary
ping	To test Nornir Proxy Minion process
queues_utils	Utility function to manage Proxy Minion queues
run	Used to run Nornir Task
shutdown	Gracefully shutdown Nornir Instance
stats	Produces a dictionary of Nornir Proxy Minion statistics
workers_utils	Utility function to manage Proxy Minion workers

refresh_nornir

salt_nornir.proxy.nornir_proxy_module._refresh_nornir(*args, **kwargs)

execute_job

salt_nornir.proxy.nornir_proxy_module.execute_job(task_fun, kwargs, identity)

Function to submit job request to Nornir Proxy minion jobs queue, wait for job to be completed and return results.

Parameters

task_fun – (str) name of nornir task function/plugin to import and run
kwargs – (dict) any arguments to submit to Nornir task **kwargs
identity – (dict) dictionary of uuid4, jid, function_name keys

identity parameter used to identify job results in results queue and must be unique for each submitted job.

grains

salt_nornir.proxy.nornir_proxy_module.grains(): Populate grains

grains_refresh

salt_nornir.proxy.nornir_proxy_module.grains_refresh(): Does nothing, returns empty dictionary

init

salt_nornir.proxy.nornir_proxy_module.init(opts, loader=None, init_queues=True, wkr_stats=None)

Initiate Nornir by calling InitNornir()

Parameters

opts – (dict) proxy minion options
loader – (obj) SaltStack loader context object
init_queues – (bool) if True, initializes multiprocessing queues, set to False if “nr.nornir refresh workers_only=True” called
wkr_stats – list of worker stats dictionaries to preserve them across Nornir refresh

kill_nornir

salt_nornir.proxy.nornir_proxy_module.kill_nornir(*args, **kwargs): This function kills Nornir process and its child process as fast as possible.

Warning

this function kills main Nornir process and does not recover it

list_hosts

salt_nornir.proxy.nornir_proxy_module.list_hosts(**kwargs)

Return a list of hosts managed by this Nornir instance

Parameters: Fx – filters to filter hosts

nr_data

salt_nornir.proxy.nornir_proxy_module.nr_data(key)

Helper function to return values from nornir_data dictionary, used by nr.cli, nr.cfg and nr.nc execution module functions to retrieve default kwargs values from respective proxy settings’ attributes.

Parameters: key – (str or list) if string return value for single key, if list return a dictionary keyed by items in given key list.

nr_version

salt_nornir.proxy.nornir_proxy_module.nr_version(): Function to return a report of installed packages and their versions, useful for troubleshooting dependencies.

ping

salt_nornir.proxy.nornir_proxy_module.ping(): Return Nornir proxy status

queues_utils

salt_nornir.proxy.nornir_proxy_module.queues_utils(call)

Function to retrieve operational data for job queues

Parameters: call – (str) utility to invoke - results_queue_dump

Supported calls:

results_queue_dump - drain items from result queue and return their content,
put items copies back into the queue afterwards

run

salt_nornir.proxy.nornir_proxy_module.run(task, loader, identity, name, nr, wkr_data, **kwargs)

Function for worker Thread to run Nornir tasks.

Parameters

task – (obj) callable task function
loader – (obj) __salt__.loader object instance
identity – (dict) Task results queue identity for SaltEventProcessor
kwargs – (dict) passed to task.run after extracting CLI arguments
name – (str) Nornir task name to run
nr – (obj) Worker instance Nornir object

shutdown

salt_nornir.proxy.nornir_proxy_module.shutdown()

This function implements this protocol to perform Nornir graceful shutdown:

Signal worker and watchdog threads to stop
Close all connections to devices
Close jobs and results queues
Kill all child processes
Delete Nornir object

Proxy Minion process keeps running afterwards, but cannot do anything.

stats

salt_nornir.proxy.nornir_proxy_module.stats(*args, **kwargs)

Function to gather and return stats about Nornir proxy process.

Parameters: stat – name of stat to return, returns all by default

Returns dictionary with these parameters:

proxy_minion_id - if of this proxy minion
main_process_is_running - set to 0 if not running and to 1 otherwise
main_process_start_time - time.time() function to indicate process start time in epoch
main_process_start_date - time.ctime() function date to indicate process start time
main_process_uptime_seconds - int, main proxy minion process uptime
main_process_ram_usage_mbyte - int, RAM usage
main_process_pid - main process ID i.e. PID
main_process_host - hostname of machine where proxy minion process is running
jobs_started - int, overall number of jobs started
jobs_completed - int, overall number of jobs completed
jobs_failed - int, overall number of jobs failed
jobs_job_queue_size - int, size of jobs queue, indicating number of jobs waiting to start
jobs_res_queue_size - int, size of results queue, indicating number of results waiting to be collected by child process
tasks_completed - overall number of completed Nornir tasks (including subtasks)
tasks_failed - overall number of failed Nornir tasks (including subtasks)
hosts_count - int, number of hosts/devices managed by this proxy minion
hosts_connections_active - int, overall number of connection active to devices
hosts_tasks_failed - overall number of hosts that failed all tasks within single job
timestamp - time.ctime() timestamp of stats function run
watchdog_runs - int, overall number of watchdog thread runs
watchdog_child_processes_killed - int, number of stale child processes killed by watchdog
watchdog_dead_connections_cleaned - int, number of stale hosts’ connections cleaned by watchdog
child_processes_count - int, number of child processes currently running
main_process_fd_count - int, number of file descriptors in use by main proxy minion process
main_process_fd_limit - int, fd count limit imposed by Operating System for minion process

workers_utils

salt_nornir.proxy.nornir_proxy_module.workers_utils(call)

Function to retrieve operational data for Nornir Worker instances

Parameters: call – (str) utility to invoke

Supported calls:

stats - return worker statistics keyed by worker id with these parameters:
- is_busy - boolean, indicates if worker doing the work
- worker_jobs_completed - counter of completed jobs
- worker_jobs_failed - counter of completely failed jobs
- worker_connections - hosts’ connections info
- worker_jobs_queue - size of the worker specific jobs queue
- worker_hosts_tasks_failed - counter of overall host failed tasks
- worker_jobs_started - counter of started jobs