Nornir Proxy module
Nornir Proxy Module is a core component of Nornir Proxy Minion. However, users rarely have to interact with it directly unless they writing their own Execution or Runner or State or whatever modules for SaltStack.
What is important to understand are configuration parameters you can use with Nornir Proxy Minion, as they can help to alter default behavior or control various aspects of Nornir Proxy Minion life cycle.
Introduction
Single Nornir proxy-minion can work with hundreds of devices as opposed to conventional proxy-minion that normally dedicated to managing one device/system only.
As a result, Nornir proxy-minion requires less resources to run tasks against same number of devices. During idle state only one proxy minion process is active, that significantly reduces amount of memory required to manage device endpoints.
Proxy-module required way of operating is multiprocessing
set to True
-
default value, that way each task executed in dedicated process.
Dependencies
Nornir 3.x uses modular approach for plugins. As a result required plugins need to be installed separately from Nornir Core library. Main collection of plugins to install is nornir-salt. Nornir Salt repository contains many function used by Salt Nornir Proxy Minion module and is mandatory to have on the system where proxy minion process runs.
Nornir Proxy Configuration Parameters
Below parameters can be specified in Proxy Minion Pillar.
proxytype
- string of valuenornir
multiprocessing
- boolean,True
by default, multiprocessing is a recommended way to run this proxy, threading mode also works, but might be prone to memory consumption issuesprocess_count_max
- int, default is-1
no limit, maximum number of processes to use to limit a number of tasks waiting to executenornir_filter_required
- boolean, default isFalse
, to indicate if Nornir filter is mandatory for tasks executed by this proxy-minion. Nornir has access to multiple devices, by default, if no filter provided, task will run for all devices,nornir_filter_required
allows to change behavior to opposite, if no filter provided, task will not run at all. It is a safety measure against running task for all devices accidentally, instead, filterFB="*"
can be used to run task for all devices.runner
- dictionary, Nornir Runner plugin parameters, default is RetryRunnerinventory
- dictionary, Nornir Inventory plugin parameters, default is DictInventory populated with data from proxy-minion pillar, pillar data ignored by any other inventory pluginschild_process_max_age
- int, default is 660s, seconds to wait before forcefully kill child processwatchdog_interval
- int, default is 30s, interval in seconds between watchdog runsproxy_always_alive
- boolean, default is True, keep connections with devices alive or tear them down immediately after each jobconnections_idle_timeout
- int, seconds, default is 1 equivalent toproxy_always_alive
set to True, if value equals 0 renders same behavior as ifproxy_always_alive
is False, if value above 1 - all host’s device connections torn down after it was not in use for longer then idle timeout value even ifproxy_always_alive
set to Truejob_wait_timeout
- int, default is 600s, seconds to wait for job return until give upmemory_threshold_mbyte
- int, default is 300, value in MBytes above each to triggermemory_threshold_action
memory_threshold_action
- str, default islog
, action to implement ifmemory_threshold_mbyte
exceeded, possible actions:log
- send syslog message,restart
- shuts down proxy minion process.nornir_workers
- number of Nornir instances to create, each instance has worker thread associated with it allowing to run multiple tasks against hosts, as each worker dequeue tasks from jobs queue, default is 3files_base_path
- str, default is/var/salt-nornir/{proxy_id}/files/
, OS path to folder where to save files on a per-host basis using ToFileProcessor <https://nornir-salt.readthedocs.io/en/latest/Processors/ToFileProcessor.html>_,files_max_count
- int, default is 5, maximum number of file version fortf
argument used by ToFileProcessornr_cli
- dictionary of default arguments to use withnr.cli
execution module function, default is nonenr_cfg
- dictionary of default arguments to use withnr.cfg
execution module function, default is nonenr_nc
- dictionary of default arguments to use withnr.nc
execution module function, default is noneevent_progress_all
- boolean, default is False, if True emits progress events for all tasks using SaltEventProcessor <https://nornir-salt.readthedocs.io/en/latest/Processors/SaltEventProcessor.html>_, per-taskevent_progress
argument overridesevent_progress_all
parameter.
Nornir uses inventory
to store information about devices to interact with. Inventory can contain
information about hosts
, groups
and defaults
. Nornir inventory
defined in proxy-minion pillar.
Nornir proxy-minion pillar example:
proxy:
proxytype: nornir
process_count_max: 3
multiprocessing: True
nornir_filter_required: True
proxy_always_alive: True
connections_idle_timeout: 1
watchdog_interval: 30
child_process_max_age: 660
job_wait_timeout: 600
memory_threshold_mbyte: 300
memory_threshold_action: log
files_base_path: "/var/salt-nornir/{proxy_id}/files/"
files_max_count: 5
event_progress_all: True
nr_cli: {}
nr_cfg: {}
nr_nc: {}
runner:
plugin: RetryRunner
options:
num_workers: 100
num_connectors: 10
connect_retry: 3
connect_backoff: 1000
connect_splay: 100
task_retry: 3
task_backoff: 1000
task_splay: 100
reconnect_on_fail: True
task_timeout: 600
creds_retry: ["local_creds"]
inventory:
plugin: DictInventory
hosts:
IOL1:
hostname: 192.168.217.10
platform: ios
location: B1
groups: [lab]
IOL2:
hostname: 192.168.217.7
platform: ios
location: B2
groups: [lab]
groups:
lab:
username: nornir
password: nornir
connection_options:
napalm:
optional_args: {dest_file_system: "system:"}
defaults:
data:
credentials:
local_creds:
username: admin
password: admin
# user_defined Nornir configuration
configuration:
any_key: any_value
To use other type of inventory or runner plugins define their settings in pillar configuration:
proxy:
runner:
plugin: threaded
options:
num_workers: 100
inventory:
plugin: SimpleInventory
options:
host_file: "/var/salt-nonir/proxy-id-1/hosts.yaml"
group_file: "/var/salt-nonir/proxy-id-1/groups.yaml"
defaults_file: "/var/salt-nonir/proxy-id-1/defaults.yaml"
Starting with version Salt-Nornir 0.15.0 Nornir user_defined
parameters
can be specified under pillar configuration
key:
configuration:
params:
foo: 123
Nornir Task plugins can access user defined configuration under
task.nr.config.user_defined
task attribute.
Starting with version Salt-Nornir 0.15.0 jinja_env
configuration honored
and sourced using config.get
execution module function, as a result
jinja_env
configuration can be specified in Master configuration or proxy
pillar. However, to be sourced from master, pillar_opts
should be set to True
in master configuration.
Sample Master jinja_env
configuration:
pillar_opts: true
jinja_env:
lstrip_blocks: true
trim_blocks: true
Sample proxy minion pillar jinja_env
configuration:
proxy:
proxytype: nornir
jinja_env:
lstrip_blocks: false
trim_blocks: false
Nornir runners
Nornir Runner plugins define how to run tasks against hosts. If no runner
dictionary provided in proxy-minion pillar, Nornir initialized using Nornir Salt
RetryRunner plugin
with these default settings:
runner = {
"plugin": "RetryRunner",
"options": {
"num_workers": 100,
"num_connectors": 10,
"connect_retry": 3,
"connect_backoff": 1000,
"connect_splay": 100,
"task_retry": 3,
"task_backoff": 1000,
"task_splay": 100,
"reconnect_on_fail": True,
"task_timeout": 600,
"creds_retry": [{"username": "admin", "password": "admin", "port": 2022}]
},
}
Nornir Proxy Module Functions
Name |
Description |
---|---|
Function to re-instantiate Nornir object instance refreshing pillar |
|
Function to place job in worker thread jobs queue |
|
Retrieve Nornir Proxy Minion grains |
|
Refresh Nornir Proxy Minion grains |
|
Initiate Nornir Proxy-module |
|
Un-gracefully shutdown Nornir Proxy Minion process |
|
Produces a list of hosts’ names managed by this Proxy |
|
To retrieve values from |
|
To test Nornir Proxy Minion process |
|
Utility function to manage Proxy Minion queues |
|
Used to run Nornir Task |
|
Gracefully shutdown Nornir Instance |
|
Produces a dictionary of Nornir Proxy Minion statistics |
|
Utility function to manage Proxy Minion workers |
refresh_nornir
- salt_nornir.proxy.nornir_proxy_module._refresh_nornir(*args, **kwargs)
execute_job
- salt_nornir.proxy.nornir_proxy_module.execute_job(task_fun, kwargs, identity)
Function to submit job request to Nornir Proxy minion jobs queue, wait for job to be completed and return results.
- Parameters
task_fun – (str) name of nornir task function/plugin to import and run
kwargs – (dict) any arguments to submit to Nornir task
**kwargs
identity – (dict) dictionary of uuid4, jid, function_name keys
identity
parameter used to identify job results in results queue and must be unique for each submitted job.
grains
- salt_nornir.proxy.nornir_proxy_module.grains()
Populate grains
grains_refresh
- salt_nornir.proxy.nornir_proxy_module.grains_refresh()
Does nothing, returns empty dictionary
init
- salt_nornir.proxy.nornir_proxy_module.init(opts, loader=None, init_queues=True)
Initiate Nornir by calling InitNornir()
- Parameters
opts – (dict) proxy minion options
loader – (obj) SaltStack loader context object
init_queues – (bool) if True, initializes multiprocessing queues, set to False if “nr.nornir refresh workers_only=True” called
kill_nornir
- salt_nornir.proxy.nornir_proxy_module.kill_nornir(*args, **kwargs)
This function kills Nornir process and its child process as fast as possible.
Warning
this function kills main Nornir process and does not recover it
list_hosts
- salt_nornir.proxy.nornir_proxy_module.list_hosts(**kwargs)
Return a list of hosts managed by this Nornir instance
- Parameters
Fx – filters to filter hosts
nr_data
- salt_nornir.proxy.nornir_proxy_module.nr_data(key)
Helper function to return values from nornir_data dictionary, used by
nr.cli
,nr.cfg
andnr.nc
execution module functions to retrieve default kwargs values from respective proxy settings’ attributes.- Parameters
key – (str or list) if string return value for single key, if list return a dictionary keyed by items in given key list.
nr_version
- salt_nornir.proxy.nornir_proxy_module.nr_version()
Function to return a report of installed packages and their versions, useful for troubleshooting dependencies.
ping
- salt_nornir.proxy.nornir_proxy_module.ping()
Return Nornir proxy status
queues_utils
- salt_nornir.proxy.nornir_proxy_module.queues_utils(call)
Function to retrieve operational data for job queues
- Parameters
call – (str) utility to invoke -
results_queue_dump
Supported calls:
results_queue_dump
- drain items from result queue and return their content,put items copies back into the queue afterwards
run
- salt_nornir.proxy.nornir_proxy_module.run(task, loader, identity, name, nr, wkr_data, **kwargs)
Function for worker Thread to run Nornir tasks.
- Parameters
task – (obj) callable task function
loader – (obj)
__salt__.loader
object instanceidentity – (dict) Task results queue identity for SaltEventProcessor
kwargs – (dict) passed to
task.run
after extracting CLI argumentsname – (str) Nornir task name to run
nr – (obj) Worker instance Nornir object
shutdown
- salt_nornir.proxy.nornir_proxy_module.shutdown()
This function implements this protocol to perform Nornir graceful shutdown:
Signal worker and watchdog threads to stop
Close all connections to devices
Close jobs and results queues
Kill all child processes
Delete Nornir object
Proxy Minion process keeps running afterwards, but cannot do anything.
stats
- salt_nornir.proxy.nornir_proxy_module.stats(*args, **kwargs)
Function to gather and return stats about Nornir proxy process.
- Parameters
stat – name of stat to return, returns all by default
Returns dictionary with these parameters:
proxy_minion_id
- if of this proxy minionmain_process_is_running
- set to 0 if not running and to 1 otherwisemain_process_start_time
-time.time()
function to indicate process start time inepoch
main_process_start_date
-time.ctime()
function date to indicate process start timemain_process_uptime_seconds
- int, main proxy minion process uptimemain_process_ram_usage_mbyte
- int, RAM usagemain_process_pid
- main process ID i.e. PIDmain_process_host
- hostname of machine where proxy minion process is runningjobs_started
- int, overall number of jobs startedjobs_completed
- int, overall number of jobs completedjobs_failed
- int, overall number of jobs failedjobs_job_queue_size
- int, size of jobs queue, indicating number of jobs waiting to startjobs_res_queue_size
- int, size of results queue, indicating number of results waiting to be collected by child processtasks_completed
- overall number of completed Nornir tasks (including subtasks)tasks_failed
- overall number of failed Nornir tasks (including subtasks)hosts_count
- int, number of hosts/devices managed by this proxy minionhosts_connections_active
- int, overall number of connection active to deviceshosts_tasks_failed
- overall number of hosts that failed all tasks within single jobtimestamp
-time.ctime()
timestamp ofstats
function runwatchdog_runs
- int, overall number of watchdog thread runswatchdog_child_processes_killed
- int, number of stale child processes killed by watchdogwatchdog_dead_connections_cleaned
- int, number of stale hosts’ connections cleaned by watchdogchild_processes_count
- int, number of child processes currently runningmain_process_fd_count
- int, number of file descriptors in use by main proxy minion processmain_process_fd_limit
- int, fd count limit imposed by Operating System for minion process
workers_utils
- salt_nornir.proxy.nornir_proxy_module.workers_utils(call)
Function to retrieve operational data for Nornir Worker instances
- Parameters
call – (str) utility to invoke
Supported calls:
stats
- return worker statistics keyed by worker id with these parameters:is_busy
- boolean, indicates if worker doing the workworker_jobs_completed
- counter of completed jobsworker_jobs_failed
- counter of completely failed jobsworker_connections
- hosts’ connections infoworker_jobs_queue
- size of the worker specific jobs queueworker_hosts_tasks_failed
- counter of overall host failed tasksworker_jobs_started
- counter of started jobs