Nornir Proxy module

Nornir Proxy Module is a core component of Nornir Proxy Minion. However, users rarely have to interact with it directly unless they writing their own Execution or Runner or State or whatever modules for SaltStack.

What is important to understand are configuration parameters you can use with Nornir Proxy Minion, as they can help to alter default behavior or control various aspects of Nornir Proxy Minion life cycle.

Introduction

Single Nornir proxy-minion can work with hundreds of devices as opposed to conventional proxy-minion that normally dedicated to managing one device/system only.

As a result, Nornir proxy-minion requires less resources to run tasks against same number of devices. During idle state only one proxy minion process is active, that significantly reduces amount of memory required to manage device endpoints.

Proxy-module required way of operating is multiprocessing set to True - default value, that way each task executed in dedicated process.

Dependencies

Nornir 3.x uses modular approach for plugins. As a result required plugins need to be installed separately from Nornir Core library. Main collection of plugins to install is nornir-salt. Nornir Salt repository contains many function used by Salt Nornir Proxy Minion module and is mandatory to have on the system where proxy minion process runs.

Nornir proxy pillar parameters

  • proxytype - string of value nornir

  • multiprocessing - boolean, set to True is a recommended way to run this proxy

  • process_count_max maximum number of processes to use to limit a number of tasks waiting to execute

  • nornir_filter_required - boolean, to indicate if Nornir filter is mandatory for tasks executed by this proxy-minion. Nornir has access to multiple devices, by default, if no filter provided, task will run for all devices, nornir_filter_required allows to change behavior to opposite, if no filter provided, task will not run at all. It is a safety measure against running task for all devices accidentally, instead, filter FB="*" can be used to run task for all devices.

  • runner - dictionary, Nornir Runner plugin parameters, default is RetryRunner

  • inventory - dictionary, Nornir Inventory plugin parameters, default is DictInventory populated with data from proxy-minion pillar, pillar data ignored by any other inventory plugins

  • child_process_max_age - int, seconds to wait before forcefully kill child process, default 660s

  • watchdog_interval - int, interval in seconds between watchdog runs, default 30s

  • proxy_always_alive - boolean, default True, keep connections with devices alive on True and tears them down after each job on False

  • job_wait_timeout - int, seconds to wait for job return until give up, default 600s

  • memory_threshold_mbyte - int, value in MBytes above each to trigger memory_threshold_action

  • memory_threshold_action - str, action to implement if memory_threshold_mbyte exceeded, possible actions: log- send syslog message, restart - shutsdown proxy minion process.

  • files_base_path - str, OS path to folder where to save ToFile processor files on a per-host basis, default is /var/salt-nornir/{proxy_id}/files/

  • files_max_count - int, maximum number of files for ToFileProcessor tf argument

  • nr_cli - dictionary of default kwargs to use with nr.cli execution module function

  • nr_cfg - dictionary of default kwargs to use with nr.cfg execution module function

  • nr_nc - dictionary of default kwargs to use with nr.nc execution module function

Nornir uses inventory to store information about devices to interact with. Inventory can contain information about hosts, groups and defaults. Nornir inventory defined in proxy-minion pillar.

Nornir proxy-minion pillar example:

proxy:
  proxytype: nornir
  process_count_max: 3
  multiprocessing: True
  nornir_filter_required: True
  proxy_always_alive: True
  watchdog_interval: 30
  child_process_max_age: 660
  job_wait_timeout: 600
  memory_threshold_mbyte: 300
  memory_threshold_action: log
  files_base_path: "/var/salt-nornir/{proxy_id}/files/"
  files_max_count: 5
  nr_cli: {}
  nr_cfg: {}
  nr_nc: {}
  runner:
     plugin: threaded
     options:
         num_workers: 100
  inventory:
     plugin: SimpleInventory
     options:
       host_file: "/var/salt-nonir/proxy-id-1/hosts.yaml"
       group_file: "/var/salt-nonir/proxy-id-1/groups.yaml"
       defaults_file: "/var/salt-nonir/proxy-id-1/defaults.yaml"

hosts:
  IOL1:
    hostname: 192.168.217.10
    platform: ios
    location: B1
    groups: [lab]
  IOL2:
    hostname: 192.168.217.7
    platform: ios
    location: B2
    groups: [lab]

groups:
  lab:
    username: nornir
    password: nornir
    connection_options:
      napalm:
        optional_args: {dest_file_system: "system:"}

defaults: {}

Nornir runners

Runners in Nornir define how to run tasks against hosts. If no runner dictionary provided in proxy-minion pillar, Nornir initialized using Nornir Salt RetryRunner plugin with these default settings:

runner = {
    "plugin": "RetryRunner",
    "options": {
        "num_workers": 100,
        "num_connectors": 10,
        "connect_retry": 3,
        "connect_backoff": 1000,
        "connect_splay": 100,
        "task_retry": 3,
        "task_backoff": 1000,
        "task_splay": 100,
        "reconnect_on_fail": True,
        "task_timeout": 600
    },
}

Nornir Proxy Module functions

salt_nornir.proxy.nornir_proxy_module.inventory_data(**kwargs)

Return Nornir inventory as a dictionary

Parameters

Fx – filters to filter hosts

salt_nornir.proxy.nornir_proxy_module.run(task, loader, *args, **kwargs)

Function for worker Thread to run Nornir tasks.

Parameters
  • task – callable task function

  • Fx – filters to filter hosts

  • kwargs – arguments to pass to nornir_object.run method

salt_nornir.proxy.nornir_proxy_module.execute_job(task_fun, args, kwargs, cpid)

Function to submit job request to Nornir Proxy minion jobs queue, wait for job to be completed and return results.

Parameters
  • task_fun – str, name of nornir task function/plugin to run

  • args – list, any arguments to submit to Nornir task *args

  • kwargs – dict, any arguments to submit to Nornir task **kwargs

  • cpid – int, Process ID (PID) of child process submitting job request

Additional execute_job arguments read from kwargs:

salt_nornir.proxy.nornir_proxy_module.stats(*args, **kwargs)

Function to gather and return stats about Nornir proxy process.

Parameters

stat – name of stat to return, returns all by default

Returns dictionary with these parameters:

  • proxy_minion_id - if of this proxy minion

  • main_process_is_running - set to 0 if not running and to 1 otherwise

  • main_process_start_time - time.time() function to indicate process start time in epoch

  • main_process_start_date - time.ctime() function date to indicate process start time

  • main_process_uptime_seconds - int, main proxy minion process uptime

  • main_process_ram_usage_mbyte - int, RAM usage

  • main_process_pid - main process ID i.e. PID

  • main_process_host - hostname of machine where proxy minion process is running

  • jobs_started - int, overall number of jobs started

  • jobs_completed - int, overall number of jobs completed

  • jobs_failed - int, overall number of jobs failed

  • jobs_job_queue_size - int, size of jobs queue, indicating number of jobs waiting to start

  • jobs_res_queue_size - int, size of results queue, indicating number of results waiting to be collected by child process

  • hosts_count - int, number of hosts/devices managed by this proxy minion

  • hosts_connections_active - int, overall number of connection active to devices

  • hosts_tasks_failed - int, overall number of tasks failed for hosts

  • timestamp - time.ctime() timestamp of stats function run

  • watchdog_runs - int, overall number of watchdog thread runs

  • watchdog_child_processes_killed - int, number of stale child processes killed by watchdog

  • watchdog_dead_connections_cleaned - int, number of stale hosts’ connections cleaned by watchdog

  • child_processes_count - int, number of child processes currently running

  • main_process_fd_count - int, number of file descriptors in use by main proxy minion process

  • main_process_fd_limit - int, fd count limit imposed by Operating System for minion process

salt_nornir.proxy.nornir_proxy_module.refresh_nornir(*args, **kwargs)

Function to re-initialise Nornir proxy with latest pillar data.

This function calls shutdown function, gets latest pillar from master and re-instantiates Nornir object.

salt_nornir.proxy.nornir_proxy_module.kill_nornir(*args, **kwargs)

This function kills Nornir process and its child process as fast as possible.

Warning

this function kills main Nornir peocess and does not recover it

salt_nornir.proxy.nornir_proxy_module.shutdown()

This function implements this protocol to perform graceful shutdown:

  1. Signal worker and watchdog threads to stop

  2. Close all connections to devices

  3. Close jobs and results queues

  4. Kill all child processes

  5. Delete Nornir object

salt_nornir.proxy.nornir_proxy_module.nr_version()

Function to return a report of installed packages and their versions, useful for troubleshooting dependencies.

salt_nornir.proxy.nornir_proxy_module.nr_data(key)

Helper function to return values from nornir_data dictionary, used by nr.cli, nr.cfg anf nr.nc execution module functions to retrieve default kwargs values from respective proxy settings’ attributes.