Nornir Netbox Pillar Module
SaltStack external pillar module to source Salt-Nornir proxy minion pillar and Nornir inventory data from Netbox.
SaltStack pillar module name - salt_nornir_netbox
Foreword
With great power comes great responsibility
Salt-Nornir Netbox Pillar strives to be as efficient as possible and uses Netbox read-only GraphQL API because of that. However, the more data sourced from Netbox the longer it takes to process it and more memory it will occupy. Moreover, Netbox infrastructure need to be scaled to match Salt-Nornir requirements capable of processing appropriate number of incoming requests.
Keep in mind that pillar retrieval happens from Salt-Master only with retrieved data pushed to proxy minions for their use. Given Netbox can supply significant amount of data, Salt-Master resources need to be sized appropriately to be able to process it in a timely fashion.
Salt-Nornir Proxy uses Nornir workers internally, each worker is an instance of Nornir with its own dedicated inventory. As a result, an independent copy of pillar data retrieved from Netbox used by each Nornir worker. This can raise memory utilization concerns and should be kept an eye on.
It is always good to test this pillar functionality to get an understanding of how much resources required before using it in scaled-out deployments.
Dependencies
Salt-Nornir Netbox Pillar module uses Netbox read-only GraphQL API, as
a result GraphQL API need to be enabled for this pillar module to work.
In other words, Netbox configuration GRAPHQL_ENABLED
parameter should
be set to True
.
Salt-Nornir Netbox Pillar module uses Netbox REST API for secrets retrieval, as a result REST API need to be enabled if secrets fetched from Netbox.
Configuration Parameters
Sample external pillar Salt Master configuration:
ext_pillar:
- salt_nornir_netbox:
url: 'http://192.168.115.129:8000'
token: '837494d786ff420c97af9cd76d3e7f1115a913b4'
use_minion_id_device: True
use_minion_id_tag: True
use_hosts_filters: True
use_pillar: True
host_add_netbox_data: True
host_add_interfaces: True
host_add_interfaces_ip: True
host_add_connections: True
secrets:
resolve_secrets: True
fetch_username: True
fetch_password: True
secret_device: keymaster
plugins:
netbox_secretstore:
private_key: /etc/salt/netbox_secretstore_private.key
If use_pillar
is True, salt_nornir_netbox additional configuration can be
defined in proxy minion pillar under salt_nornir_netbox_pillar
key:
salt_nornir_netbox_pillar:
url: 'http://192.168.115.129:8000'
token: '837494d786ff420c97af9cd76d3e7f1115a913b4'
host_add_interfaces: "nb_interfaces"
hosts_filters:
- name__ic: "ceos"
- location__nic: "south"
tag: "mytag"
role: "core"
secrets:
resolve_secrets: True
fetch_username: True
fetch_password: True
secret_device: nrp1
plugins:
netbox_secretstore:
private_key: /etc/salt/netbox_secretstore_private.key
Pillar configuration updates Master’s configuration and takes precedence. Configuration not merged recursively, instead, pillar top key values override Master’s configuration.
Warning
Salt-Nornir Netbox pillar module strives to optimize interaction with Netbox improving efficiency of data retrieval. However, the more data fetched from Netbox the longer it takes to process it.
Name |
Default |
Example |
Description |
---|---|---|---|
|
N/A |
Netbox URL |
|
|
N/A |
N/A |
Netbox API Token |
|
False |
True or False |
If True, configuration context data of device with name
equal to proxy minion-id merged with proxy minion pillar
|
|
False |
True or False |
If True, Netbox devices that have tag assigned with value equal
to proxy minion-id included into pillar data
|
|
False |
True or False |
If True, devices matched by
hosts_filters processedand included into pillar data
|
|
False |
True or False |
If True, Master’s ext_pillar
salt_nornir_netbox configurationaugmented with pillar
salt_nornir_netbox_pillar configuration |
|
False |
True, False or
String e.g.
netbox_data |
If True, Netbox device data merged with Nornir host’s data, if
host_add_netbox_data is a string, Netbox device data saved intoNornir host’s data under key with
host_add_netbox_data value |
|
False |
True, False or
String e.g.
interfaces |
If True, Netbox device’s interfaces data added into Nornir host’s data
under
interfaces key. If host_add_interfaces is a string,interfaces data added into Nornir host’s data under key with
host_add_interfaces value |
|
False |
True, False or
String e.g.
nb_connections |
If True, Netbox device’s interface and console connections data added
into Nornir host’s data under
conections key. If host_add_connections is a string, connections data added into Nornir host’s data under key with
host_add_connections value |
|
None |
“name__ic”: “ceos1” |
List of dictionaries where each dictionary contains filtering
parameters to filter Netbox devices, Netbox devices that
matched, processed further and included into pillar data
|
|
N/A |
N/A |
Secrets Configuration Parameters indicating how to retrieve
secrets values from Netbox
|
url
and token
are mandatory parameters. salt_nornir_netbox.hosts_filters
nomenclature available at Netbox
documentation.
Filters processed in sequence, devices matched by at least one filter added into
proxy minion pillar.
Name |
Default |
Example |
Description |
---|---|---|---|
|
True |
True or False |
If True, attempts to resolve secrets values defined using
URL like strings
|
|
True |
True or False |
If True, attempts to retrieve host’s username from Netbox
secrets plugins, raises error if fails to do so, removing
host from pillar data.
|
|
True |
True or False |
If True, attempts to retrieve host’s password from Netbox
secrets plugins, raises error if fails to do so, removing
host from pillar data.
|
|
N/A |
keymaster |
Name of netbox device to retrieve secrets from by default |
|
N/A |
N/A |
Netbox Secrets Plugins Configuration Parameters |
Parameter |
Default |
Example |
Description |
---|---|---|---|
|
N/A |
“/etc/salt/nb_secretstore.key” |
OS Path to file with netbox_secretstore RSA Private Key content |
Sourcing Data from Netbox
salt_nornir_netbox external pillar retrieves data from Netbox using several methods. By default none of the methods turned on. All of the methods can be used separately or simultaneously, if used simultaneously processing follows order below.
Pillar data automatically sourced from Netbox on proxy-minion process startup or
restart, to source data on demand use nr.nornir refresh
command:
salt nrp1 nr.nornir refresh
Method-1 If use_minion_id_device
is True, configuration context data of
device with name equal to proxy minion id merged into proxy minion pillar.
Sample Netbox device configuration context data that contains Salt-Nornir proxy minion pillar data:
proxy:
proxy_always_alive: true
hosts:
fceos6:
data:
secrets:
- bgp: nb://netbox_secretstore/keymaster-1/BGP/peers_pass
- snmp: nb://netbox_secretstore/keymaster-1/SNMP/community
- nb://netbox_secretstore/keymaster-1/OSPF/hello_secret
hostname: 1.2.3.4
password: nb://netbox_secretstore/keymaster-1/SaltNornirCreds/password
platform: arista_eos
port: '22'
username: nb://netbox_secretstore/keymaster-1/SaltNornirCreds/username
nornir:
actions:
foobar:
args:
- show clock
description: test action
function: nr.cli
nrp3_secret_key: nb://nrp3_key_secret
Above data recursively merged with Salt-Nornir Proxy Minion pillar by SaltStack pillar system.
Method-2 If use_minion_id_tag
set to True, devices that have tag attached with value set
equal to minion-id retrieved from Netbox and processed to merge their data into
proxy minion pillar Salt-Nornir hosts
Method-3 If use_hosts_filters
is True, devices queried from Netbox using filters from
salt_nornir_netbox.hosts_filters
list and processed to merge their data into proxy
minion pillar Salt-Nornir hosts. If use_pillar
set to True, Proxy Minion
pillar can be used to define filters list under salt_nornir_netbox_pillar.hosts_filters
key
Netbox Device Processing
Nornir host’s parameters sourced from Netbox device’s configuration context
nornir
section. Sample device configuration context nornir
section
in YAML format:
nornir:
name: lsr21-foc771
platform: cisco_xr
hostname: 1.2.3.4
port: 22
groups: ["lab", "def_creds"]
username: admin1234
password: "nb://netbox_secretstore/password"
connection_options:
napalm:
platform: iosxr
scrapli:
platform: cisco_iosxr
puresnmp:
port: 161
extras:
version: v2c
community: "nb://netbox_secretstore/keymaster/snmp/community"
ncclient:
username: "nb://netbox_secretstore/netconf-creds/username"
password: "nb://netbox_secretstore/netconf-creds/password"
port: 830
extras:
hostkey_verify: False
device_params:
name: iosxr
data:
inventory_id: FCF483551
Above data processed and included into Salt-Nornir host’s inventory following these rules:
Device configuration context
nornir
section merged with Nornir host’s inventoryIf
name
defined under Netbox device’s configuration contextnornir
section it is used as a Nornir host’s inventory name key, otherwise device name usedIf
platform
not defined in Netbox device’s configuration contextnornir
section, platform value set equal to the value of device’s platform NAPALM Driver. If Netbox device has no platform associated and no platform given in configuration contextnornir
section, KeyError raised and device excluded from pillar dataIf
hostname
parameter not defined in Netbox device’s configuration contextnornir
s action,hostname
value set equal to device primary IPv4 address, if primary IPv4 address is not defined, primary IPv6 address used, if no primary IPv6 address defined, device name is used as ahostname
in assumption that device name is a valid FQDNIf
host_add_netbox_data
is a string, Netbox device data saved into Nornir host’s data usinghost_add_netbox_data
value as a key. For example, if value ofhost_add_netbox_data
is netbox_data, Netbox device data saved into Nornir host’s data under netbox_data key. Ifhost_add_netbox_data value
is True, Netbox device data merged with Nornir host’s data parameters
Warning
device is skipped if salt_nornir_netbox
fails to identify its platform
Sample device data sourced from Netbox, host_add_netbox_data
key name equal to
netbox
string in this example:
hosts:
ceos1:
data:
netbox:
airflow: FRONT_TO_REAR
asset_tag: UUID-123451
config_context:
domain_name: lab.io
lo0_ip: 1.0.1.4
secrets:
bgp: 123456bgppeer
secret1: secret1_value
secret2: secret2_value
secret3: secret3_value
secret4: secret4_value
syslog_servers:
- 10.0.0.3
- 10.0.0.4
custom_field_data:
sr_mpls_sid: 4578
device_role:
name: VirtualRouter
device_type:
model: FakeNOS Arista cEOS
last_updated: '2022-10-01T04:43:03.890510+00:00'
location:
name: Cage-77
name: fceos4
platform:
name: FakeNOS Arista cEOS
napalm_driver: arista_eos
position: '40.0'
primary_ip4:
address: 1.0.1.4/32
primary_ip6:
address: fb71::32/128
rack:
name: R101
serial: FNS123451
site:
name: SALTNORNIR-LAB
status: ACTIVE
tags:
- name: nrp3
tenant:
name: SALTNORNIR
Sourcing Secrets from Netbox
salt_nornir_netbox supports netbox_secretstore plugin to source secrets. netbox_secretstore should be installed and configured as a Netbox plugin. RSA private key need to be generated for the user which token is used to work with Netbox, private key need to be uploaded to Master and configured in Master’s ext_pillar:
ext_pillar:
- salt_nornir_netbox:
token: '837494d786ff420c97af9cd76d3e7f1115a913b4'
use_pillar: False
secrets:
resolve_secrets: True
fetch_username: True
fetch_password: True
secret_device: keymaster
plugins:
netbox_secretstore:
private_key: /etc/salt/netbox_secretstore_private.key
Alternatively, secrets plugins configuration can be defined in Salt-Nornir
Proxy Minion pillar if Master’s ext_pillar configuration has use_pillar
set to True.
Any of inventory keys can use value of URL string in one of the formats:
nb://<secre-plugin-name>/<device-name>/<secret-role>/<secret-name>
- fully qualified path to particular key, can be used to source secrets from any devices, secrets plugins or secrets rolesnb://<secre-plugin-name>/<secret-role>/<secret-name>
-device-name
assumed to be equal to the value ofsecrets.secret_device
parameter if it is given, otherwise Netbox device name is used as suchnb://<secre-plugin-name>/<secret-name>
- same rule applied fordevice-name
as in case 2, but secret searched ignoringsecret-role
and if secret with given name defined under multiple roles, the first one returned by Netbox is used as a secret value.nb://<secret-name>
- salt_nornir_netbox attempts to search for givensecret-name``across all plugins and secret roles, uses same rule for ``device-name
as in case 2
salt_nornir_netbox recursively iterates over entire data sourced from Netbox and attempts to resolve keys using specified secrets URLs.
For example, if this is how secrets defined in Netbox:
And sample configuration context data of fceos4
device is:
secrets:
bgp: nb://netbox_secretstore/keymaster-1/BGP/peers_pass
secret1: nb://netbox_secretstore/fceos4/SaltSecrets/secret1
secret2: nb://netbox_secretstore/SaltSecrets/secret2
secret3: nb://netbox_secretstore/secret3
secret4: nb://secret4
Above secrets would be resolved to this:
secrets:
bgp: 123456bgppeer <--- resolves to key id 187
secret1: secret1_value <--- resolves to key id 178
secret2: secret2_value <--- resolves to key id 179
secret3: secret3_value <--- resolves to key id 180
secret4: secret4_value <--- resolves to key id 181
salt_nornir_netbox iterates over all key’s values and resolves them accordingly.
Sourcing Interfaces and IP addresses data
Salt-Nornir Netbox Pillar can source device’s interface data from Netbox if
host_add_interfaces
parameter given. Interfaces added to device’s data
under key with the name equal to host_add_interfaces
parameter value, by
default it is set to interfaces
.
Interfaces combined into a dictionary keyed by device interface names.
Sample device interfaces data retrieved from Netbox:
hosts:
ceos1:
data:
interfaces:
Port-Channel1:
bridge: null
bridge_interfaces: []
child_interfaces: []
custom_fields: {}
description: Main uplink interface
enabled: true
ip_addresses: []
last_updated: '2022-09-19T18:47:28.425655+00:00'
mac_address: null
member_interfaces:
- name: eth101
- name: eth102
mode: null
mtu: null
parent: null
tagged_vlans: []
tags: []
untagged_vlan: null
vrf: null
wwn: null
eth1:
bridge: null
bridge_interfaces: []
child_interfaces:
- name: eth1.11
custom_fields: {}
description: Interface 1 description
enabled: true
ip_addresses: []
last_updated: '2022-09-19T18:47:29.519124+00:00'
mac_address: null
member_interfaces: []
mode: TAGGED
mtu: 1500
parent: null
tagged_vlans: []
tags: []
untagged_vlan: null
vrf: null
wwn: null
eth1.11:
bridge: null
bridge_interfaces: []
child_interfaces: []
custom_fields: {}
description: Sub-Interface 1 description
enabled: true
last_updated: '2022-09-19T18:47:38.603688+00:00'
mac_address: null
member_interfaces: []
mode: TAGGED
mtu: 1500
parent:
name: eth1
tagged_vlans: []
tags: []
untagged_vlan: null
vrf:
name: CUST1-Flinch34
wwn: null
eth201:
bridge: null
bridge_interfaces: []
child_interfaces: []
custom_fields: {}
description: ''
enabled: true
ip_addresses: []
last_updated: '2022-09-19T18:47:29.284530+00:00'
mac_address: null
member_interfaces: []
mode: TAGGED
mtu: null
parent: null
tagged_vlans:
- name: VLAN_2
vid: 102
- name: VLAN_3
vid: 103
- name: VLAN_4
vid: 104
- name: VLAN_5
vid: 105
tags: []
untagged_vlan:
name: VLAN_1
vid: 101
vrf: null
wwn: null
If host_add_interfaces_ip
parameter set to True, interface IP addresses
retrieved from Netbox as well.
Interface IP addresses combined into a list and added under ip_addresses
key in interface data:
hosts:
ceos1:
data:
interfaces:
eth1.11:
ip_addresses:
- address: 1.0.1.4/32
custom_fields: {}
description: ''
dns_name: ''
last_updated: '2022-09-19T18:47:36.818058+00:00'
role: null
status: ACTIVE
tags: []
tenant: null
- address: fb71::32/128
custom_fields: {}
description: ''
dns_name: ''
last_updated: '2022-10-01T04:43:03.873277+00:00'
role: LOOPBACK
status: ACTIVE
tags: []
tenant: null
Sourcing Connections Data
If parameter host_add_connections
given, Salt-Nornir Netbox pillar
can fetch device interfaces connection details from Netbox and add them to
device’s data under key equal to host_add_connections
parameter value, by
default it is set to connections
.
Connections retrieved for device interfaces and console ports and combined into a dictionary keyed by device interface names.
Sample device interfaces connections data retrieved from Netbox:
hosts:
ceos1:
data:
connections:
ConsolePort1:
custom_fields: {}
label: ''
last_updated: '2022-09-19T18:47:52.533889+00:00'
length: null
length_unit: null
remote_device: fceos5
remote_interface: ConsoleServerPort1
status: CONNECTED
tags: []
tenant:
name: SALTNORNIR
type: CAT6A
eth1:
custom_fields: {}
label: ''
last_updated: '2022-09-19T18:47:48.091871+00:00'
length: null
length_unit: null
remote_device: fceos5
remote_interface: eth1
status: CONNECTED
tags: []
tenant:
name: SALTNORNIR
type: CAT6A
Reference
- salt_nornir.pillar.salt_nornir_netbox.ext_pillar(minion_id, pillar, *args, **kwargs)
Salt Nornir Netbox External Pillar
- Parameters
minion_id – proxy minion id
pillar – proxy minion pillar data
args – list of any additional argument
kwargs – dictionary of any additional argument
External pillar automatically called on proxy minion startup, to refresh pillar data on demand call command on Salt-Master:
salt nrp1 nr.nornir refresh
- salt_nornir.pillar.salt_nornir_netbox._process_device(device, inventory, params)
Helper function to extract data to form Nornir host entry out of Netbox device entry.
- Parameters
device – device dictionary
inventory – Nornir inventory dictionary to update with host details
params – salt_nornir_netbox configuration parameters dictionary
- salt_nornir.pillar.salt_nornir_netbox._resolve_secrets(data, device_name, params)
Recursive function to iterate over data dictionary and resolve secrets using Netbox secret plugins, retrieving secrets for given device by device_name. Designed so to add capability to process data for one device, while retrieving keys from other device to simplify secrets management on Netbox side - all keys can be recorded under single, master device, instead of individual devices.
- Parameters
device_name – string, name of device to retrieve secret for
data – dictionary, containing key with values to be resolved
params – salt_nornir_netbox configuration parameters
- salt_nornir.pillar.salt_nornir_netbox._resolve_secret(device_name, secret_path, params, strict=False)
Function to retrieve netbox_secretstore secret value at secret_path.
- Parameters
device_name – string, name of device to retrieve secret for
secret_path – string, path to the secret in one of the supported formats
params – dictionary with salt_nornir_netbox parameters
strict – bool, if True raise KeyError if no secret found, return None otherwise
Supported secret key path formats:
nb://<plugin-name>/<device-name>/<secret-role>/<secret-name>
nb://<plugin-name>/<secret-role>/<secret-name>
nb://<plugin-name>/<secret-name>
nb://<secret-name>
- salt_nornir.pillar.salt_nornir_netbox._fetch_device_secrets(device_name, params)
Function to retrieve all secret values for given device from all configured Netbox secret plugins.
- Parameters
device_name – string, name of device to retrieve secrets for
params – dictionary with salt_nornir_netbox parameters
- salt_nornir.pillar.salt_nornir_netbox._host_add_interfaces(device, host, params)
Function to retrieve interface and ip addresses data and add it into Nornir host’s inventory.
- Parameters
device – device data dictionary
host – Nornir host inventory dictionary
params – dictionary with salt_nornir_netbox parameters
- salt_nornir.pillar.salt_nornir_netbox._host_add_connections(device, host, params)
Function to add interfaces and console ports connections details to Nornir host data.
- Parameters
device – device data dictionary
host – Nornir host inventory dictionary
params – dictionary with salt_nornir_netbox parameters
- salt_nornir.pillar.salt_nornir_netbox._netbox_secretstore_get_session_key(params)
Function to retrieve netbox_secretstore session key