Linux Monitoring Agent Installation

Linux Monitoring Agent Installation

Nectus Linux Monitoring Agent is required to be installed on target machine to enable Nectus to monitor Linux system parameters such as CPU and RAM utilization, Storage and Network Utilization.

Agent collects all the required monitoring metrics and reports to the main Nectus Server every 5 minutes.

 

Step 1: Download Monitoring Agent Package

Download binary package appropriate to your target machine Linux flavor and version from the download page.

 

Step 2a: Installation for Debian-based systems (including Ubuntu)

After downloading the package run the following installation command:

$ sudo apt-get install -y <package-name>

This will install Nectus monitoring agent on your target machine. If everything is OK you’ll see an output like this:

$ sudo apt install -y ./nectus-agent_1.4-3_amd64.deb
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
Note, selecting ‘nectus-agent’ instead of ‘./nectus-agent_1.4-3_amd64.deb’
The following NEW packages will be installed:
nectus-agent
0 upgraded, 1 newly installed, 0 to remove and 81 not upgraded.
Need to get 0 B/843 kB of archives.
After this operation, 2,654 kB of additional disk space will be used.
Get:1 /home/oleg/nectus-agent_1.4-3_amd64.deb nectus-agent amd64 1.4-3 [843 kB]
Selecting previously unselected package nectus-agent.
(Reading database … 278414 files and directories currently installed.)
Preparing to unpack …/nectus-agent_1.4-3_amd64.deb …
Unpacking nectus-agent (1.4-3) …
Setting up nectus-agent (1.4-3) …
Created symlink /etc/systemd/system/multi-user.target.wants/nectus-agent.service → /lib/systemd/system/nectus-agent.service.

 

Step 2b: Installation for RedHat, CentOS and Amazon Linux

After downloading the package run the following installation command:

$ sudo yum install -y <package-name>.rpm

You’ll see an output like this:

$ sudo yum install -y nectus-agent-1.4-3.el9.x86_64.rpm

Last metadata expiration check: 0:00:12 ago on Wed 27 Sep 2023 10:40:52 +07.
Dependencies resolved.
==================================================================================================================
Package Architecture Version Repository Size
==================================================================================================================
Installing:
nectus-agent x86_64 1.4-3.el9 @commandline 1.0 M

Transaction Summary
==================================================================================================================
Install 1 Package
Total size: 1.0 M
Installed size: 4.3 M
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: nectus-agent-1.4-3.el9.x86_64 1/1
Installing : nectus-agent-1.4-3.el9.x86_64 1/1
Running scriptlet: nectus-agent-1.4-3.el9.x86_64 1/1
Verifying : nectus-agent-1.4-3.el9.x86_64 1/1

Installed:
nectus-agent-1.4-3.el9.x86_64

Complete!

 

Step 3: Configuration

After installation is complete you need to set up the agent’s basic configuration.

Nectus agent configuration file is located at: /etc/nectus/nectus-agent.conf.

Default configuration file content is listed in Appendix 1. Most of the values are set to defaults and don’t require a change except following two parameters:

  • SERVER_PRESHARED_KEY
  • AGENT_PORT

SERVER_PRESHARED_KEY is used to encrypt messages between the Agent and Nectus Server. Initially it’s set to value <YourSecretKey>.

Edit this parameter to match the key configured in Nectus GUI (Settings->General Settings ->Linux Monitoring Integration).

SERVER_PRESHARED_KEY is a global parameters that is used for all Linux Servers.

AGENT_PORT parameter determines the TCP port used by Agent to listen for incoming connections from the Nectus server. By default the TCP port is set to 5400.

AGENT_PORT is Server specific parameter that can be different on each Linux Server.

 

Step 4: Generate SSL Certificate (optional)

The Linux Monitoring Agent listens on pre-configured TCP port for incoming HTTPS connection and needs an SSL certificate to operate.

Default location for storing certificate files is /etc/nectus but it’s configurable (parameter SSL_CERTIFICATE_DIR in the configuration file).

You can either use any existing certificate or generate a new self-signed one.

To generate new self-signed certificate run commands:

$ cd /etc/nectus

$ openssl dhparam -out dh.pem 2048

$ openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 10000 -out certificate.pem

The directory pointed by SSL_CERTIFICATE_DIR parameter should finally contain files dh.pem, key.pem and certificate.pem.

 

Step 5: Open TCP port at the Firewall

If there is firewall between Nectus Server and Linux Monitoring Agent you need to permit connections on TCP port 5400 (Unless it is changed to a different port)

Make sure your Linux internal firewall also permits inbound connections on this port.

 

Step 6: Start Agent

Restart the agent in order for configuration changes to take effect:

$ service nectus-agent restart

Check it’s status to ensure proper operation:

$ service nectus-agent status

If case of successful start you’ll see some output like this:

$ service nectus-agent status

● nectus-agent.service – Nectus Agent

Loaded: loaded (/lib/systemd/system/nectus-agent.service; disabled; vendor preset: enabled)

Active: active (running) since Mon 2019-04-29 17:55:39 UTC; 4s ago

Process: 10247 ExecStart=/usr/bin/nectus-agent $CONFFILE (code=exited, status=0/SUCCESS)

Main PID: 10255 (nectus-agent)

Tasks: 2

Memory: 1.2M

CPU: 14ms

CGroup: /system.slice/nectus-agent.service

└─10255 /usr/bin/nectus-agent /etc/nectus/nectus-agent.conf

Apr 29 17:55:39 ubuntu-16 systemd[1]: Starting Nectus Agent…

Apr 29 17:55:39 ubuntu-16 systemd[1]: nectus-agent.service: PID file /run/nectus/nectus-agent.pid not readable

Apr 29 17:55:39 ubuntu-16 systemd[1]: Started Nectus Agent.

User account

For security reasons it’s not allowed to run Monitoring Agent as root user. If it’s trying to start as ‘root’ then the agent will drop its privileges to the user account set in configuration file

(usually /etc/nectus/nectus-agent.conf) as USER_INSTEAD_OF_ROOT.

Default USER_INSTEAD_OF_ROOT is ‘nectus’.

The agent automatically creates this user account during installation (if this user does not exist already) and provides minimal set of permissions required to run the agent code.

Troubleshooting

Nectus agent runs as a daemon process. You can use the following commands to start, stop restart and check status of the agent:

$ service nectus-agent start

$ service nectus-agent stop

$ service nectus-agent restart

$ service nectus-agent status

If there are any problems with starting the agent first check the agent’s status:

$ service nectus-agent status

Then check the log file (usually /var/log/nectus/nectus-agent.log). If it does not help then you can try to run the agent manually. Usage of the module:

nectus-agent <config-file> [-D]

config-file full path to configuration file

-D debug mode (agent starts not as a daemon but as a usual process)

Locate the executable nectus-agent in /usr/bin and try to run the agent not as a daemon:

$ /usr/bin/nectus-agent /etc/nectus/nectus-agent.conf -D

Check the output and log files at /var/log/nectus.

Upgrade

Upgrade to a new version of the agent is the same as a new installation: download the appropriate package and run either:

> sudo apt-get install -y <package-name>.deb

for Debian/Ubuntu systems or

> sudo yum install -y <package-name>.rpm

for RedHat/CentOS/Amazon Linux.

Uninstallation

To uninstall Nectus agent run the following command:

> sudo apt-get remove nectus-agent

for Debian/Ubuntu systems or

> sudo yum remove nectus-agent

for RedHat/CentOS/Amazon Linux.

Appendix 1. Default Configuration File

#########################################
####       GENERAL PARAMETERS        ####
AGENT_IP                0.0.0.0          # IP address of listening socket. 
                                         # Default 0.0.0.0 (listen on all interfaces)
 
AGENT_PORT              5400             # TCP/IP port of the listening socket. Default 5400
 
SSL_CERTIFICATE_DIR     /etc/nectus      # Path to agent’s SSL certificate. 
        # Directory should contain files:
        # dh.pem, certificate.pem and key.pem.
        # To generate those files (self-signed certificate) run commands:
        # openssl dhparam -out dh.pem 1024 
        # openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 10000 -out certificate.pem
        # Maybe you would need root privileges for this, then run: sudo openssl etc…
 
SERVER_PRESHARED_KEY    <YourSecretKey> # Pre-shared key to encrypt messages between 
                                        # the agent and Nectus monitoring server. 
                                        # Must be the same as set in Nectus GUI.
 
NUM_OF_THREADS          1               # Number of agent’s threads. Valid values are from 1 to 32.
                                        # Default value = 1.
 
USER_INSTEAD_OF_ROOT    nectus          # Drop privileges to a specific user existing in the system.
                                        # Only has effect if the agent is run as root.
                                        # Default value = nectus.
 
#########################################
######### Logging parameters ############
LOG_DIR                         /var/log/nectus  # Path to log files
 
LOG_LEVEL                       info
# Log verbosity level. Possible values are:
#       error – only errors are reported in log
#       info  – default logging level
#       debug – most verbose level including debug information.
 
LOG_FLUSH_PERIOD_SECONDS 30     # Flush log data to disk every X seconds. 
                                # Valid values are from 1 to 900. Default value = 30.
 
LOG_FILE_MAX_SIZE_MBYTES 30     # Maximum size of a single log file. After reaching this size 
                                # next log file is opened. Valid values are from 1 to 1000.
                                # Default value = 30.
 
LOG_FILE_MAX_COUNT      30      # Maximum count of log files stored. After reaching this count
                                # the oldest log files are deleted. 
                                # Valid values are from 1 to 1000. Default value = 30.
/by

ClickHouse DB Installation for Nectus Netflow & Syslog Storage

, ,

Requirements:  Operating System: Ubuntu 22.04.1 LTS       RAM: 16GB   vCPU: 2

More information about installation: https://clickhouse.com/docs/en/install/#self-managed-install

Step 1

Open terminal and install required packages:

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates dirmngr

Step 2

Import the ClickHouse public key:

sudo apt-key adv –keyserver hkp://keyserver.ubuntu.com:80 –recv 8919F6BD2B48D754

NOTE: It is recommended to import the public key if it’s a fresh Ubuntu install.

Otherwise, you may get the following error when adding the repository:

GPG error: http://repo.clickhouse.com/deb/stable/main/release:
The following signatures couldn’t be verified because the public key is not available:

NO_PUBKEY 8919F6BD2B48D754

Step 3

Add ClickHouse repository:

echo “deb https://packages.clickhouse.com/deb stable main” | sudo tee \
/etc/apt/sources.list.d/clickhouse.list

Step 4

Install ClickHouse:

NOTE: During the installation you will be asked to create a password for default user.

Generate a strong password and save it, it will be used for further configuration.

sudo apt-get update
sudo apt-get install -y clickhouse-server clickhouse-client

Step 5

Start ClickHouse server as a daemon:

sudo service clickhouse-server start

Step 6

Now after installation it is time to test connection to ClickHouse database.

Start client (use password generated on Step 4):

clickhouse-client –password

NOTE: Use Ctrl+D to exit ClickHouse client.
NOTE: TCP ports 8123 and 9000 must be open

Step 7

NOTE: Use Ctrl+D to exit ClickHouse client.

While being connected to the ClickHouse, create required Netflow & Syslog databases:

CREATE DATABASE NETFLOW;
CREATE DATABASE SYSLOG;

ClickHouse Configuration

Step 8

By default, ClickHouse allows inbound connections from localhost only.

To allow connections from remote IP-s

edit /etc/clickhouse-server/config.xml:

sudo vim /etc/clickhouse-server/config.xml

Find <listen_host> parameters (one for IPv6 and one for IPv4) allowing all connections after <logger> section, and uncomment them.

Also uncomment and change to 1 <listen_reuse_port> tag for allowing simultaneous IPv4 and IPv6 interfaces:

<listen_host>::</listen_host>

<!– Same for hosts without support for IPv6: –>
<listen_host>0.0.0.0</listen_host>

<listen_reuse_port>1</listen_reuse_port>

Step 9

Restart ClickHouse to apply new parameters:

sudo service clickhouse-server restart

Step 10

Add internal ClickHouse user “root” for Nectus server to connect to DB.

For that edit users.xml file:

sudo vi /etc/clickhouse-server/users.xml

Find <users> section right after <profiles> section.

Add code snippet with user configuration from below, right between </users> and <default> tags, as on a screenshot below.

Generate strong password, replace it in snippet below, and then save file:

<root>
<password>**********</password>
<networks incl=”networks” replace=”replace”>
<ip>::/0</ip>
</networks>
<profile>default</profile>
<quota>default</quota>
</root>

NOTE: If file was overwritten by WinSCP or any other similar tool, verify that file has correct ownership:

sudo chown clickhouse:clickhouse /etc/clickhouse-server/users.xml

Step 11

Restart ClickHouse to apply new user:

sudo service clickhouse-server restart

Step 12

Verify newly created user able to log in:

clickhouse-client -u root –password

Step 13

Disable limit of max database size to drop. By default, ClickHouse doesn’t allow to drop table bigger than 50 Gb to prevent occasional data loss.

NOTE: More info here: https://cloud.yandex.com/en/docs/managed-clickhouse/concepts/settings-list#dbms-cluster-settings

To disable it, edit config file /etc/clickhouse-server/config.xml:

sudo vi /etc/clickhouse-server/config.xml

Find and uncomment line to disable limit of max table size to drop:

<max_table_size_to_drop>0</max_table_size_to_drop>

Restart ClickHouse to apply change:

sudo service clickhouse-server restart

Nectus Configuration

Step 14

We have completed the ClickHouse installation. This last step requires login to Nectus to finish the NetFlow/Syslog integration.

Open to “Nectus Settings -> General Settings -> Netflow Integration” page:

Enter the required information and click Test DB Connection (Remote Server IP is the IP address of the Ubuntu/ClickHouse server). The result should be “Test DB Connection OK”

Click “Run Integration Scripts” and finally Save.

Step 15

Restart Nectus NetFlow and Syslog Services.

/by

Monitoring BGP Sessions with Nectus

,

Monitoring BGP Sessions with Nectus

Nectus 1.2.53 introduces ability to automatically discover and monitor iBGP and eBGP sessions across all router types and manufacturers.

Nectus scheduled network discovery automatically finds all the BGP sessions via SNMP polls and adds it to Nectus database where user can group them by type and enable active monitoring and alerting.

BGP Session Discovery

As part of regular scheduled network discovery Nectus detects all existing BGP Sessions and adds it to one of the existing BGP group which has “Default” flag set to ON.

You can see all the BGP sessions that were discovered in Inventory -> BGP Sessions or in left side panel called “BGP Sessions”

 

If you right-click on BGP Session and select “Properties”

 

.. you can assign a human friendly name or change a group membership for any of the BGP sessions.

 

You can create multiple BGP session groups and assign discovered BGP sessions to different groups according to its type or functional purpose.

To create a new Group, right-click on “BGP Sessions” line and select “Add New Group” on context menu.

 

Configure New Group name, set “Enable monitoring” to ON if you want all BGP Sessions in this group to be monitored for UP/DOWN Status.

 

If you set “Default” value to ON, this group will become a group where all newly discovered BGP sessions be added automatically. Only one group can have “Default” value set to ON

You can also assign BGP Session to a Group by right-clicking on BGP session and selecting “Move BGP Session to..” option

 

 

By clicking “View Session Info” on BGP Session context menu you can view detailed information about BGP Session

 

 

After you finished creating BGP session groups, assigning BGP sessions to corresponding groups and enabling BGP session groups for Monitoring the final step is to enable BGP Sessions monitoring in corresponding SNMP Device Monitoring Profiles that are applied to the routers that terminate all the BGP sessions.

Find required SNMP Monitoring Profile and enable “BGP Session Status” Monitoring Option

Now are you all set to start monitoring your BGP sessions line a pro.

 

/by

Features coming up in next Nectus release

  1. BGP sessions monitoring with UP/DOWN Alerting
  2. Linux Servers Monitoring (CPU, RAM,  Storage, Up/Down)
/by

UPS Health Status Monitoring with Nectus

,

UPS Status Monitoring with Nectus

Health monitoring of UPS units is one of the most critical tasks of network infrastructure monitoring.

If UPS fails during power outage it results in immediate network downtime resulting in loss of revenue and degrading user experience.

Most of the modern UPS Units perform daily self-tests and results of those self-tests are available via SNMP protocol.

Our goal is to collect those self-test results on regular basis and generate an alert or incident when self-test indicates a UPS failure.

Step 1. Determine which SNMP OID to use

Very first step before you can start monitoring UPS health status is to determine which SNMP OID contains self-test information for your UPS units.

You can open a support case with your UPS vendor and request SNNP OID string for self-test information.

For example, SNMP OID for self-test information from UPS manufactured by APC is .1.3.6.1.4.1.318.1.1.1.7.2.3.0

This OID returns following values:

1 = Ok

2 = Failed

3 = Invalid Test

4 = Test In Progress

 

Step 2. Create Device View that contain all UPS units

Next step is to create a Device View that contain all UPS units that we want to monitor.

To create a Device View go to Inventory -> Views -> SNMP Device Views.

 

Create a new Device Vice called “UPS Units”

 

 

Step 3. Create Custom SNMP Tracker for UPS Self-Test Info

Now we are ready to create a custom SNMP tracker that will be polling our UPS units every 5 min for self-test information.

In Nectus GUI go to Monitoring → SNMP Monitoring Settings → Custom SNMP Trackers

 

Press “Create” button to bring up Custom SNMP Tracker creation interface.

Complete tracker General information tab with correct SNMP OID and select Device View that contains all UPS units that we want to monitor.

 

Select Email and SMS list for Alert Recipients.

In “Alerts” Tab we need to specify which status values will be considered Normal and which values should trigger Alerts.

You can specify multiple values separated by comma.

 

Define an Alert Severity level for Alarm Values as Informational, Warning or Critical.

Define number of consecutive readings for which UPS self-test status must report an Alarm condition before formal Alert is created.

Nectus performs one SNMP poll per 5 min.

If you define value for consecutive readings as 3 it should result in Alert created after 15 minutes of True Alarm conditions.

Press “Save” to complete Custom SNMP tracker creation for UPS Status Monitoring

This completes creation of custom tracker that will be monitoring UPS self-test information on all APC UPS units in your network.

 

/by

Suppression of Frequent Identical Alerts In Nectus

,

Suppression of Frequent Identical Alerts In Nectus

Often when a network device or interface changes its state between Up and Down multiple times in a short time interval, numerous identical alerts will be sent to all the configured alert recipients.

Nectus offers the ability to suppress frequent identical alerts to reduce informational noise level by specifying a suppression timer for up to three hours.

Nectus calculates a CRC hash value from each of the email alert bodies and if hash values of two alerts are matching they are considered to be identical and can be suppressed.

Nectus excludes event timer values and specific metric values from hash calculation logic. You can configure the suppression timer from 0 to 180 minutes where 0 means no suppression.

To configure the suppression timer, go to    Monitoring -> Global Monitoring Settings

 

 

Under Monitoring Settings, select the Alerts tab.

To enable suppression specify a time interval from 1 to 180 minutes.

At the end of the time interval, alert logic will resume as normal.

Setting timer to 0 will disable the suppression.

 

 

/by

Importing WMI Servers from CSV file to Nectus

,

Importing WMI Servers from CSV file to Nectus

To import a list of Windows Servers from CSV file to Nectus prepare a CSV file with only single column containing list of IP addresses of Windows Servers.

No other information is required.

File format example in Excel.

To start import, right-click on WMI Server Group that you want to import servers into and select “Import  WMI Servers from CSV file” option

 

 

Select CSV File that you prepared.

 

 

Press “Start” button

 

Nectus will import each server individually and collect basic Server information via WMI Interface.

 

/by

Configuring WMI Integration in Nectus

,

Configuring WMI Integration in Nectus

Nectus uses WMI Interface in several of its modules.

List of modules that depends on WMI Integration:

  1. Windows Server Monitoring Module
  2. IPAM Module

In Windows Server Monitoring module Nectus uses WMI interface to poll Windows Servers for critical health metrics such as CPU and RAM Utilization etc.

In IPAM Module Nectus uses WMI interface to interact with Microsoft DNS and DHCP servers while Importing and creating reservations.

WMI Integration settings must be complete before Nectus can communicate with Windows Servers in those modules.

To complete WMI Integration go to Settings -> General Settings -> WMI Integration

Click on “WMI Integration” link and complete all the required fields for WMI Service account

that will be used by Nectus connect to Windows Servers via WMI Interface.

WMI Service account should not have password expiration policy to prevent Integration from breaking every time password has to be changed.

Use “Test” button to test Service account credential against any of Windows Servers of your choice.

After connection “Test” is passed WMI Integration is complete.

 

/by