Technical Method of Procedure (MOP) for Cacti Network Monitoring Disaster Recovery (Backup and Restoration)

Disaster Recovery (DR) is a fundamental component of an organization’s overall business continuity strategy. Its explicit focus is on ensuring that the IT systems supporting critical business functions remain operational, or can be restored to an operational state, as swiftly as possible following any disruptive event. Every enterprise, regardless of its industry or scale, must possess the capability to recover rapidly from incidents that halt day-to-day operations. Failure to implement a robust disaster recovery plan can lead to significant data loss, diminished productivity, unforeseen expenses, and severe reputational damage. Network monitoring systems, such as Cacti, serve as the vital eyes and ears of an organization’s IT network infrastructure. They provide continuous visibility into network health, performance, and resource utilization, enabling proactive problem detection and resolution. The uninterrupted availability of such a monitoring system is therefore not merely beneficial but non-negotiable for maintaining overall business resilience.

Prerequisites and Pre-Backup Checklist

Before initiating any backup or recovery procedures for Cacti, it is essential to ensure that the target environment meets the necessary system requirements and to identify all critical components that must be included in the backup. A crucial preparatory step involves temporarily halting Cacti’s data collection processes to ensure data consistency during the backup.

Understanding the System Requirements for Cacti

A successful Cacti installation, and by extension a successful recovery, depends on the presence and correct configuration of several core software components:

Component	Requirement
Web Server	Compatible with PHP: Apache (httpd) or NGINX
RRDTool	Recommended: Version 1.5 or greater
Database	MySQL (MariaDB Server) 5.5+
PHP	Recommended: Version 5.5 or greater
NET-SNMP	Recommended: Version 5.7.3 or greater

Identifying Critical Cacti Components for Backup

A comprehensive Cacti disaster recovery plan necessitates backing up more than just the database. Cacti’s functionality relies on a complex ecosystem of data and configuration files. The following components are critical for a complete and successful restoration:

Component	Typical Location(s)	Description/Importance
Cacti Database (MariaDB)	/var/lib/mysql/cacti (data files)	Core configuration, device definitions, graph settings, user accounts, and historical data pointers. Essential for Cacti’s operational metadata.
RRD (Round Robin Database) Files	/var/lib/cacti/rra/	Stores actual time-series performance data for all graphs. Critical for historical graphing. Can grow very large.
Cacti Application Files	/var/lib/cacti (or /usr/share/cacti)	Core application binaries, libraries, and scripts (e.g., poller.php). Required for Cacti to run.
Cacti Configuration File	/usr/share/cacti/include/config.php or /etc/cacti/db.php	Contains database connection details, paths, and other critical Cacti-specific settings.
Spine Poller Configuration	/etc/spine.conf (if Spine is used)	Configuration for the high-performance Spine poller.
Custom Scripts	/var/lib/cacti/scripts/ (or custom paths)	Any custom scripts used for data collection or processing.
Custom Templates/Plugins	Varies (e.g., /var/lib/cacti/resource/templates/, plugin directories)	Custom XML templates for devices/graphs, and any installed plugins.
Web Server Configuration	/etc/httpd/conf.d/cacti.conf (Apache), /etc/nginx/sites-available/cacti (Nginx)	Web server configuration specific to Cacti’s virtual host or alias.
PHP Configuration	/etc/php.ini, /etc/phpX/apache/php.ini	PHP runtime settings, including date.timezone and module configurations.
Cacti Cronjob/Systemd Unit	/etc/cron.d/cacti or /etc/systemd/system/cactid.service	Defines the schedule and execution parameters for the Cacti poller.

The importance of backing up RRD files, in addition to the database, cannot be overstated. While the database stores configurations and metadata, the actual time-series data for historical graphing resides exclusively in the RRD files. A common pitfall in Cacti disaster recovery is focusing solely on the database, which, while critical, leaves the core monitoring data unrecoverable. Furthermore, a simple copy of RRD files is often insufficient for restoration; they typically require a specific conversion to XML format and then restoration from XML to ensure data integrity and compatibility across different environments. This multi-component backup strategy is essential for a truly comprehensive Cacti recovery.

Pre-Backup Operational Steps: Disabling the Cacti Poller

To guarantee data consistency and minimize any gaps in graphs during the backup and subsequent migration process, it is absolutely critical to disable the Cacti poller cron job or systemd service on the source server before initiating any backup procedures. This step ensures that no new data is written to the Cacti database or the RRD files while the backup is in progress. By creating a static state, the backup captures a consistent snapshot of the monitoring data. If the poller were to remain active during the mysqldump or RRD XML conversion, new data could be written, leading to a temporal mismatch between the database and RRD files, which could result in an inconsistent or corrupted backup that is difficult or impossible to restore reliably. This directly mitigates the risk of a “dirty” backup, a common cause of failed disaster recovery attempts.

For cron-based poller (CentOS/Ubuntu/Debian example): Edit the Cacti poller cron file, typically located at /etc/cron.d/cacti, and comment out the line that executes poller.php.

For systemd-based poller (if cactid.service is used): Stop and disable the Cacti poller service.

# systemctl stop cactid
# systemctl disable cacti

After disabling, verify that the poller is no longer running by checking Cacti’s log file or monitoring system processes.

---

Phase 1: Cacti Database (MariaDB) Backup Procedure

The Cacti database, typically MariaDB, holds all the configuration, device information, graph definitions, and historical data pointers that are fundamental to Cacti’s operation. A consistent backup of this database is paramount for a successful disaster recovery.

mysqldump is the standard and most widely used utility for performing logical backups of MySQL and MariaDB databases. This tool generates a set of SQL statements that can recreate the database objects (tables, views, stored procedures) and insert the data into them. This approach offers significant advantages in terms of portability and ease of restoration, as the backup is essentially a text file that can be imported into any compatible MariaDB/MySQL instance. mysqldump can support database backups of up to 150GB.

Step-by-Step: Performing a Full Cacti MariaDB Database Dump

(i) Access the Cacti Server: Establish an SSH connection to your Cacti server.

(ii) Navigate to a Secure Backup Location: Create a dedicated directory for backups with sufficient disk space. For instance, /opt/backups/cacti/db/ is a suitable location.

# mkdir -p /opt/backups/cacti/db
# cd /opt/backups/cacti/db

(iii) Execute the mysqldump Command: Run the following command to create a compressed dump of your Cacti database.

# mysqldump -u cactiuser -p'your_cacti_db_password' \
  --single-transaction --skip-lock-tables cacti | \
  gzip > cacti_db_backup_$(date +"%Y%m%d_%H%M%S").sql.gz

🔒 Note: Replace:

• cactiuser with your cacti database username
• 'your_cacti_db_password' with the actual cacti database password
• cacti with your cacti database name

Options Explained:

• --single-transaction: Ensures data consistency during the dump.
• --skip-lock-tables: Prevents mysqldump from explicitly locking tables during the backup process. This is essential for minimizing disruption and downtime in production environments.
• gzip: Compresses the output to save space.
• $(date ...): Adds a timestamp to the filename for versioning.

(iv) Verify the Backup File Integrity: After the mysqldump command completes, it is crucial to verify the existence and integrity of the created .sql.gz file. A backup that cannot be restored is worthless. This verification step, though seemingly small, is critical because a backup process might appear successful, but the file itself could be corrupted during creation or compression. Without this check, such corruption would remain undetected until a disaster strikes, rendering the DR plan ineffective.

Step-by-Step: Backing Up Cacti RRD Files (Convert RRD files to XML)

Simply copying RRD files directly is often problematic for Cacti migration or restoration, as they might contain platform-specific metadata or internal pointers that are not preserved across direct copies, especially between different OS versions or hardware. The recommended and most reliable method involves converting them to XML format on the source server, and then archiving these XML files for transfer. This XML conversion acts as a universal serialization format for RRD data, ensuring the integrity and usability of the historical graph data post-restoration. Without this crucial step, the restored Cacti instance might appear operational but fail to render accurate historical graphs, rendering the monitoring data useless.

(i) On the Source Cacti Server: Create a temporary directory to store the XML files

# mkdir -p /opt/backups/cacti/xml_rrd_dump

(ii) Convert all .rrd files to .xml using rrdtool dump:

# cd /var/lib/cacti/rra
# for i in $(find ./ -name "*.rrd"); do rrdtool dump "$i" > /opt/backups/cacti/xml_rrd_dump/$i.xml; done

(iii) Create a compressed tar archive of the XML files for efficient transfer:

# cd /opt/backups/cacti/xml_rrd_dump/
# tar cvfz cacti_rrd_xml_$(date +"%Y%m%d_%H%M%S").tar.gz *

(iv) Clean up temporary XML files (optional, but recommended to save space):

# cd /opt/backups/cacti/xml_rrd_dump/
# rm -rf /opt/backups/cacti/xml_rrd_dump/*.xml

---

Phase 2: Secure Transfer to Remote Storage

Once the Cacti database backup, configuration files, scripts, RRD XML archives and any other important files have been created, securely transferring them to a remote storage location is a critical step in any disaster recovery plan. This ensures that backups are protected from localized failures affecting the primary Cacti server. Both SCP (Secure Copy Protocol) and SFTP (Secure File Transfer Protocol) are network protocols designed for securely copying files between systems. They leverage the underlying SSH (Secure Shell) connection, which encrypts both the files being transferred and any authentication credentials (like passwords). For automated backup transfers, SCP’s straightforward syntax often makes it the preferred choice.

Step-by-Step: Transferring Backup Files to Remote Storage

(i) Ensure SSH Connectivity: Verify that SSH is properly configured and accessible between your Cacti server (the source of the backup) and the designated remote storage server (the destination).

(ii) Remote Storage Server Authentication (Using SSH Keys): For automated and secure transfers, it is highly recommended to use SSH key-based authentication instead of relying on passwords. This involves generating an SSH key pair on the Cacti server and adding the public key to the authorized_keys file for the target user on the remote storage server. Relying on passwords for automated scripts introduces both a security risk (passwords embedded in scripts) and a reliability issue (password changes breaking automation). SSH keys, conversely, enable passwordless automation, which is essential for scheduled backups. This approach transforms a “secure transfer” into a “secure, automated, and reliable transfer,” which is fundamental to a scalable disaster recovery strategy. The command examples below will generate ssh key pair on the cacti server and copy the public key to remote server.

# ssh-keygen -t rsa -b 4096
# ssh-copy-id username@remote_storage_server

(iii) Copy the Cacti Database and RRD Backup Files on the local Cacti server to Remote Storage Server using SCP: Because we already setup ssh-keys in step (ii) above, the following command will not ask for a password.

# scp /opt/backups/cacti/db/cacti_db_backup_YYYYMMDD_HHMMSS.sql.gz username@remote_storage_server:/path/to/remote/backup/location/
# scp /opt/backups/cacti/xml_rrd_dump/cacti_rrd_xml_YYYYMMDD_HHMMSS.tar.gz username@remote_storage_server:/path/to/remote/backup/location/

Where:
 - username is the user account on the remote storage server.
 - remote_storage_server is the IP address of the remote storage server.
 - /path/to/remote/backup/location/ is the specific directory on the remote server where the backup files will be stored.

(iv) Verify Transfer: After the scp command reports completion, it is essential to log in to the remote storage server and verify the presence and integrity of the transferred file. This ensures that the backup copy is indeed available and not corrupted.

# ssh username@remote_storage_server
# ls -lh /path/to/remote/backup/location/cacti_db_backup_YYYYMMDD_HHMMSS.sql.gz
# ls -lh /path/to/remote/backup/location/cacti_rrd_xml_YYYYMMDD_HHMMSS.tar.gz

---

Phase 3: Cacti Database and Application Restoration

Restoring Cacti involves more than just importing a database dump; it requires a meticulous process to bring all components back online, including the RRD files and various configuration elements. A clean, pre-configured Cacti installation on the target server is a fundamental prerequisite for restoration. This involves installing all necessary software components, including a compatible web server, PHP with all required modules, RRDTool, and MariaDB. It is crucial that the version of Cacti installed on the new server matches the version of the original Cacti server to ensure full compatibility and prevent unforeseen issues. The recommendation to set up the new Cacti server before stopping the old one, if possible, is a crucial operational strategy. This allows for parallel setup and testing, significantly reducing the actual downtime window during the cutover. By pre-staging the new server, the migration window (when the poller is stopped and data is transferred) is drastically shortened, directly improving the Recovery Time Objective (RTO) and minimizing service interruption. Furthermore, this approach serves as a risk mitigation strategy: if unforeseen issues arise with the new server setup, the old Cacti instance remains operational, providing a fallback and preventing a complete loss of monitoring. During the setup of the new MariaDB instance, ensure that the Cacti database user is created with the necessary privileges, including SELECT access on the mysql.time_zone_name table, and that this table is populated with time zone information. Crucially, the Cacti poller must remain disabled on the new server until all restoration steps are fully completed and verified.

Step-by-Step: Restoring the Cacti MariaDB Database

(i) Create the Cacti Database and User on the Target Server: If the database does not exist or needs to be recreated (e.g., for a clean slate restoration), execute the following commands. Replace cactiuser and your_cacti_db_password with the actual credentials used on the original server.

# mysql -u root -p
CREATE DATABASE cacti;
CREATE USER 'cactiuser'@'localhost' IDENTIFIED BY 'your_cacti_db_password';
GRANT ALL PRIVILEGES ON cacti.* TO 'cactiuser'@'localhost';
GRANT SELECT ON mysql.time_zone_name TO 'cactiuser'@'localhost';
FLUSH PRIVILEGES;
EXIT;

(ii) Transfer the Database Backup File to the Target Server: Use scp to retrieve the gzipped SQL dump from your remote storage to a temporary directory on the target Cacti server (e.g., /tmp/).

# scp username@remote_storage_server:/path/to/remote/backup/location/cacti_db_backup_YYYYMMDD_HHMMSS.sql.gz /tmp/

(iii) Decompress the Database Backup File:

# gzip -d /tmp/cacti_db_backup_YYYYMMDD_HHMMSS.sql.gz

(iv) Import the SQL Dump File into the Cacti Database: This command will populate the newly created or existing cacti database with all the configuration and metadata from your backup.

# mysql -u cactiuser -p'your_cacti_db_password' cacti < /tmp/cacti_db_backup_YYYYMMSS_HHMMSS.sql

(v) Verify Database Population (Optional but Recommended): A quick check can confirm that data has been imported successfully. Verify if the hosts are listed successfully.

# mysql -u cactiuser -p'your_cacti_db_password'
MariaDB [(none)]> USE cacti;
MariaDB [cacti]> SELECT description FROM host LIMIT 20; # Verify if hosts are listed
MariaDB [cacti]> EXIT;

Step-by-Step: Restoring Cacti RRD Files From XML Archive

The RRD files, which contain the actual time-series performance data, are restored from the XML archive created during the backup phase. This method ensures data integrity and compatibility across different environments.

(i) On the Target Cacti Server: Restore RRDs from XML Archive: Transfer the RRD XML Archive to the Target Cacti Server: Use scp to retrieve the compressed XML archive from your remote storage to a temporary directory on the target Cacti server (e.g., /tmp/).

# cd /tmp/
# scp username@remote_storage_server:/path/to/remote/backup/location/cacti_rrd_xml_YYYYMMDD_HHMMSS.tar.gz /tmp/

(ii) Copy the archive to the Cacti RRD directory and extract its contents:

# cp /tmp/cacti_rrd_xml_YYYYMMDD_HHMMSS.tar.gz /var/lib/cacti/rra/
# cd /var/lib/cacti/rra/
# tar zxvf cacti_rrd_xml_YYYYMMDD_HHMMSS.tar.gz

(iii) Restore the RRD files from the extracted XML files using rrdtool restore:

for i in $(find . -name "*.xml"); do 
  out_file="$(echo "$i" | sed 's/.xml$//')"
  echo "Restoring $i to $out_file"
  rrdtool restore "$i" "$out_file"
done

(iv) Clean up temporary files:

# cd /var/lib/cacti/rra/
# rm -rf cacti_rrd_xml_YYYYMMDD_HHMMSS.tar.gz
# rm -rf /var/lib/cacti/rra/*.xml 

(v) Change Ownership of RRD Files: It is crucial to ensure that the Cacti poller user (e.g., www-data, apache, or cactiuser depending on your setup) has appropriate read and write permissions to the /var/lib/cacti/rra/ directory and its contents. Incorrect permissions are a common cause of poller failures post-restoration.

# chown -R apache:apache /var/lib/cacti/rra/ # Adjust user/group as per your Cacti installation

Step-by-Step: Restoring Cacti Configuration Files and Custom Assets

(i) Transfer and Restore config.php: First, back up the default config.php file on the new server, and then transfer your backed-up config.php from remote storage to /var/www/html/cacti/include/ or /var/lib/cacti/include or /etc/cacti/db.php depending on your cacti installation. Ensure its permissions are correct (e.g., chmod 644 config.php).

(ii) Transfer and Restore Custom Scripts, Templates, and Plugins: Copy any custom scripts (e.g., those from /var/www/html/cacti/scripts/) to their corresponding directory on the new server. Transfer custom XML templates. While some templates are stored in the database, others (especially custom ones) might be external. These can often be re-imported via the Cacti web interface (Console > Import Templates). Restore any custom plugins to their respective installation directories.

(iii) Restore Web Server and PHP Configuration (if applicable): If the original Cacti server had specific Apache/Nginx or PHP configurations tailored for Cacti (e.g., virtual host definitions, PHP memory limits), transfer and apply these configurations to the new server. After applying any web server or PHP configuration changes, restart the web server service for the changes to take effect.

(iv) Restore Cacti Cronjob: Re-create the Cacti poller cron entry in /etc/cron.d/cacti or verify the systemd service configuration (cactid.service) on the new server. Ensure the path to poller.php (or spine) is correct for the new Cacti installation.

---

Phase 4: Post-Recovery Verification and Activation

The restoration process is not complete until the Cacti instance is fully verified for functionality and actively collecting new data. A methodical approach to post-recovery checks is essential to ensure operational readiness.

Step-by-Step: Verifying Database Integrity and Cacti Configuration

(i) Database Connectivity: Confirm that Cacti can successfully connect to the MariaDB database using the credentials specified in the config.php file.

(ii) Cacti Web Interface Access: Attempt to log in to the Cacti web interface (typically at http://your_cacti_server_ip/cacti). Verify that user accounts and their permissions are correctly restored.

(iii) Configuration Check: Navigate to Console > Settings within the Cacti web interface. Systematically verify that all paths (e.g., RRDTool Binary Path, PHP Binary Path, Poller Path, RRDTool Defaults) are correctly set for the new server environment.

(iv) Host and Graph Definition Verification: Check Console > Devices to ensure all previously monitored hosts are listed. Browse through Graphs to confirm that graph trees and definitions are present and appear as expected.

(v) Check RRD File Permissions: It is critical to verify that the Cacti poller user (e.g., www-data, apache, or cactiuser) has the necessary read and write permissions to the /var/lib/cacti/rra/ directory and its contents. Incorrect permissions are a frequent cause of poller failures post-restoration.

# ls -l /var/lib/cacti/rra/

(vi) Initial Graph Population: After the poller is re-enabled, monitor the graphs for the appearance of new data points. A key consideration for administrators is that it may take some time (e.g., 15 minutes or more) for graphs to start populating with fresh data. In a high-stress disaster recovery situation, not seeing graphs populate immediately can trigger panic and lead to unnecessary, potentially disruptive troubleshooting steps. This understanding promotes a methodical, calm approach to post-recovery verification, allowing focus on actual issues rather than perceived ones.

(vii) Re-enable the Cacti Poller and Services: Once all components are restored, verified, and permissions are correctly set, the Cacti poller can be re-enabled. For cron-based poller, uncomment the Cacti poller line in /etc/cron.d/cacti. For systemd-based poller, start and enable the Cacti poller service. Restart the web server (Apache/Nginx) if any configuration changes were made during the restoration process, to ensure all changes are loaded.

Step-by-Step: Initial Troubleshooting Steps Incase of any Cacti Issues

If Cacti does not function as expected after restoration and poller activation, a systematic troubleshooting approach is necessary. The verification steps highlight that Cacti is not just a database application but a complex ecosystem where the database, RRD files, poller, and web server must all interact correctly. A break in any link (e.g., wrong RRD permissions, incorrect poller path) will prevent graph population, even if the database is perfectly restored. This underscores the need for a comprehensive “systems thinking” approach to disaster recovery.

(i) Check Cacti Log File: The Cacti log file is the primary source of information for troubleshooting. It is highly recommended to set the poller logging level to at least ‘Medium’ or ‘Debug’ (via Console > Settings > General > Poller Logging Level) for detailed diagnostic information. Typical Location: /var/log/cacti/cacti.log or similar.

(ii) Verify Cron Daemon: Ensure that the cron daemon is actively running on the server, as it is responsible for executing the poller script.

# sudo systemctl status crond

(iii) Poller Type Consistency: Confirm that the Cacti poller type configured in Console > Settings > Poller (either cmd.php or spine) matches what was used on the original server and is correctly installed and configured on the new server.

(iv) Database Permissions: Double-check the MariaDB user permissions for the Cacti database to ensure the poller and web interface have full access.

(v) Firewall Rules: Verify that necessary firewall rules are in place to allow communication between Cacti components (e.g., MariaDB port 3306) and between the Cacti server and its monitored devices (e.g., SNMP ports, ICMP).

---

Disaster recovery is not a static plan; it is a dynamic and continuous process of planning, implementation, and iterative refinement. A well-defined Method of Procedure, such as the one detailed here, provides the essential blueprint. However, the true value and effectiveness of this blueprint are realized only through consistent practice and rigorous testing. Regular testing and monitoring are vital to identify potential issues and assess the effectiveness of any disaster recovery plan. In a high-stress disaster scenario, even the most meticulously documented procedures can be challenging to execute flawlessly. This is where the human element becomes critical. Drills provide the necessary practice, building muscle memory and reducing the likelihood of errors under pressure. This practice transforms a theoretical plan into a practical, executable capability. Organizations that regularly drill their DR plans demonstrate a higher level of operational maturity, which not only mitigates risks but also enhances their ability to adapt to unforeseen disruptions.

Furthermore, drills serve as an invaluable feedback mechanism for continuous improvement. They expose weaknesses in the plan, reveal unexpected challenges, and highlight outdated procedures, allowing for iterative refinements to the MOP, the tools used, and team coordination. It is imperative that systems administrators make disaster recovery drills a fundamental, non-negotiable part of their operational workflow. Do not let your critical network monitoring capabilities go dark precisely when they are needed most. Schedule these drills, execute them with discipline, learn from every outcome, and relentlessly refine your organization’s resilience. Your business uptime, data integrity, and reputation depend on this unwavering commitment to preparedness.

---