Using High Availability Capabilities to Make Migration of the Monitoring System Simple

The Need for Data Migration

Monitoring platforms such as eG Enterprise collect large numbers of metrics and data points about the applications and infrastructure being monitored. As the complexity of the applications, the number of tiers and the scale of the infrastructure grows, so do the number of metrics that need to be analyzed. Even in a mid-sized IT infrastructure, there may be over 100s of thousands of metrics collected and analyzed over time.

Another key variable in determining the volume of monitoring data stored is the data retention period. Do you want the raw metrics stored for 6 weeks, 3 months or 1 year? The greater the retention period, the larger the database used to store data by the monitoring tool.

From time to time, organizations deploying the monitoring tool may have to migrate the tool – to a different network, to a different data center, or even to a different cloud provider. Here are some reasons why organizations may be forced to or even proactively choose to migrate their monitoring tool deployment:

• The monitoring tool is deployed on-prem and they may look to move it to the cloud, as part of a drive to reduce on-premises infrastructure.
• The monitoring tool needs to be migrated from one cloud provider to another because the organization is consolidating systems with one provider.
• Migration of the monitoring tool can also happen from one cloud region to another – e.g., because a new data center in a new region geographically closer to the enterprise has become available and this offers better latency.
• The database of the monitoring tool may need to be migrated – from a VM-hosted model to a cloud managed database service model (e.g., from an Amazon AWS EC2 instance and to AWS RDS, see How to choose between AWS RDS and EC2 Hosted Database?).
• Migration may also be required to move to a modern operating system or database version. For example, Microsoft SQL standard edition did not support database partitioning until SQL server 2016 (prior to 2016 only Enterprise version supported partitions and the database had been configured without partitions). With the new version and improved use of database partitioning, performance of the monitoring tool could significantly improve.
• Migrations may also be forced for scalability – e.g., the physical server the monitoring tool is installed on does not have the capacity to support additional workloads. Migration to a larger system may be required to handle the increased workload.

Data Migration Can be Time Consuming

One of our managed service provider partners had a need to migrate their deployment of eG Enterprise from a hosted database to a cloud managed database service. During the process, they were also looking to migrate the operating system of the VM hosting the eG manager as well. With many monitoring platforms, a migration like this would have involved reinstalling the manager and database on the new servers/cloud environment and then manually migrating the data from the old to the new platform. Manually migrating several GBs of data from one database to another is time consuming. Furthermore, since the eG Enterprise manager makes use of database partitioning techniques to make database access fast and efficient, this would have required that a database admin (DBA) spend several days to perform the migration. Instead of this manual approach, we used the eG redundant manager configuration to perform the migration without requiring extensive manual effort during the migration.

What is eG Enterprise Redundancy?

eG Enterprise includes a manager redundancy option for ensuring high availability of the monitoring service. In this configuration, two eG Enterprise managers are configured in a redundant cluster. Agents can report to either manager and data is replicated to the other manager in the cluster in real-time. Admins can login to either manager in the cluster and track the current state of the infrastructure being monitored. Since redundancy is implemented at the application level, in this configuration, no special hardware is required. Additionally, the managers can even be geographically distributed – e.g., in different AWS availability zones.

If one manager in the cluster goes down or is not available for any reason, the other manager in the cluster takes responsibility for the monitoring. Any metrics collected during this period can be saved and uploaded to the other manager when it comes back up again.

While customers typically deploy the manager redundancy for ensuring that their monitoring service is available 24×7, we used this configuration in an interesting way for migrating the customer’s eG Enterprise deployment.

Leveraging eG Enterprise Redundancy for Hassle Free Migrations

To migrate the customer’s eG Enterprise deployment, the following steps were performed:

• A second manager instance with its own database was configured. This manager was configured on the new servers and was set up to use the new managed database instance.
• Manager redundancy was configured so live data from the original eG manager was replicated automatically to the new eG manager. Configurations were automatically synchronized across both managers without needing any manual effort.
• The redundant set up was left running for the period for which the customer needed data to be retained (e.g., 6 weeks). During this time, the new manager’s database was automatically populated with metrics and appropriate partitions for data storage were created automatically to facilitate time-based data purging in the future.
• At the end of this period, the redundant manager configuration was removed, the initial manager was decommissioned and the new eG manager took over all the monitoring load.

This novel use of our eG redundancy configuration highlights how organizations can migrate their eG Enterprise deployment easily without requiring several man days of administrative effort. This same process can be employed by any organization that requires to move their on-prem monitoring solution to the cloud, or to migrate from one cloud provider to another.

For customers with large historical databases and long data retention times, options to back-up the database and migrate with a short maintenance window may offer better value in terms of timescales and storage / data costs.

Further Information

• Our SaaS offerings are currently available in several geographic AWS regions, including Australia, Singapore, Europe (Germany), and USA; enabling customers to comply with government and industry regulatory requirements, such as the European GDPR, and the Australian Privacy Act. Read more: Deploying eG Enterprise Monitoring – SaaS or On-Premises (eginnovations.com)
• You can read more about ensuring your monitoring is resistant and robust to failure in: Data Center and IT Infrastructure Redundancy Management (eginnovations.com)
• More information on our solutions for MSPs hosting multi-tenant environments is available on our MSP Solutions page.
• AWS Database Types – Aurora vs DynamoDB vs RDS – How do they compare? (eginnovations.com)
• How to choose between AWS RDS and EC2 Hosted Database? (eginnovations.com)