Welcome to LWN.net
The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider accepting the trial offer on the right. Thank you for visiting LWN.net!
Free trial subscription
Try LWN for free for 1 month: no payment or credit card required. Activate your trial subscription now and see why thousands of readers subscribe to LWN.net.
March 16, 2016
This article was contributed by Geert Jansen
ManageIQ is an open-source project that allows administrators to control and manage today’s diverse, heterogeneous environments that have many different cloud and container instances spread out all over the world. It can automatically discover these environments wherever they are running and bring them all under one management roof. Beyond that, it can simplify life for users by allowing them to choose new virtual machines (VMs) and containers and have them immediately "spun up" and available for use.
The first step in managing a complex environment is to discover what is actually there. ManageIQ does this by accessing the APIs of the virtualization systems, public clouds, and other management systems that make up the environment. Using the APIs it will download the lists of VMs, hypervisors, containers, networks, and whatever else is relevant to the system in question. All these "things" it discovers are called "managed elements" and are stored and tracked in the Virtual Management Database (VMDB). Currently the VMDB schema consists of over 200 entities and relationships. It defines elements such as "Virtual Machine" and "Hypervisor", ensures that a "Virtual Machine" has a "name" attribute, and that a "Virtual Machine" is related to a "Hypervisor" by a "runs on" relationship. The individual management systems are called "element managers" in ManageIQ parlance, and the pieces of code that connect to the APIs are called "Providers".
After initial discovery, ManageIQ uses the APIs to listen for events that might indicate a managed element has changed, and uses those to refresh the VMDB. The result is that the ManageIQ VMDB is almost always up to date with respect to what is actually present in the environment, even if changes are made outside of ManageIQ. Most of today’s APIs (but unfortunately not all) support these change notifications. In addition to the on-demand refresh, a full refresh is also scheduled every 24 hours.
The discovered inventory is visualized through the ManageIQ web interface, which shows all of the discovered elements and their relationships. For example, when used together with VMware, it will show a list of virtual machines, their attributes, the hypervisor they run on, the connected networks, etc. The inventory can also be visualized in reports, which can be scheduled and emailed, or displayed as dashboards.
One interesting thing about the VMDB is that it allows an abstract approach to management. The advantage of that was seen recently when support for containers was added. It involved extending the VMDB schema with elements including "Container" and "Pod", and creating a provider that connects to the container management system (Kubernetes in this case).
After discovery, ManageIQ provides for ongoing operational management. This covers quite a few disciplines, and we’ll look at the most important ones here below.
ManageIQ provides control actions for the things it manages. For example, VMs and Instances have "power on" and "power off" actions. Not every possible action is covered, but the goal is to expose the most common actions so that the usual day-to-day management can be completely done within ManageIQ.
Change management is another operational management discipline. ManageIQ can show reports of what attributes (e.g. memory, disks, network devices of a VM, or even installed software versions when using SmartState Analysis, which is described below) of an entity have changed, and when. Attributes can be compared across different objects of the same type, for example, to compare the state of a VM against a golden image. It can also compare the configuration of the same object against itself from an earlier time. This is called drift tracking.
A third discipline is capacity management. ManageIQ providers track various utilization metrics such as CPU, memory, and disk. These metrics can be visualized in charts, and aggregated to understand when capacity will run out. Modeling "what if" scenarios is possible as well.
Financial management is another area where ManageIQ can help operations staff. It can be used to create a cost model for elements that it discovers. For example, a certain cost can be allocated to VM memory and disk. Reports can then be generated to show the total cost of the various groups in the system.
Self-service allows an administrator to maintain a catalog of requests that can be ordered by regular users, for example, to provision a single VM or an application stack. Self-service is good for both the administrator and the end user: it saves a lot of time for the administrator while end users get their service going much faster.
Self-service is one of the more powerful use cases of ManageIQ. It starts with an administrator creating a "service bundle," which is a collection of "service items". Each service item is a "thing" that ManageIQ knows how to create, for example, a VM or a container. There is also a generic service item that can call into the ManageIQ workflow engine (more on that below), and can be used to provision arbitrary things by invoking arbitrary actions. The order in which items in a bundle are provisioned is specified by the administrator.
Services typically require some amount of input. For example, if the request is to provision a VM, then a typical question would be the size of the memory and the disk. This information can be requested from the user through a dialog, which can be created using a built-in dialog editor.
Once the service bundle and the dialog are created, they need to be associated with an "entry point" in the ManageIQ workflow engine (called "Automate"). The entry point defines the process to provision the bundle. There is a default entry point to provision bundles, but this entry point can be changed so that custom logic can be invoked. Workflows are Ruby-based and can be edited through a built-in integrated development environment. (Many actions in ManageIQ are actually workflows in Automate; they can be inspected and also modified by the administrator.) Aside from provisioning, the workflow engine is also used to run an approval process before the provisioning takes place.
With the bundle definition, dialog, and entry point, the request can be published in a service catalog, which then enables users to order the service.
Once a service is deployed, the user will see it under the "Services" tab in the web interface. While a service is operational, a user can interact with it. For example, if configured to do so, ManageIQ will allow a user to start and stop VMs comprising the service, or to get a console for them. Custom actions can also be created by adding menu items that can be connected to entry points in Automate. An example of a custom action would be to backup a service, or to run it on more nodes (i.e. scale it out).
The self-service model in ManageIQ also includes a process for termination. The administrator can specify a lifetime for the service. Once the lifetime has expired, the service can (optionally) be decommissioned automatically. Users can be given the privilege to extend the lifetime and to get warnings about upcoming expirations via email.
ManageIQ allows administrators to define compliance policies and apply those against elements that are discovered. This is especially useful when users are deploying their own systems through self-service as it gives a certain amount of control back to the administrator.
Compliance policies consist of a number of rules, and are enforced by the ManageIQ policy engine, which is called "Control". The policy engine is modeled on the Event-Condition-Action model. If a certain event happens, a condition is evaluated, which, if true, results in an action. For compliance purposes, the event is usually "element discovered" or "element updated", the conditions are the sets of rules to enforce, and the action is "update compliance status", optionally combined with an automated remediation workflow in Automate. Control can also be used to invoke automation based on any type of event. For example, a high load can trigger a scale-out action.
A nice thing about compliance in ManageIQ is that it works on more than just the metadata of the items discovered through the various APIs. It is also possible to define rules for the contents of VMs, hypervisors, and containers. Extracting these contents is done by a process called SmartState Analysis (SSA).
SSA can discover configuration files, event logs, and package databases; it stores that information in the VMDB. Interestingly, SmartState is a fully agent-less technology. It works by accessing the disks remotely over platform-specific APIs, usually snapshot and/or backup APIs. As the disks are untrusted and potentially concurrently updated, they cannot be safely mounted by a Linux kernel. To get around this, ManageIQ contains Ruby-based read-only filesystem and volume manager implementations that access the disks from user space.
The benefits of the agent-less approach is that it doesn’t require cooperative guests, which means that it also works with VMs that are deployed through self-service, vendor provided "black box" appliances, or VMs that predate the implementation of the cloud management platform. Another benefit of being agent-less is that it also works for VMs that are shut down.
SmartState can give a lot of insight into the environment. A nice example of a compliance policy based on data from SmartState is this policy that checks if a Red Hat operating system is vulnerable to the recently discovered DROWN attack .
ManageIQ ships with a number of providers that are listed below. If there is a commercial variant of an open-source project it is given in parentheses.
- VMware vSphere
- oVirt (Red Hat Enterprise Virtualization)
- OpenStack and TripleO (Red Hat Enterprise Linux OpenStack Platform)
- Microsoft System Center Virtual Machine Manager
- Amazon Web Services
- Microsoft Azure
- Kubernetes (Red Hat Enterprise Linux Atomic and Red Hat OpenShift)
- The Foreman and Katello (Red Hat Satellite)
Other providers are in progress in the master branch, such as providers for the Google Cloud Platform, Ansible Tower, and software-defined networking.
ManageIQ is developed by the ManageIQ community. Development happens on GitHub using a pull-request-based development model. Discussions of development topics happens on Gitter , and users interact with each other on the talk.manageiq.org forum. ManageIQ is available under the Apache 2 license.
Despite its young age as an open-source project, ManageIQ has a large and mature code base. The code for ManageIQ was originally developed by ManageIQ Inc., starting in 2006. This company was acquired by Red Hat in December 2012, which released the ManageIQ code in June 2014. The code base weighs in at over 200,000 lines of code excluding tests and gem -ified components, and is written in the Ruby on Rails framework.
Since being released as open source, almost 6,000 requests have been merged, by over 100 contributors. While the majority of contributions are from Red Hat staff, the project is actively growing and seeking outside contributions. Recently, companies like Booz Allen Hamilton, Produban/Banco Santander, and Google have been making contributions to the project.
Releases happen approximately every 6 months and are named after chess grandmasters. Most recently the project made its third release named "Capablanca." ManageIQ is also the upstream project for the Red Hat CloudForms product.
The project also holds an annual design summit where users and developers from all over the world come together to exchange ideas and establish the development roadmap. The second ever design summit will be held June 6-7 in Mahwah, NJ .
ManageIQ is distributed as a Linux-based virtual appliance that is a little over 1GB in size. After downloading it and importing it into a supported solution (e.g. QEMU/KVM on Linux), the web interface will start up. The first task is to configure a provider by connecting to an element manager such as oVirt, OpenStack, or Amazon Web Services, then waiting a couple of minutes for discovery to complete. The steps are documented in the ManageIQ documentation , with video walkthroughs of the Top Tasks . After the basic inventory has been discovered, it is possible to create custom dashboards, define offerings for self-service, or create compliance policies.
ManageIQ is a big project with a lot of features. It is quite powerful, and is even fun to use. It is best to start simply by focusing on a single objective, for example self-service or reporting. If you get stuck, or even just want to say hello, please contact the community at the talk.manageiq.org forum.
[Geert Jansen is the manager of the CloudForms product at Red Hat.]
to post comments)