Thoughts on…

When you first download and install RHQ, you’ll log in to the web console and notice that there are two different types of grouping constructs for resources – mixed and compatible. In short, compatible groups must contain the same types of resources, whereas mixed groups do not. Under the covers, these are implemented by the exact same construct, but how meaning has been applied to them, and what you can do with each of them, is why this blog got the title it did.

Mixed groups are predominantly used for security, in particular, authorization. With them you can put all sorts of resources together – Windows and Linux platforms, IIS and Apache servers, etc. Then, you can attach that mixed group to a role, and any users in that role will be able to see those resources.

If you want to be able to give someone access to an entire box, then create a mixed group with the “recursive” option enabled. By turning that option on, any resource you add to the group automatically adds all descendant resources to the group as well. For instance, if you add a platform, it will indirectly add all servers under that platform, as well as all services under all of those servers, and so on.

While mixed groups have one thing they’re good at, compatible groups have an array of functionality they excel at providing. First and foremost is their “compatibility” with all of the other subsystems RHQ provides: monitoring, configuration, operations, etc.

For monitoring, RHQ shows aggregate and average metrics across the group members. For configuration, RHQ enables you to change the configured connection properties across everybody in the group at the same time. For operations, RHQ allows you to execute the same operation against all resources in the group – at the same time, or serially (one after the other, in rolling fashion).

Very recently, a customer pointed out to me how groups – mixed and compatible – can be used in a novel way. Their question was simple: what’s the easiest method to see all of the resources in their environment that are down?

In order to do this today, you have to use the Browse Resources page, go to the each tab in turn – platforms, servers, and services – and sort on the availability column. Granted, this is fairly easy to do and doesn’t take all that long, but wouldn’t it be nice to be able to automatically create a group that contained any and all resources that were down in the system.

OK, maybe you’re initial thought is “why not just use the Problem Resources portlet?” Well, a ‘problem’ resource isn’t necessarily one that is down. If you have ANY alerts, or if you have metrics that are more than 5% outside of their baseline range (a running average calculated over time automatically by RHQ), the resource will also show up in this portlet. This customer JUST wanted the unavailable resources.

Alright, and maybe your second thought was “well, why not use alerts?” Today, we can fire alerts when a resource goes down, and you CAN use the notification mechanism so that you get an email when this happens. However, there are at least two problems with this strategy:

Problem 1

Alerts are only good at telling you what JUST happened in the system. Alerts will be created as the result of some agent sending data up to the server, such as an availability report or the results of an operation. So, if you already have resources that are down before you set up your alert definitions, you will not be notified because those resources were already down.

Problem 2

Setting up availability alerts across ALL resources in the system will take a while. A lot of time could be saved by using the alert templates feature (Administration > Monitoring Defaults), which would make sure that all existing resources (and any resources that are imported in the future) automatically have alert definitions created for them. However, you’d still have to set up one template across every single resource type in the system, and so depending on how many plugins you have installed could be several dozen templates to create. Also, for each of those alert templates, you’d have to setup identical notification rules too, which takes more time still.

Interestingly enough, before I could even reply to the customer, they suggested a solution – a feature enhancement, to be precise – which would do the trick. They wanted to extend DynaGroups to be able to aggregate resources by availability.

I was floored by the simplicity of this suggestion. In fact, I sort of recall rubbing my eyes looking to wake up from a dream, because I thought it was so incredible that the development team hadn’t thought of this before. And I wasted no time creating the issue in JIRA to track this request.

Anyone that knows me probably already guessed I had the fix locally within an hour, but because the request came in during the final seconds just before the 1.1 release I held off on committing it. Though, as soon as SVN was unlocked for 1.2 development it was one of the first commits.

If you’re building off of trunk (or running anything rev1730 or greater), it’s easy to create a Group Definition that will always keep a DynaGroup populated with the resources that are unavailable.

resource.availability = DOWN

But let’s say you are monitoring a very large inventory, and want to break things down further to keep the groups more granular. For example, let’s say you wanted to create different DynaGroups for each type of resource that’s down. This way you can look at your IIS servers that have failed, independent from your Apache vhosts that aren’t up, separate from your File Systems that aren’t at their expected mount points. That expression set would be as follows:

resource.availability = DOWN
groupby resource.type.plugin
groupby resource.type.name

But maybe that creates too many groups, or gives you results for resource types you aren’t interested in. Let’s say you want to focus your search because you only care about one specific type of resource failing, maybe just your Apache servers. Instead of grouping by the plugin and resource type, specify those pieces of information exactly:

resource.availability = DOWN
resource.type.plugin = Apache
resource.type.name = Apache HTTP Server

Thus, in a roundabout way, resources groups can actually be used as indirect tools for monitoring the health of your platforms, servers, and services.

This, however, just scratches the surface in terms of how groups can be used to monitor your enterprise. One major focus for the 1.2 release of RHQ is going to be on cluster management. Remember, compatible groups serve as a natural way of exposing RHQ subsystems at the group-level. So expect to see lots of new group-level services and UI functionality.

At the time of this writing, the requirements for cluster support were in their infancy, but we encourage you to read the latest requirements and post your ideas back to the resource clustering thread in the forums.

For developers that know multiple programming languages, that can understand Java generics, who’ve play around with ANTLR, that have read about closures, and who generally automate a lot of their day-to-day work through various scripting languages, picking up a new syntax is a piece of cake. But this is *certainly* not the case for everyone.

If a new feature in a software product, for example, is driven through familiar controls such as drop-down menus, checkboxes, and input text fields, then chances are pretty high that they’ll figure out how to use it naturally (or eventually by way of trial and error). However, if you were to provide a text-only interface that required knowing some not-so-obvious syntax, the user has no choice than to turn to the manual, and continually reference it for details, pointers, guidelines, and grammar.

As in most things, “know your audience” comes into play here too. It is absolutely crucial to take into consideration who your user population is when deciding how to provide interfaces for new software functionality. The wrong interface, no matter how great the feature, could render it absolutely useless to some. And, as experience dictates, first impressions tend to stay with those users even in the face of improved controls in newer versions of the product.

Despite having knowledge of all of this, yours truly (due to time constraints in the development cycle) had to introduce a less then ideal interface to interact with what is otherwise a rock-solid and incredibly powerful feature. Prior to the 1.1.0 release of RHQ, the group definition / dynagroups feature could only be manipulated via a raw syntax.

Not only did this make using the feature cumbersome (even for skilled users), but it made for a rather high learning curve. I had an inclination of this as development was commencing. I sort of knew this as QA was underway. And I definitively knew this when it came time to write the end user documentation. Despite having this knowledge, there simply wasn’t enough time in the cycle to provide a better interface.

Well, as expected, we took flak for it through our customer support channels. More than once did people have trouble understanding what a group definition was, and balked at using the feature. Some of these were customers that had read all of the provided documentation, but still didn’t quite understand how to use it properly. It just goes to show you that no matter how great you think a feature is, the interface to that feature is so crucial to it penetrating your user community and becoming a tool that lessens their burden, as opposed to another “system” they have to learn.

Luckily, a solid support team coupled with a rather understanding and forgiving customer base helped to eventually smooth things over. The more and more examples we threw their way, the better they understood what group definitions were trying to accomplish. In fact, two customers eventually liked it so much that they started suggesting improvements for it, so as to make dynagroups even more useful in the future.

So, perhaps we got lucky. Maybe the utility of the feature, in this case, outweighed the annoyance of having to learn its syntax. The feature *was* introduced to automate a large portion of what was once dozens and dozens of hours of manual work. But providing a poor interface just because we can get away with it doesn’t make it right, and it doesn’t give us an excuse to sit on our hands either.

To remedy this, we made providing a newer and better interface for group definitions / dynagroups a priority in the 1.1.0 release of RHQ. As a result, it was shipped with a wizard formally titled the DynaGroup Expression Builder. Now when you create a new group definition (or edit an existing one) you’ll see a little icon at the upper right-hand portion of the input box. Click it. You’ll find it provides all of the familiar controls we should have started with in the first place: drop-down menus, checkboxes, and input text fields. Several non-developers used the interface before it was released, and commented on how much easier it was to understand.

So does this mean, assuming you’ve already gotten used to the raw syntax from pre-1.1.0 versions, that you’ll now need to learn yet another new interface? Nope. We’ve left the old interface there for two reasons: 1) backwards compatibility (for those already familiar with the syntax, as just discussed) and 2) to provide an expert interface. See, we feel that once you use the expression builder enough times, you’re going to learn the syntax naturally by example and, once you know it sufficiently well, you might want a more direct interface instead of being forced to use the wizard every time.

With the newer and better interface firmly in place, there’s no reason NOT to use dynagroups. So try ‘em out, and let us know what you think, either through the customer support portal, a user thread in the forums, a comment on a JIRA, or even (as this author would prefer) a post back to this blog.

* For more information about the Dynagroups feature, please visit the docs.
* RHQ is an extensible, open source management and monitoring infrastructure. For more information see the project site.

Here’s an interesting use case for DynaGroups.

One customer might have a few dozen/hundred JBoss AS instances, but only a handful of them have a web app deployed called, say, “someapp.war”. They want to know what set of expressions they could use to maintain a group of all appservers that have that webapp deployed.

This wasn’t possible in the 1.0.0 version of the platform, so in the upcoming 1.0.1 release of RHQ you will see DynaGroups support a new “child” expression token. Using it, you can easily satisfy the above use case. The expression set looks like:

resource.type.plugin = JBossAS
resource.type.name = JBossAS Server
resource.child.name.contains = someapp

Up until now, RHQ only supported “resource.parent” and “resource.grandParent” sub-expressions, which meant that you could only traverse up the resource hierarchy. This new expression token really helps to round out hierarchical navigation using expressions, allowing traversal in either direction.

If you have suggestions for other DynaGroups features, don’t hesitate to postback to the RHQ development forums here — http://forums.rhq-project.org/