Thoughts on…

Java Middleware & Systems Management

GWT Performance Monitoring

with 12 comments

Profiling a web application for performance bottlenecks requires a balance between focusing on the minutia and taking a step back and looking at the bigger picture. Since nearly all software projects deal with time constraints, a team must wisely choose where to spend its efforts tweaking an application to yield the best experience for its end users.

However, before diving into the code looking for optimizations, there are a few key things that one should understand about the performance profile of an application: Critical Path, Outliers, Variances, and Environment.

Critical Path

In a typical user session, which are the most frequented pages and/or business services? Determining these can help generate a prioritized list of services requiring optimization. For instance, decreasing the running time of a service that is called 50 times during a typical browsing experience is likely to yield significantly more perceived benefit than it would if the service were only called once.


It may be the case that one service takes significantly more time to load than any other service on the page. If any of those other services were written in such a way where their execution is contingent on the data returned by the slow service, then the effects of the slow service become magnified. Paying attention to outliers will help one understand whether the perceived performance of one or more pages can be significantly improved by the optimization of just a single service.


Typically, a service will not execute in the exact same number of milliseconds each each time it is invoked, but it will execute within some tolerance, e.g. 1100-1300ms 98% of the time. Tracking these variances will help determine whether some service will cause random lags in your application, that may be otherwise difficult to track down because they occur sporadically. A service that will take 5 times as long to execute in the worst case may indicate the need to rework that implementation.


Often times programmers are tempted to introduce optimizations because of something they saw on their local box. However, these development environments often differ significantly from how an application will perform once it’s deployed to production or shipped to a customer. Application load is another important factor to consider; an application with one logged-in user will not necessarily have the same bottleneck(s) as it would if there were dozens, hundreds, or thousands of logged-in users. Thus, one must consider where an application might be deployed, and use a similar test environment to profile an application for performance issues.

With these principles in mind, I wrote an extension to the GWT client-side RPC mechanism to capture data that will assist in making better informed decisions about which services need optimization. The first thing we have to do is inject a custom ProxyGenerator into the GWT compilation process.

<generate-with class="...TrackingServiceInterfaceProxyGenerator">
    <when-type-assignable class=""/>

The implementation for that custom generator is as follows:

package org.rhq.enterprise.gui.coregui.user.rebind.rpc;


public class TrackingServiceInterfaceProxyGenerator extends ServiceInterfaceProxyGenerator {
    protected ProxyCreator createProxyCreator(JClassType remoteService) {
        return new TrackingProxyCreator(remoteService);

I sub-classed the GWT default ServiceInterfaceProxyGenerator implementation so that the only thing I had to do is override the one method that returns the ProxyCreator. In this case, it returns a custom implementation that will be responsible for reassigning the super class of all generated proxies:

package org.rhq.enterprise.gui.coregui.user.rebind.rpc;

import org.rhq.enterprise.gui.coregui.client.util.rpc.TrackingRemoteServiceProxy;

public class TrackingProxyCreator extends ProxyCreator {
    public TrackingProxyCreator(JClassType type) {

    protected Class getProxySupertype() {
        return TrackingRemoteServiceProxy.class;

Note: make sure to keep your custom ProxyCreator and ProxyGenerator classes outside of the client source directory because these classes can not be cross-compiled to javascript. If you forget to do this and attempt to compile anyway, module validation will fail with a message like: “[ERROR] Line XX: No source code is available for type; did you forget to inherit a required module?”

The TrackingRemoteServiceProxy is responsible for the actual RPC instrumentation. This class will wrap the original asynchronous callback method in one that performs statistics tracking.

package org.rhq.enterprise.gui.coregui.client.util.rpc;


public class TrackingRemoteServiceProxy extends RemoteServiceProxy {
    protected  RequestCallback doCreateRequestCallback(ResponseReader reader, 
        String methodName, int invocationCount, AsyncCallback callback) {
        RequestCallback original = 
            super.doCreateRequestCallback(reader, methodName, invocationCount, callback);
        TrackingRequestCallback trackingCallback = 
            new TrackingRequestCallback(methodName, original);
        return trackingCallback;


The implementation of the TrackingRequestCallback is fairly straightforward – all it has to do is notify the class responsible for statistics tracking each time a response is received for any service, whether that invocation was successful or returned an error:

package org.rhq.enterprise.gui.coregui.client.util.rpc;


public class TrackingRequestCallback implements RequestCallback {

    private String name;
    private long start = System.currentTimeMillis();
    private RequestCallback callback;

    public TrackingRequestCallback(String name, RequestCallback callback) { = name;
        this.callback = callback;

    public void onError(Request request, Throwable exception) {
        RemoteServiceStatistics.record(getName(), getAge());
        callback.onError(request, exception);


    public void onResponseReceived(Request request, Response response) {
        RemoteServiceStatistics.record(getName(), getAge());
        callback.onResponseReceived(request, response);

    public String getName() {
        return name;

    public long getAge() {
        return System.currentTimeMillis() - start;

Finally we have the RemoteServiceStatistics singleton, which is responsible for keeping track of all invocations for all services across the application. When the statistics are queried, it can use this full audit trail of invocations to generate things like min/max/avg running times and standard deviations:

package org.rhq.enterprise.gui.coregui.client.util.rpc;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import org.rhq.enterprise.gui.coregui.client.util.rpc.RemoteServiceStatistics.Record.Summary;

public class RemoteServiceStatistics {

    public static class Record {
        private String serviceName;
        private String methodName;
        private List data = new ArrayList();

        private Record(String remoteService, long millis) {
            // remoteService format "{ServiceName}_Proxy.{MethodName}"
            this.serviceName = remoteService.substring(0, remoteService.indexOf("_Proxy"));
            this.methodName = remoteService.substring(remoteService.indexOf('.') + 1);

        private void record(long millis) {

        public Summary getSummary() {
            return new Summary(serviceName, methodName, data);

        public static class Summary {
            public final String serviceName;
            public final String methodName;
            public final int count;
            public final long slowest;
            public final long average;
            public final long fastest;
            public final long stddev;

            private Summary(String serviceName, String methodName, List data) {
                this.serviceName = serviceName;
                this.methodName = methodName;
                this.count = data.size();

                if (!data.isEmpty()) {
                    int i = 0;
                    long total = 0, slow = 0, fast = 0;
                    for (long next : data) {
                        if (i++ == 0) {
                            total = slow = fast = next;
                        } else {
                            total += next;
                            if (next > slow) {
                                slow = next;
                            } else if (next < fast) {
                                fast = next;
                    slowest = slow;
                    fastest = fast;
                    double avg = (total) / (double) count;

                    double sumOfSquares = 0;
                    for (long next : data) {
                        sumOfSquares += Math.pow(next - avg, 2);
                    stddev = (long) Math.pow(sumOfSquares / count, 0.5);
                    average = (long) avg;
                } else {
                    slowest = average = fastest = stddev = 0;


    private static Map statistics = new HashMap();
    private RemoteServiceStatistics() {
        // static access only

    public static void record(String remoteService, long millis) {
        Record record = statistics.get(remoteService);
        if (record == null) {
            record = new Record(remoteService, millis);
            statistics.put(remoteService, record);
        } else {

    public static Summary get(String remoteService) {
        Record stat = statistics.get(remoteService);
        if (stat == null) {
            stat = new Record(remoteService, 0);
        return stat.getSummary();

    public static List getAll() {
        List stats = new ArrayList();

        for (String remoteService : statistics.keySet()) {

        return stats;

I wrote a quick test page in the user interface to expose the statistics data. Below is a cropped summary of what you’ll see:

By putting the statistics into a table with sortable columns, it makes analyzing the data much simpler. Below is a quick list of the indicators that can be derived from the data set above:

  • ResourceBossGWTService.getInventorySummaryForLoggedInUser – high average and low variance implies often/always slow, look for possible improvements such as query rewrites, database indexes, or pre-computes.
  • AlertGWTService.findAlertCountByCriteria/findAlertsByCriteria – high variance, look for either possible database table contention issues that would lead to periodic query slowness or possibly eagerly/conditionally loaded data not necessary to render the view
  • TagGWTService.findTagReportComositesByCriteria – since this is on the critical path (invoked every time a user is looking at a resource / group / bundle / etc), must ensure page rendering isn’t contingent on this data; in other words, it should be OK to lazy load this data onto the view

After you have the prioritized list of services that might require optimization, I would then open up Speed Tracer to see just how these network figures relate to pure client-side figures such as parsing, layout, style calculations, timer firings, events, etc.

It may be the case that most of the perceived end-user time is spent taking the RPC results and rendering the DOM, not in the network request itself. If that’s the case, then Speed Tracer will very likely help you to pinpoint the exact client-side issue(s) you’re dealing with. On the other hand, if you determine that the bottleneck is on the server, you can follow the steps in my other blog entry on Hibernate Performance Monitoring to help get to the bottom of those issues.

Keep in mind, this solution is only meant to be a starting off point. It’s doesn’t tell you precisely why the services are slow, but does give you an indication of which are the worst offenders and how frequently they are invoked. One thing I’ll probably be implementing in the near future is payload response tracking. Tracking this data and correlating it with the rest of the remote service statistics would help distinguish between network slowness caused by slow queries on the server versus too much data being returned in the response to the client. I encourage you to experiment with this solution, and expose other statistics that you feel are useful in getting a quick overview of where your GWT-based application performance bottlenecks may lie.


Written by josephmarques

November 29, 2010 at 5:18 pm

Posted in gwt

Tagged with

12 Responses

Subscribe to comments with RSS.

  1. Nice – I look forward to giving this a try.

    Chris Lowe

    November 30, 2010 at 8:45 am

  2. hi there,

    nice article.

    the link to hibernate monitoring is broken. You need to remove the last 2 characters and then it will be OK.


    anjan bacchu

    November 30, 2010 at 1:50 pm

    • thanks. links are fixed.


      November 30, 2010 at 6:06 pm

  3. “You need to remove the last 2 characters and then it will be OK.”
    You can read more?


    December 19, 2010 at 10:06 pm

    • The ‘href’ attribute wasn’t properly terminated when copying / pasting from my local editor, so the links were initially broken. I edited the post and fixed both the “Speed Tracer” and “Hibernate Performance Monitoring” links.


      December 19, 2010 at 10:39 pm

  4. I will be giving this a try. The alternative approach I took is I extended an already excellent statistic collection tool called ‘Javamelody’ to support GWT-RPC and track based on usage ‘heavy hitters’, as well as monitor deviations/variances for any that are significant to get them more predictable (paginate results when they are not for example).

    The challenge I haven’t solved from a client-facing side (the users browser) is showing server time vs latency time, particularly when latency may include SSL/TLS encryption on many small ajax/GWT-RPC requests.



    February 15, 2011 at 2:22 pm

  5. I took this stuff and built upon it. I made it so you can put these stats in a Window component thus allowing you to watch the stats as you poke around the UI (the Window remains in the UI as you walk around the pages – very handy because you can now watch the stats live as they change based on what you click in the UI). I also added a “auto-refresh” option so you can refresh the stats table every N seconds or every time you navigate to a new page. Finally, I added a button that lets you see the data in a CSV text so you can cut-n-paste the data in a spreadsheet for later analysis.

    Very very useful when you need to perform performance analysis on your app.

    John Mazz

    February 25, 2011 at 4:37 pm

  6. Thanks dude, you saved me a lot of time!


    December 24, 2011 at 6:28 pm

  7. Hurrah! After all I got a webpage from where I be capable of actually obtain useful facts concerning my study and knowledge.

    trojan zbot

    May 2, 2013 at 6:06 pm

  8. Awesome! What license is this code under? Apache 2.0?


    May 14, 2013 at 12:05 pm

  9. Howdy! I know this is kind of off topic but I was wondering
    which blog platform are you using for this site?

    I’m getting fed up of WordPress because I’ve had issues with hackers and I’m looking at options for another platform. I would be awesome if you could point me in the direction of a good platform.


    June 15, 2013 at 6:31 pm

  10. Hello there, I found your site by means of Google while looking
    for a comparable matter, your website got here up,
    it appears to be like good. I have bookmarked it in
    my google bookmarks.
    Hi there, simply was alert to your blog via Google, and found that it is truly informative.

    I’m gonna watch out for brussels. I will appreciate should you proceed this in future. A lot of folks shall be benefited out of your writing. Cheers!

    kansas city royals mlb

    June 26, 2013 at 3:46 pm

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: