Thoughts on…

Java Middleware & Systems Management

Archive for November 2010

GWT Performance Monitoring

with 12 comments

Profiling a web application for performance bottlenecks requires a balance between focusing on the minutia and taking a step back and looking at the bigger picture. Since nearly all software projects deal with time constraints, a team must wisely choose where to spend its efforts tweaking an application to yield the best experience for its end users.

However, before diving into the code looking for optimizations, there are a few key things that one should understand about the performance profile of an application: Critical Path, Outliers, Variances, and Environment.

Critical Path

In a typical user session, which are the most frequented pages and/or business services? Determining these can help generate a prioritized list of services requiring optimization. For instance, decreasing the running time of a service that is called 50 times during a typical browsing experience is likely to yield significantly more perceived benefit than it would if the service were only called once.

Outliers

It may be the case that one service takes significantly more time to load than any other service on the page. If any of those other services were written in such a way where their execution is contingent on the data returned by the slow service, then the effects of the slow service become magnified. Paying attention to outliers will help one understand whether the perceived performance of one or more pages can be significantly improved by the optimization of just a single service.

Variances

Typically, a service will not execute in the exact same number of milliseconds each each time it is invoked, but it will execute within some tolerance, e.g. 1100-1300ms 98% of the time. Tracking these variances will help determine whether some service will cause random lags in your application, that may be otherwise difficult to track down because they occur sporadically. A service that will take 5 times as long to execute in the worst case may indicate the need to rework that implementation.

Environment

Often times programmers are tempted to introduce optimizations because of something they saw on their local box. However, these development environments often differ significantly from how an application will perform once it’s deployed to production or shipped to a customer. Application load is another important factor to consider; an application with one logged-in user will not necessarily have the same bottleneck(s) as it would if there were dozens, hundreds, or thousands of logged-in users. Thus, one must consider where an application might be deployed, and use a similar test environment to profile an application for performance issues.


With these principles in mind, I wrote an extension to the GWT client-side RPC mechanism to capture data that will assist in making better informed decisions about which services need optimization. The first thing we have to do is inject a custom ProxyGenerator into the GWT compilation process.

<generate-with class="...TrackingServiceInterfaceProxyGenerator">
    <when-type-assignable class="com.google.gwt.user.client.rpc.RemoteService"/>
</generate-with>    

The implementation for that custom generator is as follows:

package org.rhq.enterprise.gui.coregui.user.rebind.rpc;

import com.google.gwt.core.ext.typeinfo.JClassType;
import com.google.gwt.user.rebind.rpc.ProxyCreator;
import com.google.gwt.user.rebind.rpc.ServiceInterfaceProxyGenerator;

public class TrackingServiceInterfaceProxyGenerator extends ServiceInterfaceProxyGenerator {
    @Override
    protected ProxyCreator createProxyCreator(JClassType remoteService) {
        return new TrackingProxyCreator(remoteService);
    }
}

I sub-classed the GWT default ServiceInterfaceProxyGenerator implementation so that the only thing I had to do is override the one method that returns the ProxyCreator. In this case, it returns a custom implementation that will be responsible for reassigning the super class of all generated proxies:

package org.rhq.enterprise.gui.coregui.user.rebind.rpc;

import com.google.gwt.core.ext.typeinfo.JClassType;
import com.google.gwt.user.client.rpc.RemoteService;
import com.google.gwt.user.client.rpc.impl.RemoteServiceProxy;
import org.rhq.enterprise.gui.coregui.client.util.rpc.TrackingRemoteServiceProxy;

public class TrackingProxyCreator extends ProxyCreator {
    public TrackingProxyCreator(JClassType type) {
        super(type);
    }

    @Override
    protected Class getProxySupertype() {
        return TrackingRemoteServiceProxy.class;
    }
}

Note: make sure to keep your custom ProxyCreator and ProxyGenerator classes outside of the client source directory because these classes can not be cross-compiled to javascript. If you forget to do this and attempt to compile anyway, module validation will fail with a message like: “[ERROR] Line XX: No source code is available for type com.google.gwt.user.rebind.rpc.ProxyCreator; did you forget to inherit a required module?”

The TrackingRemoteServiceProxy is responsible for the actual RPC instrumentation. This class will wrap the original asynchronous callback method in one that performs statistics tracking.

package org.rhq.enterprise.gui.coregui.client.util.rpc;

import com.google.gwt.http.client.RequestCallback;
import com.google.gwt.user.client.rpc.AsyncCallback;
import com.google.gwt.user.client.rpc.impl.RemoteServiceProxy;
import com.google.gwt.user.client.rpc.impl.Serializer;
import com.google.gwt.user.client.rpc.impl.RequestCallbackAdapter.ResponseReader;

public class TrackingRemoteServiceProxy extends RemoteServiceProxy {
    ...
    @Override
    protected  RequestCallback doCreateRequestCallback(ResponseReader reader, 
        String methodName, int invocationCount, AsyncCallback callback) {
        RequestCallback original = 
            super.doCreateRequestCallback(reader, methodName, invocationCount, callback);
        TrackingRequestCallback trackingCallback = 
            new TrackingRequestCallback(methodName, original);
        RPCTracker.getInstance().register(trackingCallback);
        return trackingCallback;
    }

}

The implementation of the TrackingRequestCallback is fairly straightforward – all it has to do is notify the class responsible for statistics tracking each time a response is received for any service, whether that invocation was successful or returned an error:

package org.rhq.enterprise.gui.coregui.client.util.rpc;

import com.google.gwt.http.client.Request;
import com.google.gwt.http.client.RequestCallback;
import com.google.gwt.http.client.Response;

public class TrackingRequestCallback implements RequestCallback {

    private String name;
    private long start = System.currentTimeMillis();
    private RequestCallback callback;

    public TrackingRequestCallback(String name, RequestCallback callback) {
        this.name = name;
        this.callback = callback;
    }

    public void onError(Request request, Throwable exception) {
        RemoteServiceStatistics.record(getName(), getAge());
        callback.onError(request, exception);

    }

    public void onResponseReceived(Request request, Response response) {
        RemoteServiceStatistics.record(getName(), getAge());
        callback.onResponseReceived(request, response);
    }

    public String getName() {
        return name;
    }

    public long getAge() {
        return System.currentTimeMillis() - start;
    }
}

Finally we have the RemoteServiceStatistics singleton, which is responsible for keeping track of all invocations for all services across the application. When the statistics are queried, it can use this full audit trail of invocations to generate things like min/max/avg running times and standard deviations:

package org.rhq.enterprise.gui.coregui.client.util.rpc;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import org.rhq.enterprise.gui.coregui.client.util.rpc.RemoteServiceStatistics.Record.Summary;

public class RemoteServiceStatistics {

    public static class Record {
        private String serviceName;
        private String methodName;
        private List data = new ArrayList();

        private Record(String remoteService, long millis) {
            // remoteService format "{ServiceName}_Proxy.{MethodName}"
            this.serviceName = remoteService.substring(0, remoteService.indexOf("_Proxy"));
            this.methodName = remoteService.substring(remoteService.indexOf('.') + 1);
            this.data.add(millis);
        }

        private void record(long millis) {
            this.data.add(millis);
        }

        public Summary getSummary() {
            return new Summary(serviceName, methodName, data);
        }

        public static class Summary {
            public final String serviceName;
            public final String methodName;
            public final int count;
            public final long slowest;
            public final long average;
            public final long fastest;
            public final long stddev;

            private Summary(String serviceName, String methodName, List data) {
                this.serviceName = serviceName;
                this.methodName = methodName;
                this.count = data.size();

                if (!data.isEmpty()) {
                    int i = 0;
                    long total = 0, slow = 0, fast = 0;
                    for (long next : data) {
                        if (i++ == 0) {
                            total = slow = fast = next;
                        } else {
                            total += next;
                            if (next > slow) {
                                slow = next;
                            } else if (next < fast) {
                                fast = next;
                            }
                        }
                    }
                    slowest = slow;
                    fastest = fast;
                    double avg = (total) / (double) count;

                    double sumOfSquares = 0;
                    for (long next : data) {
                        sumOfSquares += Math.pow(next - avg, 2);
                    }
                    stddev = (long) Math.pow(sumOfSquares / count, 0.5);
                    average = (long) avg;
                } else {
                    slowest = average = fastest = stddev = 0;
                }
            }
        }

    }

    private static Map statistics = new HashMap();
    private RemoteServiceStatistics() {
        // static access only
    }

    public static void record(String remoteService, long millis) {
        Record record = statistics.get(remoteService);
        if (record == null) {
            record = new Record(remoteService, millis);
            statistics.put(remoteService, record);
        } else {
            record.record(millis);
        }
    }

    public static Summary get(String remoteService) {
        Record stat = statistics.get(remoteService);
        if (stat == null) {
            stat = new Record(remoteService, 0);
        }
        return stat.getSummary();
    }

    public static List getAll() {
        List stats = new ArrayList();

        for (String remoteService : statistics.keySet()) {
            stats.add(get(remoteService));
        }

        return stats;
    }
}

I wrote a quick test page in the user interface to expose the statistics data. Below is a cropped summary of what you’ll see:

By putting the statistics into a table with sortable columns, it makes analyzing the data much simpler. Below is a quick list of the indicators that can be derived from the data set above:

  • ResourceBossGWTService.getInventorySummaryForLoggedInUser – high average and low variance implies often/always slow, look for possible improvements such as query rewrites, database indexes, or pre-computes.
  • AlertGWTService.findAlertCountByCriteria/findAlertsByCriteria – high variance, look for either possible database table contention issues that would lead to periodic query slowness or possibly eagerly/conditionally loaded data not necessary to render the view
  • TagGWTService.findTagReportComositesByCriteria – since this is on the critical path (invoked every time a user is looking at a resource / group / bundle / etc), must ensure page rendering isn’t contingent on this data; in other words, it should be OK to lazy load this data onto the view

After you have the prioritized list of services that might require optimization, I would then open up Speed Tracer to see just how these network figures relate to pure client-side figures such as parsing, layout, style calculations, timer firings, events, etc.

It may be the case that most of the perceived end-user time is spent taking the RPC results and rendering the DOM, not in the network request itself. If that’s the case, then Speed Tracer will very likely help you to pinpoint the exact client-side issue(s) you’re dealing with. On the other hand, if you determine that the bottleneck is on the server, you can follow the steps in my other blog entry on Hibernate Performance Monitoring to help get to the bottom of those issues.

Keep in mind, this solution is only meant to be a starting off point. It’s doesn’t tell you precisely why the services are slow, but does give you an indication of which are the worst offenders and how frequently they are invoked. One thing I’ll probably be implementing in the near future is payload response tracking. Tracking this data and correlating it with the rest of the remote service statistics would help distinguish between network slowness caused by slow queries on the server versus too much data being returned in the response to the client. I encourage you to experiment with this solution, and expose other statistics that you feel are useful in getting a quick overview of where your GWT-based application performance bottlenecks may lie.

Written by josephmarques

November 29, 2010 at 5:18 pm

Posted in gwt

Tagged with