ADF performance tuning: Whitepaper published on the ADF Performance Monitor

The ADF Performance Monitor is an application performance management tool (APM) tool that really understands Oracle ADF applications. Web applications often contain frustrating blind spots and mysterious, recurring problems that are often difficult to identify, diagnose, and fix. The ADF Performance Monitor helps enterprises by delivering insight into real end-user experiences. It helps development, QA teams and administrators detect, analyze and resolve common and less common issues in response times and resource usage of ADF applications.


This blog publishes a new whitepaper that gives detailed information about the architecture and implementation of the ADF Performance Monitor.

History

The first version of the ADF Performance Monitor was created by Frank Houweling and released in 2009. This was first an ADF 10g version; this version consisted of printing performance metrics in JDevelopers console log only (what methods, operations and queries were executed, when and how often). The urgent need of performance visibility came back every time in somewhat all Oracle ADF projects; nearly all ADF applications had performance problems in some way and extreme difficulty in identifying these bottlenecks.

One year later (2010) the second version was released including saving metrics to the database. In 2011 a dashboard application was added that visually reported the performance metrics saved in the database. Last years the monitor has been extensively improved and extended with many new and advanced features.

Over last years the ADF Performance Monitor has been implemented in more than 50 Oracle ADF business production applications – in more than 25 countries in the world; for example in the Netherlands, Germany, Belgium, United Kingdom, France, Spain, Canada, United States, Switzerland, Austria, Romania, Poland, Libanon, Sultanate of Oman, India, China, Australia, Cuba, Mexico, e.g.

New whitepaper published

Whitepaper ADF Performance Monitor - this document gives more information about the architecture and implementation of the ADF Performance Monitor.

Content of the whitepaper:

• Executive overview
• Introduction
• Oracle ADF applications and performance
• ADF Performance Monitor overview
• Use in JDeveloper
• Use in test and production environment
• Dashboard reporting application
• Summary and details HTTP response times
• ADF framework call stack
• Warnings and suggested solutions
• Worst performing executions in ADF BC and model layer
• Error stacktraces
• JVM performance
• Product architecture
• Configuration
• Turn on/off at all times
• Prerequisites
• Monitored Events

Functionality

In development, test and production environments, the ADF Performance Monitor provides similar functionality as the callstacks of the Oracle ODL Analyzer (by ADF request). The Oracle ODL Analyzer can be useful in the development stage, but can’t be used in test en product environments because of the amount of logging generated and the significant performance overhead. The ADF Performance Monitor records the performance with a very low overhead (less than 4%, caused by the detailed collection of performance metrics). An example of a callstack in the ADF Performance Monitor is shown in the image below. In this case the bottleneck is a slow ViewObject query of 18033 milliseconds (with usagename HRService.EmployeesView1):

In addition to that, the monitor reports overviews of the worst performing ADF Business Components (shown in the image below), BindingContainer and webservice executions and the possibility to drill down. Extensive help is available in the monitor on how to interpret the metrics. Also JVM metrics and application errors are reported. SLA monitoring (load and HTTP request response times) is possible. Because of the low performance overhead, it is safe to use the ADF Performance Monitor in production environments. The metrics collection can be dynamically turned on and off (at runtime) at all times. When the monitor is turned off there is no performance overhead because the metrics classes are not active. More detailed information is available in the whitepaper published in this blog.

With the ADF Performance Monitor, development, QA and operation teams get insight into what is happening inside their ADF application throughout the whole application lifecycle. With this insight they can circumvent frequent performance problems, use best practices and deliver a responsive and scalable ADF application.

More information

More information is available on www.adfpm.com.

ADF Performance Tuning: Tips on How to Limit ADF BC Memory Consumption

This blog contains tips and tricks on to how limit the JVM memory consumption of ADF applications. Like other web applications, Oracle ADF applications potentially use a lot of JVM memory. Many times, the root cause of a high memory usage is that application data retrieved from the database into memory is not properly bounded; too many rows are fetched and held in ADF BC memory. This can lead to memory over-consumption, very long running JVM garbage collections, a freeze of all current requests or even OutOfMemoryErrors. To make matters worse, these rows and their attributes are frequently passivated and activated for an unnecessary long period of time. The solution to this problem can be found in reducing the size of sessions by decreasing of the amount of data loaded and held in the session. With a low memory consumption, a more responsive, stable and scalable ADF application can be delivered.

Long JVM garbage collections

A ‘happy JVM’ is important. If garbage collections are running very long (for example more than 30 seconds), this is often an indication of a problem – such as a freeze of all current requests because the JVM cannot clear enough memory. In the image below an example is shown of very long running garbage collections (red line). The heap space (blue) over time evolves more into a horizontal line than the saw-tooth shaped line a healthy JVM would be characterized by. In this case an OutOfMemoryError occurred and the server needed to be restarted. This should be a trigger to investigate whether the JVM heap space is set too low or that the ADF application consumes too much memory.

Root cause of memory overload

When the JVM heap space is set at a reasonable value, the root cause of memory over-consumption is often that the ADF application fetches far too many rows (and too many attributes – I will address this subject in another blog).

This can happen when the ViewObject SQL query potentially has access to thousands or even hundreds of thousands of database rows and it does not have (or not enough) restricting bind variables (or when no ViewCriteria is applied).

Further (one or more):

• An af:table, af:treeTable, af:tree component is used in a page and the underlying ViewObject’s Access Mode is Scrollable (this is the default value at the ViewObject tuning section)

• The iterators RangeSize property (in the PageDefinition file) is set to -1

• The ViewObject fetchSize is set to All Rows – (All at Once)

• A ViewObject is only used for inserting new records (and its fetchsize is still set at the default value All Rows – (As Needed) - it can still execute its SQL query (and unintentionally fetch all database rows that are accessible)

• Iterating in Java through all ViewObject rows

Limiting rows fetched in ADF BC memory

Range paging

ADF ViewObjects provide a mechanism to page through large data sets so that a user can navigate to a specific page in the results. Range Paging fetches and caches only the current page of rows in the ViewObject row cache (at the cost of another query execution) to retrieve each page of data. Range paging is not beneficial for small data sets where it is better to have all fetched rows in the ViewObject row cache. With Range paging you can say: “I would like to see page 9 of the search results (and 10 per page)”:

• R = Range size (rows per page)
• P = Page number of the result

You can set the ViewObject Access Mode to Range paging in the ViewObject tuning section:

Maximum fetchsize on ViewObject

It is very important to limit the impact of non-selective queries that may return thousands of rows is setting a maximum fetchsize on the ViewObject instance. Recommend values are:

• Table-layout:    Only up to row number:  Set this to ± 250 rows and set a proper fetchSize (or use Range Paging)
• Form-layout:    At Most One Row (for dedicated ‘single row’ ViewObjects, when the ViewObject is being used only in the context of a single row (like the form part of a table/form layout). When a table row of a table/form layout is selected or clicked on, the form layout ViewObject is queried  with its ID value as bind parameter (for example with the executeWithParams operation). For table/form layout it is a good practice to create two separate fysical ViewObjects for table and form, especially when the table can fetch more than ± 250 rows database rows.
• Insert-layout:  No Rows (for Insert-only View Objects, when you know the Viewobject is being used only in insert mode)

An excellent way to prevent loading too much (database) rows is to set a global maximum fetchsize. You can set a global Row Fetch Limit in ADF META-INF/adf-config.xml:

If you do want to set a higher or smaller maximum fetchsize for individual ViewObject instances, you can still set a different value at the tuning section of an instance and override the global threshold. It is recommended to think carefully for each ViewObject what the best value is.

Extra bind variables on ViewObject

If your ViewObject fetches/loads a large dataset, you can limit it with extra bind parameters (or extra ViewCriteria). For example, on a search screen end-users must fill in extra mandatory search fields that correspond to extra bind parameters, to make the database query resultset smaller.

Alternatively, before you execute the ViewObject query of an import (search) screen that has access to thousands of rows you can first execute a select COUNT that gets the number of rows returned by the query.

If the number of rows exceeds a certain threshold (for example 250 rows), a message can be shown to the end user that too many results are found and that more refined, specific search terms or date restrictions are required. Only if the number of found results does not exceed the threshold, the query is executed, the database rows are fetched/loaded and the user gets the results.

Read-only ViewObjects

If a ViewObject does not have to insert or update data, use a more memory friendly read-only ViewObject;

• Non updatable EO based ViewObject (deselect updatable in the ViewObject editor)
• Expert-mode read-only ViewObject

ApplicationModule lazy loading

It is recommended to defer the runtime instantiation of ViewObject and nested ApplicationModule instances until the time they are used. You can set this in the ApplicationModule tuning section:

Shared ApplicationModule

Use a shared ApplicationModule to group view instances when you want to reuse lists of static data across the application.

ApplicationModule Pooling

When a ViewObject is properly tuned and limited in its fetching, only the rows really needed are being kept in memory. This is very important for performance and to keep the memory low. This will also make ApplicationModule pooling activations and passivations much faster. ApplicationModule pooling parameter settings are also very important for performance, I will address this in another blog.

Detecting an overload of ViewObject rows

One of the new features of the ADF Performance Monitor is a kind of ADF BC memory recorder and analyzer. It detects and warns when too many ViewObject (database) rows are being fetched/loaded into memory. Also, for a selected time range (month, week, day, e.g.), the tool gives aggregated overviews for each ViewObject instance how many rows have been fetched/loaded into ADF BC memory (ViewObject row cache or EntityCache). ViewObjects that frequently fetch hundreds or thousands of rows – and waste memory – can be detected. Also individual high fetches/loads can be analysed.

With this information, ViewObject instances that frequently fetch too many database rows can be addressed by the suggestions written in this blog; adding extra bind parameters, setting a (better) maximum fetchsize or using Range Paging.

ADF Performance Monitor: Detecting and Analyzing a High ADF BC Memory Consumption

Like other web applications, Oracle ADF applications potentially use a lot of JVM memory. Many times, the root cause of a high memory usage is that application data retrieved from the database into memory is not properly limited; hundreds or thousands of rows (with too many attributes) are fetched and held in ADF BC memory. This can lead to memory over-consumption, very long running JVM garbage collections, a freeze of all current requests or even OutOfMemoryErrors. To make matters worse, these rows and their attributes are frequently retained (passivated and activated) in the session for an unnecessary long period of time. The solution to this problem can be found in reducing the size of sessions by decreasing of the amount of data loaded and held in the session. With a low memory consumption, a more responsive, stable and scalable ADF application can be delivered.

This blog describes one of the new features of the ADF Performance Monitor; a kind of ADF BC memory recorder and analyzer. It detects and warns when too many rows are being fetched (from the database or webservice) and held in ADF BC memory.

ADF BC memory load analyzer of the ADF Performance Monitor

For a selected time range (month, week, day, hour, 5 minute), the tool gives aggregated overviews for each ViewObject instance how many rows have been fetched/loaded into ADF BC memory (ViewObject row cache and entity cache). ViewObjects that frequently fetch hundreds or thousands of rows – and potentially waste memory – can be detected. Individual high fetches/loads can be further analyzed. For each ViewObject, for a selected time range (month, week, day, e.g.), there are the following overviews:

• The total (sum) of all the ViewObject rows (by ViewObject) that are created in ADF BC memory and fetched from the database. In this overview (screenshot below) ViewObject instances that frequently fetch too many database rows can be detected. The metrics of this example overview is coming from a test server. In this case we can see that ViewObject instance Pagg009ShowAM.GbsVslVO is fetching/retrieving by far the most rows – frequently thousands of database rows. We can see that from mid November 2013 to the end of January 2014 it fetched in total 184.000 database rows (and created 184.000 ViewRowImpl objects). And the highest fetch was 7000 rows in a single HTTP request. This should be a trigger for further investigation.

• The maximum ViewObject rows (by ViewObject) that are fetched from the database and created (cached) in ADF BC memory in a single HTTP request. In general, to avoid memory over-consumption, very long running JVM garbage collections, a freeze of all current requests or even OutOfMemoryErrors, we should not allow ViewObjects to fetch and load more than ±500 rows. All ViewObject instances that are able to fetch and cache more than ±500 rows should be a candidate for further investigation. In this case (see image below) the same ViewObject instance Pagg009ShowAM.GbsVslVO has fetched the most rows of all ViewObjects in a single HTTP request: 7000 rows. The other ViewObjects that have fetched more than 500 rows should also be investigated.

With these two overviews, ViewObject instances that frequently fetch and load too many database rows can be addressed.
The tool can zoom in and analyze the individual high load (fetch) occurrences of a ViewObject (click on the ViewObject instance name). In the detail overview we can analyze when the high load occurrences happened and how many rows were fetched each time:

Drill down to cal stack of ADF request

We can drill down further to the call stack of the HTTP request. It can be useful to analyze, and find out more details about the circumstances of the high load like: What were the bind parameter values of the ViewObject query? Which ViewCriteria were applied (if one was applied)? Did it happen during ApplicationModule activation? Which taskflow was involved ?

With all this detail information, ViewObject instances that frequently waste ADF BC memory can be addressed. For example by adding extra bind parameters, setting a (better) maximum fetchsize or using Range Paging.

Detecting an overload of ViewObject rows

The metrics classes of the ADF Performance Monitor make use of extension points in the ADF framework in order to measure the time every action/step takes in a http request. It makes use of ADF BC framework extension classes. To measure how many rows are fetched from the database and how many ViewObject rows are created and cached, there are several possibilities:

• Invoking the getFetchedRowCount() in the BaseViewObjectImpl after the fetch is complete:

                int fetched = viewObjectImpl.getFetchedRowCount();

• Measuring the population of ViewRowImpl objects in the base BaseViewObjectImpl

And counting them in the BaseViewRowImpl. Before a query is executed (again), you should reset the counter.  You can do this in the executeForCollection(), the method that is invoked before each database query.

• Overriding the method createRowFromResultSet() and counting them in the BaseViewObjectImpl. This is how the ADF Performance Monitor keeps track of the number of fetched rows and number of ViewRowImpl objects created. Reset the counter before a query is executed (again):

A high load can now be detected in JDevelopers console log. You might want to not log every single row that is fetched but only when you detect a high load.

More information

More details about the ADF Performance Monitor are available on www.adfpm.com.

ADF Performance Tuning Video (42 min): Improve Your Oracle ADF App Response Time by as Much as 70 Percent

Performance needs to be ingrained in your application - it cannot be added in during the last stages of development. In this video I discuss how you can optimize the performance of your Oracle ADF Fusion application, diagnose and solve typical performance problems, and build an efficient, responsive, scalable ADF application that circumvents common bad practices. This video was originally presented as part of the Oracle ACE Track during the Oracle Technology Network Virtual Developer Day event "Oracle ADF Development - Web, Mobile and Beyond." Last month the video was published on OTNArchBeat (Oracle Architect Community Video Channel).

Important: to watch in high quality at YouTube, select HD quality (720p):

Improve Your Oracle ADF App Response Time by as Much as 70 Percent: 

Content

In the video I discuss several categories of things that are happening frequently too slowly, too often, too little, too soon and too big in ADF applications, and solutions.

Agenda of video session - things that are happening in ADF applications:

• Too slowly (too slow ViewObject queries, EntitObject DML operations, ApplicationModule pooling passivations and activations)
• Too often (too many (mini) queries, too many database round-trips, too many HTTP Requests, too many full HTTP Requests, too frequent ApplicationModule passivation & activation, unintentionally left iterators in PageDefs)
• Too big (too much data in ADF BC memory, too big scope for managed beans, too much HTML to the browser, too much logging on)
• Too soon (too soon executed taskflows, too soon instantiated ApplicationModules, ADF UI components that are loaded too soon (Immediate versus lazy))
• Too little (too little Caching, too little JVM Heap size)

We use the ADF Performance Monitor to detect, analyze and help resolve the problems discussed in this video:

ADF performance tuning: Overview Video published on the ADF Performance Monitor

A good performance is the key to the success of a web application. Oracle ADF applications are no exception to this rule. ADF performance tuning can be time intensive, costly and quite a challenge when performance issues require developers to delve deep into the inner workings of the ADF framework.

The ADF Performance Monitor is an advanced tool specifically designed for measuring, analyzing, tuning, and checking the performance of Oracle ADF applications. The tool can track and collect crucial (production) performance information of the application's runtime ADF components that are not standard provided by Oracle. It helps development, QA, and operation teams to detect, analyze and resolve common and less common issues in response times and resource usage of ADF applications. 

This blog publishes an overview video ADF Performance Monitor (13 min).

The first version of this tool was already released in 2009 (ADF 10g). One year later (2010) the second version (ADF 11g) was released. Last years the monitor has been improved and extended with many new and advanced features. Currently the ADF Performance Monitor has been implemented in more than 15 Oracle ADF applications over the world and has been proven to be very useful. Read the quotes of ADF experts and managers.

Oracle ADF applications and performance

ADF is a powerful, advanced and highly configurable framework that is very performing and scalable if the ADF developer chooses the right combination of parameter settings. However, web applications in general and ADF applications in particular have many pitfalls that can be circumvented by choosing the correct performance configuration parameter settings. In most cases, unfortunately, the default values are not the most optimal values.

Frequently even experienced ADF developers cannot pinpoint why an ADF application is slow. In this case information of what is happening behind the scenes would be very useful in order to get a better understanding of their ADF application.

In development, test and production environments, the ADF Performance Monitor provides similar functionality as the callstacks of the Oracle ODL Analyzer (by ADF request). The Oracle ODL Analyzer can be useful in the development stage, but can’t be used in test en product environments because of the amount of logging generated and the significant performance overhead. The ADF Performance Monitor records the performance with a very low overhead (less than 4%, caused by the detailed collection of performance metrics).

Detecting and Analyzing a High ADF BC Memory Consumption

Recently a new feature was added: a kind of ADF BC memory recorder and analyzer. It detects and warns when too many rows are being fetched (from the database or webservice) and held in ADF BC memory. With this feature you can investigate and address memory over-consumption. Memory over-consumption can lead to very long running JVM garbage collections, a freeze of all current requests or even OutOfMemoryErrors.

Development, QA and operation teams

With the ADF Performance Monitor, development, QA and operation teams get insight into what is happening inside their ADF application throughout the whole application lifecycle. With this insight ADF developers can diagnose and solve performance problems already in an early stage, make better (architecture) decisions and can build more responsive and scalable ADF applications. With the warnings and suggested solutions of the ADF Performance Monitor they can circumvent frequent performance problems, use best practices and deliver a higher quality. This will lead to an application that is more consistent and better to maintain. They can reduce the utilization of infrastructure, hardware and licenses. End-users will be more happy.

A whitepaper on the ADF Performance Monitor is available. More information is available on www.adfpm.com.

ADF Performance Monitor: Customer Case Video

The ADF Performance Monitor is designed for measuring, analyzing, tuning, and checking the performance of Oracle ADF applications. The monitor can track and collect crucial (production) performance information of the application’s runtime ADF components that are not standard provided by Oracle. It helps development, QA, and operation teams to detect, analyze and resolve common and less common issues in response times and resource usage of ADF applications.

Recently an overview video was published. This blog publishes a customer case video of the Dutch Ministry of Justice.

Customer case video (4 min)

This video shows how the ADF Performance Monitor helped the Dutch Ministry of Justice to solve their performance problems. The primary information system for the Dutch judiciary is replaced with a new system based on Oracle ADF. Some facts about this project:

• Very large ADF application (1000+ ViewObjects, 15+ ApplicationModules)
• 15+ developers
• 5000+ end-users
• 100+ concurrent requests
• production environment: 4 Weblogic nodes with each 16 GB JVM memory
• Frequently response times over 20 seconds
• Many times long running JVM garbage collections over 60 seconds

View Video (4 minutes)

More information

More details are available here.