I wrote the presentation for several reasons:-
1. To capture key lessons learned on the project from a technical point of view.
2. To capture information on batch process design considerations as there is not much information in the public domain.
3. To produce some "joined up" information on tuning J2EE software using Oracle, mainly because most J2EE material stops short of the database at the JDBC level and most database tuning material stops short of application server at JDBC but from the other direction. There is material "out there" but not in one place presented in a manner that covers the whole software stack.
In this day and age most databases are 'fronted' by an application server and we have long since moved on from all the business logic residing within the database and the database being the main focal point of integration. Very few applications are islands and the 'glue' used to join applications up is almost as important as the applications themselves, therefore a more all encompassing approach needs to be taken when tuning and performance monitoring the software stack.
These are the main points I will cover:-
These are the main points I will cover:-
1. Batch process architecture and design considerations.
2. The batch process design used on my project
3. Available monitoring and tuning tools for tuning J2EE architectures using Oracle.
4. Tuning findings
5. Conclusions
As everyone always likes to skip to the end of a book before reading the rest, I will touch on some of the conclusions from this work, namely:-
1. Why Martin Fowler's first law on distributed Object architectures is well founded.
2. Dangers of swamping a J2EE application server with too many threads.
3. Avoiding 'Chatty' designs.
4. Although row by row processing is a concern, it is not always the end of the world in terms of performance and only a genuine concern if it prevents performance and scalability targets from being reached.
5 comments:
I'm very interested in your thoughts on tuning J2EE batch for Oracle. Many of the customer I work with are using WebSphere XD Compute Grid (aka WebSphere Batch) with an Oracle DB.
In terms of batch processing architectures (both from an infrastructure point of view as well as an application design point of view), I suggest you look at some of the material Chris Vignola and I have put together for Compute Grid. I'm working on an application design best practices document, which I will post in the CG dicussion forum soon. There may be some points in our work that will be relevant to your presentation.
Thanks,
Snehal Antani
Links:
Compute Grid Runtime: http://www-128.ibm.com/developerworks/websphere/techjournal/0804_antani/0804_antani.html
Compute Grid Applications: http://www.ibm.com/developerworks/websphere/techjournal/0801_vignola/0801_vignola.html
SwissRe's use of Compute Grid: http://www-01.ibm.com/software/tivoli/features/ccr2/ccr2-2008-12/swissre-websphere-compute-grid-zos.html
Compute Grid Discussion Forum: http://www-128.ibm.com/developerworks/forums/forum.jspa?forumID=1240&start=0
Snehal,
Now there is a funny thing, when I was looking for material on J2EE and batch processing, developer works articles with your name came up a lot and I was indeed thinking of contacting you. But you have beaten me to it. I also saw the thread on the TheServerSide where there was a lively discussion between yourself and another chap on batch versus asynchronous processing.
There is a batch process architecture on devx which is not that dis-similar to the one we use, by playing around with things such as:-
1. Setting pass by reference to on.
2. The threading settings of the client JVM application that spawns the worker threads.
3. The server JVM heap configuration and ergonomics, we use WebSphere on Solaris, which is why I mention JVM ergonomics.
4. As per point 3 but for the client JVM.
5. Index compression in the database.
6. Changing the database block size.
7. Use of the aggressive optimizations flag on the server JVM.
I was able to get significant performance improvement out of our software.
We also found that 'Chatty' designs can kill batch processes, i.e. you end up spending most of your time on the network between the J2EE app server(s) and the database server(s). Therefore, try and use the database with a minimal amount of network roundtrips.
Also, if your clients and using genuine batch processes, they will get great mileage from using the Oracle array interface, this will:-
1. Minimize network chatter between the app server and the database.
2. Minimize parsing within the database.
3. Allow bulk operations within the database to be leveraged.
Refer to this link from the Oracle Real World performance group:-
http://structureddata.org/2007/04/30/upscaling-your-database-application-performance-the-array-interface/
I would presume that the projects you are working on use Oracle 10g. If this is the case, I would strongly urge you to use something called the Oracle 10g time model.
The time model includes something called "db time", which is not to be confused with wall clock time. Put simply, the higher the db time is, the more the database is being utilized.
Not that I want to teach anyone how to suck eggs, but low db time means that there is something in the J2EE tier causing poor utilization on the database. In fact when capturing performance statistics for the entire application I would recommend capturing the following as a bare minimum:-
1. App server CPU utilization.
2. Database server CPU utilization
3. Database db time
4. Batch run time obviously.
The problem here is that if you are not careful, you can end up drowning in statistics. This simplifies the tuning process thus:-
1. If you lower db time and the bottleneck is on the database, CPU usage on the J2EE app server / cluster should go up and the end to end run time should come down.
2. Tuning efforts on the J2EE app server, should make db time on the database server go up. If a 'tweak' made on the J2EE application server does not result in a reduction in batch run time, the bottleneck is on the database (or the the network) but most likely on the database.
If db time comes down, but not the batch run time, then the bottleneck is on the app server (or network) but most likely on the J2EE app server.
Using this technique worked very very well. Here is a link to a presentation on db time from the chap at Oracle accredited with inventing it:-
http://www.ctoug.org/files/DB%20Time.pdf
Above and beyond this, for me to pass any further comment, I would need to know a bit more about the design and architecture of the software, i.e.:-
1. Composition of the ORM layer ?, JPA, raw JDBC are stored procedures being used etc ?.
2. Is the batch processing, 'Genuine batch', i.e. are arrays of jobs being processed at the same time.
3. Is Oracle Rac being used
I don't like to openly publish my contact details on the internet, therefore if you wish to contact me directly, please send me a linkedin invite. I work in the Uk for a large German company, you should be able to easily spot this. As my company already does business with IBM add me in as a business contact. Jonathan Marshall from IBM in the Uk also has my email address.
I am not an expert on the grid and caching products, suffice it say if they front a database, I am adament that there will come a point in scaling the software, at which you will get diminishing returns unless things like the array interface are leveraged.
If you contact me I will gladly send you a draft version of my presentation.
Thanks for dropping by my blog.
Snehal,
I just realsised, as WollatonDba is a "nom du plume", you need my real name; Chris Adkin.
Regards,
Chris
excellent summary! If you have some cycles, it would be great to see some posts in the Compute Grid discussion forum on performance hints and tips for J2EE Batch and Oracle. There is a fairly large technical community that monitors the forum, many would be interested in this.
Snehal,
My next presentation will be of interest to you then. I will pop by the compute grid forum in the very near future.
Chris
Post a Comment