System iNetwork

April 2007

April 29, 2007 6:00 PM

Java data queues 211 -- refactoring the thread based console application view-controller into parts which can be reused

Now we have a
- demonstration model for MVC running on the System i written in RPGLE
- demonstration view-controller for MVC on the System in written in CL
- demonstration console based view-controllers for MVC written in Java which locks the application while waiting to read the output of the model (DtaQExample1 and 2)
- demonstration console based view-controllers for MVC written in Java where a Thread waits to read the output of the model (DtaQExample 3a, 3b and 3c).

What about Eclipse and Rich Client Platform? Remember, the subject of this blog. Actually we are closer than you might think. In this blog entry DtaQExample3a and DtaQExample3Thread will be refactored into parts which can be re-used. We will take those parts to Eclipse RCP and continue their development.

Refactor DtaQExample3Thread into a generalized, generic form

The new class is ModelOutputDataQueue. This class will handle the data queue output of a Model (MVC), any Model of which follows the design presented. In order to do this anything which ties the model's output data queue handling to the specific Demonstration Model has to be moved out of this class. This class will work with the demonstration Model running on the System i and future Models, given some help outside the new generalized class.

In fact this is a very generalized keyed data queue reading class which can be run in a Thread. Considered in isolation, it is independent of the whole notion of MVC. It could be used anywhere that a keyed data queue needs to be read from a Thread.

Many of the design aspects for DtaQExample3Thread are retained. The changes are:

1. The new class doesn't assume anything about the data being read from the data queue. Reading the keyed data queue returns a KeyedDataQueueEntry, a generic object which assumes nothing about the data or its format. Taking the data from the KeyedDataQueueEntry and putting it into a Record is done outside the new class (by the Listener). A generalized, reusable class won't know what the data which the model will send looks like, the data structures may not have even been designed.

2. Give the class a way to stop the run() method short of crashing the data queue. This is done by the method stopReadingDq(). This method sets a flag which breaks the run loop then writes a dummy entry to the data queue which the run() loop is reading. The run() loop reads the dummy entry, sees the flag is set and breaks out without firing the PropertyChangeEvent.

A side note: A refinement would be for the PropertyChangeEvent to fire a "dqEnded" event before breaking the loop. That way the PropertyChangeListener could act on the fact that the program is no longer reading the Model's output data queue.

3. Every 10 minutes the read() times out then re-read the data queue. This gives the program a chance to throw an error if the data queue isn't available.

4. Create Exception stubs for everything the System i could throw. Later fill in as needed.

5. Consolidate exception reporting into one method. Right now this method simply does a printStackTrace(). In future implementation this method will be expanded to log errors and to take some action like cause the program to terminate. Remember the run() method of a Thread can't throw an exception. The reporting method will send a dqError PropertyChangeEvent to the PropertyChangeListener so it can notify the rest of the program something happened.

6. The class implements Runnable instead of extending Thread. The user of this class will create its own Thread, instantiating it with this class.

Refactor DtaQExample3a to put the data queue writing logic into a separate class

The new class is ModelInputDataQueue. This class separates the function of writing to the Model's Input Data Queue from the main application DtaQExample3d. The main application performs the functions from a specific implementation of a Model, such as the Demonstration Model running on the System i. This class only writes the data queue as DtaQExample3a did, but without any model specific details. The changes to generalize the class were done for the same reason as the Output class:

1. The new class doesn't assume anything about the Record which will be written. Instead the Record will be passed to the write() method.

2. Create Exception stubs for everything the System i could throw. Later fill in as needed. . The write() method can throw exceptions so the caller of the write() method will have to be prepared to handle (or ignore) them.

3. Consolidate exception reporting into one method. Right now this method simply does a printStackTrace(). In future implementation this method will be expanded to log errors and to take some action like cause the program to terminate.

DtaQExample3aListener becomes DtaQExample3dListener

DtaQExample3dListener required only one change. DtaQExample3aListener received a Record from the Event fired by the Thread. Record is specific to a RecordFormat which is specific to a System i file. DtaQExample3dListener receives a KeyedDataQueueEntry which is generic. The class builds a model output Record from the KeyedDataQueueEntry sent in the Event.

Only a few lines of code are changed to accomplish this. The Listener becomes very specific for the data it listens for while the Thread that is being listened to knows nothing about the data. It just reads a KeyedDataQueueEntry and passes it on to the Listener.

The PropertyChangeListener which will be attached to ModelOutputDataQueue can't be standardized into a generic class because what is does is specific to
- the Model being implemented,
- the data sent to the listener in the PropertyChangeEvent object, and
- possibly the name of the event fired.

DtaQExample3a becomes DtaQExample3d

DtaQExample3d is the main application. The ModelInputDataQueue class now has no dependencies on the specific Model that it is used with. This means that their caller (DtaQExample3d) must creates the model input Record required for the implemented Model and pass it to the write() method.

The code to write the model input data queue is removed from the main application (DtaQExample3a) and replaced by calling the write(Record) method of ModelInputDataQueue.

A Thread is created to run ModelOutputDataQueue.run() in place of DtaQExample3Thread.

To shut down the program call ModelOutputDataQueue.stopReadingDq() to end the Thread. After calling this method wait for the thread to end with Thread.join(milliseconds). After the Thread ends or the join times out close the connections to the System i and end the program with System.exit(0).

Replace Me.getKEY() with the class IpDtaQKey

IpDtaQKey generates a key based on IP address. The key represents a View-Controller which communicates with the Model. Often the VC will be running on multiple desktops. IpDtaQKey creates a key for the model data queues from the IP address of the desktop client. This will be unique per desktop.

Another side note: One caveat though! This breaks down if two or more clients were to run separately on the same desktop using the same set of data queues. In this case the key would have to be "uniquified" for each client (on the same desktop using the same data queues).

Java is not well suited for this. Each JVM lives in its own world and is unaware of any others which may be running on the same system. The JVM can't make use of the operating system resources for coordination because the JVM is intentionally blind to operating system specifics.

To solve this I'd use this approach.

1. Decide on a folder any JVM on the desktop can find. Make sure it is not dependent on the user logged on, that is not dependent on the users home directory. For example C:\TEMP.

2. Create a Collection object of "unique" values of 0 to 255 which can be represented by x00 to xFF. The unique value will be the look up if you choose a Map type Collection. If you choose an Array type collection than the unique value will be the index. I'd also put a timestamp on it so old value could be recycled if they hadn't been removed.

3. Each time the program which needs a key starts up it
- calls the IpDtaQKey for the base key,
- deserializes the Collection,
- finds a unique value and convert it to hex (2 characters),
- updates the Collection with the unique value used,
- cleans up any obsolete used unique values (from the timestamp),
- then serializes the Collection.
The data queue key for this instance is IpDtaQKey.getDqKey() + uniqueValue

4. When the program ends deserailze the Collection, release the unique value so it can be re-used and serialize the Collection. The clean up in the previous step is a back up because programs crash and you can't depend upon this step being executed each time a new unique value is put in the Collection.

Like I said, Java doesn't make it easy. Please let me know if you have better ideas. This is an on-going problem for me. What I have described is the best solution I have come up with to date.

If you run these examples in QSHELL on the System i then you need to do something else for the key. I have started a dummy job, gotten its number, ended the dummy job and used the dummy job's job number for the key.

Try to avoid a design which requires the key value to be created by the user in a property or environment variable. This works great during development but when you start distributing the application you will find that maintaining a unique key per installation is a real pain. Been there, done that. Bad!

Play with running both the CL VC and DtaQExample3d!

Actually you can do this with any of the DtaQExample programs. The CL demo View-Controller can write an entry to the demo Model output data queue without reading it back. If you do this while DataQExample3# is running, with the same key the instance of DtaQExample3# is using, the instruction and data will pop up on the console.

All the entries on the demo model data queue are being placed on it by the demo Model running in batch on the iSeries. The difference is where the input to the Model came from, the System i CL program on the desktop Java application. DtaQExample3# shows both and doesn't treat them any differently.

The thread to read the KeyedDataQueue starts to show its power when messages pop up on the console from the model while the console is reading stdin for your input.

The code is in the CVS

The new classes have been added to the dataqueue package except for IpDtaQKey which was added to the common package.

Taking some time off

The next blog entry set will be about writing an Eclipse RCP demonstration View-Controller that has been implemented as
- a CL program
- as a Java console program (6 different ways now)

I am experimenting with moving my Eclipse 3.2 work into WDSc 7. If these experiments are successful then the next exercise will be done in WDSc 7 and include notes on how to do it in Eclipse 3.2.

Right now my personal preference is native Eclipse. However WDSc 7 has advanced to the point that managing two separate development environments may outweigh the extra capabilities of native Eclipse. I will see and get back to you on this.

I am taking some time of posting this blog to take care of this and other things.

In the mean time please write if you have questions or comments. Please write to tell me what you'd like to see in an Eclipse RCP application version of this Java console program. Tell me what parts you would like explored in detail.

Thanks all. Enjoy the spring -- that is part of my "other things" :)

Posted by Bill Blalock on April 29, 2007 at 6:00 PM | Comments (1)

April 21, 2007 1:00 PM

Java data queues 202 -- the listener side of a thread based console application view-controller for System i demonstration model

A previous blog entry "Java data queues 201 -- the thread side of a thread based console application view-controller for System i demonstration model" solved the problem of the java application being stuck waiting to read a data queue by putting the read data queue part of the program into a separate thread. Each time a data queue read occurs the thread fires a PropertyChangeEvent. The event contains the information read from the data queue.

This blog entry address the rest of the example application. A large part of the main() portion of this application, DtaQExample3#, is the same as Examples 1 and 2 in that they
- get input from the console and
- write it to the model input data queue
(remember that demo model running on the iSeries?). See the early discussion.

The different (new) parts of DtaQExample3# is that
- the model input data queue (read) is started as a thread (DtaQExample3Thread)
- PropertyChangeListener is implemented
Three examples (3a, 3b and 3c) show different ways to implement PropertyChangeListener so you can experiment with the differences.

DtaQExample3a implements PropertyChangeListener as a separate class.
DtaQExample3b implements PropertyChangeListener as an anonymous inner class.
DtaQExample3c implements PropertyChangeListener in its declaration and provides the methods of PropertyChangeListener

The PropertyChangeListener (PCL) in these examples listens for the PropertyChangeEvent (PCE) named "readDq". When it receives on it creates Strings from the data (Record) contained in the PCE and displays the Strings on the console.

If you look at the propertyChange() method of PCL in all the three examples you will see they are identical.

Example3a implements PCL as a separate class, DtaQExample3aListener. This isolates what is going on the in PCL from the rest of the application. You choose what to provide this class in the constructor design and the setter methods, and choose what to expose with the getter methods.

Example3b implements PCL as an anonymous inner class. An inner class is a separate class that has limited assess to data in the class which creates it and does not exist outside that class.

Example3c implements PCL in the class declaration, at the very top of the program. Because the Example3c class formally implements the PropertyChangeListener class it must provide the required method
public void propertyChange( PropertyChangeEvent event )
The method has full access to the class variables of Example3c and easily interacts with the rest of the class.

The choice of how to implement PropertyChangeListener depends on the design.

The advantages of a separate PCL class (3a) are
- it can be re-used
- it puts the PCL and what it does into a black box
The downside is more design work and coding is required.

The advantage of an inner class to implement PCL (3b) is that it is quick to write and can interact with the data in the parent class. It is a good choice if the inner class does very little and won't be reused.

The downside, at least for me, is that inner classes can be messy, hard to read and have a tendency to grow as the design changes.

The advantage of implementing PCL at the class level (3c) is that the work of the listener is put into the propertyChange() method and that method can easily communicate with the rest of the class.

I really don't see a downside when this approach fits with the design. That is the PCL is used only by, and communicates with, the one class which implements it.

Posted by Bill Blalock on April 21, 2007 at 1:00 PM | Comments (0)

April 15, 2007 12:00 PM

Use WDSc 7 for Eclipse RCP Development!

WDSc 7.0 has been available for a month now and it lives up to the expectations. System i Network has articles on the new version and more are on the way.

WDSc 7.0 can be used for Rich Client Platform development! The prior version of WDSc (6.01) did not completely support RCP development because it was based on Eclipse 3.0 and RCP development really needs Eclipse 3.1. WDSc 7.0 is based on Eclipse 3.2.1 ... close enough!

The RCP introductory tutorials which used Eclipse 3.2.x can now be done with WDSc 7. The JT Open toolbox can be used as a plug in instead of an external jar file. That is very neat!

Try these tutorials from earlier in this blog with WDSc 7:
Hands on -- Getting started with Eclipse RCP.
How to modify the Eclipse RCP tutorial (part 1) to connect to a System i.
Continuing to modify Eclipse RC Tutorial (part 1) into a System i client

There are no changes in Ed's tutorial, the first one. In the second tutorial is simplified by using jt400.jar as a dependent plugin (read on). The third tutorial has no changes.

So now you can have it all for a while under WDSc 7. Well at least until Europa rises ... that databinding is something else....

In "How to modify the Eclipse RCP tutorial (part 1) to connect to a System i" jt400.jar was treated as an external jar and added through the classpath, see section "Add jt400.jar to the project classpath".

Instead of following those instructions, do this to add the JT Open Toolbox to the list of required plug-ins.

- open META-INF/MANIFEST.MF with PDE (double click on it)

- select "dependecies" tab, third from left (see tutorial)

- in "Select a plug-in" type
com.ibm.etools.iseries.toolbox
two plugin will be listed
-- com.ibm.etools.iseries.toolbox
-- com.ibm.etools.iseries.toolbox.doc
highlight the first one (not .doc) and press the Add button.

- save the XML document

Now the toolbox is in the list of required plug ins. When you run the tutorials in WDSc the JT Open jar will be found. If you build an application and deploy it the JT Open jar file will be a part of it, just like the other Eclipse plugins.

Continue with the tutorials. That is the only change to do these RCP examples in WDSc 7.

Note that this change was not necessary to make these examples work in WDSc 7, the original instructions would have worked fine. We are benefiting from JT Open being a part ot WDSc and available as a plug-in.

Posted by Bill Blalock on April 15, 2007 at 12:00 PM | Comments (0)

April 8, 2007 8:00 PM

Java data queues 201 -- the thread side of a thread based console application view-controller for System i demonstration model

The prior blog entry presented a Java console application which performs the same function as the CL view-controller -- write a data queue then read. The examples used the techniques covered in IBM's documentation. That was a starting place -- putting the classes explained in the Java toolbox documentation into a console view-controller.

The write-then-read pattern does not work for GUI clients. The desktop application has to be free to service inputs from the user while waiting for the data queue read to happen. The program can't be locked waiting for model output on the data queue.

The Java console application of the last entry is re-written to place the data queue read portion of the write then read pattern in a separately running thread. The main program spawns a thread which waits for the model output data queue and reads it. After the thread is spawned the main program is free to do whatever … in this case wait for input from the user and write to the model input data queue.

I haven't found examples of using JTOpen Java toolbox data queue classes in threads. Over the years I have developed several techniques of running data queue classes in threads.

dataqueues.DtaQExample3Thread.java is a class that reads the model (or server) output data queue. When a data queue entry is read the class passes the Record object read to be processed. In this case the Record object, that represents the model output data structure, is turns it into Java variables (String objects) and written to the console.

The operation of this class is narrowly defined:
- wait for data on the model output data queue
- when the data is read package it into a Record object
- fire a PropertyChange event which contains the Record object which was just read and wait for the next data queue entry
- handle any exceptions
- when told to clean up and end the thread

My experience is that the thread which reads a data queue should do as little as possible but do it well. Most of its time is spent waiting on a data queue.

Important points in this class are
a) the AS400 object is instantiated and passed to the thread object when it is created by the constructor
b) the class handles its data queue exceptions
c) the class cleans up after itself and shuts the thread down (thus shutting itself down)
d) the java bean PropertyChange classes are used by the thread to communicate

a) The AS400 object is a shared resource, shared by all the parts of the Java application running in one instance of the JVM. It is created and managed outside the thread. The thread merely uses it and handles any exceptions which result from the threads use of the AS400 object. That is why it is passed to the thread in the constructor.

The AS400 object will used for other services between the Java client and the System i server. Reading the model output data queue is just one use. For this example the AS400 object will also be used to write the input received through the console to the model's input data queues. In a real MVC scenario the AS400 object would also be used by message queues and perhaps JDBC or record access services.

b) The thread waits to read the data queue. What if something goes wrong? The thread has to handle the error or at least pass it back to the program. My experience is that it is best if the thread handles the errors. Most of the errors are extremely fatal. To keep this simple I just dump the stack trace and crash (System.exit(0)).

c) The thread spends most of its time waiting for a data queue entry. It can't do anything else while it is waiting. How do you tell the thread to stop waiting and do something else?

In this example I took the cheap way. One thing that the thread will notice while it is waiting for a data queue read is if the data queue goes away! The thread specifically traps for the condition that the data queue went away. When that happens the thread cleans up and terminates its self. The caller ends the data queue services, gives the thread a few moments to notice and end itself, then goes about its business.

I have used two other methods, both more sophisticated than pulling the rug out from under the data queue thread.

1) Send the thread a message to shut down through the data queue it is reading. Remember, the thread is reading a data queue. It doesn't know where the message originated. If the thread processes a "terminate" message then either the main Java application or the System i server can send this message. In the case of a Java rich client the main Java application would send the message to "terminate" during the shut down process. In a more traditional client server program the server might send the terminate message and the thread pass the information to the main program, perhaps shutting it down.

2) Instead of waiting forever for the data queue read waits a short period of time. When the read times out the program looks to see if it should continue reading the data queue. I use this in conjunction with a shutdown() method in the thread.
- The main program call DtaQThread.shutdown().
- shutdown() sets a shutdown flag in the thread
- the next time the read times out the program sees that shutdown flag is set and acts. Otherwise it goes into another read cycle.

Waiting forever on a data queue read has subtle problems for long running client server programs. Basically the code reading the data queue will never get data, or see that there is a problem, if the data queue goes away. Seriously, the JTOpen toolkit read data queue class won't notice that the System i has been IPLed if the data queue read timeout is a -1 (forever). If the data queue read times out and then read again for N seconds it will see that something happened to the data queue (like it is no longer there or was recreated) and can handle the error.

For simplicity sake in this example I took the easy way out. When the main program is stopped by the user typing *STOPITNOW the data queue service is disconnected. The thread will take that error as a signal to clean up and die.

d) Data is read from the model output data queue. Now what? How is that communicated to the main program? For this example I chose java been PropertyChange classes but there are several ways it could have been done.

I don't use the DataQueueListeners which can be attached to a DataQueue object because they don't work well to my mind. If you want to discuss it leave a comment.

The Java bean PropertyChange classes are a good choice. The thread can read the data, do whatever processing it needs to do, and fire a PropertyChangeEvent that contains the data read from the data queue. The code controls the point the change event is fired and what is communicated to the listeners.

The PropertyChangeListener "hears" the fired event and acts on the data. In this case the PropertyChangeListener unravels the Record object from the data queue read and writes it to the console.

There are several other ways to do this. I have tried a few of them. Please leave a comment if you have another way to do it.

Also note that a GUI client is a multi-threaded program by nature and may already provide a means for the thread to notify the rest of the client that a data queue entry has been read. Remember, the purpose of this example is to give an alternative example of data queue use for a rich client. This is just a starting point, something simple to explore and experiment with.

Enough for now -- perhaps too much, certainly not enough.

The other parts of this example, Example3, will be addressed in the next blog entry. There are three version of Example3, all use the same DtaQExample3Thread class. Here is the code for the rest of the example if you want to try DtaQExample3Thread.

DtaQExample3a implements PropertyChangeListener as a separate class.
DtaQExample3b implements PropertyChangeListener as an anonymous inner class.
DtaQExample3c implements PropertyChangeListener in its declaration and provides the methods of PropertyChangeListener

Enjoy!

Posted by Bill Blalock on April 8, 2007 at 8:00 PM | Comments (0)

April 1, 2007 6:00 PM

Java data queues 101 -- a console application view-controller for System i demonstration model

This set of blog entries focuses on using data queues in Java. Data queues, and later message queues, are the foundation for the model <> view-controller interchange between a System i and a Java application (Eclipse RCP or otherwise) developed in this blog series.

The last blog entry implemented an RPG System i demonstration model using data queues. The demonstration model reads an input data queue and writes to an output data queue. As a bonus the entry provided a CL view-controller to test the model. Now focus goes to the Java side, putting data on the input data queue for the model and reading the response from the model on the output data queue.

This blog entry presents two versions of a Java console application which performs the same function as the CL view-controller. The examples use the techniques covered in IBM's documentation. I look at this as a starting place -- putting the classes explained in the Java toolbox documentation into a console view-controller application.

Consider this a review of data queue related classes and examples provided by IBM. That documentation is found in the iSeries Information Center, under Programming > Java > Toolbox section. You can compare the Java code to the data queue examples which IBM provides.

In the last blog entry the CL view-controller gave you a means to exercise the demonstration model which was running in batch with a couple of options (write to the model but don't read and read from the model but don't write). This blog entry provides the same basic tool in Java. You can run this on the System i (in QSH) or on your desktop.

The code for this is in the CVSDude repository, in the project "info.billblalock.mvcintro.java_apps". The programs are
- dataqueue.DtaQExample1.java and
- dataqueue.DtaQExample2.java.
The programs are similar with one significant difference I will get to later.

The logic of these programs is like the CL view-controller in that they write an entry to the demonstration model's input data queue and wait to read an entry from the demonstration model's output data queue. This pattern is similar to EXFMT of a 5250 display, write the display file and wait for a read.

The console prompts the user for the instruction to send to the model. If the instruction is *STOPNOW the program terminates. If not the program prompts the user for the data to send with the instruction. The instruction and data is written to the model input data queue. The program waits to read the model output data queue.

The class common.Me.java defines constants and getters for your AS400, data queue names, etc. That is the only thing you should need to change to get these programs to run on your system.

These two Java client programs will run either on the System i as Java programs or on your desktop. There is no GUI involved, all the IO is through stdin and stdout.

The only difference between Example1 and Example2 is that Example1 dynamically builds the Record object and Example2 uses a static pre-compiled Record object.

Example1 doesn't know anything about the data structures for the input and output data queues that are expressed as System i files (external data structures). It dynamically retrieves the files from the System i and learns the structure of the data from the files. Each time the program is run the data structures are resolved.

Example2 uses precompiled classes to represent the data structures. It doesn't have to retrieve this information from the System i.

Example1 shows the time it takes to retrieve the information during the startup of the program. Example2 shows that virtually no time (0 milliseconds) is required to use the precompiled information.

I did this comparison because while the Java toolbox documentation mentions the two methods it doesn't give you a good feel for the real difference.

Example1, the dynamic example, limits the impact of changes in data structures. Dynamic resolution means your program won't care if new fields are added to the data structures, if lengths of fields are changed or if the order of field names is changed. But you pay for this flexibility each time the Java program dynamically fetches the information. Of course the program is still going to tank if the names of the fields or their data types are changed.

Example 2 is fast. If the external files on the System i which represent the external data structures are changed then the pre-compiled classes will have to be re-generated.

That is pretty much gives you all you need to know to decide between the two methods for your given circumstances.

The class common.CreateModelRecordFormats.java is the code to generate the static, or pre-comiled classes to represent the data structures based on System i files. See the toolbox documentation for details.

I am writing less blog and putting the time into the code. If I missed something or there are questions leave a comment.

Enjoy!

Posted by Bill Blalock on April 1, 2007 at 6:00 PM | Comments (0)