System iNetwork

March 2007

March 25, 2007 3:15 PM

Demonstration data queue model (server) and test view-controller (client) on the System i

This entry describes how to create a demonstration data queue model (server) in RPG and a test view-controller (client) in CL. This is straight System i commands, CL and RPG. When you are finished you will have a demo model running which will read an input data queue and echo the formatted input to a keyed output data queue. You will also have a test client that works from the command line which you can use to
- test the demonstration server, write its input and read its output
- read any entries on the output queue
- force input to the demonstration server for another client to pick up

The Java programs that explain using data queues in Java in future entries will use the demonstration model. You can use the CL client to read input from the Java examples or put input into the demonstration server to be read by the Java examples.

The Java examples will be presented first as console applications which can be run on a PC or on the System i. They will go beyond the basic data queue operations described in the Java toolbox documentation.

If you are interested in the Java parts take the time to build the demonstration server and test client on your System i.

The IBM Redbook Who Knew You Could Do That with RPG IV? A Sorcerer’s Guide to System Access and More, section titled Data Queue APIs - Programming with Data Queue APIs (5.2.3) explains implementing a client / server model using data queues. When possible this material is referenced for the System i side of the Eclipse RCP client data queue examples.

The steps to create a demonstration model for the future Java work are:

1. Create the data queues
2. Build a demonstration model / server on the System i
3. Build a test client / view-controller on the System i

We will move through the System i side of this very quickly and at a high level. This is very simple and basic for the experienced readers of System i network and the topic is thoroughly covered in the IBM Redbook referenced.

Step 1.

This design uses two data queues. One data queue is for the Model (server) to receive data from the View-Controller (client), call it MDLI (MoDeL Input). The second data queue is for the Model to send data to the View-Controller, call it MDLO (MoDeL Output). Feel free to pick your own names. This short name leaves 6 characters to identify the specific model when implementing.

MDLI (Model in) is a FIFO data queue. The Model gets instructions from any View-Controller through this data queue. The data structure of MDLI is
- identifier of the View Controller (6 characters)
- instruction (16 characters)
- data which is defined in an external data structure

CRTDTAQ DTAQ(your_lib /MODLI) MAXLEN(64512) AUTORCL(*YES)
TEXT('Template input to Model of MVC')

MDLO (Model out) is a keyed data queue. The Model returns results to any View-Controller that sent the request, identified by the key, through this data queue.
- the key is 6 characters, the identifier of the View-Controller
- the data structure is
- instruction (16 characters), echo back the instruction the model is responding to
- data which is defined in an external data structure

CRTDTAQ DTAQ(your_lib/MODLO) MAXLEN(64512) SEQ(*KEYED)
KEYLEN(16) AUTORCL(*YES) TEXT('Template output to Model of MVC')

The source for step one:
MDLDMODQCL
The library to create the data queues, your_lib, is passed to this CL as a parameter.

Step 2

Create a test System i server / model which reads MDLI and writes MDLO. We are demonstrating how client applications interact with data queues. We don't need much from the Model but we need something to prove the clients works and debug them.

The test server / model needs to read the input data queue (MDLI) and write to the output queue (MDLO). At this point all we need is something the client can write to and get a response.

This server / model runs in batch in a simple tight loop
- read MDLI (QRCVDTAQ API)
- build a reply with the info read from MDLI
- write the input from MDLI and the response to an audit file
- write MDLO (QSNDDTAQ API)
- if the instruction in MDLI is '*ENDIT' set on LR otherwise repeat

The test model / server is similar to the test data queue server described in Data Queue APIs - Programming with Data Queue APIs (5.2.3) of Who Knew You Could Do That with RPG IV? A Sorcerer’s Guide to System Access and More .

The differences between the test program for this project and the server described in the Redbook are:
- the data queues use data structures that are defined externally where the IBM example defines the necessary fields as program variables.
- the model / server loops continuously until *ENDIT is received as an instruction where the IBM example executes one cycle
- an audit file records the model / server function for testing and debugging

The source for step two:
MDLIDS01 - model input data queue external data structure
MDLODS01 - model output data queue external data structure
MDLAFL01 - audit file to show what goes in and out of the demo server
MDLDMOSRRG - source for the demo server
MDLDMOSRCL - starts the demo server in batch, modify this to meet your needs

Hint: Change the parameter of the audit file of "Records to force a write" (FRCRATIO) to 1. This forces the demo server to write each cycle to disk as it happens. That way you can check on the server's operation which query or DSPPFM while it is running.

Step 3

Create a test client or view-controller on the System i. This is done to test the server or model. It also demonstrates that the server or model will serve clients other than a desktop Java application.

We need a simple client, one that takes input from the user, sends it to the model / server, and shows the user the results from the model. These simple tasks can be done with CL. The command prompter can accept the users input. SNDUSRMSG can display the output.

You can also display the audit file to see that the model is working.

The steps are

Define a CMD to accept the three fields in the MDLI data structure
VCID - view controller ID
INSTRCTN - instruction to the Model
MESSAGE - message to the Model

To make the command more useful I defined two logical variables. One to indicate if CL program should only read the output data queue (MDLO) and the other to indicate if the CL program should only write to the input data queue. With these additions the CL client can
- test the demo server by writing to its input and reading its output
- read the output of the server (for an input put by another client)
- write to the input of the server (for another client to read as server output)

Write a CL program to accept these as parameters. The key points are

- Concatenate the three fields into a single field to send to the demo server

- Use QSNDDTAQ API to send the single field to the MDLI data queue

- Use QRCVDTAQ API to received the results from the model, MDLO data queue. Only wait a reasonable amount of time.

- Use SNDUSRMSG to display the output of the model. If the length returned by QRCVDTAQ is zero then give the user an error message otherwise give the user the results from the model.

The source for step three:
MDLDMOVCCL
MDLDMOVC

The source in HTML has been linked to this entry. The source is also in the CVS provided by CVSDude. CVS is an open source change management system supported by Eclipse and WDSc. The CVS can also be accessed through the web.

The instructions for accessing CVSDude repository are in a link at the top right of the main page for this blog, beneith the blog mission link. I can be reached at b_blalock@comcast.net.

Posted by Bill Blalock on March 25, 2007 at 3:15 PM | Comments (0)

March 15, 2007 2:00 PM

Adding message queues to the Model (server)

It would be nice if the model (server) programs always do their "normal" or "expected" action for instructions received, but they don't. In the pure System i environment these exceptions are communicated via messages. The model (server) can also use messages and message queues to tell the view-controller (client) about these "abnormal" or "unexpected" events.

Consider a simple case, a model which adds a record to the data base.

A monolithic 5250 program would write the record and trap for an error. If an error occurred it would display information about the and let the user correct it. An error indicator would come on if the record couldn't be added (for example a second instance of a unique key), a decimal data error, NULL data was put into a non-NULL capable field, perhaps an invalid date or time value for a DATE, TIME or TIMESTAMP field.

Suppose a model program is passed the instruction to add a record to a database but encounters any of these problems. There is no interactive user to hand the error back to as in the monolithic 5250 program. How would a service program handle this situation? Message queues! The service program would probably send a message to the caller.

This data queue communicating model (server) design proposed uses message queue to communicate back to the view-controller (client). The Java toolbox does not support program to program messages queues unless the System i program was called by Java. However the Java toolbox does support external message queues.

The model program writes a message with SNDUSRMSG or SNDPGMMSG to the message queue. The Java client reads the message, can even respond to it.

That is the quick version of my model concept for MVC. If you want more information and examples please see the IBM Redbook.

This discussion is not limited to Java PC clients. The clients (view-controller) can be developed with any method, on any platform, which supports data queues and message queues. In this blog the intended clients are Eclipse RCP applications. But the "clients" could be
- 5250 applications running on the same system a different system
- Java desktop clients developed with tools other than Eclipse RCP
- C based desktop clients (Client Access provides DLLs for data queue and message queue)
- some type of web application server which communicates with the business application model though queues
- a peer-to-peer application which needs the services of your model

If your application already separates the business logic into programs you can write a data queue layer to that business logic for the Eclipse RCP client to use.

If your application doesn't already separate the business logic from the display a data queue based model then this is a good general purpose way to get the separation.

Posted by Bill Blalock on March 15, 2007 at 2:00 PM | Comments (0)

March 8, 2007 9:30 PM

MVC for Java using data queues

The last blog entry covered the two ways Java has to call System i programs. The Java tool kit classes to call System i programs and commands are easy to use. For implementing MVC, they are too slow. The other way, JNI/RMI, is fast enough but is not easy to use. A better way is to use data and message queues to communicate between the Model (System i) and View-Controller on the desktop. This performs better than the Java client calling System i programs for each transaction. This is much simpler and faster to develop than JNI-RMI access to System i programs. In addition the data queue enabled Model will be able to work with other clients that communicate with data and message queues.

The Model is data queue enabled with one or more programs (servers) running on the System i. The data queue handling programs
- monitor input data queues for instructions from the view-controller (client), for example "read" a record, "update" a record, "execute" a transaction.
- call existing programs (the Model) to perform the business functions.
- send the results to the client (view-controller) on output data queues, for example the "record read", "status of the update", results of the "executed transaction."

One input and one output data queue is used. The input data queue is a non-keyed FIFO. The data on this queue is
- an identifier of the view-controller or client
- an instruction to the model
- data which the model is to act

The output data queue is keyed, the key is the identifier of the view-controller or client. The data on this queue is
- the instruction echoed back as confirmation
- the data that is the result of the model's action

The view-controller (client) sends the model (server) the request on the model input data queue and then reads the model output data queue by its identifier (key), waiting for the response.

Ideally business logic is isolated in callable programs, service or OPN-like. The data queue handling programs callses these programs that support the application business logic. The program processing the data queues accesses the business logic in whatever way makes sense.

The data queue handlers could be implemented as more than one program. Each program "peeks" at the input data queue (checking incoming messages) and the program which processes the "instruction" reads and removes it from the queue. An alternative would be to make the input data queues keyed and the different programs read for the specific keys that they operate on.

The model (server) program is "stateless". It executes with just the information from the input data queue, does it works, writes the output data queue, and waits for the next data queue entry. The program has no "state" from previous executions. The view-controller or client does all the remembering. This is similar to a CGI program for a HTML server and web page.

A very thorough explanation with examples of the data queue server model is presented in the IBM Redbook "Who Knew You Could Do That with RPG IV? A Sorcerer’s Guide to System Access and More" (SG24-5402-00). See section 5.2. The source for the examples can be downloaded. The instructions to download the source are in Appendix 1.

This design is scalable. One "server" program (or set of programs) can handle many clients. Additional instances of the "server" program (or set of programs) can be started if needed.

Remember, the "clients" or "view-controllers" can be anything which can communicate with the System i with data queues. Another System i, 5250 display programs, web services running on the System i and even C++ desktop clients. Use your imagination. The data queue handling programs won't know the difference nor will the model / business logic.

This design overcomes the performance problems of calling programs or commands through the Java toolbox. The model programs are running in batch on the System i. There is no start up penalty each time the client "communicates" with them.

What kind of performance can be expected?. The stock answer is "it depends."

For this blog I developed
- a demonstration model/server for the System i
- a demonstration Java view-controller for the desktop using the console (DOS prompt window)
The code for both of these will be posted in the future.

The System i model (server) is running on a Model 170. The PC Java view-controller connects to the System i through the Internet (cable modem). The model is like the data queue server described in the Redbook, it only move fields from one data queue to the other and doesn't do any processing.

The best times I have observed from the view-controller side is less than 100 milliseconds both to send the instruction to the model running on the 170 and to received the response from the model. That is under 100 ms (70-80 actually) each way. Not bad.

One of the console Java applications (see the code for this blog) shows the time in milliseconds it takes to send the message to the data queue and then to receive the result from the demonstration server. Test it yourself!

You can expect better performance in a local area network than over the Internet. However the performance will drop when the model actually does something, has to compete for resource, etc.

Posted by Bill Blalock on March 8, 2007 at 9:30 PM | Comments (0)

March 4, 2007 9:00 PM

Pitfalls of implementing MVC in a Java desktop with calls to System i programs

The MVC design pattern separates business logic and the user interface. For the System i a common way to separate the business logic is to put it into programs that are called by the user interface programs -- all of which reside on the System i. Most non System i clients can't directly call System i programs but Java can!. Java can execute (call) programs on the System i in two different way. Both of these ways have pitfalls for Java, Eclipse RCP or other desktop clients.

A side note ....

Actually I am already here. I hope to learn more about WDSc 7 amoung other things. Maybe even meet some of the developers.

Back to our regualarly schedule program .....

The usual way to isolate the data and business logic from the display is by coding it in programs, particularly service programs. This is not very useful to clients which aren't running on the System i. Non System i clients require a layer between the client and the programs which implement the business model. For example, web pages require the CGI layer between the HTML (web browser) and the service program.

Java can directly execute the programs which implement the business model, but not very well or very easily. The Java toolbox provides classes to execute commands and programs as well as a flavor of XML to simplify executing programs.

It most cases it takes too long to call a command or program through the Java toolbox each time the application interacts with the System i. The user won't be happy. Execution of these classes and methods has a lot of over head. The overhead is mostly on the System i which has to do all the work to prepare for the command or program, then execute it.

I am not criticizing the Java toolkit. These classes and methods are incredibly useful, and extend the capabilities of Java both on the System i and on clients connecting with a System i. But it is my experience they take too long when called each time the user does the equivalent of pressing Enter.

Java Native Interface (JNI) is a much more efficient method for Java to run programs that implement the business model. The Java JNI classes running on the System i execute programs natively. These programs can be business models, APIs, anything that could be called with CL, C or RPG! With JNI a Java client can have comparable performance to ILE in calling sub procedures, service programs, APIs, etc.

To access these JNI classes from the desktop client (or other platform) you have to use Remote Method Invocation (RMI).

Step by step it looks like this:
- the desktop application calls a local look-a-like method whose matching method is on the System i running as a RMI
- the network layer communicates with the matching RMI running on the System i
- the RMI natively uses JNI to call the business logic program or whatever written in RPG, C, CL, etc.
- the business logic program or whatever does its work and returns its results
- through JNI the results get back into the RMI program
- the network layer communicates the results to the look-alike method on the desktop
- the desktop application gets the results from the method

From the application side it is pretty simple:

results = MyRmiClass.foobar(param1, param2);

The above steps are hidden from the application programmer. The devil is in getting to the point where you are ready to write that line.

This is really a very efficient and robust method for Java on a remote system to execute programs on the System i. Really! "Remote system" can be a GUI client, another System i, a peer-to-peer application running on any platform which supports Java. Cool stuff.

And it is almost as complicated as it sounds -- but worth it if you have a Java something which needs fast and robust access to System i programs.

In summary, Java has two ways to call System i programs. The Java tool kit classes to call System i programs and commands are easy to use. For implementing MVC, they are too slow. The other way, JNI/RMI, is fast enough but is not easy to use. There is a better way.

Posted by Bill Blalock on March 4, 2007 at 9:00 PM | Comments (0)