zCBT© – Automation Toy

zCBT© - Rexx Functions for Automation

zCBT© is an automation toy for z/OS, the simplest solution for z/OS system event management at no cost. You don’t need special skill to automate your system using zCBT, except for Rexx programming. Once zCBT is properly setup, you will very able to manage your system events using very simple rexx scripting. All you need is rexx skill.

Although a toy, zCBT must run on z/OS as a subsystem, instead of an address space. zCBT subsystem supports 5 types of events:

  1. Messages (MSG)events for both WTO and WTOR. Message is trapped before sent to console, hence you can optionally suppress it. By trapping substring of message text, you can do several actions.  For WTOR message, you can reply it.
  2. Command (CMD) events.  Command is trapped before sent console, hence you can optionally suppress it.  This is an opportunity for you to have your own console commands.  By preparing rexx routine to trap certain command verbs (regardless they are valid commands) and their associated actions, you will have your own commands.
  3. End-of-job-step (EOS) eventsfor both jobs and STCs.  EOS event is trapped at almost the time of its occurrence and reporting condition codes. If you are a smart programmer, by using zCBT you can develop your own scheduler in rexx language.
  4. End-of-job (EOJ) eventsfor jobs and STCs.  EOJ event is simulated from all related EOS previously occurred.  Hence, there is a short time delay (less than 1 sec).
  5. Time-of-day (TOD) events.  TOD is very common event people can trap.  Internally it just STIMER or STIMERM macro which doesn’t need authorization as privileged routine.  Since rexx able to obtain date and time, zCBT only support TOD time event for both clock and interval time.

The way zCBT subsystem supports system automation is by providing some rexx functions. Via these functions, you can ask zCBT to capture any of the above 5-type of events you want to and notify your rexx program immediately when they occur. For a single request, you can optionally use cbevent() function. It will send requested event to zCBT and wait until the event is occurred.

For multiple events, you can't use cbevent() as it blocks your rexx program since the first issue. Your rexx program has to send all requests to zCBT before entering wait state. To do that, use cbset() function instead, to send each request to zCBT. Then, issue cbevent() without argument to go to wait state until any one of requested events are occurred. You rexx program can iterate to reissue cbevent() until all request events have occurred.

Your rexx program can run on TSO TMP session, TSO batch or non-TSO job. To run CBTIVP, you can start JCBTEST1 member of JCLLIB as either normal STC or under MSTR subsystem. If you need zCBT to automate your system startup, you must run your rexx program as STC under MSTR subsystem.

 

zCBT Supported Rexx Functions

You can easily handle 5 types of events, message, command, end-of-jobstep (EOS), end-of-job (EOJ) and time-of-day (TOD) by using ordinary rexx scripting or programming. To do so, zCBT provides 7 rexx functions, these are:

  1. cbcmd() - to issue system command
  2. cbevent() - to request and/or listen system events
  3. cbset() - to request system events
  4. cbstate() - to return state onformation of a workload (JOB, STC or TSU)
  5. cbwait() - to wait time of day
  6. cbwto() - to issue messages to console
  7. cbwtor() - to issue message to console and wait for reply

Cbcmd(), cbstate(), cbwto() and cbwtor() are nothing to do with event. They just ordinary tools to help user doing something easier with rexx.  Whereas cbevent(), cbset() and cbwait() are basic function to construct an automation framework.

 

Example of Automation Case

In standard rexx, you won't be able to issue system commands and WTO/WTOR messages. With zCBT active on the system, you can do it. For example, you want to start CICS1 and warn operator to do something when CICS1 is up.    Without zCBT, normally you do the following steps:

  1. Open SDSF panel and issue /START CICS1
  2. Keep scrolling SDSF log to see CICS1 is really up, indicated by message "DFHSI1517 Control is being given to CICS"
  3. Send message to operator by issueing TSO SEND command

With zCBT rexx, let say you won't to bother with automation.    Just to get the above 3 steps simpler.     You can code 5 rexx instructions or less like this:

a = cbcmd('START CICS1')
b = cbwait('+15')
c = cbstate('CICS1')
if word(c,10) = 'UP' then,
   d = cbwtor('Please do blah blah blah...')

The above rexx assumes CICS1 is ready in 15 seconds. The last instruction is issueing WTOR in case you need to know whether operator is responding your request. But during that, your terminal is suspended until this WTOR is replied. If you don't want your terminal held, use cbwto() instead.

The above example is not an automation. Just doing smarter job - something that impossible to happen without zCBT or other similar tool. CICS1 ready in 15 seconds is just guessing, isn't it? To ensure you notify operator after CICS1 up, you can change your code into 2 simple instructions like this:

a = cbcmd('START CICS1')
b = cbevent('MSG','DFHSI1517',,'MSG=Please do blah blah blah...')

The above example is simpler than your code, isn't it? Nevertheless, it is a real automation. Once it is executed, it issue START CICS1 on console, then going to sleep. Of course your terminal is suspended unless you do it as background task. Soonest the message DFHSI1517 is occurred, notification is sent to console as WTO message. Then this rexx is ended and your terminal is resumed.

Automation is simpler, right? Not just simplify your activities. It simplifies the codes too 🙂

 

How to Get All cbxxx() Functions Work?

All cbxxx() function calls are coded in ordinary standard Rexx. It is an IBM standard Rexx. So, the program must be placed as a member of a patitioned dataset (PDS) in either SYSEXEC or SYSPROC concatenation. If you run in background (batch JOB or STC), you can use IKJEFT01 or IRXJCL as usual. Existing functions also still available as usual. The syntax and coding rules also must comply IBM Rexx standards as usual. The only things affected are your local and user function you have installed in IRXFLOC and IRXFUSER, if any. If you don't have them, you have to decide which one is published. If you choose to publish your local/user functions, zCBT load library must be accessed as JOBLIB or STEPLIB in only jobs and/or TSO users who need zCBT supports. Conversly, your IRXFLOC and/or IRXFUSER are accessed as JOBLIB or STEPLIB, so public can use zCBT supported functions.

The thing you must do before run your Rexx program is to initialize zCBT. All cbxxx() functions will only work under support of zCBT subsystem. You don't need to initialize zCBT subsystem in IEFSSNxx. All you need is to start the following procedure under master scheduler subsystem (START ZCBT,SUB=MSTR).

//ZCBT PROC SSN=ZCBT,HLQ=SYS5                            
//INITSSI EXEC PGM=CBTEMS,REGION=0M,TIME=1440,   
//  PARM='SSN=&SSN'                                      
//STEPLIB DD DISP=SHR,DSN=&HLQ..CBT.LINKLIB  

This procedure will run as an STC for few seconds, then ended. Just to install zCBT module in the common memory segment called extended common storage area (ECSA) and hook up some certain z/OS control pointers called subsystem interface (SSI) to switch to this module, that's it. We say "hook up" because some of them are not published by IBM. ZCBT STC is ended. However, zCBT services remain active until next IPL. Actually, the services active even after its STC is ended, Because, zCBT is not a system task which run under task management control. Rather, it becomes a part of z/OS kernel (nucleus) which is in mainframe terminology called as a subsystem. Something a bit wider than system exit routine.

zCBT STC doesn't need JES to start. It can be started any time before or after JES up. Once it's started, zCBT subsystem then active and all cbxxx() functions work for you until next IPL.

 

Constructing Automation Framework using zCBT Toy

The use of cbevent() on the above example is an automation case. Just a simple rule of automation. Why? Because it only deal with one event and work once only. Automation framework means, an automation which contains a set of rules and works continously until a certain period or even up to next IPL. So, it may deal with several system events.

To catch more than one event, you can not specify event and action in a single cbevent() function call as shown in the above example. Rather, you need to use paired of cbset() cbevent(). Actually it is not really paired, rather, one cbset() call for each event, and one cbevent() call to wait all events requested by all cbset() calls. Cbevent() will bring your rexx into wait state and will wake up soonest either one of events requested by cbset() calls is occurred. So, you need to repeat cbevent() call in order to process all events you have requested in previous cbset() calls. Do not worry, this is not a must. zCBT will smartly abort all pending request when requester is prematurely ended.

For example, if you want to build a rule to (1) prevent anyone stop RMF and (2) restart RMF in case it's down. In this case, you need at least 2 events, (1) 2 command event of 'STOP RMF' and 'P RMF', and (2) an EOJ event of RMF termination. So, your codes will be like this:

a = cbset('CMD','STOP RMF','SUPPRESS') /* reject 'STOP RMF' command when issued */
b = cbset('CMD','P RMF','SUPPRESS')    /* reject 'P RMF' command when issued    */
c = cbset('EOJ','RMF')                 /* wake me up when RMF is down           */
Do forever
   x = cbevent()                /* i am sleeping until you (zCBT) ring the bell */
   if word(x,1) = 'EOJ' &,      /* When ringed, expected info in x variable is  */
      word(x,2) = 'JOB=RMF',    /* a string of 'EOJ JOB=RMF SCC=nnn MAXCC=nnn'  */
      then,                     /* informing that RMF just ended.               */
      y = cbcmd('START RMF')    /* If so, issue START RMF to bring it back up   */
End

In the above rexx codes, cbevent() call will be ringed by zCBT based on previous 3 cbset() calls. However, for the first 2 cbset() calls, no action is required. Actions for both calls are command suppression and done by zCBT based on 'SUPPRESS' word in 3rd argument. Nevertheless, cbevent() call will always be ringed. So, when ringed, returned information in x can be any one of 3 possible text string, (1) 'EOJ JOB=RMF SCC=nnn MAXCC=nnn', (2) 'CMD STOP RMF' or (3) 'CMD P RMF'. That's why value of x must be evaluated to avoid wrong action.

When this rexx is running, no one can terminate RMF although granted by security. Do you know how to beat it? Use CANCEL or FORCE command or stop the rexx. But, you can also make this rexx to suppress CANCEL and FORCE commands. You can also make it unstopable, by suppressing any termination command to this rexx. Such rexx program normally run as a system task (STC).

Another sample case probably interesting. To have optimum automation, some CICSes were prepared to accept console command without password. This is to avoid writing password in the script hardcodedly. Seems to be unsafe, right? Of course in such case, security is giving up, and now become zCBT responsibility.

Although now, by security, everyone allowed to issue CICS command from console, the guard from zCBT is coming. Now is your turn to think how to make it happen. All F or MODIFY commands for CICS are suppressed all the time except for certain time range, let say 00:00:00 to 00:15:00, in which some certain F commands to CICS will be issued for automation purposes. Soonest all the command done, the gate is closed again until the same time range of next day. So, only that time range, the situation is really unsecure. But, you must understand that the hole is only for command, instead of doing business transaction. So, don't need to worry too much. Beside, we can also shorten the time range and add some smart filterings to make it more secure.

Typically, to make it happen, we need more than one rexx program running on separate STCs. The following rexx codes describe how the guard works. With the end of the guard rexx below, the guard STC (let say named as GUARD) must be ended too, to releave F CICS command suppression. So the automated F CICS commands must be handle by other rexx program in other STC, let say named as FCICS. Soonest all F CICS commands done, rexx in FCICS must issue cbcmd('START GUARD') to reactivate CICS protection.

a = cbset('CMD','F CICS','SUPPRESS')  /* suppress all F CICS* commands */ 
Do forever                            /* start loop forever            */ 
   b = cbevent()                      /* wait for F CICS* commend      */
   if time('S') >= 0 &,               /* Only at 00:00:00 to           */ 
      time('S) < 900 then,            /* 00:15:00 the gate is open     */
      leave                           
   c = cbwto(b 'is not allowed')
End
c = cbwto('Please reenter' c 'command')
exit    

Although these examples are not an automation framework, we expect the illustration of events trapping and action firing mechanism using cbset() and cbevent() calls at least bring you to understanding how automation framework is supposed to be.      Based on this example, you may imagine whether this rexx program can be enhanced to accommodate all possible automation rules in your z/OS system.    Of course it depends on how complex your system is.     The thing you should understand is the limitation of this toy.     Cbevent() is for all events except TOD.   Whereas cbwait() is only for TOD.    So, you should not mix cbevent() and cbwait() in a single rexx program unless you really understand what you expect.     The other limitation is that zCBT require rexx skill.     If you have quite rexx skill and you sure zCBT can handle the whole automation rules needed in your system, you are starting to audit yourself why spending hundreds of thausands dollars for a certain vendor.

 

Automating System Startup and Shutdown

Normally in z/OS, the first system tasks required to up following NIP are JES (let say JES2), VTAM, TSO, SDSF, TCPIP and TN3270. Unfortunately, we can't just put them together in COMMNDxx for automatic start. Any one of them started before JES up will be prevented by system and fail because primary job entry gate (JES) is not active. Shilly, isn't it? Why don't just put them in a queue to wait for JES?

Well, we don't want to discuss it, instead, we want to resolve such given situation. The point is how to start VTAM soonest JES up, and start the rest after VTAM up, except for TN3270, will be started after TCPIP up. There are many tools to do it, such as VTAMAPPL. But, normally what they do is just using time estimation like the first above example. Moreover, they also need JES to up. Unfortunately, sometime JES need certain replies to complete its initialization. You got to do it manually, right? So, although it is put in COMMNDxx to get up automatically, JES must also be put in COMMNDxx for automatic start too. Otherwise, it won't work and system never establish.

Here is the solution... using a free toy, zCBT. You need to prepare 2 simple rexx programs, one for system startup and one for shutdown. For system startup, we only work with message events to build the rule. First is starting JES2. Soonest after message $HASP492 occurred, which indicates that JES ready, start VTAM and some other tasks. VTAM readiness is indicated by message IST020I. So use the occurrence of that message to start TSO. At this time, trigger for TCPIP is readiness of USS indicated by message EZZ9291I. Then TCPIP readiness, indicated by message EZAIN11I is used to trigger TN3270 and other TCP/IP applications. As an automation, this rexx must also be able to do other things to guarantee this start up procedure to complete the work unattendedly. In this example, we prepare some JES2 replies as necessary. So, the codes will be like this, more or less.

/* Show in console it is starting */
 x = cbwto('CBTIPL: Startup procedure in progress...')     

/* Start JES */ 
 x  = cbcmd('START JES2')      

/* Ask zCBT to wake me up if the following msgs occurred */                         
     x  = cbset('MSG','$HASP426')                           
     x  = cbset('MSG','$HASP492')                           
     x  = cbset('MSG','$HASP454')                           
     x  = cbset('MSG','$HASP441')                           
     x  = cbset('MSG','$HASP471')                           
     x  = cbset('MSG','IST020I')                            
     x  = cbset('MSG','EZZ9291I')                           
     x  = cbset('MSG','EZAIN11I')                           
 
/* Main logic of the rule */
 jes2  = 0                                                  
 jrpy  = 0                                                  
 jreq  = 0                                                  
 vtam  = 0                                                  
 omvs  = 0                                                  
 tcp   = 0                                                  
 all   = 0                                                  
 Do forever                                                 
     event  = cbevent()                                     
     evtype = strip(word(event,1))                          
     If evtype = 'MSG' then do                              
         info = strip(word(event,2))                        
         Select                                             
             When info = '$HASP492' then,                   
                  Do                                        
                      jes2 = 1                              
                      x  = cbcmd('START VTAM')              
                      x  = cbcmd('START SDSF')              
                      x  = cbcmd('START IRRDPTAB')          
                      x  = cbcmd('START EZAZSSI,P=&SYSNAME')
                  End                                       
             When info = 'IST020I' then,                    
                  Do                                        
                      vtam = 1                              
                      x  = cbcmd('START TSO')               
                  End                                       
             When info = 'EZZ9291I' then,                   
                  Do                                        
                      omvs = 1                              
                      x  = cbcmd('START TCPIP')             
                  End                                       
                                                            
             When info = 'EZAIN11I' then,                   
                  Do                                        
                      tcp  = 1                                        
                      x  = cbcmd('START TN3270')                        
                      x  = cbcmd('START HTTPD1')                      
                  End                                                 
                                                                      
             Otherwise,                                               
                  Do                                                  
                      jeswtor = strip(word(event,3))                  
                      Select                                          
                         When jeswtor = '$HASP441' |,                 
                              jeswtor = '$HASP454' |,                 
                              jeswtor = '$HASP471' then,              
                             Do                                       
                                Parse var info "(REPLYID=" rpyid ")" .
                                rpytxt = "REPLY " || rpyid || ",Y"    
                                x  = cbcmd(rpytxt)                    
                                jrpy = 1                              
                             End                                      
                         When jeswtor = '$HASP426' then,              
                             Do                                       
                                Parse var info "(REPLYID=" rpyid ")" .
                                rpytxt = "REPLY " || rpyid ||,        
                                         ",NOREQ"                     
                                x  = cbcmd(rpytxt)                    
                                jreq = 1                              
                             End                                      
                         Otherwise NOP                                
                      End                                             
                  End                                                 
         End                                                          
         all = jes2 * vtam * omvs * tcp                               
         if all = 1 then LEAVE                                        
     End                                                              
 End                                                                  
 x = cbwto('CBTIPL: Startup procedure complete')                      
 Exit                                                                 

For system shutdown, the first thing to trigger is a shutdown request. It can be anything, let say a command SHUTDOWN. This command is not provided in z/OS, and so far, no subsystem use this word as its command. So we can use it freely. The rexx must require zCBT to notify when a word SHUTDOWN is entered as a command. Once this word is trapped, broadcast warning that system is going to shutdown shortly followed by count down for certain amount of time. The rexx codes typically look like this:

a = cbevent('CMD','SHUTDOWN','SUPPRESS') /* wait for SHUTDOWN is issued and */
                                         /* suppress it as it is actually   */
                                         /* unknown command                 */
b = "SEND 'Please pack up and logout immediately, ",
    "system is going to shutdown shortly'")
c = cbcmd(b)                             /* execute the text in var b       */
d = cbwait('+01:00')                     /* sleep 1 minutes                 */
Do i = 5 to 1 by -1                      /* starting count down             */
   b = "SEND 'System is going to shutdown in" || i || "minutes.'"
   c = cbcmd(b)                          /* count down warning...           */
   d = cbwait('+01:00')                  /* sleep 1 minutes                 */ 
End

After certain amount of time, then start the true shutdown procedure steps. First round is stopping all highest level tasks, the tasks which no other tasks depend on them. They are typically CICS transactions, users batch jobs, TSO users (TSU) and so forth. This vary per environment. Certain sites even presume this round must be done manually by users in respond to the previous warning. Whereas in some other sites, this round require to be automated. Idea of the above F CICS commands automation can be adopted in this case.

Second round is stopping all the rest which become the highest level after all the real highest level tasks are down. Typically, CICS, initiators, TSO TCAS, FTP server, RMF and so on. Repeating such rule, next round must be TN3270, HTTP server. Afterward, based on the same rule, subsequent round must be TCPIP, then VTAM and BPXOINIT. So, the final round is only JES.

At the moment, we don't plan to discuss shutdown processing deeper, as it is usually not so important. Let it be a homework for those who are interested.

 

How to Activate System Startup?

One thing that we think important is how to execute system startup rexx program soonest zCBT subsystem is initialized. In old version of zCBT, such thing can be done by specifing its STC name in IEFSSNxx. Current zCBT is using dynamic SSI, so you shouldn't use IEFSSNxx. Specifing it as START command in COMMNDxx along with START command for zCBT initialization, no guarantee its execution will be done after zCBT initialization is completed. In the meantime, if it is in fact, is executed before zCBT completely initialized, of course all its cbxxx() calls are failed.

So, the best way is, put it in second step in zCBT initialization procedure. If you have system shutdown procedure, put it in third step. Let say, the procedure name is ZCBT, put 'START ZCBT,SUB=MSTR' in your COMMNDxx and remove all other START commands, except for system tasks that run under master scheduler, such as LLA, VLF, RRS and so forth. Prepare ZCBT procedure in 3 steps like this:

//ZCBT PROC SSN=ZCBT,HLQ=DERU                            
//* --------------------------------------
//*       zCBT Initialization 
//* --------------------------------------
//INITSSI EXEC PGM=CBTEMS,REGION=0M,TIME=1440,          
//  PARM='SSN=&SSN'                                      
//STEPLIB DD DISP=SHR,DSN=&HLQ..CBT.LINKLIB              
//* ---------------------------------------
//*    Automated System Startup Procedure
//* ---------------------------------------
//STARTUP EXEC PGM=IRXJCL,TIME=1440,REGION=0M,           
//  PARM='STARTUP'                                
//STEPLIB  DD DISP=SHR,DSN=&HLQ..CBT.LINKLIB             
//SYSTSPRT DD DSN=&&SYSUT1,UNIT=VIO,SPACE=(1024,(50,20)),
//            DCB=(RECFM=FB,LRECL=121,BLKSIZE=1210)      
//SYSEXEC  DD DISP=SHR,DSN=NSI.REXXLIB,                  
//            UNIT=3390,VOL=SER=&NSIVOL                  
//SYSTSIN  DD DUMMY                                      
//* ---------------------------------------
//*   Automated System Shutdown Procedure
//* ---------------------------------------
//SHUTDOWN EXEC PGM=IRXJCL,TIME=1440,REGION=0M,           
//  PARM='SHUTDOWN'                                
//STEPLIB  DD DISP=SHR,DSN=&HLQ..CBT.LINKLIB             
//SYSTSPRT DD DSN=&&SYSUT1,UNIT=VIO,SPACE=(1024,(50,20)),
//            DCB=(RECFM=FB,LRECL=121,BLKSIZE=1210)      
//SYSEXEC  DD DISP=SHR,DSN=NSI.REXXLIB,                  
//            UNIT=3390,VOL=SER=&NSIVOL                  
//SYSTSIN  DD DUMMY                                      

 

Read zCBT manual to learn the detail of each function.

Download Rexx Functions for Automation

Product Summary
Name zCBT Automation
Type Subsystem and Rexx Functions Package
Environment V1.10 (new) for z/OS
Special Sensitive, privileged, APF authorized
Coding Assembler H
Author Deru Sudibyo
License Freeware
Note Need subsystem name
Download link