Before using this information, be sure to read the general information under Notices IBM Rational Developer for System z.
This edition applies to IBM Rational Developer for System z Version 8.5 (program number 5724-T07) and to all subsequent releases and modifications until otherwise indicated in new editions.
Order publications by phone or fax. IBM Software Manufacturing Solutions takes publication orders between 8:30 a.m. and 7:00 p.m. eastern standard time (EST). The phone number is (800) 879-2755. The fax number is (800) 445-9269. Faxes should be sent Attn: Publications, 3rd floor.
You can also order publications through your IBM representative or the IBM branch office serving your locality. Publications are not stocked at the address below.
IBM welcomes your comments. You can send your comments by mail to the following address:
When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.
Note to U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
You may see various error messages while running Rational® Developer for System z®. This section lists some common messages, their causes, and what you can do to correct the problem.
For explanations of the purposes of the different log files mentioned in this document, see the Rational Developer for System z Host Configuration Reference Guide (SC14-7290), chapter "Troubleshooting configuration problems", section "Log files".
This section describes errors you might see in the rseserver.log.
ERROR RseDaemon: server failure: error binding socket
RSE server is not allowed to bind to a port. In this case, _RSE_PORTRANGE is probably defined, and the ports in this range are secured in TCPIP PROFILE, but with an incorrect filter. RSED* must be allowed to bind to these ports. Note: The * is important.
None.
ERROR RseDaemon: getServerPort(): result=-1;No Access Authority (FEKAPPL)
This problem can be caused by faulty security setup for the APPL class (as indicated in the message). But missing program control status for LE modules (see RACF® setup) can also result in this error, because Developer for System z is not able to load the code needed to do the security checks. In that case, the system log will show RACF messages like the following messages:
None.
ERROR RseDaemon: userid: User has insufficient permission to profile FEKAPPL in the APPL class, RACF return and reason code=00d8
This problem can be caused by faulty security setup for the APPL class (as indicated in the message), but it can also indicate a range of other problems.
The listed return code is the z/OS® UNIX reason code (errnojr), and is documented in the UNIX System Services Messages and Codes manual.
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/BPXZA8A0/3.3
The following table lists some of the possible reason codes:
00d8 |
JRNotServerAuthorized The calling address space is not permitted to the BPX.SERVER facility class or the BPX.SERVER facility class is undefined and caller not a superuser (UID=0). |
02af |
JREnvDirty The specified function is not supported in an address space where a load was done that is not program controlled. |
None.
This section describes errors you might see in the syslog.
USER COMPLETION CODE=4093 REASON CODE=00000090
ABEND issued during initialization when errors were detected.
See Language Environment Run-Time Messages (SA22-7566), section "Language Environment® Abend Codes", code U4093 (X'FFD').
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/CEEA9190/SPTCS00020 (z/OS 1.10)
90x - Condition management for MVS™ could not be initialized
Explanation: This abend and reason code, issued during JMON startup, usually indicates that an OMVS RACF segment has not been defined for the user ID associated with the JMON started task.
Additional information: None.
IEF451I IBMUSER1 PLI STP0000 - ENDED BY CC 1016
For PL/I compiles, the PL/I listing data set (DD PLI.SYSPRINT) has the wrong characteristics. A valid combination is:
DSORG=PO,RECFM=VBA,LRECL=137,BLKSIZE=27998
None.
CRRZI0277E System ABEND E37-4 CRRZI0282E A data set opened for output used all space available to or on the current volume, and no more volumes were available.
In most cases, an E37-4 does not mean the volume is out of space, but rather that the target data set is out of extents. But Developer for System z shows the official MVS message text, so making the situation clear is difficult.
For the explanation of message IEC032I E37-4, see:
http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/iea2m7a0/3.18
A data set opened for output used all space available to or on the current volume, and no more volumes were available. Change the JCL to specify more volumes.
None.
This section describes errors you might see in the rsecomm.log.
ERROR MVSFileSystemMiner: java.nio.BufferUnderflowException
java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:419)
at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:351)
at com.ibm.fmi.server.dataprovider.fm.FMConnection.readSocketMessage
(FMConnection.java:178)
CRRZF6001E Error checking IBM® File Manager Version
File Manager (FM) is not running APF-authorized. FM Version 10 does not have PTF UK54428 applied. The manual bypass is to copy module FMN0POPT from SFMNMOD1 to SFMNMODA.
(For this situation, Rational Developer for System z Version 7.6 client shows error message: Error checking IBM File Manager Version with CRRZF6001E.)
None.
ERROR FFSServerImpl: java.lang.RuntimeException: No lock daemon port
java.lang.RuntimeException: No lock daemon port
at
com.ibm.ftt.rse.mvs.server.filesystem.impl.FFSServerImpl.sendCommandToLockDaemon
(FFSServerImpl.java:325)
rsed.envvars is not updated to support the lock daemon. This situation applies to Rational Developer for System z Version 7.5.0.1 and higher only.
Update rsed.envvars to support the lock daemon.
None.
ERROR FFSServerImpl: java.net.ConnectException: EDC8128I Connection refused.
java.net.ConnectException: EDC8128I Connection refused.
at java.net.PlainSocketImpl.socketConnect(Native Method)
This error message means that the lock daemon (FFSServerImpl) or JMON (JESMiner) is not listening on the designated port.
See Language Environment Run-Time Messages (SA22-7566), message DC8128I.
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/CEEA9190/SPTCS00425 (z/OS 1.10)
ERROR logHandler: java.lang.UnsatisfiedLinkError: IRRRacf (Not found in java.library.path)
/usr/lib/libIRRRacf.so is not in LIBPATH, or /usr/lib/libIRRRacf.so is not accessible.
This message is informational (INFO) in Developer for System z Version 7.5. The message is an error (ERROR) in Version 7.6.
ERROR class com.ibm.etools.zos.server.ZosSystemService:
java.lang.RuntimeException: pThreadSecurity failed, rc=7
at com.ibm.etools.zos.server.ZosSystemService.setThreadSecurity
If unsuccessful, pthread_security_np() sets errno to one the following values.
RC |
Errno |
Explanation |
1 |
EACCES |
The password or PassTicket, or the password phrase provided is not valid for the passed userid. |
2 |
EINVAL |
A parameter is not valid. |
3 |
EMVSERR |
An MVS environmental error or internal error occurred. |
4 |
EMVSEXPIRE |
The password or PassTicket, or the password phrase provided has expired. |
5 |
EMVSSAF2ERR |
The SAF call to the security product incurred an error. |
6 |
EMVSSAFEXTRERR |
The SAF call to the security product incurred an error. |
7 |
EPERM |
1a. The process does not have appropriate privileges to set a thread-level security environment. Note: /usr/lib/libIRRRacf.so not being program controlled will result in case 1a. 1b. The caller is not permitted to the BPX.SERVER FACILITY class profile or BPX.SERVER is not defined and the caller is not a superuser. 1c. No password or PassTicket, or password phrase is provided and the caller is not defined as a surrogate of the passed user ID. 2. The caller is not a superuser and permitted to the BPX.DAEMON FACILITY class profile or BPX.DAEMON is not defined and the caller is not a superuser. |
8 |
ESRCH |
The user ID provided as input is not defined to the security product or does not have an OMVS segment defined. |
10 |
ENOSYS |
The function is not implemented. |
100 |
? |
new error, not mapped, see C/C++ Run-Time Library Reference (SA22-7821) |
See C/C++ Run-Time Library Reference (SA22-7821), section discussing pthread_security_np()
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/EDCLB190/3.740 (z/OS 1.10)
pthread_security_np() is a wrapper for z/OS UNIX service BPX1TLS/BPX4TLS. See http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/BPXZB1B0/2.142 (z/OS 1.12).
cmd=java -D...
When the RSE daemon starts, it lists the content of rsed.envvars in rsedaemon.log (line=''). This is followed by the thread pool startup information (cmd=java env[] ). Thread pools started dynamically to satisfy a connection request show up with s-markers (scmd=java senv[]).
None.
INFO logHandler: 66107:smile:4464456:8108-8118:9828744
The message is logged at the start of RSE server initialization, before the client connects. The message is part of a sequence of messages produced during logon. Example:
INFO logHandler: 66107:smile:4464456:8108-8118:9828744
INFO logHandler: ProcessId=66107: port 8108 assigned to smile
INFO logHandler: ProcessId=66107: smile logged on
None.
INFO logHandler: ProcessId=66107: port 8108 assigned to smile
The message is logged after the RSE server thread is initialized and the port number is sent back to the client. The message is not related to the client actually connecting.
Note: This message does not exist in Developer for System z Version 7.5.x.x. In that version, the message logHandler: ProcessId=pid: userid logged on is issued instead.
The message is part of a sequence of messages produced during logon. Example:
INFO logHandler: 66107:smile:4464456:8108-8118:9828744
INFO logHandler: ProcessId=66107: port 8108 assigned to smile
INFO logHandler: ProcessId=66107: smile logged on
None.
INFO logHandler: ProcessId=66107: smile logged on
Developer for System z Version 7.5: The message is logged after the RSE server thread is initialized and the port number is sent back to the client. The message is not related to the client actually connecting.
Developer for System z Version 7.6: The message is logged after the client actually connects.
logHandler: ProcessId=pid: userid logged on
with
the user logs location of userlog.The message is part of a sequence of messages produced during logon. Example:
INFO logHandler: 66107:smile:4464456:8108-8118:9828744
INFO logHandler: ProcessId=66107: port 8108 assigned to smile
INFO logHandler: ProcessId=66107: smile logged on
None.
INFO RseDaemon: startProcesses(): spawn command result=156:132
The message is logged when the daemon starts a thread pool.
Note:
The errno value picks up bogus values when the thread pool creation is successful. This problem is resolved in Developer for System z Version 8.0.2.0 (37988).
The layout of the message is changed in Version 8.0.2.0.
INFO RseDaemon: startProcesses(): spawn command: parentPID(672) errno(0)
dataFd(144) ctlFd(150)
The message is part of a sequence of messages produced during thread pool startup. Example:
INFO RseDaemon: startProcesses(): spawn command result=156:132
INFO RseDaemon: startProcesses(): send Query to the Server process
INFO RseDaemon: startProcesses(): Server process started. (processId=157)
INFO : ProcessId=157, Memory_Usage=1%, Used_Memory=987KB, threads=3, TCBs=12
None.
INFO RseDaemon: startProcesses(): spawn command: parentPID(672)
errno(0) dataFd(144) ctlFd(150)
The message is logged when the daemon starts a thread pool.
Layout of the message is different before Developer for System z Version 8.0.2.0.(37988). Example:
INFO RseDaemon: startProcesses(): spawn command result=ppid:errno
The message is part of a sequence of messages produced during thread pool startup. Example:
INFO RseDaemon: startProcesses(): spawn command: parentPID(156)
errno(0) dataFd(144) ctlFd(150)
INFO RseDaemon: startProcesses(): send Query to the Server process
INFO RseDaemon: startProcesses(): Server process started. (processId=157)
INFO : ProcessId=157, Memory_Usage=1%, Used_Memory=987KB, threads=3, TCBs=12
None.
INFO RseDaemon: getServerPort(): select timeout
The message is logged when the thread pool is unresponsive (normally during client logon) and is flagged with timeout status.
INFO logHandler: pid:userid:ticket:port:clientid
The message is part of a sequence of messages produced during a client logon that is encountering problems. Example:
INFO logHandler: pid:userid:ticket:port:clientid
INFO RseDaemon: getServerPort(): select timeout
INFO RseDaemon: selectThread poolProcess(): timeout, process=pid
INFO LogHandler: Thread poolMonitor:pid: command=TIMEOUT
INFO logHandler: ProcessId=pid: port port assigned to userid
None.
INFO RseDaemon: getServerPort(): select timeout
A server process to handle this new client connection was selected, but it does not respond in time. After three attempts, the process is deemed unusable and another or new server process is selected.
None.
This section describes errors you might see in the syslog.
USER COMPLETION CODE=4083 REASON CODE=00000003
The back chain was found in error.
See Language Environment Run-Time Messages (SA22-7566), section "Language Environment Abend Codes", code U4083 (X'FF3').
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/CEEA9190/8.13?SHELF=&DT=20080602034358 (z/OS 1.10)
03x - The save area chain terminated with a 0 chain pointer
Explanation: This abend and reason code, give during client connect or daemon IVP, usually indicates that APAR QO94976 for CA's ACF2 is not installed
Additional information: None.
ICH408I USER(JOHN) GROUP(SYSPROG) NAME(JOHN DOE)
Upon permission failure, RACF issues message ICH408I in the system log. RACF shows you what resource was requested on line 2, but with USS based resources, RACF does not tell you where exactly the permission failure occurred. The failure could be anywhere in the path leading to the requested resource. In the following example, /u/smile is allocated with 700, denying user JOHN access to /u/smile/.eclipse/RSE/SMILE.
ICH408I USER(JOHN) GROUP(SYSPROG) NAME(JOHN DOE)
/u/smile/.eclipse/RSE/SMILE
CL(DIRSRCH) FID(01D5C9D4E5C8C200020B000000000003)
INSUFFICIENT AUTHORITY TO LSTAT
ACCESS INTENT(--X) ACCESS ALLOWED(GROUP ---)
EFFECTIVE UID(0096061081) EFFECTIVE GID(0999999999)
Use the looppath.sh shell script to list every directory of the path (looppath.sh <path> <cmd>).
$ /usr/lpp/rdz/bin/looppath.sh /u/smile/.eclipse/RSE/SMILE ls -dog
drwxr-xr-x 104 3904 Sep 9 11:41 /
dr-xr-xr-x 25 0 Oct 27 11:19 /u/
drwx------ 3 8192 Oct 2 2008 /u/smile/
drwxrwxrwx 3 8192 Jul 1 2008 /u/smile/.eclipse/
drwxrwxrwx 3 8192 Sep 3 2008 /u/smile/.eclipse/RSE/
drwxrwxrwx 2 8192 Jan 11 2009 /u/smile/.eclipse/RSE/SMILE/
None.
RSEG1057 Connect failed. Host <host> not found or not responding
Second logon with same user ID kills the first one (from Developer for System z Versions 7.5.1.1. and 7.6.0.1). See Multiple logons by a single userid.
None.
RSEG1242 Daemon failed to launch server on <host> using port <port>.
Connection refused.
"Connection refused" is a TCP/IP message saying that nobody is listening on the port you are trying to connect to. Here are the possible reasons no one is listening:
telnet <host> <port>
RSEG1242 Daemon failed to launch server on <host> using port <port>
server failure: <message>
The details show the error message shown in rseserver.log (only if the RSE daemon or server was reached). Host logs are probably needed to see what is going on and resolve the issue. See the rseserver.log section for more information.
RSEG1242 Daemon failed to launch server on <host> using port <port>
userid: <message>
The details show the error message shown in rseserver.log (only if RSE daemon or server was reached). Host logs are probably needed to see what is going on and resolve the issue. See rseserver.log section for more information.
None.
RSEG1243 Failed to connect to the daemon on <host> using port <port> with an unexpected exception.
This message indicates that there was an exception while launching the server using the daemon. The exception is shown in the detailed message.
java.net.SocketTimeoutException: Read timed out
The client was able to connect to RSE daemon at TCP/IP level, and it was able to send user ID or password, but the RSE daemon is not able to send something back (server port and ID token). This is probably a firewall issue, prohibiting outbound traffic. Or there are two client PCs on the network with the same IP address, and the answer is routed back to the other machine.
None.
ERROR class com.ibm.etools.zos.server.ZosClient: (R_ticketserv)
SafRc=8, racfRc=8 racfRsn=16
at com.ibm.eserver.zos.racf.IRRPassTicket.generate
ERROR class com.ibm.etools.zos.server.ZosSystemService:
SafRc=8, racfRc=8 racfRsn=16 at
com.ibm.eserver.zos.racf.IRRPassTicket.generate
When this error occurs during logon, it is due to a security setup error (or missing ACF2 or TSS maintenance). See PassTicket - Access not authorized (8/8/16) when it occurs randomly with RC 8/8/16 and the security product is ACF2 or TSS.
See Security Server RACF Callable Services (SA22-7691), section R_ticketserv (IRRSPK00)
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/ICHZD190/2.48.7?SHELF=ICHZBK90&DT=20080520114546 (z/OS 1.10)
R_ticketserv (IRRSPK00) return codes |
|||
SAF return code |
RACF return code |
RACF reason code |
Explanation |
0 |
0 |
0 |
The service was successful. |
4 |
0 |
0 |
RACF is not installed. |
8 |
8 |
0 |
Invalid function code. |
8 |
8 |
4 |
Parameter list error. |
8 |
8 |
8 |
An internal error was encountered. |
8 |
8 |
12 |
A recovery environment could not be established. |
8 |
8 |
16 |
Not authorized to use this service. |
8 |
8 |
20 |
High order bit was not set to indicate last parameter. |
8 |
12 |
8 |
Invocation of the Security Server Network Authentication Service Program Call (PC) interface failed with a 'parameter buffer overflow' return code. This indicates an internal error in IRRSPK00. |
8 |
12 |
12 |
Invocation of the Security Server Network Authentication Service Program Call (PC) interface failed with an 'unable to allocate storage' return code. The region size for the Security Server Network Authentication Service started task (SKRBKDC) should be increased. |
8 |
12 |
16 |
Invocation of the Security Server Network Authentication Service Program Call (PC) interface failed with a 'local services are not available' return code. This indicates that the Security Server Network Authentication Service started task (SKRBKDC) address space has not been started or is terminating. |
8 |
12 |
20 |
Invocation of the Security Server Network Authentication Service Program Call (PC) interface failed with an 'abend in the PC service routine' return code. The symptom record associated with this abend can be found in the logrec data set. |
8 |
12 |
24 |
Invocation of the Security Server Network Authentication Service Program Call (PC) interface failed with an 'unable to obtain control lock' return code. This can occur if the task holding the lock is not being dispatched (for example, a dump is in progress). |
8 |
16 |
X'nnnn nnnn' |
The Security Server Network Authentication Service was not able to successfully extract the client principal name from the supplied Kerberos V5 ticket. X'nnnnnnnn' is the Kerberos return code. Refer to the Security Server Network Authentication Service documentation for more information. |
8 |
16 |
28 |
Unable to generate PassTicket. |
8 |
16 |
32 |
PassTicket evaluation failure. Possible reasons include: PassTicket to be evaluated is not a successful PassTicket. The PassTicket to be evaluated was already evaluated before and replay protection is in effect. No PTKTDATA profile exists to match the specified application. An internal error occurred. |
This section describes errors you might see in the stderr.log.
pthread: EDC5163I SAF/RACF extract error.
See Language Environment Run-Time Messages (SA22-7566), section discussing XL C/C++ Run-Time Messages, message EDC5163I.
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/CEEA9190/SPTCS00264 (z/OS 1.10)
An authorization failure occurred when attempting the service. This message is equivalent to the z/OS UNIX System Services errno, EMVSSAFEXTRERR.
This message is probably linked to the pThreadSecurity failed message in rsecomm.log.
None.
pthread: EDC5111I Permission denied.
See Language Environment Run-Time Messages (SA22-7566) , section discussing XL C/C++ Run-Time Messages, message EDC5111I.
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/CEEA9190/SPTCS00217 (z/OS 1.10)
An attempt was made to access a file in a way that violates its file access permissions. This message is equivalent to the POSIX.1 EACCES errno.
This message is probably linked to the pThreadSecurity failed message in rsecomm.log.
None.
TSS7250E 151 J=<job/AS name> A=<userid> TYPE=<security class> RESOURCE=<resource to be checked>
Unlike ICH408I, the message does not tell which security profile blocked the access, or which access was requested.
None.
SAFOETKT U4083
SAFOETKT U4083 abends in the syslog with client connects and tests with fekfivpd.
The following fixes are known to help with PassTicket problems when using ACF2/TSS.
None.
INFO LogHandler: ThreadSecurity error: rc=5
ERROR RseDaemon: getServerPort(): result=-1;RACF Internal Errors (FEKAPPL)
The following fix resolves the ACF2 error:
The following fix resolves the TSS error:
This problem occurs in Developer for System z Version 7.5.1.1, Version 7.5.1.2, Version 7.6.0.0, Version 7.6.0.1 and Version 7.6.1.0 only.
syslog:
ACF04056 ACCESS TO RESOURCE IRRPTAUTH.FEKAPPL.userid TYPE
RPTK BY userid NOT AUTHORIZED
rseserver.log:
ERROR class com.ibm.etools.zos.server.ZosClient: (R_ticketserv)
SafRc=8, racfRc=8 racfRsn=16 at com.ibm.eserver.zos.racf.IRRPassTicket.generate
Any command can cause this PassTicket generation error with ACF2 or Top Secret. The initial report was for JESminer but the problem may still exist for other Rational Developer for System z host code.
The JESMiner thread is running continuously until the client disconnects from the RSE Server. The JESMiner thread generates a PassTicket at the time of CONNECT request. When the same thread tries to generate a PassTicket again (for example, for reconnect), ACF2/TSS issues error message "Access not authorized" for the PassTicket function. This is because ACF2/TSS changes the effective ID to the client ID after the thread-level security is established for the client. On the other hand, RACF does not change the effective ID even if the thread-level security is changed for the client. So, RACF always generates the PassTicket under the authority of the started task ID, but ACF2/TopSecret tries to generate the PassTicket under the authority of the client ID after the thread-level security is established. The client ID does not have any authority to generate a PassTicket, and so SafRc=8, racfRc=8, racfRsn=16 (codes for "no authority") are issued.
The following Rational Developer for System z fix resolves the ACF2/TSS-specific problem:
In another occurrence, a change was made in Developer for System z Version 8.0.1 that resulted in two pthread_secutity_np() calls in a single thread. Due to the effective ID switching done by ACF2/TSS, the second call fails with return codes 8/8/16.
The following Developer for System z fix resolves this ACF2/TSS-specific problem:
None.
ERROR ServerThread: 517: SafRc=8, racfRc=8 racfRsn=4
com.ibm.eserver.zos.racf.IRRPassTicket.generate(IRRPassTicket.java:226)
ERROR RseDaemon: getServerPort(): result=0;server failure: SafRc=8,
racfRc=8 racfRsn=4
com.ibm.etools.zos.server.ZosSystemService.generatePassTicket
(ZosSystemService.java:96)
com ibm.etools.zos.server.ServerThread.run(ServerThread.java:83)
java.lang.Thread.run(Thread.java:736)
Due to a bug in TSS and ACF2, Developer for System z cannot generate a PassTicket when 64-bit Java™ is used, and the call ends with SafRc=8, racfRc=8, racfRsn=4 (Invalid parameter list). The following PTFs resolve this issue:
None.
INFO LogHandler: ThreadSecurity error: rc=1
ERROR RseDaemon: getServerPort(): result=-1;PassTicket Generation Error
A customer using 31-bit Rational Developer for System z Version 7.6.2.1 host got this error after upgrading TSS from Release 14 to Release 15.
The ThreadSecurity error: rc=1 indicates that our pthread_security_np() call failed with return code EACCES, which means "Permission is denied; the specified password is incorrect." This indicates things fail in the usage of the generated PassTicket, not the generation itself.
The SIGNMULTI flag (which is the TSS equivalent of the NO REPLAY PROTECTION used by RACF) was good:
TSS ADD(NDT) PSTKAPPL(FEKAPPL) SESSKEY(KEY16 ) SIGNMULTI
CA TSS support indicated that a PTF will be supplied.
None.
ck_process_owner <userid> <groupid> <uid> <gid> 8 8 4
mm/dd/yy yy.ddd h.mm.ss <jobname> <source> <sysname> <cpu>
Failed - Caller not authorized to use this callable service
Real Uid : 0 Effective UID: 0
Saved UID : 0
Target PID : 0
Signal code : Signal n/a for type 3 (getpsent) requests
Signal code : 0
The ISPF Client Gateway checks the active processes to see if there is already a server active from a previous invocation of the service. To do so, it analyses the output of the getpsent callable service. z/OS UNIX will try to collect information for every active process (including the ones the requester has no privileges for) and return only those for which the requestor is authorized.
So the error message (which is only visible in trace-like security reports) is related to how z/OS UNIX works. It is NOT hindering normal operation.
For information, see http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/BPXZB190/2.248?SHELF=BPXZSH91&DT=20080521092140 (z/OS 1.10).
JM227I Unexpected RACF error. SAF 8, RACF 28, Reason 0, User GP1TAJ
See Security Server RACROUTE Macro Reference (SA22-7692), section RACROUTE REQUEST=VERIFY
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/ICHZC690/3.61.2?SHELF=ICHZBK90&DT=20080603040614 (z/OS 1.10)
08 means that the requested function has failed.
28 means that the OIDCARD (operator ID) parameter is required but not supplied.
OIDCARD=oid addr specifies the address of the currently defined operator identification card of the user who has entered the system. The address points to a field that is 1 byte in length, followed by the operator ID card.
OIDCARD is used for operators logging on without password, and is a password substitution for them.
A possible scenario for getting this error is if JMON received a null password.
None.
This section describes errors you might see in the syslog.
ABEND=SEC6 U0000 REASON=0F01C008
An error occurred during processing of a callable service to z/OS UNIX System Services.
See MVS System Codes (SA22-7626), section "System Completion Codes", code EC6.
Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/IEA2H790/SPTM014775 (z/OS 1.10)
Explanation: This message, issued during RSE daemon startup, usually indicates that an OMVS RACF segment has not been defined for the user ID associated with the RSED started task.
Additional information: None.
Explanation: This message, given during improper connection termination, usually indicates the ISPF Client Gateway caused RSE to crash. Check for ISPF APAR OA27174 - PTF UA45333 (for z/OS 1.10), UA45332 (for z/OS 1.9), or UA45242 (for z/OS 1.8).
Additional information: ISPF APAR OA29821 is also known to resolve a SEC6 ABEND in RSE. (Another symptom is never-ending Client Gateway address spaces.) Check for PTF UA49277 (for z/OS 1.11), PTF UA49276 (for z/OS 1.10), PTF UA49275 (for z/OS 1.9), or PTF UA49274 (for z/OS 1.8).
This chapter presents some useful hints and tips for setting up Rational Developer for System z.
In Version 7.1.0.0, Developer for System z stopped using APPC as the default method for accessing MVS based resources. The default now is to use an SCLMDT (FMID HSD3310) function.
In Version 7.5.0.0, the ISPF Client Gateway replaced the SCLMDT function used in Version 7.1.x.x. The code for the ISPF Client Gateway started as an exact copy of the SCLMDT code.
Until Version 7.5.1.1, if RSE was not able to generate a PassTicket, and thus not able to switch the security context of a user-specific thread to that of the owning user, processing continued using the server's credentials. In Version 7.5.1.1, this behavior changed and a connection is now terminated if a PassTicket cannot be generated. The client gets a clear error message when this happens.
Until Version 7.5.1.1, if a user was not allowed explicit READ permit to the FEKAPPL profile in the APPL class, processing continued. In Version 7.5.1.1, this behavior changed, and a connection is now terminated if a user does not have an explicit permit to FEKAPPL. The client gets a clear error message when this happens.
As a bypass for a TSS bug, the explicit check for FEKAPPL is removed in Version 8.0.1.0, and Developer for System z relies on the judgment of RACF. (RACF works the other way around: Explicit NONE permits are needed to refuse access.)
In Versions 7.5.1.1 and 7.6.0.1, there is a behavior change for multiple logons by a single userid. Until those release versions, Rational Developer for System z did not check for multiple logons by the same user, because only the user-specific log files are affected. However, some customers have users who regularly lose the client-host connection (usually because they are tired of waiting for a huge download and shut down the client). When those users log on again, the time-outs of the first session have not yet expired, and the host thinks the first session is still active. This is not a problem, except that the first session still has the lock on all the files the user was editing, and the users are not willing to wait for the host to realize that the session is gone and free the locks.
Lowering the time-out values was a first attempt to resolve the problem (in Version 7.6 GA), but this is a delicate balance, as Developer for System z should not terminate a session just because there is a network slowdown.
As a solution, the single.logon option was created (activated with v7511 UK51499 or v7601 UK54177) that cleans up any existing session if a user ID logs on twice. This is now the default behavior, because the benefit of ensuring all locks are freed outweighs the problem for the few users that log on multiple times. You can change this behavior with the new single.logon directive in rsed.envvars, which is documented in the PTF hold information.
The new directive looks like this:
#_RSE_JAVAOPTS="$_RSE_JAVAOPTS -Dsingle.logon=false"
The purpose is to allow a user ID to log on multiple times. The default value is false. Uncomment to allow a user ID to log on multiple times to a single RSE daemon.
In Version 7.5.0.0, daemon.log was introduced in rsed.envvars (commented out) to specify where rsedaemon.log and rseserver.log are located. If not specified, the home directory of the user assigned to the RSED started task is used. Note that daemon.log is relative to the configuration directory (/etc/rdz) when it is not an absolute path.
In Version 7.5.1.1, user.log and DSTORE_LOG_DIRECTORY were introduced in rsed.envvars, (commented out) which specify where the user-specific logs are located: user.log/DSTORE_LOG_DIRECTORY/$LOGNAME/*.log.
In Version 7.6.0.0, Developer for System z uncommented the mentioned variables and routed all logs to /var/rdz/logs. Example:
_RSE_JAVAOPTS="$_RSE_JAVAOPTS -Ddaemon.log=/var/rdz/logs"
_RSE_JAVAOPTS="$_RSE_JAVAOPTS -Duser.log=/var/rdz/logs"
_RSE_JAVAOPTS="$_RSE_JAVAOPTS -DDSTORE_LOG_DIRECTORY="
In previous releases, RSE server bind to IP address 0, which results in a bind to every available TCPIP stack. This breaks Distributed Dynamic VIPA setups, and requires special TCPIP configuration to bypass the problem.
In Version 8.0.1.0, RSE server (active inside the threadpool) will analyze the IP address used by the client when it connected to RSE daemon. RSE server will then do a bind to just that address to start listening on the port. This way DDVIPA will automatically route the server connection request to the same system it sent the initial daemon connection request to.
In previous releases, Rational Developer for System z supported host based property groups through a set a configuration files, /etc/rdz/propertiescfg.properties, /var/rdz/properties/propertygroups.xml, and /var/rdz/properties/defaultvalues.xml.
In Version 8.0.1.0, all this is now managed on a bigger scale through push-to-client, using /etc/rdz/pushtoclient.properties and /var/rdz/pushtoclient/keymapping.xml.
In previous releases, Rational Developer for System z supported host based projects through a set a configuration files, /etc/rdz/projectcfg.properties and /var/rdz/projects/USERID/project.instance.
In Version 8.0.1.0, this is managed on a bigger scale through push-to-client, using /etc/rdz/pushtoclient.properties and /var/rdz/pushtoclient/keymapping.xml.
When you want to send files to IBM using FTP, use the following process:
XMIT DUMMY.DUMMY DS('data.set.name(member)') outdsn(userid.PMR#.XMIT')
//TERSE EXEC PGM=AMATERSE,PARM=PACK
//*TERSE EXEC PGM=TRSMAIN,PARM=PACK
//*STEPLIB DD DISP=SHR,DSN=SYS1.MIGLIB
//SYSPRINT DD SYSOUT=*
//INFILE DD DISP=SHR,DSN=data.set.name
//OUTFILE DD DISP=(NEW,CATLG),SPACE=(CYL,(250,250),RLSE),
// UNIT=SYSALLDA,DSN=data.set.name.TRS
Because RSE is a z/OS UNIX process, the OMVS data is often needed to get the complete picture. RSE also contains multiple address spaces (daemon and thread pools), so wildcards are required.
DUMP COMM=(description or dump name)
R xx,SDATA=(PSA,SQA,LSQA,RGN,TRT,LPA,CSA,SUM,ALLNUC),CONT
R xx,JOBNAME=(OMVS,RSED*),DSPNAME=('OMVS'.*,'RSED*'.*),END
CD SET,SDUMP=(CSA,LPA,LSQA,NUC,PSA,RGN,SQA,SUM,SWA,TRT),Q=YES
CD SET,SDUMP,MAXSPACE=2000M
SLIP SET,ID=RSES,MSGID=FEKxxxI,JL=(OMVS,RSED*),DN=('OMVS'.*,'RSED*'.*),END
The APPC white paper, Using APPC to provide TSO command services (SC14-7291-00), holds more information on setting up APPC itself, setting up the APPC transaction program (TP), modifying an existing TP, and basic problem solving.
When a TP executes, the TP runtime messages, such as allocation and termination messages, go to a log named by the MESSAGE_DATA_SET keyword in its TP profile. The default value in sample JCL FEK.SFEKSAMP(FEKAPPCC) is &SYSUID.FEKFRSRV.&TPDATE.&TPTIME.LOG. This log file does not appear unless the KEEP_MESSAGE_LOG(ALWAYS) keyword is added to the transaction definitions. The default value in sample JCL FEK.SFEKSAMP(FEKAPPCC) is ERROR.
For actual debugging (only on request of IBM support), you can trap the SYSTSPRT output of the transaction. The default output location in sample JCL FEK.SFEKSAMP(FEKAPPCC) is SYSOUT=*.
When writing to a dataset, be sure you use a user-specific naming convention (for example, use &SYSUID. in the DSN), because there will be one APPC transaction per Rational Developer for System z user.
When writing the output to SYSOUT, be aware that JES treats APPC's SYSOUT output as special:
To perform an APPC API trace to a text file, use the following process:
EX 'SYS1.SBLSCLI0(ATBTRACE)' 'START DSN(userid.APITRACE)
LU(base_lu) TP(FEKFRSRV)'
EX 'SYS1.SBLSCLI0(ATBTRACE)' 'STOP DSN(userid.APITRACE)'
The trace is case-sensitive.
To perform an APPC CTRACE(SVCDUMP), use the following process:
Add isptrace=nullfile to ISPF.conf. This will tell ISPF to start tracing. Output is written to userid.ISPVCALL.TRACE ( or to userid.CALL.TRACE if the z/OS release level is before 1.9).
To test if ISPVCALL works as expected, issue: TSO ISPVCALL STATUS. (No DD is needed.)
Location of documentation for this trace: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/F1A1C590/3.33.2?SHELF=F1A1BKB1&DT=20090402133646 (z/OS 1.11).
Add the following variable setting to rsed.envvars:
RESOLVER_TRACE=/var/rdz/logs/resolver.trace
You can use RESOLVER_TRACE=stdout when you are working from the command line.
To perform a TCP/IP and RESOLVER CTRACE, use the following process:
TRACE CT,WTRSTART=CTWTR1
TRACE CT,ON,COMP=SYSTCPRE,SUB=(resolverprocname)
R xx,WTR=CTWTR1,OPTIONS=(ALL),END
TRACE CT,ON,COMP=SYSTCPIP,SUB=(tcpipprocname)
R xx,WTR=CTWTR1,OPTIONS=(TCP,PFS,UDP),END
TRACE CT,ON,COMP=SYSTCPRE,SUB=(resolverprocname)
R xx,WTR=DISCONNECT,END
TRACE CT,OFF,COMP=SYSTCPRE,SUB=(resolverprocname)
TRACE CT,ON,COMP=SYSTCPIP,SUB=(tcpipprocname)
R xx,WTR=DISCONNECT,END
TRACE CT,OFF,COMP=SYSTCPIP,SUB=(tcpipprocname)
TRACE CT,WTRSTOP=CTWTR1,FLUSH
The SAF trace documented here requires a diagnostic driver for Developer for System z that creates a console message just before and after the targeted security call, in this case PassTicket generation.
The diagnostic driver issues the following messages when a PassTicket is generated:
Follow these instructions to activate the test driver outside SMP/E.
mv /usr/lpp/rdz/lib/zosserver.jar /usr/lpp/rdz/lib/zosserver.jar.bu
chmod 755 /usr/lpp/rdz/lib/zosserver.jar
ls -lE /usr/lpp/rdz/lib/zosserver.jar
-rwxr-xr-x --s- 1 IBMUSER SYS1 59779 Dec 17 12:04 zosserver.jar
The next thing to do is set up the system dump and SAF trace. Note that RACF relies on GTF to create a SAF trace.
See (for z/OS 1.10)http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/IEA2V191/10.0?SHELF=IEA2BK91&DT=20090220211519
//*
//* GTF TRACE
//*
//* START COMMAND : S GTF.GTFUSER
//* STOP COMMAND : P GTFUSER
//*
//GTF PROC TRACE=USER.$$$$$GTF.TRACE,
// DSP=NEW,
// SIZE='CYL,10',
// PLIB=USER.$$$$$GTF.PARMLIB,
// MEMBER=GTFPARM
//*
//IEFPROC EXEC PGM=AHLGTF,TIME=1440,REGION=4096K,
// PARM='MODE=EXT,DEBUG=NO,TIME=YES'
//IEFRDER DD DSN=&TRACE,
// DISP=(&DSP,CATLG),
// UNIT=SYSALLDA,
// SPACE=(&SIZE)
//SYSLIB DD DISP=SHR,DSN=&PLIB(&MEMBER)
//*
TRACE=USRP
USR=(F44)
END
SLIP SET,ID=RDZ1,MSGID=FEKP02I,JL=(RACF,RSED*),DN=(1.IRR*),END
START GTF.GTFRDZ,MEMBER=GTFSAF
R xx,U
@SET TRACE( JOBNAME(RSED*) RACROUTE(TYPE(1)) CALLABLE(TYPE(43)) ) LIST
For more information about this, see http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/ICHZA290/4.6.1.1?SHELF=ICHZBK90&DT=20080521121606&CASE= (z/OS 1.10). RACF support will tell you the desired trace options for the specific problem.
Recreate the problem. (Only logon, do not disconnect to avoid trace pollution.)
Finally, disable the trace, collect the data, and undo the changes for test driver.
Disable the trace as soon as possible after the problem has been recreated, to prevent the data from "rolling out" of the trace file.
@SET TRACE( NOJOBNAME NORACROUTE NOCALLABLE ) LIST
STOP GTFRDZ
mv /usr/lpp/rdz/lib/zosserver.jar.bu /usr/lpp/rdz/lib/zosserver.jar
ls -lE /usr/lpp/rdz/lib/zosserver.jar
-rwxr-xr-x --s- 2 IBMUSER SYS1 59669 Dec 16 18:37 zosserver.jar
The SDSF PS view is an easy way of seeing z/OS UNIX processes (active and zombie), but there are other options.
A super user (UID 0) can run ps -ef from the OMVS shell to see all active processes, or ps -fu <userid> for those processes belonging to a specific user. For more information, see http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/BPXZA5B0/PS) (z/OS 1.12).
Operator commands D OMVS,A=ALL and D OMVS,U=<userid> give similar output as the ps shell command. See http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/IEA2G1B1/4.10.48 (z/OS 1.12).
To show the z/OS UNIX reason code, add the following variable setting to rsed.envvars:
_EDC_ADD_ERRNO2=1
For more information, see the XL C/C++ Programming Guide (SC09-4765), section about _EDC_ADD_ERRNO2. (Location: http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/CBCPG190/4.12.2.7?SHELF=CBCBS190&DT=20080616063107 for z/OS 1.10)
When _EDC_ADD_ERRNO2 is set to 1, errno2 is added to perror(), strerror(), and strerror_r() messages. Example:
+FEK117E socketpair() failed.
reason=(EDC5124I Too many open files. (errno2=0x050B0146))
GSK SSL (also known as System SSL) is used by the RSE daemon. The RSE server uses Java SSL.
Set the following variables in rsed.envvars:
GSK_TRACE_FILE=/var/rdz/logs/gskssl.trc
GSK_TRACE=0xFF
Recreate the problem.
Format the trace. (Trace is sent to DD STDOUT.)
//GSKTRACE JOB CLASS=A,MSGCLASS=A,MSGLEVEL=(1,1),NOTIFY=&SYSUID
//*
//* SHOW FORMATTED GSKTRACE
//*
// SET FILE='/var/rdz/logs/gskssl.trc'
//*
//FORMAT EXEC PGM=BPXBATCH,REGION=0M,TIME=NOLIMIT,
// PARM='SH ls -l &FILE;echo;gsktrace &FILE'
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*
//*
Expected output:
08:40:26.523 [corejni.c:1036 0xdf45ac0]
Java_com_ibm_etools_zos_server_CoreJNI_createSocket() Entered
key=enable_ssl
value=true
key=daemon_keydb_file
value=rsessl.racf
key=daemon_key_label
value=rsehost
key=server_keystore_file
value=rsessl.racf
key=server_keystore_label
value=rsehost
key=server_keystore_type
value=JCERACFKS
key=enable_ssl, value=true
Java_com_ibm_etools_zos_server_CoreJNI_createSocket() Exited
08:40:35.418 [corejni.c:1103 0xdf45ac0]
Java_com_ibm_etools_zos_server_CoreJNI_acceptSocket() Entered
08:41:09.968 [corejni.c:1117 0xdf45ac0]
Java_com_ibm_etools_zos_server_CoreJNI_acceptSocket() Exited
key=daemon_keydb_file, value=rsessl.racf
key=daemon_keydb_password, value=
key=daemon_key_label, value=rsehost
Java_com_ibm_etools_zos_server_CoreJNI_readSocket() Entered
Java_com_ibm_etools_zos_server_CoreJNI_readSocket() Exited
Wed Jan 06 08:40:21 EST 2010
INFO RseDaemon: startProcesses(): spawn command result=16777795:132
INFO RseDaemon: startProcesses(): send Query to the Server process
---------------------------------------------------------------
Wed Jan 06 08:40:35 EST 2010
INFO RseDaemon: startProcesses(): Server process started. (processId=583)
INFO LogHandler: Thread poolProcess:583: KeyStore=JCERACFKS, rsessl.racf,
INFO : ProcessId=583, Memory_Usage=4%, Used_Memory=2771KB, threads=3, TCBs=14
---------------------------------------------------------------
Wed Jan 06 08:41:11 EST 2010
INFO LogHandler: 583:smile:8070256:0:6291103
INFO LogHandler: ProcessId=583: port 1837 assigned to smile
INFO : ProcessId=583, Memory_Usage=6%, Used_Memory=4028KB, threads=6, TCBs=18
INFO LogHandler: ProcessId=583: smile logged on
Alert messages, as in the following sample, are documented in RFC 2246 (http://www.ietf.org/rfc/rfc2246.txt).
ERROR read_v3_alert(): SSL V3 alert 46 received from <ip> ERROR
gsk_secure_socket_init(): SSL V3 server handshake failed with ...INFO
default_setsocketoptions(): TCP_NODELAY restored for socket ...EXIT
gsk_secure_socket_init(): <--- Exit status 0x0000019e (414)
ENTRY gsk_strerror():
---> Thd-0 EXIT gsk_strerror(): <--- Exit status 0x00000000 (0)
Alert |
Description |
Explanation |
0 |
close_notify |
This message notifies the recipient that the sender will not send any more messages on this connection. The session cannot resume if any connection is terminated without proper close_notify messages with level equal to warning. |
10 |
unexpected_message |
An inappropriate message was received. This alert is always indicates an unrecoverable error and should never be observed in communication between proper implementations. |
20 |
bad_record_mac |
This alert is returned if a record is received with an incorrect MAC. This message always indicates an unrecoverable error. |
21 |
decryption_failed |
A TLSCiphertext decrypted in an invalid way. Either it was not an even multiple of the block length or its padding values, when checked, were not correct. This message always indicates an unrecoverable error. |
22 |
record_overflow |
A TLSCiphertext record was received which had a length more than 2^14+2048 bytes, or a record decrypted to a TLSCompressed record with more than 2^14+1024 bytes. This message always indicates an unrecoverable error. |
30 |
decompression_failure |
The decompression function received improper input (for example, data that would expand to excessive length). This message always indicates an unrecoverable error. |
40 |
handshake_failure |
Receiving of a handshake_failure alert message indicates that the sender was unable to negotiate an acceptable set of security parameters with the available options. This is an unrecoverable error. |
41 |
? no_certificate ? |
See SSL support. |
42 |
bad_certificate |
A certificate was corrupt, contained signatures that did not verify correctly, or had other problems. |
43 |
unsupported_certificate |
A certificate was of an unsupported type. |
44 |
certificate_revoked |
A certificate was revoked by its signer. |
45 |
certificate_expired |
A certificate has expired or is not currently valid. |
46 |
certificate_unknown |
Some other (unspecified) issue arose in processing the certificate, rendering it unacceptable. |
47 |
illegal_parameter |
A field in the handshake was out of range or inconsistent with other fields. This always indicates an unrecoverable error. |
48 |
unknown_ca |
A valid certificate chain or partial chain was received, but the certificate was not accepted because the CA certificate could not be located or could not be matched with a known, trusted CA. This message always indicates an unrecoverable error. |
49 |
access_denied |
A valid certificate was received, but when access control was applied, the sender decided not to proceed with negotiation. This message always indicates an unrecoverable error. |
50 |
decode_error |
A message could not be decoded because some field was out of the specified range or the length of the message was incorrect. This message always indicates an unrecoverable error. |
51 |
decrypt_error |
A handshake cryptographic operation failed, including being unable to correctly verify a signature, decrypt a key exchange, or validate a finished message. |
60 |
export_restriction |
A negotiation not in compliance with export restrictions was detected. For example, attempting to transfer a 1024 bit ephemeral RSA key for the RSA_EXPORT handshake method. This message always indicates an unrecoverable error. |
70 |
protocol_version |
The protocol version the client has attempted to negotiate is recognized, but not supported. (For example, old protocol versions might be avoided for security reasons.) This message always indicates an unrecoverable error. |
71 |
insufficient_security |
Returned instead of handshake_failure when a negotiation has failed specifically because the server requires ciphers more secure than those supported by the client. This message always indicates an unrecoverable error. |
80 |
internal_error |
An internal error unrelated to the peer or the correctness of the protocol makes it impossible to continue (such as a memory allocation failure). This message always indicates an unrecoverable error. |
90 |
user_canceled |
This handshake is being canceled for some reason unrelated to a protocol failure. If the user cancels an operation after the handshake is complete, just closing the connection by sending a close_notify is more appropriate. This alert should be followed by a close_notify. This message is generally a warning. |
100 |
no_renegotiation |
Sent by the client in response to a hello request or by the server in response to a client hello after initial handshaking. Either of these would normally lead to renegotiation. When renegotiation is not appropriate, the recipient should respond with this alert. At that point, the original requester can decide whether to proceed with the connection. One case where this would be appropriate would be where a server has spawned a process to satisfy a request; the process might receive security parameters (key length, authentication, and so forth) at startup, and it might be difficult to communicate changes to these parameters after that point. This message is always a warning. |
Java SSL is used by the RSE server. The RSE daemon uses GSK SSL (also known as System SSL).
Set the following variables in rsed.envvars:
_RSE_JAVAOPTS="$_RSE_JAVAOPTS -Djavax.net.debug=true"
_RSE_JAVAOPTS="$_RSE_JAVAOPTS -Denable.standard.log=true"
The output is written to stdout, so enable.standard.log must be set to trap it. The trapped output will go to /var/rdz/logs/stdout.n.log ( where n=1, 2, 3, ...).
You can use the method trace to trace the NIO (network I/O) code within the Java Virtual Machine (JVM). Add the following parameters to the JVM:
-Xtrace:methods={java/nio/channels/SocketChannel.*},output=/var/rdz/logs/nio.trc
This will produce a binary file nio.trc, which needs to be formatted using the following command:
java com.ibm.jvm.format.TraceFormat
/var/rdz/logs/nio.trc /var/rdz/logs/nio_trace.txt
The tracing suggested will trace only the SocketChannel class. If you want to trace the entire network I/O code, you need to limit the tracing to java/nio/* in the Xtrace command.
This trace requires Developer for System z Versions 7.1.1.4, 7.5.1.1, 7.6.0.1 or later.
CARMA can run with an LE diagnosis heap manager, which will allow development to find the cause of a memory leak. Upon termination, this heap manger writes a memory allocation report to the CARMALOG DD. See http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/CEEA1190/2.1.10?SHELF=CEE2BK90&DT=20080530025919 (z/OS 1.10) for more information.
Batch submit invocation CRASUBMT:
//CARMALOG DD SYSOUT=*
//SYSTSIN DD * ISPSTART PGM(CRASERV) +
PARM('ENVAR("_CEE_HEAP_MANAGER=CEL4MCHK")/&PORT &TIMEOUT')
/*
CRASTART invocation in crastart.conf (indentation indicates line continuation):
-COMMAND=ALLOC FI(CARMALOG) MOD CATALOG DSORG(PS) RECFM(F,B)
LRECL(133) BLKSIZE(27930) SPACE(5,5) TRACKS UNIT(SYSALLDA)
DA(&CRAUSER..&SYSNAME..CRA.CARMALOG)
PROGRAM=IKJEFT01 ISPSTART PGM(CRASERV)
PARM('ENVAR("_CEE_HEAP_MANAGER=CEL4MCHK")/&CRAPRM1. &CRAPRM2.')
For information about how to set up and configure Rational Developer for System z, see the IBM Rational Developer for System z Host Configuration Guide (SC23-7658).
Running the RSE Daemon with IVP, the FEKFIVPT gets the following error in the syslog:
TCP/IP IVP test...
Initialize failed, rc= 1004
This error code is generated by the following invocation in FEKFIVPT:
parse value socket('Initialize', 'RSCLIENT') with rc .
if rc <>0 then call SayError 'Initialize' rc 0
The rc=1004 for a SOCKET API is documented in the TCPIP errno.h file, and it means:
1004 EIBMIUCVERR All The request failed because of an IUCV error.
This error is generated by the client stub code. Ensure IUCV/VMCF is functional.
Here is a link to the z/OS Communications Server Version 1.9 that contains this socket return code information: http://publib.boulder.ibm.com/infocenter/zos/v1r9/index.jsp?topic=/com.ibm.zos.r9.cs3cod0/syserret.htm
You should ensure that the TCP/IP IUCV/VMCF subsystem is configured and started according to the TCP/IP stack that is used. Here is a link to the z/OS Communications Server Version 1.9 Information Center. The information there explains how to configure and start this subsystem: http://publib.boulder.ibm.com/infocenter/zos/v1r9/index.jsp?topic=/com.ibm.zos.r9.halz002/f1a1b37042.htm. Here is a link to the IUCV/VMCF considerations information: http://publib.boulder.ibm.com/infocenter/zos/v1r9/index.jsp?topic=/com.ibm.zos.r9.halz002/f1a1b37042.htm
Attempts to run an IVP test using IBM Rational Developer for System z using the command fekfivpd port username results in the error message:
CEE3201S The system detected an operation exception (System Completion Code=OC1)
The full error message is:
CEE3201S The system detected an operation exception
(System Completion Code=OC1).
From entry point main at compile unit offset -33C7C3B0 at entry offset
-33C7C3B0 at address 00000000.
Possible Bad Branch: Statement: Offset: +000000E8
41 *-* 'set -a;.' path'rsed.envvars;fekfdivp'
port NoDollar(uid) +++ RC(132) +++
The TSO region size is too small.
None.
Attempts to start the RSE daemon task on the host results in the following message in the job output:
BPXM009I BPXBATCH FAILED BECAUSE OPEN (BPX1OPN) FOR STDIN FAILED
WITH RETURN CODE 0000009C REASON CODE 0B0C00FC
Look at the last 2 bytes of the REASON CODE, which are 00FC. See "Reason Codes" in UNIX System Services Messages and Codes.
JRSAFNoUID: The user ID has no UID.
Create an OMVS segment with a UID. The problem is the user ID for the RSE started task has no group ID, and the OMVS segment is not defined correctly.
None.
After installing Rational Developer for System z Version 7.5 and running the Installation Verification Program (IVP) the following output is generated:
*** CHECK : USS MODULES Checking ISPF Directory :
/usr/lpp/ispf Checking modules in /usr/lpp/ispf/bin directory RC=8
MSG: ISPZINT not found
MSG: Remove suffix /bin from _CMDSERV_BASE_HOME setting in rsed.envv
Checking for ISPF configuration file ISPF.conf
*** ERRORS FOUND IN INSTALLATION VERIFICATION CHECK ***
IVP processing terminated until errors corrected
Re-run IVP process once errors above rectified
Host installation verification terminated with Errors
The ISPF SISPLPA data set could not be found or was not APF-authorized.
The SISPLPA data set must be:
None.
When the RSE started task is not active, you might see the following messages:
connect() failed: EDC8128I Connection refused.
43 *-* 'set -a;.' path'rsed.envvars;fekfdivp' port NoDollar(uid)
+++ RC(255) +++
You might see the following error messages in an "insufficient memory" situation:
CEE3201S The system detected an operation exception (System Completion Code=0C1)
From entry point main at compile unit offset -33C7C3B0 at entry offset -33C7C3B0
at address 00000000.
Possible Bad Branch: Statement: Offset: +000000E8
<> LEAID ENTERED (LEVEL 10/15/2007 AT 09.21)
<> LEAID PROCESSING COMPLETE. RC=0
41 *-* 'set -a;.' path'rsed.envvars;fekfdivp' port NoDollar(uid)
+++ RC(132) +++
CA TSS or ACF is unable to generate a pass ticket using 64-bit Java.
The full error message is:
ERROR ServerThread: 517: SafRc=8, racfRc=8 racfRsn=4
ERROR RseDaemon: getServerPort(): result=0;server failure:
SafRc=8, racfRc=8 racfRsn=4
com.ibm.eserver.zos.racf.IRRPassTicket.generate(IRRPassTicket.java:226)
com.ibm.etools.zos.server.ZosSystemService.generatePassTicket
(ZosSystemService.java:96)
com.ibm.etools.zos.server.ServerThread.run(ServerThread.java:83)
java.lang.Thread.run(Thread.java:736)
Maintenance for Computer Associates Top secret or ACF2 is needed.
View the file rseserver.log to see if the above messages exist:
CA TSS R14.0 use PTF RO31780
CA TSS R15.0 use PTF RO30836
CA ACF2 R14.0 use PTF RO31548
CA ACF2 R15.0 use PTF RO30898
None.
Similar error messages occur in various tests. RSED start with IVP parameter is failing with access error to IRRPTAUTH.FEKAPPL.* profile in the PTKTDATA security class.
JES Job Monitor test:
executed on SYSTEM -- Mon May 16 14:46:35 EDT 2011
executed by uid=130154(STCRSE) gid=839(STARTASK)
using /etc/rdz/rsed.envvars
current address space size limit is 1084203008 (1034.0 MB)
maximum address space size limit is 1084203008 (1034.0 MB)
testing JES Job Monitor on port 6715...
Initialize failed, rc= 1004
Lock Daemon test:
executed on SYSTEM -- Mon May 16 14:46:36 EDT 2011
executed by uid=130154(STCRSE) gid=839(STARTASK)
using /etc/rdz/rsed.envvars
current address space size limit is 1084203008 (1034.0 MB)
maximum address space size limit is 1084203008 (1034.0 MB)
testing RSE Lock Daemon on port 4036...
Initialize failed, rc= 1004
TCP/IP IVP test:
Initialize failed, rc= 1004
PassTicket IVP test:
the default applid=FEKAPPL
Access Error to IRRPTAUTH.FEKAPPL.* profile in PTKTDATA security class:
SAF RC=4 ; RACF RC=4 ; RACF Reason=0
RSE daemon IVP ended -- return code 40 --
The RC=1004 is caused because the TCPIP stack could not be found.
Access Error to IRRPTAUTH.FEKAPPL.* profile in PTKTDATA is caused because PTKTDATA was not in the GENERIC PROFILE CLASS.
RC=1004 can be resolved by changing the socket initialization in FEKFIVPJ.
Parse value socket('Initialize', 'RSCLIENT') with return code. Parse value socket('Initialize', 'RSCLIENT',40,TCPIP stack name) with return code.
To make the client connection work, _BPXK_SETIBMOPT_TRANSPORT must be set in rsed.envvars pointing to the correct TCPIP stack.
The Access error can be resolved with: SETR GENERIC(PTKTDATA), followed by SETR RACLIST(PTKTDATA) REF.
None.
During execution of the fekfivpt IVP or when starting the RSE daemon, using parameter IVP=IVP, you might see the wrong name for this host system.
host IP address:
------------------
hostName=NODENAME
That NODENAME is wrong. The NODENAME should be the name of the user’s mainframe. The result should look similar to the example in the Host Configuration Guide (SC23-7658), "Installation verification" chapter, section "Verify started tasks", subsection "RSED, RSE daemon", in the STDOUT example, near the end of the example. (In the example, CDFMVS08 appears in place of NODENAME.).
First, type this command at a USS prompt: hostname -r
If that gives the wrong value, then Rational Developer for system z will not work either. You might (wrongly) see:
EZZ8342I gethostbyname(NODENAME): Unknown host
Second, read Appendix B of the Host Configuration Reference Guide (SC14-7290). That tells how to set up TCP/IP and explains how the searching works.
If nothing in Appendix B helps, then view your SYS1.PARMLIB(COMMNDxx) for this line:
S EZAZSSI,P=NODENAME
If that line is present, then change NODENAME to the correct one. You might use the symbolic value &SYSNAME.:
S EZAZSSI,P='&SYSNAME.'
None.
When using SSL with IBM Rational Developer for System z, the client connection fails. The security configuration of this customer included an unknown Certificate Authority (an internal created CA, not Thawte or Verisign). This is similar to using a self-signed certificate and requires that the public portion of the CA certificate must be extracted and added to the keyring as well as the requested certificate.
A stack trace will show:
java.net.SocketTimeoutException: Read timed out
*
In rseserver.log, this error will be seen:
ERROR CertificateValidator:
java.io.FileNotFoundException: RSEDRING (EDC5129I No such file or
directory. (errno2=0x05620062))
at java.io.FileInputStream.<init>(FileInputStream.java:123)
at java.io.FileInputStream.<init>(FileInputStream.java:83)
at com.ibm.etools.zos.server.ZosCertificateValidator.<init>
(ZosCertificateValidator.java:174)
at com.ibm.etools.zos.server.RseDaemon.main(RseDaemon.java:358)
and
INFO LogHandler:
ServerKeyStore:java.io.IOException:
Failed validating certificate paths
The RSE Server performs a validation of the certificate. In this case validation was failing. Add the internal issuing Certificate Authority certificate into the KEYRING used for RSED as well as the requested certificate.
None.
In IBM Rational Developer for System z, the message
RSEG1242 Daemon failed to launch server on <host> using port <port>
can have the following details:
When you see CONNECTION REFUSED, then RSE (Remote System Explorer) daemon is not active.
When you see SERVER FAILURE, with a message, then your client has contacted the host but became disconnected.
For CONNECTION REFUSED you should do the following things:
telnet <host> <port>
The
client can access RSE daemon, but not the RSE server due to firewall
issues. (This is typically indicated by a non-zero port number in
the RSEG1242 message.)For SERVER FAILURE: <message>: The details show the error message shown in rseserver.log, but only if the RSE daemon or server was reached:
ERROR RseDaemon: server failure: password revoked
F RSED,APPL=RSL ON
and
F RSED,APPL=RCL ON
None.
The RSED started task running with the IVP=IVP argument failed.
Access Error to IRRPTAUTH.FEKAPPL.* profile in PTKTDATA
security class : SAF RC=8 ; RACF RC=8 ; RACF Reason=0
Security setup is not complete. With this customer, the started task user ID OMVS segment did not have a GID assigned, and SETROPTS LIST showed
ATTRIBUTES = INITSTATS NOWHEN(PROGRAM) SAUDIT CMDVIOL NOOPERAUDIT
with NOWHEN(PROGRAM) program control is not active. z/OS UNIX requires program control for using privileged functions.
Generate GID for Rational Developer for System z users and activate basic program control support. Both actions are documented in the Security setup documentation of Developer for System z :
ATTRIBUTES = INITSTATS WHEN(PROGRAM -- BASIC) SAUDIT CMDVIOL
NOOPERAUDIT
Access Error to IRRPTAUTH.FEKAPPL.* profile in PTKTDATA
security class : SAF RC=4 ; RACF RC=4 ; RACF Reason=0
The access error to the IRRPTAUTH.FEKAPPL.* profile in PTKTDATA is caused because PTKTDATA was not in the GENERIC PROFILE CLASS.
The Access error is resolved by doing the following things:
RDELETE (PTKTDATA IRRPTAUTH.FEKAPPL.*)
SETROPTS GENERIC(PTKTDATA)
RDEFINE PTKTDATA IRRPTAUTH.FEKAPPL.* UACC(NONE
SETROPTS RACLIST(PTKTDATA) REFRESH
None.
This section discusses some of the problems that can occur during CARMA installation and setup.
You get a return code -3 when starting the CARMA server.
Look in the CARMA server job output in SDSF - cra&port# and see:
--- CRANDVRA CMD START 16:46:22 -----------------
invoking NDVRC1 PGM(CRASERV) PARM(13935 420)
136 *-* Cmd
+++ RC(-3) +++
--- CRANDVRA CMD END 16:46:22 -----------------
Usually, the invocation fails with RC -3 when there is an APF authorization issue. Endevor must be APF-authorized, and when it is accessed using STEPLIB or TASKLIB, all other libraries in STEPLIB or TASKLIB, such as data set hlq.SFEKLOAD, must also be APF-authorized or z/OS removes the APF authorization of those libraries that do have the authorization. See the code example in the figure "crastart.endevor.conf - CRASTART with CA Endevor® SCM RAM" in chapter "(Optional) Common Access Repository Manager (CARMA)", section "CRASTART with CA Endevor® SCM RAM", subsection "Customize crastart.endevor.conf" of the IBM Rational Developer for System z Host Configuration Guide (SC23-7658). This is where you customize the file crastart.endevor.conf. There is a line
TASKLIB = dataset1,dataset2,...
All of those data sets must be APF authorized.
None.
Attempts to install or set up CARMA result in this error message when using IBM Rational Developer for System z.
ERROR com.ibm.carma.rse.server.miners.CARMAMiner:
CARMA error occured RDZ - ENDEVOR error 500
ERROR com.ibm.carma.rse.server.miners.CARMAMiner: CARMA error occured
during command processing
com.ibm.carma.rse.server.transport.CARMAException: 500:
The 500 error code is returned from Endevor, so you are already in Endevor.
MSG3FILE might be empty because CA Endevor never gets past an access error. Check BSTERR in case it contains the information that could be key to solve this problem.
None.
The CARMA server is not starting up in timely manner.
com.ibm.carma.CARMAException: CRRZC2200E An error was received from the
CARMA Host [E09Z]:
[406] 406: Server could not be loaded in appropriate timeframe. Contact
system programmer.
at com.ibm.carma.client.subsystem.CARMASubSystem.processErrorMessage
(Unknown Source)
Rational Developer for System z cannot connect to CARMA.
Possible causes:
Solutions:
None.
Copyright IBM Corporation - 2012
U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Programming interfaces: Intended programming interfaces allow the customer to write programs to obtain the services of .
For license inquiries regarding double-byte (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to:
The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.
IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.
Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact:
Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee.
The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us.
Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment.
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.
All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only.
This information is for planning purposes only. The information herein is subject to change before the products described become available.
This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.
This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are provided "AS IS", without warranty of any kind. IBM shall not be liable for any damages arising out of your use of the sample programs.
Each copy or any portion of these sample programs or any derivative work, must include a copyright notice as follows:
If you are viewing this information softcopy, the photographs and color illustrations may not appear.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at www.ibm.com/legal/copytrade.shtml.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.
Other product and service names might be trademarks of IBM or other companies.