hp e3000 programming and posix programming and posix presented by Mark Bixby mark_bixby@hp.
hp e3000 contents programming and posix • Migration from MPE to Unix Platforms • Getting Started • The Hierarchical File System (HFS) • Files and Directories - A Review • A Simple Program and a CGI Program • Creating and Linking with Libraries • POSIX API Topics • POSIX Shell Topics • Additional Programming Topics Solution Symposium April 3, 2002 Page 2
hp e3000 migration from mpe to unix platforms programming and posix • POSIX originated in the Unix world • your MPE POSIX applications will be 105% compatible with Unix platforms, due to Unix having a more complete implementation of the following areas: • terminal I/O • process handling • uid/gid security • consider developing any new MPE business logic with POSIX in order to facilitate your future migration! Solution Symposium April 3, 2002 Page 3
hp e3000 getting started programming and posix • Logon: :hello . • Enter the POSIX shell: :xeq sh.hpbin.
hp e3000 the hierarchical file system (hfs) programming and posix / (the root) APACHE PUB SYS bin lib usr PUB NET cgi-bin htdocs Absolute path: /APACHE/PUB/cgi-bin/hwcgi Relative path: .
hp e3000 working with files - a review programming and posix • Naming a file • 16 character limit if below an MPE account or group • 255 character limit otherwise • File Types - bytestream vs.
hp e3000 programming and posix organizing files with directories - a review • Displaying your current directory - pwd • Absolute and relative pathnames • /an/absolute/pathname • a/relative/pathname • ./another/relative/pathname • ..
hp e3000 file and directory security programming and posix • each object is owned by a POSIX user (UID) and a POSIX group (GID) • POSIX UID maps to an MPE USER.
hp e3000 permission mode bits programming and posix • User Group Other rwx rwx rwx • Specified in chmod command symbolically or as 3 octal digits: • chmod u=rwx,g=rx,o=x file • equivalent to chmod 751 file • The umask command and function specifies a mask of permission modes to be disabled when files are created • umask 007 denies all access to “other” • remains in effect until another umask or logoff Solution Symposium April 3, 2002 Page 9
hp e3000 file security example programming and posix shell/iX> chmod 751 file shell/iX> ls -l file -rwxr-x--x 1 MANAGER.SYS SYS 0 Jan 3 13:29 file SYS 0 Jan 3 13:29 file shell/iX> chmod 644 file shell/iX> ls -l file -rw-r--r-- 1 MANAGER.
hp e3000 vi editor programming and posix • the only bytestream file editor supplied by CSY • hated by many, but standard on all Unixes • command mode vs.
hp e3000 vi command quick reference programming and posix • a - append to the right of cursor (data entry mode) • i - insert to the left of cursor (data entry mode) • o - add a new line below cursor (data entry mode) • O - add a new line above cursor (data entry mode) • dd - delete the current line • x - delete the current character • r - replace current character with next typed character • cw - change current word (data entry mode) • dw - delete current word • .
hp e3000 vi command quick reference (cont.
hp e3000 compiling - gcc vs. c89 programming and posix • Use gcc if: • you’re porting an open-source application • you want to write portable code that will run on other platforms • support contracts available from http://www.gccsupport.
hp e3000 programming and posix a simple program and a cgi program • A Simple Program • Create the file • Compile and link • Run it • A CGI Program • Create the file • Compile and link • Test it • Run it from a web browser Solution Symposium April 3, 2002 Page 15
hp e3000 a simple program - 1 programming and posix • Create the source file hw.c: #include
hp e3000 a simple program - 2 programming and posix • Compile and link the source file: shell/iX> gcc -o hw -D_POSIX_SOURCE hw.
hp e3000 a cgi program - 1 programming and posix • Edit the source file: shell/iX> cp hw.c hwcgi.c shell/iX> vi hwcgi.c #define _POSIX_SOURCE /* instead of -D */ #include main() { printf("Content-type: text/plain\n\n"); printf("hello world\n"); } • Compile and link the program: shell/iX> gcc -o hwcgi hwcgi.
hp e3000 a cgi program - 2 programming and posix • Test the CGI program: shell/iX> echo foo | hwcgi | cat Content-type: text/plain hello world • Copy CGI program to cgi-bin directory: shell/iX> cp hwcgi /APACHE/PUB/cgi-bin • Point browser at: http://systemname/cgi-bin/hwcgi Solution Symposium April 3, 2002 Page 19
hp e3000 a cgi program - 3 programming and posix Solution Symposium April 3, 2002 Page 20
hp e3000 creating an nmrl archive library - 1 programming and posix • Write new helloworld() function in helloworld.c: #define _POSIX_SOURCE #include helloworld() { printf("hello world\n"); } shell/iX> gcc -c helloworld.c • -c generates an NMOBJ instead of NMPRG • Create the NMRL archive library: shell/iX> ar -rv libhw.a helloworld.
hp e3000 creating an nmrl archive library - 2 programming and posix • Have our main program in hwcgimain.c call an external: #include extern void helloworld(void); main() { printf("Content-type: text/plain\n\n"); helloworld(); } shell/iX> gcc -c -D_POSIX_SOURCE hwcgimain.c • Link the program: shell/iX> gcc -o hwcgi hwcgimain.o -L. -lhw • -L. specifies library search directory (. is CWD) • -lhw refers to libhw.
hp e3000 creating an nmxl shared library programming and posix • Create the NMXL shared library: shell/iX> gcc -Wl,-b -o libhw.sl \ helloworld.o -nostdlib • -Wl - pass parameters to linker • -b - create an NMXL shared library • Link with the shared library: shell/iX> gcc -o hwcgi hwcgimain.
hp e3000 linking with system libraries programming and posix • libc is included in link by default shell/iX> gcc -o hwcgi hwcgi.c • System libraries located in /lib and /usr/lib • libc, libsvipc are in /lib • libsocket is in /usr/lib • System libraries exist in both archive and shared form (as of MPE 6.0). During link, • NMRL archive library (.a suffix) merged into program • NMXL shared library (.
hp e3000 linking with libraries - syntax programming and posix • -lfoo means link with library libfoo.a • -lc is included in link by default • -Lpath tells where library is located • -L/lib -L/usr/lib is included in link by default • Link with libsvipc archive library shell/iX> gcc -o hwcgi hwcgi.c -lsvipc • Link with libsvipc shared library shell/iX> gcc -o hwcgi hwcgi.
hp e3000 gcc vs.
hp e3000 make utility programming and posix • Rebuilds only those components which need rebuilding based on which dependent files have newer timestamps • A simple Makefile: all: hwcgi hwcgi: hwcgimain.o libhw.a $(CC) -o $@ hwcgimain.o -L. -lhw libhw.a: helloworld.
hp e3000 posix api topics programming and posix • Program Parameters • Environment Variables • File Management • Process Management • User and Group Management • InterProcess Communication • Sockets • Signals • Error handling Solution Symposium April 3, 2002 Page 28
hp e3000 program parameters programming and posix • int argc - number of parameters • char **argv - pointer to list of parameter pointers • argv[0] - name of program as invoked by user int main(int argc, char **argv) { int i; /* print all parameters */ for (i=0; i < argc; i++) { printf("argv[%d] = %s\n",i,argv[i]); } } Solution Symposium April 3, 2002 Page 29
hp e3000 environment variables programming and posix • special string variables copied from parent to child when new processes are created • the POSIX shell will only copy exported variables – foo=bar; export foo – export foo=bar • static copies of CI variables are exported by the shell • the environment is a process-local structure; your parent and any already running children won’t see any environment changes that you make Solution Symposium April 3, 2002 Page 30
hp e3000 environment variables - reading programming and posix #include #include
hp e3000 environment variables - writing programming and posix #include #include
hp e3000 file management programming and posix • POSIX file descriptors instead of MPE file numbers • a 32-bit integer field • just a stream of bytes - no record boundaries! • certain reserved file descriptors are always opened when a process is created: • 0 - stdin (program input) • 1 - stdout (program output) • 2 - stderr (program error message output) Solution Symposium April 3, 2002 Page 33
hp e3000 file management - open()/close() programming and posix • int open (const char *path, int oflag, int modes); • returns -1 for error, else returns file descriptor • ONE of O_RDONLY, O_WRONLY, O_RDWR • O_APPEND - file offset set to EOF prior to writes • O_CREATE - opt.
hp e3000 file management - read()/write() programming and posix • ssize_t read (int fildes, void *buffer, size_t nbyte); • returns number of bytes actually read or -1 if error • can also be used on socket descriptors • ssize_t write (int fildes, const void *buffer, size_t nbyte); • returns number of bytes actually written or -1 if error • can also be used on socket descriptors Solution Symposium April 3, 2002 Page 35
hp e3000 file management - lseek() programming and posix • off_t lseek (int fildes, off_t offset, int whence); • changes the current file position • returns the new file offset or -1 if error • offset - number of bytes • whence - how the offset is applied to the current position: – SEEK_SET - Set new offset to offset. – SEEK_CUR - Set new offset to offset plus the current offset. – SEEK_END - Set new offset to offset plus the current file size.
hp e3000 file management - stat() programming and posix • int stat (const char *path, struct stat *buffer); • int fstat (int fildes, struct stat *buffer); • int lstat (const char *path, struct stat *buffer); • reports on the symlink instead of the target file • obtains file attributes • some struct stat fields from
hp e3000 file management - dup()/dup2() programming and posix • duplicates file descriptors • commonly used with fork()/exec() to create pipes • int dup (int fildes); • int dup2(iint fildes, int fildes2); • filedes2 specifies the desired new descriptor number • commonly used to redirect stdin/stdout/stderr Solution Symposium April 3, 2002 Page 38
hp e3000 file management - fcntl() programming and posix • int fcntl(int *fildes, int cmd, ...
hp e3000 process management - fork() - 1 programming and posix #include
hp e3000 process management - fork() - 2 programming and posix • Compile & link sample program shell/iX> gcc -o forkt forkt.
hp e3000 process management - exec() programming and posix • 6 forms: execl(), execve(), execvp(), execv(), execve(), execvp() • replaces the current process with a newly spawned one if ( (pid = fork()) < 0) perrror("fork"); else if (pid == 0) { /* child */ if (execl("/bin/echo", "echo", "child:", "hello", "world", (char *) 0) < 0) perror("execl"); printf("child: this never prints\n"); } Solution Symposium April 3, 2002 Page 42
hp e3000 process management - execl() programming and posix • Compile & link sample program shell/iX> gcc -o execlt execlt.
hp e3000 programming and posix process management getpid()/getppid() • int getpid (void) • returns the POSIX PID of the calling process • pid_t getppid (void); • returns the POSIX PID of the parent of the calling process Solution Symposium April 3, 2002 Page 44
hp e3000 user management - getuid()/setuid() programming and posix • uid_t getuid (void); • returns the POSIX UID of the calling process • int setuid(uid_t uid); • changes the POSIX UID of the calling process • requires GETPRIVMODE() • if you change to a UID in another MPE account, the POSIX GID will also be changed to match the new account Solution Symposium April 3, 2002 Page 45
hp e3000 programming and posix user management getpwnam()/getpwuid() • struct passwd *getpwnam(const char *name); • struct passwd *getpwuid(uid_t uid); • reads from virtual /etc/passwd user directory file • /etc/passwd does not exist on MPE • selected struct passwd fields from /usr/include/pwd.h • pw_name - user name (USER.
hp e3000 group management - getgid()/setgid() programming and posix • gid_t getgid (void); • returns the POSIX GID of the calling process • int setgid(gid_t gid); • exists but isn’t implemented • MPE forces your GID to correspond to the MPE account of your UID anyway Solution Symposium April 3, 2002 Page 47
hp e3000 programming and posix group management getgrgid()/getgrnam() • struct group *getgrgid(gid_t gid); • struct group *getgrnam(const char *name); • reads from virtual /etc/groups group directory file • /etc/groups does not exist on MPE • selected struct group fields in /usr/include/grp.
hp e3000 programming and posix interprocess communication (ipc) • Pipes • pipe(fd[2]) • popen()/pclose() • FIFOs • mkfifo(pathname) • Message queues (in libsvipc) • Semaphores (in libsvipc) • Shared memory (in libsvipc) Solution Symposium April 3, 2002 Page 49
hp e3000 interprocess communication - pipes • Pipes are easy to demonstrate in the shell: programming and posix shell/iX> who am i STEVE,CGI.APACHE@SYSTEMNAME ldev5 TUE 1:04P shell/iX> who am I | cut -f1 -d' ' STEVE,CGI.
hp e3000 pipes the easy popen() way programming and posix #include
hp e3000 pipes the hard pipe() way programming and posix #include #include
hp e3000 sockets programming and posix • InterProcess communciation via socket address: • Internet (32-bit IPv4 address, port number) • Unix (local file name) • Functions • socket() - create socket descriptor • connect() - connect to a remote socket • bind() - to use specific listening socket (i.e.
hp e3000 sockets - server example programming and posix mysock = socket(AF_INET, SOCK_STREAM, 0); bind(mysock,
); listen(mysock, queuedepth); …begin main loop… remotesock = accept(mysock, ); read request with read() or recv() write response with write() or send() close(remotesock); …end main loop… close(mysock); Solution Symposium April 3, 2002 Page 54hp e3000 sockets - client example programming and posix mysock = socket(AF_INET,SOCK_STREAM,0); connect(mysock,); write() or send() the request to the server read() or recv() the response from the server close(mysock); Solution Symposium April 3, 2002 Page 55
hp e3000 inetd socket applications programming and posix • MPE INETD invokes socket server programs with redirection: • fd 0 (stdin) redirected to the accept()ed socket • fd 1 (stdout) redirected to JINETD $STDLIST • fd 2 (stderr) redirected to JINETD $STDLIST • dup2(0,1) for a more typical inetd environment • just do your normal terminal I/O to stdin and stdout which are really network sockets Solution Symposium April 3, 2002 Page 56
hp e3000 signals programming and posix • signal() & raise() are ANSI C, not POSIX.
hp e3000 signals - kill programming and posix • int kill (pid_t pid, int sig); • sends a signal to another process • kill shell command which calls the kill() function Solution Symposium April 3, 2002 Page 58
hp e3000 error handling programming and posix • errno is a global variable defined in
hp e3000 miscellaneous - system() programming and posix • int system(const char *command); • passes command to the shell for execution • all shell metacharacters will be acted upon, so use EXTREME caution when passing user-supplied data to system()! Note that popen() has the same issue.
hp e3000 mpe intrinsics vs.
hp e3000 programming and posix mpe intrinsics vs. posix functions (cont.
hp e3000 additional programming topics programming and posix • Debugging Your Application • Shell Scripts • Regular Expressions • Awk • Security Pitfalls • Development Tools • GNU Tools • Porting Wrappers Solution Symposium April 3, 2002 Page 63
hp e3000 debugging your application - 1 programming and posix • Add printf() statements in your code • use #ifdef DEBUG compile directive • Add perror() statements in your code • use #ifdef DEBUG compile directive if ( (fd = open(pathname, O_RDWR)) < 0) { /* errno already set by open() */ #ifdef DEBUG sprintf(msg, "functionX(): open(%s, O_RDWR)", pathname); perror(msg); #endif return -1; } Solution Symposium April 3, 2002 Page 64
hp e3000 programming and posix debugging your application - 2 • MPE System Debugger shell/iX> callci "run ./program ;debug" • Symbolic debugger - xdb (does not support gcc) • use -g switch during compile shell/iX> c89 -g ... • link with /SYS/LIB/XDBEND – first, as MANAGER.SYS: shell/iX> cd /SYS/LIB; ln -s XDBEND end.o shell/iX> c89 -o … /SYS/LIB/end.
hp e3000 diff and patch commands programming and posix • diff - compares two files and reports on differences • -r option recursively compares two directory trees • -u option on GNU diff for making open-source patches • patch - modifies files using diff output • can modify entire directory trees • saves rejected diff code in *.
hp e3000 shell programming programming and posix • Automate steps with a shell script hwcgi.sh #!/bin/sh gcc -c helloworld.c ar -rv libhw.a helloworld.o gcc -c hwcgimain.c gcc -o hwcgi hwcgimain.o -L. -lhw • Execute permission required to execute shell/iX> chmod u+x hwcgi.sh shell/iX> hwcgi.sh • Special scripts: /etc/profile and .
hp e3000 shell interpreters programming and posix • the first line of a shell script specifies the interpreter to be run and the parameters if any, I.e.
hp e3000 posix shell command syntax programming and posix • (cmd) - execute cmd in a subshell • cmd1 | cmd2 - pipe cmd1 stdout to cmd2 stdin • cmd1 && cmd2 - execute cmd2 only if cmd1 returns zero exit status (true) • cmd1 || cmd2 - execute cmd2 only if cmd1 returns non- zero exit status (false) • cmd1; cmd2 - execute cmd1, then cmd2 • cmd & - execute cmd asynchronously Solution Symposium April 3, 2002 Page 69
hp e3000 posix shell flow of control programming and posix • case word in pattern1) command1 ;; pattern2) command2 ;; esac execute the command of the first pattern matching word • for variable in word1 word2 …; do command done for each word, set the variable to the word and execute the command(s) Solution Symposium April 3, 2002 Page 70
hp e3000 posix shell flow of control (cont.
hp e3000 posix shell flow of control (cont.
hp e3000 posix shell functions programming and posix • function name { command } or name() { command } • treated just like any other command or script • has a separate list of positional parameters • may declare local variables Solution Symposium April 3, 2002 Page 73
hp e3000 posix shell variables and arrays programming and posix • variable=value to assign a scalar variable • variable=value command to make a variable assignment visible only to the new command env • variable[index]=value to assign an array item • $variable or ${variable} to dereference a variable • $variable[index] to deference an array item • ${variable:-default} for the non-null value of variable else default • plus about 15 additional variations… • $? - last exit status value • $$ - POSIX PID of t
hp e3000 posix shell parameters programming and posix • $0 - name of the shell script • $n where n=1..
hp e3000 posix shell command substitution programming and posix • `command` (backquotes) or $(command) • replaces the command substitution expression with the stdout output from the execution of command • TIMESTAMP=$(/bin/date) echo "$TIMESTAMP log event" >>logfile Solution Symposium April 3, 2002 Page 76
hp e3000 posix shell file/directory substitution programming and posix • ~user replaced by user’s home directory • cd ~MGR.APACHE/htdocs = /APACHE/PUB/htdocs • * ? [] - pathname wildcards replaced by possibly multiple files/dirs • * - zero or more characters • ? - one character • [] - group or range (first-last) of characters • /*/PUB/foo.bar - foo.
hp e3000 posix shell i/o redirection programming and posix • file - write stdout to file • >>file - append stdout to file • 2>file - write stderr (2) to file • 2>&1 - write stderr (2) to the same place as stdout (1) • <file here is some data to be copied to a file ALLDONE Solution Symposium April 3, 2002 Page 78
hp e3000 posix shell escaping and quoting programming and posix • \ - disregard the special meaning of the next character • 'string' - disregard the special meaning of all characters in the string • "string" - disregard all special meanings except for command substitution and variable dereferencing • bad: callci run foo;info="bar" • good: callci run foo\;info=\"bar\" • good: callci 'run foo;info="bar"' Solution Symposium April 3, 2002 Page 79
hp e3000 posix shell callci command programming and posix • callci command_string • used to invoke CI commands from the shell • command_string gets passed to HPCICOMMAND • callci uses CI I/O redirection in certain situations including batch jobs, so MPE commands that don't work with CIOR will fail • fails: callci setvar variable value • workaround: callci mysetvar variable value Solution Symposium April 3, 2002 Page 80
hp e3000 posix shell test command programming and posix • test expression or [ expression ] • exit status indicates the result of the expression: • 0 - true • 1 - false • 2 - expression syntax error • -f file - true if the file exists • -d file - true if the file is a directory • string1 = string2 - true if strings are identical • number1 -eq number2 - true if numbers are equal • expr1 -a expr2 - AND relationship • expr1 -o expr2 - OR relationship • and many more...
hp e3000 regular expressions (regexp) programming and posix • the language of pattern matching • man regexp for full syntax • .
hp e3000 awk programming - /bin/awk programming and posix • powerful regexp-based pattern matching and string manipulation • great for file parsing and reformatting • specify search patterns and associated actions • full if-then-else logic and more • better performance in certain applications compared to the POSIX shell because no forking will be done Solution Symposium April 3, 2002 Page 83
hp e3000 potential posix security pitfalls programming and posix • loose or missing umask resulting in world- or groupwriteable security • files and directories rely on ACDs to implement security, and many MPE utilities may still result in ACDs being deleted • setuid/setgid executables • shell metacharacters like > or | or ` being parsed by popen() and system() • user-supplied file names containing multiple upward directory references to reach the root and then downward to any file on the machine, I.e. ..
hp e3000 development tools programming and posix • Edit files from another system • Samba - http: //jazz.external.hp.com/src/samba/ • Development Environments • Whisper Technology http://www.programmerstudio.
hp e3000 gnu tools programming and posix • Downloadable software from: http://jazz.external.hp.com/src/gnu/gnuframe.
hp e3000 porting wrappers programming and posix • Downloadable software from: http: //jazz.external.hp.com/src/px_wrappers/index.
hp e3000 error handling with mpe intrinsics programming and posix • _mpe_errno, _mpe_intrinsic are global variables defined in • Requires _MPEXL_SOURCE compile directive to use • Porting Wrappers has functions pmpeerror() & strmpeerror() plus header file #include
hp e3000 additional resources programming and posix • MPE/iX manuals: http://docs.hp.com/mpeix/all/index.html • HP C/iX Library Reference Manual - function man pages • MPE/iX Developer’s Kit Reference Manual - function man pages • MPE/iX Shell and Utilities User’s Guide - commands, shell, vi, make • New Features of MPE/iX: Using the Hierarchical File System - commands • Programming with examples: • “Advanced Programming in the UNIX Environment” by W. Richard Stevens http://www.kohala.com/start/apue.
hp e3000 additional resources (cont.) programming and posix • POSIX • “POSIX Programmer's Guide” by Donald Lewine http://www.oreilly.com/catalog/posix/ • “The POSIX.1 Standard - A Programmer’s Guide” by Fred Zlotnick • POSIX Specifications from IEEE - very detailed http://standards.ieee.org/ • make • “Managing Projects with make” by Andrew Oram and Steve Talbott http://www.oreilly.
hp e3000 additional resources (cont.) programming and posix • :XEQ POSIXCBT.LSN.SYS - a basic POSIX tutorial bundled in FOS since 5.0 (primarily covers HFS topics) • Invent3k public access development system with preinstall GNU and other POSIX tools – http://jazz.external.hp.
hp e3000 join the hp3000-l community! programming and posix • Available as a mailing list and as the Usenet newsgroup comp.sys.hp.mpe • In-depth discussions of all things HP e3000 • Talk with other people using POSIX on MPE • seek advice, exchange tips & techniques • Keep up with the latest HP e3000 news • Interact with CSY • http://jazz.external.hp.com/papers/hp3000-info.
