HP Tru64 UNIX and TruCluster Server Version 5.1.B-4 Patch Summary and Release Notes (13156)

3.2.1.7 Protection Against Buffer Overflow Exploitation Added
This kit provides a security feature to prevent the execution of instructions that reside in heap
or other data areas of process memory. The result is additional protection against buffer overflow
exploits. This feature is similar in concept to Tru64 UNIX executable stack protection.
This feature is implemented as a dynamic sysconfig tunable variable, executable_data, in
the proc subsystem. The supported settings allow system administrators to cause requests from
privileged processes for writable and executable memory to fail, or to be treated as a request for
writable memory, and to optionally generate a message when such a request occurs.
In a buffer overflow exploitation, an attacker feeds a privileged program an unexpectedly large
volume of carefully constructed data through inputs such as command line arguments and
environment variables. If the program is not coded defensively, the attacker can overwrite areas
of memory adjacent to the buffer.
Depending upon the location of the buffer (stack, heap, data area), the attacker can deceive these
programs into executing malicious code that takes advantage of the program's privileges or alter
a security-sensitive program variable to redirect program flow.
With some expertise, such an attack can be used to gain root access to the system.
Enabling the executable_data tunable changes a potential system compromise into, at worst,
a denial-of-service attack. A vulnerable program may still contain a buffer overflow, but an
exploit that writes an instruction stream into the buffer and attempts to transfer control to those
instructions will fail, because memory protection will prohibit instruction execution from that
area of memory.
Many applications never execute from the memory even though they unnecessarily request
write-execute memory directly or as a result of an underlying function acting on their behalf. By
substituting writable memory for the requested write-execute memory, the executable_data
tunable allows such applications to benefit from the additional protection without requiring
application modification. See sys_attrs_proc(5) for more information.
Before enabling executable_data (changing it from the default value of 0), you must run the
/usr/sbin/javaexecutedata script. Otherwise, privileged Java™ applications will fail in
unpredictable ways. See javaexecutedata(8) for more information.
NOTE: The Java language interprets byte code at runtime. Unless marked as exempt, privileged
applications written in Java will receive an error when they attempt to execute instructions
residing in the unexecutable memory. The manner in which these errors are handled is
application-specific and thus unpredictable. This is why you must run the
/usr/sbin/javaexecutedata before you enable executable_data.
The following example demonstrates the failing behavior to expect for a privileged process if
execute_data is set to 53 but runs the /usr/sbin/javaexecutedata script. Other Java
applications run with privilege may exhibit different (but still failing) behavior.
# java -classic -jar SwingSet2.jar
Process 1185 Invalid write/execute mmap call denied.
Process 1185 Invalid write/execute mmap call denied.
Process 1185 Invalid write/execute mmap call denied.
(...)
Process 1185 Invalid write/execute mmap call denied.
Process 1185 Invalid write/execute mmap call denied.
**Out of memory, exiting**
The following example demonstrates the failing behavior to expect for a privileged processes if
execute_data is set to 37 but runs the /usr/sbin/javaexecutedata script. Other java
applications run with privilege may exhibit different (but still failing) behavior.
# java -classic -jar SwingSet2.jar
Process 1185 Invalid write/execute mmap call modified.
46 Tru64 UNIX Patches