Distributed Systems Administration Utilities User's Guide, Linux, March 2009

traffic within a Serviceguard cluster (member to member). Standard TCP or UDP is used for
intra-cluster log traffic.
ssh port forwarding is transparent to syslog-ng. The /etc/syslog-ng.conf.client file on
Red Hat or /etc/syslog-ng/syslog-ng.conf.client file on SLES is configured so that
syslog-ng forwards log traffic to a local port managed by ssh. The local ssh connects to the
remote sshd on the log consolidation server and sshd opens the standard syslog-ng TCP
port. The remote log consolidation believes it has a standard log forwarding client and is unaware
of the tunneling taking place.
One problem with ssh tunneling is failure of the log consolidation server. If the syslog-ng
server stops or crashes, the remote ssh tunnels disconnect. The client ssh tunnels will try to
reconnect at one minute intervals. The reconnect time is configurable.
Each failed reconnect attempt is logged to the client’s local syslog.log. During this time,
syslog-ng’s client log (/var/log/syslog-ng.log) will show it trying to reconnect to the
tunnel. The default reconnect time is 10 seconds. This is configurable using syslog-ng’s global
setting "time_reopen(<seconds>)" parameter. See the syslog-ng open source reference
manual (/opt/dsau/doc/syslog-ng) for details.
3.5.2.1 ssh Port Forwarding to a Serviceguard Cluster Log Consolidator
When using ssh port forwarding with a Serviceguard cluster as the log consolidation server, a
special ssh configuration is required.
In general, using ssh port forwarding requires that the log consolidation server perform a key
exchange with the log forwarding client. Specifically, the ssh public key for the remote log
forwarding client must be added to the consolidation server’s authorized keys file. Also, the
fingerprint for the log consolidation server is added to the log forwarding client’s /.ssh/
known_hosts file. The client log forwarder is a trusted system after this key exchange, and the
consolidation server does not need to prompt for any ssh passwords at this point.
Since the consolidation server is a package, it can potentially run on every member of the cluster.
This key exchange between the remote log forwarding client and a cluster member must be
replicated for each cluster member. Each cluster member has to establish the same trust
relationship to the log forwarding clients.
A problem can arise with the log forwarding client’s known_host fingerprints. When using a
package’s relocatable IP address for the initial ssh key exchange, the client will have the adoptive
node’s fingerprint added to its local /.ssh/known_hosts file. When the package fails over
and the ssh connection is re-established, the new adoptive node will have a different fingerprint
and ssh will detect this as a man-in-the-middle attack and refuse to re-establish the ssh tunnel.
In order to prevent this, each cluster member must look like the same system from the perspective
of the log forwarding clients. This can be achieved by having each cluster member use an identical
host key. The ssh host keys are located in /etc/ssh and contained in the following files:
• ssh_host_key
• ssh_host_key.pub
• ssh_host_dsa_key
• ssh_host_dsa_key.pub
• ssh_host_rsa_key
• ssh_host_rsa_key.pub
Pick one of the cluster members and copy these files to the same directory on the other cluster
members. Using the “cluster copy” or ccp tool is a quick way to do this, using the following
commands:
# cd /etc/ssh
# ccp ssh_host_* /etc/ssh/cexec /etc/init.d/sshd restart
3.5 Securing Consolidated Logs 79