DataLoader/MX Reference Manual (G06.24+)

Recovery Strategies
DataLoader/MX Reference Manual—525872-002
7-2
Design Time Considerations
DataLoader/MX can be used to load a SQL/MX or SQL/MP database in many ways. It
can be as simple as a single DataLoader/MX process putting data directly into a single
table. It can be as complex as dozens, or hundreds, of DataLoader/MX processes
arranged in multiple stages, with these processes taking input from multiple sources,
processing it, and using it to load many different tables, some directly and some
through import, LOAD, or APPEND. Clearly, it is impossible to describe a specific
recovery plan for each possible scenario. Following are a number of different general
approaches you can use to develop an overall recovery plan for your loading scenario.
Design Time Considerations
You should consider how to load your database at the time you design it, along with
other considerations, such as query performance and online transaction processing
performance. Along with those considerations, consider how you will recover if the load
process fails. You must ensure that DataLoader/MX will be able to read the data again,
possibly in the same order as it read it on its first run.
The database design process is a trade-off between a design that can be loaded
quickly and recover from failures during loads and a design that is optimized for other
functions. For example, putting all of the new data in a new partition or at the end of an
existing partition is good for load performance and recovery but can result in poor
application performance because placing all of the new data on a single disk eliminates
the possibility of using parallelism in the normal use of the database. No fixed set of
priorities exist, because different types of applications have different considerations.
Tools
The tools used to implement loading are:
•
DataLoader/MX: This tool uses single threaded or parallel processing to load
databases.
•
import: This tool inserts or appends data in a partition of a SQL/MX table. Note
that import does not delete existing data.
Some Basic Rules
In every case where you must perform recovery of the load process, recovery requires
that the DataLoader/MX process read the input data again, and in some cases, it must
be in the exact same order, so you must determine if this is possible. Whatever the
data source, you must make sure that the data, or a copy of it, remains unmodified
until the load is successfully finished. If necessary, you must add steps to your load
scenario to make this occur.
For example, data in a disk file can easily be read again. However, if the data is being
received by FTP from another system, when you rerun the load after a failure, the data
might no longer be available, or might not be available in the same order. In that case,
you might need to add a step to the normal loading scenario to send the data by using