Dataloader/MP Reference Manual

Running DataLoader/MP
DataLoader/MP Reference Manual—424148-003
3-13
Taking Advantage of Parallelism
Taking Advantage of Parallelism
When parallelism exists, whether is has been created by using the methods described
previously or is the result of having multiple, simultaneous input streams,
DataLoader/MP can continue to process in parallel.
Each of the now-parallel streams does inserts, updates, or deletes, each of the
DataLoader/MP processes continue, and the parallelism inherent in the underlying
system (like NonStop SQL/MP’s sophisticated locking techniques) enables them to
process in parallel.
Alternatively, if each of the now-parallel streams involves performing a load, each of
the DataLoader/MP processes can supply records to its own SQLUTIL process, which
would be performing a PARTONLY load, and parallelism is maintained.
Building Your Loading Application
A typical loading problem cannot be solved by running a single DataLoader/MP
process. Instead, loading is performed by an application consisting of multiple
DataLoader/MP processes, some customized, others standard, and perhaps SQLCI
LOAD or APPEND commands. The next two sub-sections provide suggestions for
configuring and tuning your loading application.
You can to create customized versions of DataLoader/MP with combinations of user
routines appropriate to what you want to do. As you customize each version with the
appropriate user exits, you should give these versions names that indicate their
purpose or their position in the application hierarchy.
Configuration Recommendations
Many factors influence the design of your loading application. The following two factors
will dominate most configuration decisions; you should consider them first:
•
Your data source. Your data can come from a single source (a tape drive) or from
multiple sources (multiple drives or communication lines). In either case, exactly
one DataLoader/MP process is required for each data source.
•
Your data destination. Usually, your data is to be loaded into multiple partitions of
an SQL table using SQLCI LOAD or APPEND. When this is the case, you must
configure exactly one LOAD process for each partition.
The preceding guidelines suggest an initial configuration depending on your situation.
The following suggestions will help when your requirements differ from those listed
previously, or when you are refining and tuning your initial configuration:
•
Loading with SQL Insert in DataLoader/MP. You can write your own user exits to
perform table inserts or deletes in the DataLoader/MP process. In this case, there
is no need to maintain a one-to-one relationship between processes and partitions.
•
Load balancing. Under some conditions, you might be able to split or to combine
the work done by the loading processes for better balance. You will need to