Apple Qmaster 3 User Manual
Copyright © 2009 Apple Inc. All rights reserved. Your rights to the software are governed by the accompanying software license agreement. The owner or authorized user of a valid copy of Final Cut Studio or Logic Studio software may reproduce this publication for the purpose of learning to use such software. No part of this publication may be reproduced or transmitted for commercial purposes, such as selling copies of this publication or for providing paid for support services.
Contents Preface Chapter 1 5 5 5 6 7 8 10 14 15 16 29 31 33 Welcome to Apple Qmaster About Apple Qmaster About the Apple Qmaster Documentation Additional Resources Using Apple Qmaster 3 The Apple Qmaster Distributed Processing System The Apple Qmaster Interface Installing Apple Qmaster, Shake, and Other Applications Setting Up Rendering Services and Shared Storage Submitting Batches for Apple Qmaster Processing Using Nodes Without Apple Qmaster Installed Setting Environment Variables in Apple Qmaster Us
Preface Welcome to Apple Qmaster The Apple Qmaster system provides automated work distribution and processing for high-volume projects created with digital visual effects software packages such as Shake. This preface covers the following: • About Apple Qmaster (p. 5) • About the Apple Qmaster Documentation (p. 5) • Additional Resources (p.
• Batch Monitor User Manual: This brief document describes how to use Batch Monitor to monitor the transcoding progress of batches and jobs. Additional Resources Along with the documentation that comes with Apple Qmaster, there are a variety of other resources you can use to find out more about Apple Qmaster. Compressor Website For general information and updates, as well as the latest news on Compressor, go to: • http://www.apple.
Using Apple Qmaster 3 1 The Apple Qmaster application is the client software you use to submit jobs and batches (for Shake and related software) to the larger Apple Qmaster system, an automated work distribution and processing solution for high-volume projects. This chapter covers the following: • The Apple Qmaster Distributed Processing System (p. 8) • The Apple Qmaster Interface (p. 10) • Installing Apple Qmaster, Shake, and Other Applications (p.
The Apple Qmaster Distributed Processing System Computers that submit batches to Apple Qmaster are called clients. An Apple Qmaster job is a processing task in the form of a Shake file, or other file or commands, that uses UNIX commands to specify settings such as rendering instructions and file locations and destinations. Network Batch of processing jobs Client computer Jobs are submitted. Processed files Apple Qmaster cluster Processing is performed by cluster.
Clusters When a client sends batches to the Apple Qmaster system, all the processing and subsequent moving of any output files is performed by a group of Apple Qmaster–configured computers called a cluster. To set up Apple Qmaster services, you use the Apple Qadministrator application to create one or more clusters of service nodes, with one cluster controller included in each cluster. Each computer in the cluster is connected to the other computers in the cluster through a network connection.
The Apple Qmaster Interface The Apple Qmaster application described in this document is one part of the larger distributed processing system described in The Apple Qmaster Distributed Processing System. The Apple Qmaster application is the client software you use to submit jobs and batches to the system. Both this application and the larger system are known by the same name (Apple Qmaster).
• Distributed Compressor service: servicecontroller:com.apple.stomp.transcoderx • Distributed Apple Qmaster service: servicecontroller:com.apple.qmaster.executor • Batch Monitor: Opens Batch Monitor, which allows you to view the status of all batches being processed. Batch Table This is the large middle area of the Apple Qmaster window that lists the individual jobs in a batch. The Batch table includes the following columns. • Type: One of four job types: Shake, Maya, Generic Render, Shell (UNIX).
• Maya: For more information, see Submitting Maya Batches. • Shake: For more information, see Submitting Shake Batches. • Shell: For more information, see Using Apple Qmaster to Submit UNIX Commands. • Add button: Use the Add (+) button to enter a job (with the selected job type) in the Batch table. Add button • Remove button: Use the Remove (–) button to delete the selected job from the Batch table.
• Delay: Use these text fields to set a delay in hours or minutes for any job in the batch. Click in either text field (left field = hours, right field = minutes) to enter the corresponding value. Click the arrows to step through minutes in increments of 5 at a time, or, if the Hour text field is selected, in increments of 1 hour at a time. • Submit: Click the Submit button to process the batch using the parameters set in the Apple Qmaster window.
Remote Computer Address Dialog This dialog is displayed when you click the Add/Remove button in the main Preferences dialog. Use this dialog to enter IP addresses or ranges for manually selected computers. • Host/Host IP Address Range buttons: These buttons control whether this dialog is in IP Address mode (in which you enter a specific address) or IP Address Range mode (in which you enter a range of addresses). • Host: Use this mode to enter a host name and IP address for a specific remote computer.
To use Apple Qmaster with Shake, or other frame-based rendering applications, you must follow these installation and licensing guidelines. • To submit batches for Shake: A copy of Shake must be installed on every node in the cluster. For command-line rendering on Mac OS X, a license is not needed. • To submit batches for Maya: Install and license a copy of Maya on every node in the cluster.
Shared Volume Media Management with Shake It may be necessary to relink your assets once you move your project to the shared volume. This varies depending on the types of files, plug-ins, fonts, and environmental requirements. Things that commonly need to be changed are the FileIn and FileOut paths, and env variables such as NR_INCLUDE_PATH and NR_FONT_PATH. UNC should be disabled, as these paths are typically not resolvable to nodes in the cluster when arbitrary host names are used.
• To submit batches for any application that can be executed from the UNIX command line: See Using Apple Qmaster to Submit UNIX Commands. Submitting Shake Batches Using Apple Qmaster, you can quickly create and submit a batch of Shake jobs. Note: You can also submit Shake batches as described in Using Apple Qmaster to Submit UNIX Commands, but the following method is the easiest way to submit Shake batches with Apple Qmaster. To submit Shake batches with Apple Qmaster 1 Open Apple Qmaster.
Each setting corresponds to a commonly used Shake command, with the exception of the Minimum setting. See the Shake documentation for further information about the Shake settings in this dialog. See About the Shake Plug-in Dialog for more information about the the Minimum setting. With the exception of the Minimum setting, each setting in the dialog corresponds to a commonly used Shake command or option. Refer to the Shake documentation for more information.
5 To add more jobs to this batch, repeat steps 3 and 4. 6 As needed, make any final changes to the batch, such as either of the following. • To edit a command in the list: Double-click the job. The command dialog in step 4 appears. • To delete a job from this batch (before you submit the batch): Select the job you want to delete and click the Remove (–) button. 7 As needed, set the following options for this batch. • Submit To: Choose a cluster to process this batch.
About the Shake Plug-in Dialog The Shake Plug-In dialog contains the following settings. With the exception of the Minimum setting, each setting in the dialog corresponds to a commonly used Shake command or option. Refer to the Shake documentation for more information. Basic Shake Plug-in Dialog Details • Shake field: The Shake field shows the location of the Shake application on your computer. • Script field: The Script field shows the location of the Shake project file.
• Minimum field: You can change the value in the Minimum field to change the minimum number of frames in each Apple Qmaster segment. For example, with a minimum value of 10 (the default), Apple Qmaster would divide a 100-frame job into at least 10 segments of 10 frames each. If you changed the minimum value to 5, Apple Qmaster would divide a 100-frame job into at least 20 segments of 5 frames each.
Additional Shake Plug-in Dialog Details • Monitor pop-up menu: Though the Monitor pop-up menu corresponds to a Shake command option, it requires some explanation in the context of this dialog. The pop-up menu lets you choose whether or not to display each frame as it renders. If you don’t want to see the frames as they render, choose None. To see the frames as they render, choose a display from the Monitor pop-up menu, then specify the FileOut node in the Options field. For example: -node nodename.
• In the Frames section, enter a Start and End frame for the job. The available settings correspond to commonly used Maya command options. You can also type additional command options in the Options field. See the Maya documentation for more information about all the Maya command settings. 3 Click OK when you have finished creating the full command. 4 To add more jobs to this batch, repeat steps 1 through 3.
Another option is -rt. This specifies the number of rendering threads. By default, the number of rendering threads is 2. You might want to change this value depending on the type of computers in your cluster. In a homogeneous cluster environment, say of Mac Pro or Power Mac G5 quad systems, you would change this value to 4, as there are four processors per node, and some Maya operations might be faster with more threads. This option is passed to all the nodes in the cluster.
2 In the dialog that appears, enter the relevant information in the required and optional fields. The full command is displayed here. Some fields and settings in this dialog are optional, but the following are required: • The Executable field requires the location of the frame-based rendering application—make sure it is correct. • Use the Choose buttons to navigate to and choose the location of the project and scene files for this job. • In the Frames section, enter a Start and End frame for the job.
5 Verify the full command in the Command text box and click OK. You can further customize the batch instructions by adding command-line options here. The full command is displayed here. 6 To add more jobs to this batch, repeat steps 1 through 5. 7 As needed, set the following options for the batch in the Apple Qmaster window: • Submit To: Choose a cluster to process this batch.
[START FRAME] = 0 [END FRAME] = 544 [STEPS] = Not Used [ITEM] = /SHARED_VOLUME/Projects/AEProjs/Wine_Country/output In the example above, “Comp 1” is the name of the composition in your After Effects project and “test[####].psd” is the output filename. Since you cannot select an output filename in the template’s Item field before the output has been rendered, first select the output directory, then add the filename to the Item field. Note: In the example, /SHARED_VOLUME/Projects/ is the shared volume.
Note: Apple Qmaster does not currently support SMTP servers that require authentication. Using Apple Qmaster to Submit UNIX Commands You can use Apple Qmaster for distributed processing of any UNIX commands, including any application that can be operated from the command line. Apple Qmaster provides a convenient way to issue commands, shortening the amount of typing you need to do by providing an interface for the global parts of the command.
Note: If you need to delete a job from this batch (before you submit the batch), select the job you want to delete and click the Remove (–) button. 5 As needed, set the following options for this batch. • Submit To: Choose a cluster to process this batch. • Working Directory: If relevant, enter the working directory (from which you want the command to be executed). • Priority: Choose the priority level from the pop-up menu.
To configure an intermediary service node to run Apple Qmaster commands on extended nodes 1 Open the Apple Qmaster pane in System Preferences. 2 If this intermediary node will be part of a mixed cluster, with a different computer acting as the cluster controller, skip this step and go to step 3.
b Edit the Host name, Username, and (if applicable) Password columns so that they describe each computer you want to use as an extended node, then click OK. Edit the Host name, Username, and Password columns to specify the extended nodes you are adding. 6 In the Apple Qmaster preferences pane, click Start Sharing. If you configured this intermediary node as the cluster controller, your work is done and the extended node cluster is ready to start receiving and processing batches submitted by Apple Qmaster.
About Environment Variables and Preflight Scripts You can streamline your workflow with environment variables and preflight scripts. What’s an Environment Variable? Environment variables are settings within a shell, from which commands are issued that affect the way a program operates. People who regularly execute commands from a Terminal shell, or write shell scripts, are usually familiar with environment variables.
4 Click OK. Your environment variables take effect immediately, and are retained only for the selected command. Using the Command Line in Apple Qmaster If you are accustomed to doing your work from Terminal shells, and you need or prefer to run Apple Qmaster from the command line with minimal use of application interfaces, this section is for you. Note: On any given Final Cut Studio system or network, the Compressor and Apple Qmaster versions (numbers) and the version of QuickTime must all match.
Synopsis Below is a synopsis of the command for enabling and disabling Apple Qmaster services on a computer. The qmasterprefs command is located in /usr/sbin. qmasterprefs [options] [-cluster [options]] [-service [options]] Command Options This table provides information about each of the enabling and disabling service-node and cluster-controlling services on a computer.
Note: For a complete list of shell command options, enter qmasterprefs -help. Example of the qmasterprefs Command In the following example, the computer is set to be a cluster controller, bound to the primary network interface (en0), with logging set to verbose, with the logs not overwritten. The Rendering service is on, with 4 managed services and logging set to verbose, with the logs not overwritten. The Compressor Processing service is configured with the same settings, with two instances.
µ To configure a service-only node Enter the following: qmasterprefs -stopSharing -sharingType servicesOnly -startSharing µ To create a QuickCluster Enter the following: qmasterprefs -stopSharing -sharingType servicesAndQuickCluster -startSharing Shell Commands for Submitting Apple Qmaster Jobs You can use the Apple Qmaster command, Apple command-line options for submitting jobs.
Submission command option Description -command Specifies the kind of command you are entering: Shell, Shake, Maya, or other command, depending on the application you want to use for distributed processing. -options Specifies the command with XML code. Enter the XML code after -options, with the necessary qualifiers for entering it in a shell. See Example of Shake Command XML, next, for an example.
Compressor [-clustername ][-clusterid ] [-batchfilepath ] [-batchname ] [-priority ] [-jobpath ] [-settingpath ] [-destinationpath ] [-info ] [-timeout ] [-help] [-show] In this example, -jobpath, -settingpath, and -destinationpath can be repeated as many times as the number of jobs you want to submit. Note: Not all the options are necessary.
Submission command option Description -jobpath Specifies the location of the source file. -settingpath Specifies the location of the settings for the job. -destinationpath Specifies the destination file URL for the job. -info Gives detailed information for a batch or a job. -timeout Use to specify the number of seconds before Compressor can quit when looking for a cluster.
Batch Monitor [-clustername ] [-clusterid ] [-jobid -batchid ] [-timeout ] [-query ] [-help] µ To cancel a job or batch Use the following command: Batch Monitor [-clustername ] [-clusterid ] -kill -jobid -batchid Command Options This table provides information about each of the command options for monitoring batches.
Using Scripts to Run Apple Qmaster, Compressor, and Batch Monitor To simplify your use of the command line, you can add command-line-friendly scripts to /usr/bin. Each script sets an env variable for location and then executes the binary. Using the following scripts would be useful if you frequently use the command line for submissions or automated submissions. You do not need to specify the full path, and you can add frequently used env variables and options to speed up the submission process.
Batch Monitor Script Example #!/bin/csh -f # # set env var, BATCHMONITOR_LOCATION if not set # if ${?BATCHMONITOR_LOCATION} == 0 then pushd `dirname $0` >& /dev/null setenv BATCHMONITOR_LOCATION "/Applications/Utilities/Batch Monitor.
