Avid Media | Index Configuration Guide Version 2.
Legal Notices Product specifications are subject to change without notice and do not represent a commitment on the part of Avid Technology, Inc. This product is subject to the terms and conditions of a software license agreement provided with the software. The product may only be used in accordance with the license agreement. This product may be protected by one or more U.S. and non-U.S patents. Details are available at www.avid.com/patents. This guide is protected by copyright.
THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED WARRANTY. IN PARTICULAR, NEITHER THE AUTHOR NOR AT&T MAKES ANY REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE MERCHANTABILITY OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR PURPOSE. This product includes software developed by the University of California, Berkeley and its contributors. The following disclaimer is required by Paradigm Matrix: Portions of this software licensed from Paradigm Matrix.
Contents Using This Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Symbols and Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 If You Need Help. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Avid Training Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Registering a Provider with the Production Services Engine . . . . . . . . . . . . . . . . . . . . . . . . . 50 Creating a Consolidate Profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Chapter 7 Configuring iNEWS for Media | Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Editing iNEWS Site Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Monitor the Re-index Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Using This Guide This guide is intended for users of Avid MediaCentral Platform Services (MCS) with the Media | Index feature. This guide describes the configuration procedures for Media Index, such as configuring Media Index for single-server and cluster installations. For information on installing MCS, see the MediaCentral Platform Services Installation and Configuration Guide. For administrative information for MediaCentral | UX, see the Avid MediaCentral | UX Administration Guide.
Symbols and Conventions Symbols and Conventions Avid documentation uses the following symbols and conventions: Symbol or Convention Meaning or Action n A note provides important related information, reminders, recommendations, and strong suggestions. c A caution means that a specific action you take could cause harm to your computer or cause you to lose data. w A warning describes an action that could cause you physical harm.
Avid Training Services Avid Training Services Avid makes lifelong learning, career advancement, and personal development easy and convenient. Avid understands that the knowledge you need to differentiate yourself is always changing, and Avid continually updates course content and offers new training delivery methods that accommodate your pressured and competitive work environment. For information on courses/schedules, training centers, certifications, courseware, and books, please visit www.avid.
1 Media | Index Configuration Overview This document describes the tasks required to configure a Media Index system. Media Index comprises the following components: • Elasticsearch® service, which manages data storage and runs the search engine. • Elasticsearch tribe service, the federated client that enables multi-zone search functionality. • Data Import API, a bus-based service for metadata indexing, search, configuration, status, permissions, and updates for Media Index.
Checklist for Configuring Media | Index For more information on hardware requirements, see the MediaCentral Platform Services Hardware Guide. Media Index is installed with MediaCentral Platform Services by default, but the Media Index services are not started by default. Once configured, Media Index services start automatically when the server is powered-on. Media Index can be configured in both standalone and multi-zone environments.
Checklist for Configuring Media | Index Task Section Reference b If you are integrating with an Interplay Production “Installing and Configuring Production Services system that includes Production Services such as Interplay Delivery, configure additional provider services used with Media Index workflows. for MediaCentral Delivery” on page 45. b If you are integrating with an Avid iNEWS system, “Configuring iNEWS for Media | Index” on configure the iNEWS servers for Media Index.
2 Configuring Media | Index for a Single Server When you set up Media Index, you enable Elasticsearch and several other services related to Media Index. Elasticsearch and Elasticsearch Tribe are used to index, store and search assets while the Media Index services add specific Avid logic to Elasticsearch. Elasticsearch indexes are split into “shards” and “replicas”. Each shard contains indexed data and each replica contains a copy of that data.
Configuring Media | Index 3. Delete the hostname value and type the value you want to use to assign to node. n If the node name variable is not set, Elasticsearch automatically generates node names when your node starts. If it is set to hostname, Elasticsearch uses the hostname of the node where the service is running. 4. Press the Escape key to return to command mode, then type :wq and press Return to write and quit the text editor. 5.
Configuring Media | Index • Wait Time — Sets the number of minutes for the cluster to wait for the ideal number of active nodes. The default value for this field is 5 minutes. However, this option is actually not used in single-server configurations. • Supported languages — Lists the languages supported by Media Index. Media Index supports multiple languages for your search. However, enabling all languages can slow down searches. You should enable only those languages your users require.
Configuring Media | Index Enabled Languages Language Selected for Search Languages in Results Zone 1: English, French French Zone 1: French Zone 2: English, Korean Zone 1: English, French, Korean Zone 2: [no results] French Zone 1: French Zone 2: English, French, Korean c Zone 2: French If you change your language settings after you have configured Media Index, you must reset Media Index. This deletes all indexes, which you then need to re-index. To configure Media Index: 1.
Validating the Media | Index Configuration Validating the Media | Index Configuration You can validate your setup by checking the status of the Media Index services and ensuring that they are running. Using the Cluster State pane in MediaCentral UX, you can view your node with core services running and see that the system data index is available. To check the current status of Media Index: 1. Sign in to MediaCentral UX as an administrator. 2. Select System Settings from the Layout selector.
Continuing the Installation Continuing the Installation Depending upon your workflow, proceed to one or more of the following sections as applicable: • If you are installing Media Index in a multi-zone configuration, see “Setting Up Multi-Zone Search” on page 29. • If you are integrating with and Interplay Production or Interplay Archive system, you need to configure and start the Production Engine Bus Connector (PEBCo) service.
3 Configuring Media | Index for a Cluster When you set up Media Index, you enable Elasticsearch and several other services related to Media Index. Every node in an MCS cluster runs an Elasticsearch instance as well as an Elasticsearch-tribe instance. Elasticsearch and Elasticsearch Tribe are used to index, store and search assets while the Media Index services add specific Avid logic to Elasticsearch. Tribe instances allow you to perform read operations against the nodes in multiple zones.
Shard Count and Usage If you are installing a net-new Avid asset management system and MediaCentral and you cannot anticipate the number of assets, configure the primary shard value as “1”. n You can reconfigure the primary shard count at a later date, after having a better idea of the average number of assets for your workflow. However, reconfiguring the primary shard count requires a re-index of all indexed databases - which can be time consuming. Replica shards can be reconfigured without a re-index.
Shard Count and Usage As previously stated, this configuration spreads the load across multiple systems which increases database efficiency and enables additional redundancy. Notice that the primary shard and its replica are never co-located on the same server. The number of primary shards in your configuration affects the amount of data that can be stored in your index, although the type of data and metadata affects the size of the index as well.
Preparing for the Media | Index Configuration Note that in this configuration, the system is still fully redundant. Each primary shard has a replica on another node. Node 3 now hosts two primary shards. Example 3: In this final example, MediaCentral has been configured for 3 primary shards and 1 replica: If Node 1 goes offline, Replica 1 on Node 3 becomes a primary shard and the index remains fully functional.
Adding Resources to the Cluster To close the Cluster Resource Monitor, press CTRL-C on a Windows keyboard or CMD-C on a Mac keyboard. n For more information on using the cluster resource monitor utility, see “Cluster Resource Monitor” in the MediaCentral Platform Services Concepts and Clustering Guide. Adding Resources to the Cluster When you install a new MediaCentral system, all Media Index services are stopped except for the “avid-acs-media-index-configuration” service.
Configuring Media | Index Configuring Media | Index You configure Media Index in the MediaCentral | UX System Settings. You must provide the following properties in your settings: • c c Cluster Name — This name has no direct relation to the virtual host name given to the Corosync cluster. The “Cluster Name” is simply an identifier for the Media Index configuration on this system. In a standalone configuration (non-multi-zone), Avid recommends using the Corosync cluster hostname or site location (e.g.
Configuring Media | Index • Minimum active nodes — This is the minimum number of nodes that must be active for the Elasticsearch cluster to start. The correct and default value for a two-node cluster is 1. If you have more than two nodes, this value should be set to two-thirds of the total number of nodes. For example in a three node cluster, this value should be set to 2. • Ideal active nodes — This is the ideal number of nodes that should be active before the Elasticsearch cluster starts.
Configuring Media | Index Enabled Languages Language Selected for Search Languages in Results Zone 1: English, French English Zone 1: English Zone 2: English, Korean Zone 1: English, French Zone 2: English French Zone 1: French Zone 2: English, Korean Zone 1: English, French, Korean Zone 2: [no results] French Zone 1: French Zone 2: English, French, Korean c Zone 2: French If you change your language settings after you have configured Media Index, you must reset Media Index.
Validating the Media | Index Configuration 7. Click Apply. The system will display a message on the screen indicating “Update in progress...”. After a few moments, a second message should appear indicating that Index has been configured successfully: 8. Click Close. Validating the Media | Index Configuration You can validate your setup by checking the status of the Media Index services and ensuring that they are running.
Continuing the Installation To monitor the fail count within the cluster: 1. Log in to any node in the cluster as root and verify the cluster fail counts by typing the following command: crm_mon -f When configuring a cluster for Media Index a number of new resources are added to Pacemaker, making it difficult to see the fail-counts at the end of the output.
4 Setting Up Multi-Zone Search In a multi-zone MCS configuration, each zone runs its own Elasticsearch cluster with its own Elasticsearch Cluster Name. This is true for MCS single-servers as well as MCS servers in a Corosync cluster configuration. Single servers run a Elasticsearch cluster consisting of one node. These different Elasticsearch clusters are then connected to each other using the Elasticsearch tribe service.
Setting the Zone Bindings To update the Elasticsearch tribe configuration: 1. Using the Linux text editor, vi, open the Elasticsearch tribe configuration file for editing: vi /etc/elasticsearch-tribe/elasticsearch.yml 2. Add the following lines in the configuration file to add the binding information that allows the tribe to connect to another Media Index cluster: tribe..cluster.name: tribe..transport.tcp.
Setting the Zone Bindings Example from the master zone of the multi-zone configuration: [root@wavd-doc01 ~]# cat /etc/elasticsearch-tribe/elasticsearch.yml discovery.zen.ping.multicast.enabled: false tribe.local.cluster.name: Master_Zone tribe.local.transport.tcp.port: 9312 tribe.local.discovery.zen.ping.unicast.hosts: - "wavd-doc01:9300" http.port: 9201 transport.tcp.port: 9305 node.name: wavd-doc01-tribe tribe.Slave_Zone.cluster.name: Slave_Zone tribe.Slave_Zone.transport.tcp.port: 9313 tribe.Slave_Zone.
Setting the Zone Bindings http.port: 9201 transport.tcp.port: 9305 node.name: wavd-news-tribe tribe.Master_Zone.cluster.name: Master_Zone tribe.Master_Zone.transport.tcp.port: 9313 tribe.Master_Zone.discovery.zen.ping.unicast.hosts: - "wavd-doc01:9300" tribe.Slave_Zone.cluster.name: Slave_Zone tribe.Slave_Zone.transport.tcp.port: 9314 tribe.Slave_Zone.discovery.zen.ping.unicast.
Validating the Elasticsearch Configuration Validating the Elasticsearch Configuration When you finish updating the Elasticsearch configuration file, you can validate your setup by checking the Elasticsearch information for each server.
Validating the Elasticsearch Configuration 2 The import client listens for updates from the PEBco service. There is one instance of this service in each zone. In cluster configurations, the import client may or may not reside on the same server as the master. 3 Each remote zone tribe is bound to the local cluster. This this example, two remote zones are shown. Note that the tribe name reflects the local zone. 4 The local zone’s data node is listed.
Validating the Elasticsearch Configuration 5. Open a browser and navigate to http://:9201/_plugin/head, where sever_name is the host name or IP address of a server in a zone in your configuration — for example, the local zone — and the port number is configured for the tribe node of the zone. The Elasticsearch head plug-in Web page for the tribe node opens in your browser. 6. Verify that the tribe node lists the all indexes in your configuration as well as the local and remote data nodes.
5 Configuring the Production Engine Bus Connector The Production Engine Bus Connector (PEBCo) queries the change table of the Interplay Production Engine and delivers that data to the Avid Common Services (ACS) bus where the import service takes these messages and sends the data to the Elasticsearch index. A timestamp is saved on the Interplay Production Engine, and the PEBCo service queries the Interplay Production Engine regularly and uses the timestamp to determine which assets have been already indexed.
Enabling Sync Events in the Interplay Administrator To enable the sync events (Media Index v2.3 or later): 1. Start Interplay Administrator and log on to the Interplay Production or Interplay Archive database that you want to be indexed. 2. In the Server section of the Interplay Administrator, click the Server Settings icon. The Server Settings view opens. 3. In the Update Tracking section, select Enabled to enable sync events. n You do not need to restart the Interplay Production or Archive Engine. 4.
Creating a PEBCo Instance on the MCS Server Creating a PEBCo Instance on the MCS Server The Production Engine Bus Connector (PEBCo) is automatically installed during the MCS installation. Although before the service can be used, it must first be configured. Proceed to one of the following sections: • Creating a PEBCo Instance on a Single MCS Server • Creating a PEBCo Instance on an MCS Cluster Creating a PEBCo Instance on a Single MCS Server To create a PEBCo instance: 1.
Creating a PEBCo Instance on the MCS Server Creating a PEBCo Instance on an MCS Cluster To create a PEBCo instance: 1. Type the following command on any cluster node to create a PEBCo instance on the MCS cluster. pam-agent-ctrl cluster add can be any meaningful name.
Configuring the ACS Bus Service in Interplay Administrator Configuring the ACS Bus Service in Interplay Administrator The PEBCo configuration requires a series of bus calls to the MCS server. For the configuration to be successful, the Interplay Production and /or Interplay Archive Engines must have the proper bus URL so that the Interplay Administrator can configure the service correctly. To configure the ICS bus service: 1.
Configuring and Using the PEBCo Service Configuring and Using the PEBCo Service This section provides information about the Production Engine Bus Connector (PEBCo) Service. After you have reviewed the information in “About the PEBCo Service”, continue with the processes for “Configuring the Canonical Data” on page 43 and “Starting the PEBCo Service” on page 44. About the PEBCo Service Every PEBCo instance is bound to no more than one Interplay Production or Interplay Archive database.
Configuring and Using the PEBCo Service • n The Canonical Data tab lets you select the Properties used in the indexing process. These properties can be selected by users when performing an indexed search in MediaCentral UX. Some properties are part of the standard data model. These properties are enabled by default and cannot be deselected. Before you can start the first indexing operation, you must specify the Canonical Data set in the Canonical Data tab.
Configuring and Using the PEBCo Service Interplay Production Property Media Index Field Comments Description Mime Type Type Due to differences in the properties and fields used by Interplay Production and Media Index, some duplication appears in the criteria available for indexed searches with both Interplay Production and Media Index fields appearing. If you add the wrong fields as criteria, your search might not return any usable results.
Configuring and Using the PEBCo Service Starting the PEBCo Service The final step in the process is to start the PEBCo service and initiate the indexing process. To configure an instance of the PEBCo service: 1. Start Interplay Administrator and log on to the Interplay database that you want to be indexed. 2. In the Site Settings section of the Interplay Administrator, click the Production Engine Bus Connector icon. n The Production Engine Bus Connector view is not available on Macintosh clients. 3.
6 Installing and Configuring Production Services for MediaCentral Delivery The Production Automation service and the Interplay Consolidate service are used for the MediaCentral Delivery feature.
Check List for Installing and Configuring Automation and Consolidate Services The following illustration shows the Production Services components and a Delivery workflow. MediaCentral Client (Browser) q Production Services Engine w e e w Media Services Automation r w Interplay Consolidate Interplay Delivery 1. The user initiates a Delivery request. 2. The Production Services Engine forwards the request to the Production Services Automation service, which determines the next step. 3.
Prerequisites for Installing and Configuring Automation and Consolidate Services Production Services Check List(Continued) Task Section Reference b Connect the Avid shared storage client to the Avid shared storage client documentation. System Director and mount the required workspaces. b Create at least one Interplay Consolidate profile. “Creating a Consolidate Profile” on page 54 No profile is needed for Production Services Automation.
Installing the Automation and Consolidate Services c When you create a Delivery profile for MediaCentral, use a network path to an Avid shared storage share for the Temporary Workspace on Sender parameter. For example, use \\avidstorage_server\Delivery_temp. This storage space is used for temporary media files created by the Interplay Consolidate service. If you use a local folder, the large amount of media created can cause system problems for the Delivery server.
Registering the Automation and Consolidate Services To install service descriptions: 1. Start the Production Services Engine and log in as administrator. 2. Click the Admin Tool button. The Interplay Production Services and Transfer Status tool opens. 3. Click the Services tab. The Services page displays the currently configured services. 4. Click Install/Upgrade. The Install/Upgrade Service dialog box opens. 5. Click the Browse button and navigate to the folder containing the service package (.
Registering a Provider with the Production Services Engine 8. Click Install/Upgrade. The service and its description appear on the Services page. The following illustration shows both services after they are registered. Registering a Provider with the Production Services Engine A service that is installed and registered is referred to as a provider. Starting with Interplay Production Services v3.1, some service providers operate as Windows services, and others operate as Windows applications.
Registering a Provider with the Production Services Engine 2. Substitute the hostname of the Production Services Engine. Following is an example of an Automation service .ini file with the default hostname “localhost”: #Mon May 05 11:34:54 EDT 2014 @5%?MaxJobs=1 @2%tMedia_Services_Engine_Hostname=localhost @1%tProvider_Name=a51-WG6_Consolidate_0123 Following is the same .ini file, edited to use the hostname “a51-MSE.
Registering a Provider with the Production Services Engine Registering and Connecting a Consolidate Provider Interplay Consolidate runs as a Windows application. You need to register the provider by connecting to the Production Services Engine. n If you try to connect to the Production Services Engine before the latest service is registered, the Status line in the Transcode Service dialog box reads: Error From Broker! UNKNOWN_SERVICE.
Registering a Provider with the Production Services Engine 3. Do the following: a. Provider Name — A default name for the provider is automatically supplied. Accept the default name or type a new name. In this example, the name is Consolidate_01. b. Production Services Engine Host Name — Type the name of the system running the Production Services Engine application. c. Automatically Connect — Select Yes to automatically connect the provider to the Production Services Engine when the application starts.
Creating a Consolidate Profile Creating a Consolidate Profile If users in your MediaCentral zone are going to perform MediaCentral Delivery operations that use “Deliver from Mark In to Mark Out,” an administrator must create at least one Consolidate profile. This default profile must be named “default_for_delivery.” The target video quality for this profile is displayed in the Deliver or Deliver To dialog box, and is used if the user selects “Deliver from Mark In to Mark Out.
Creating a Consolidate Profile Other options in the Consolidate profile, such as Handle and Interplay Checkin Folder, are also used for Deliver from Mark In to Mark Out operations. However, they are not shown in the Deliver To or Deliver to Me dialog box. To create a Consolidate profile: 1. Start the Production Services Engine. 2. Click the Admin Tool button. The Interplay Production Services and Transfer Status tool opens. 3. Click the Profiles tab. 4.
Creating a Consolidate Profile Consolidate Profile Settings(Continued) Parameter Description TargetAudioQuality Select an audio resolution for the consolidate operation: • None • MPEG1 Layer 2: Digital audio compressed to the MPEG1 Layer 2 specification at 96 Kb/s. • Uncompressed PCM: 16-bit 48-kHz digital audio • Uncompressed PCM 24-bit: 24-bit 48-kHz digital audio TargetVideoQuality Select a video resolution for the consolidate operation.
7 Configuring iNEWS for Media | Index When integrating iNEWS with Media Index, the iNEWS system relies on the sync agent in the Mediaindex utility program to push stories to Media Index. It must be configured to control what data gets indexed. To specify which Media Index the iNEWS system should use also requires configuration within iNEWS. This integration first appeared with the release of iNEWS v5.
iNEWS v5.2.x iNEWS v5.
Editing iNEWS Site Files Editing iNEWS Site Files The following site files must be edited on the iNEWS servers to enable Media Index functionality: • /site/dict/words • /site/dict/queues • /site/config The Mediaindex program must be pointed to the message broker by adding a definition in the /site/ dict/words file: • W_BINDBUS – Identifies the binding address for the MediaCentral platform’s bus server port, which is necessary for central indexing as part of iNEWS integration with Media Index.
Editing iNEWS Site Files To edit the iNEWS configuration file: 1. From the PuTTYCS application, select the PuTTY Filter created for sending server commands to all servers, such as iNEWS Consoles. c Always back up the /site/config file before making any changes. 2.
Manually Installing the log4cpp File For more information on the Q_MEDIA_INDEX token in /site/dict/queues, see Appendix C “Standard Dictionaries,” in the Avid iNEWS Setup and Configuration Guide. 3. Reconfigure the system. Manually Installing the log4cpp File The log4cpp library package for RHEL 6.5 is required for iNEWS to work with Media Index. It is necessary for the sync agent in the Mediaindex program to function with the Avid Common Services (ACS) bus.
Assigning the Media | Index Attribute Assigning the Media | Index Attribute Define the queues or folders in the iNEWS directory that will be monitored by Media Index by applying the Media Index attribute (+mi). To apply the Media Index attribute to a queue or folder: t From the console, type the dbtraits command in the following format: dbtraits +mi For example: dbtraits SHOW.
8 Setting Up Interplay | MAM Sync Service The following topics provide basic information to help you get started using Avid Interplay | MAM Sync Service Administrator: • About Interplay | MAM Sync Service Administrator • Opening Sync Service Administrator • Understanding the Sync Service Administrator Layout • Quitting Sync Service Administrator For more information on using the Interplay MAM Sync Service Administrator, see the Interplay | MAM Sync Service Administrator User’s Guide.
About Interplay | MAM Sync Service Administrator Note the following limitations: • All Attributes marked as “searchable” in the MAM data model are always sent to Central Index during synchronization; attributes marked as “not searchable” are not sent to Central Index and are not available for searching or in search-result lists. • Multi-value (MV) attributes and multi-value compound (MVC) attributes cannot be used for searching in Central Index and cannot be displayed in search-result lists.
About Interplay | MAM Sync Service Administrator Sync Service Administrator Configuration Options Some Sync Service Administrator features are subject to configuration. The default MAM installation comes with a full set of default configuration settings for Sync Service Administrator.
Opening Sync Service Administrator 7. Select SyncCentralIndex > Toggles and check if the value of the key EnableNewThesauriIds has the required value: n - True: Enables the new ID format for thesauri and legal lists. Recommended for new installations of Interplay MAM v5.4/MediaCental v2.5 or higher, or updating an installation with a small amount of data. Setting the value to “true” requires rebuilding the index for all assets. - False: This is the default value; allows the use of MAM v5.
Understanding the Sync Service Administrator Layout Understanding the Sync Service Administrator Layout Once you have logged in, Sync Service Administrator opens and displays the Monitor view. Sync Service Administrator includes the following areas and views: Area Function Title bar The Show mapping info link provides information on attribute mapping between Interplay MAM and Central Index.
Understanding the Sync Service Administrator Layout The Monitor View In most cases, your first experience with Sync Service Administrator will be in Monitor view, where the basic Sync Service Administrator features are close at hand. The Monitor view is arranged in six sections that provide information and controls to set up basic Sync Service Administrator functions. The following illustration and table describe the layout of the Monitor view.
Understanding the Sync Service Administrator Layout Item Function 3 Details Displays detailed information about the entry selected in the Recent events list. 4 Water mark indicator Indicates the load of the synchronization queue as a traffic light. Sync Index Administrator checks the number of high-priority entries in the DM_Sync table and shows a traffic light depending on water mark settings. You can update the water mark indicator manually by clicking the Refresh button.
Understanding the Sync Service Administrator Layout Item 1 Refresh controls Function Sync Service Administrator updates the display of recently synchronized objects every five seconds if the Auto refresh check box is checked. If you uncheck the Auto refresh check box you can update the display manually by clicking the Refresh button. 2 Synchronized objects list Displays a configurable number of recently synchronized objects (default is 3,000).
Understanding the Sync Service Administrator Layout The MAM / CI Compare View The MAM / CI Compare view provides statistics on the number of objects in MAM and Central Index per object class, and differences between them. Information is provided for object (asset) and EDL (sequence) classes. The information is displayed in an overview table, as shown in the following illustration. It shows the number of objects for each object and EDL class in MAM and Central Index.
Quitting Sync Service Administrator The Errors View The Errors view provides statistics on synchronization errors, and controls to reset or purge erroneous synchronization entries in Central Index.The following illustration and table describe the layout of the Errors view. Function 1 Refreshes the entries in the Object class table. 2 Displays the number of synchronization errors for each affected object class.
A Upgrading Media | Index The procedures for upgrading your MediaCentral system configured with Media Index depend on which version of MediaCentral Platform Services (MCS) you are upgrading from. For example, if you upgrade from v2.0 to v2.3, you must reset the Media Index indexes before the upgrade begins. If you upgrade from v2.2 to v2.3, you only need to migrate the Media Index schema data. Upgrading Media Index from v2.
Upgrading Media Index from v2.0.x or v2.1.x To reset Media Index to prepare for an upgrade: 1. If you are indexing an Interplay Production database, you must first determine the name of the PEBCo instance running on the MediaCentral server. Enter the following command on either a single MediaCentral server or any node in a cluster configuration: /opt/avid/bin/pam-agent-ctrl list The server returns the name of your PEBCo instance. 2.
Upgrading Media Index from v2.0.x or v2.1.x Editing the Elasticsearch-tribe Configuration File In multi-zone configurations, you also must edit the Elasticsearch-tribe configuration file in all zones to remove information about the zone or zones that you are upgrading. For example, in a three-zone configuration where you start by upgrading Zone_1, you must remove Zone_1’s information from the elasticsearch-tribe configuration file on Zone_2 and Zone_3.
Upgrading Media Index from v2.0.x or v2.1.x 6. In one of the remaining zones, use the Linux text editor, vi, to edit the elasticsearch.yml configuration file: vi /etc/elasticsearch-tribe/elasticsearch.yml 7. Delete the binding information for the zone that is being upgraded. The example below highlights the information for Zone_1 which is the zone being upgraded: [root@wavd-news ~]# vi /etc/elasticsearch-tribe/elasticsearch.yml discovery.zen.ping.multicast.enabled: false tribe.local.cluster.
Media | Index Backup Locations 2. Check to make sure the system_data index and index information are not listed. When Media Index is reset for your upgrade process, you can install the latest version of MediaCentral Platform Services. For more information, see one of the following topics: Once the MediaCentral upgrade is complete, you must reconfigure your Media Index configuration.
B Working with Media Index This section covers the additional commands and utilities that are used with systems configured for Media Index.
Displaying Information about Initialized Systems Displaying Information about Initialized Systems You can use the info command to display the following data for all available systems within all configured Elasticsearch clusters: • System id • System type • System display name • Number of assets To display system information, do the following: t Type the following command: /opt/avid/bin/avid-acs-elastic-index-setup -m info The following is an example of Media Index configured for an Avid Interpla
Checking the Current Index Status Checking the Current Index Status The Indexes pane in the MediaCentral UX System Settings can be used to view the status of the indexes in your configuration. n The Indexes pane displays only the indexes in the local zone. To display indexes of other zones, connect directly to the MediaCentral UX of that zone. To check the current status of your indexes: 1. Sign in to MediaCentral UX as an administrator. 2. Select System Settings from the Layout selector.
Using the Media | Index Restart Command Line Tool Using the Media | Index Restart Command Line Tool When Media Index is configured on a MediaCentral system, multiple new services are enabled. In the case of a cluster, many of these resources are added as resources to the Corosync / Pacemaker cluster. In an effort to simplify working with Media Index, Avid includes a utility that allows you to start, stop, or restart many of the services (or resources) in a single command.
Using the Media | Index Restart Command Line Tool • n [command] — Specify one of the following commands to interact with the services: - start — start services - stop — stop services - restart — restart services - status — get the status To get command line tool help information, run the command tool without parameters.
Restarting the Search and PEBCo Services Restarting the Search and PEBCo Services If you find that the indexed search has stopped working in your MediaCentral environment, you can restart the search service on any node that has a problem and then resync your index. To check the status of the search service: 1. Sign in to MediaCentral UX as an administrator. 2. Select System Settings from the Layout selector. The System Settings layout opens. 3. In the Settings pane select Media Index > Cluster State.
Re-Sharding a Media Index Database 7. Click the Manage Status tab. 8. In the Manage section, click Resync to update the whole database content. Resyncing the database might take a long time to complete and invalidates the existing content of your index. During this process, the search only returns assets that are already processed within this run. Re-Sharding a Media Index Database Elasticsearch indexes are split into “shards” and “replicas”.
Taking Nodes Offline in a MediaCentral Cluster Alternatively, the /opt/avid/bin/avid-acs-elastic-index-setup -m info command can also be used to determine the System ID. - is the number of total shards that you want the index to be split into. The following command is provided as an example: /opt/avid/bin/media-index-reshard --systemID=BC2F27F5-DBC5-4611-869A-F7A43690EE57 --shards=5 For more information on shard and replica configurations, see “Shard Count and Usage” on page 19.
Removing Zones from a Multi-Zone Configuration Removing Zones from a Multi-Zone Configuration If you want to remove a zone from your MediaCentral multi-zone configuration, you must remove the bindings for that zone in the Elasticsearch tribe configuration file on all other zones. This process only removes the affected zone from the Media Index multi-zone configuration, it does not disable Media Index on any zone.
Disabling Media Index 7. Delete the binding information for the zone that is being removed from the configuration. The example below highlights the information to be removed from zone name “Master_Zone”: [root@wavd-doc01 ~]# cat /etc/elasticsearch-tribe/elasticsearch.yml discovery.zen.ping.multicast.enabled: false tribe.local.cluster.name: Master_Zone tribe.local.transport.tcp.port: 9312 tribe.local.discovery.zen.ping.unicast.hosts: - "wavd-doc01:9300" http.port: 9201 transport.tcp.port: 9305 node.
Disabling Media Index To completely disable Media Index: 1. Delete all Indexes from MCUX System Settings > Media Index > Indexes. 2. Click Reset in MCUX System Settings > Media Index > Settings. 3. If the Media Index is configured with Interplay Production, complete the following in the Interplay Administrator: a. Under Server > Server Settings, disable “Update Tracking” b. Under Site Settings > Production Engine Bus Connector, Stop the PEBco service and Unassign the PEBCo service 4.
C Media | Index Custom Configuration The Configuring Media | Index for a Single Server and Configuring Media | Index for a Cluster chapters in this document detail the most common Media Index deployments. In some instances, you might want to customize or need to modify the installation.
Configuring Media Index Fields The search criteria at the top of the pane show Media Index and TBMD (time-based metadata) values that are common to all Indexed searches. The sections below TBMD list the databases that have been indexed by the system. The image above shows a single indexed Interplay Production database. The System ID of that index is shown in the header of the section. By individually listing each system, administrators can customize the available search values on a per-database level.
Modifying the Media | Index System Settings Modifying the Media | Index System Settings The Media Index section of the MediaCentral UX System Settings allows you to monitor and modify some settings used by Media Index. You use some of these settings to configure Media Index and others to monitor the systems and the indexes used by Media Index.
Modifying the Media | Index System Settings Area or Component Description System Name (optional) The system name is set by the indexing agent. For Interplay Production, the system name is set by the Production Engine Bus Connector (PEBCo) and uses the Interplay Production Engine’s host name or virtual cluster host name. Other indexing agents — for example, the iNEWS sync agent — might not assign a system name by default.
Modifying the Media | Index System Settings Modifying the Settings Pane You can use the Settings pane in the System Settings to change the cluster name and to delete all systems and restore Media Index into the initial state. To change the cluster name: 1. Sign in to MediaCentral UX as an administrator. 2. Select System Settings from the Layout selector. The System Settings layout opens. 3. In the Settings pane select Media Index > Settings. The Media Index Settings display in the Details pane. 4.
Modifying the Media | Index System Settings To delete all systems and restore Media Index into the initial state: 1. Sign in to MediaCentral UX as an administrator. 2. Select System Settings from the Layout selector. The System Settings layout opens. 3. In the Settings pane select Media Index > Settings. The Media Index Settings display in the Details pane. 4. In the Reset Media | Index section, click Reset.
Configuring Full Text Search for MAM Thesaurus Configuring Full Text Search for MAM Thesaurus An Interplay MAM thesaurus is a list of values defined by the system administrator. The values are displayed in the MediaCentral UX Metadata pane as a pull-down menu which can be selected by the user to quickly populate a metadata field. These values are stored in the MAM database as integers which makes it very difficult for users to find the values during a plain text (full text) search.
Configuring Full Text Search for MAM Thesaurus The following is presented as an example of the above command: scp /etc/user/temp/custom.file wavd-mcs02:/etc/user/temp/ If prompted, enter the password for the root user of the destination server. 5.
Adjusting Settings Through the ACS Monitor Adjusting Settings Through the ACS Monitor Many of the services used to create Media Index are governed by configuration files that contain parameters which can be adjusted to enhance or limit the operation of the system. While these files can be edited manually through Linux, Avid provides a user interface called the Avid ACS Monitor which can be used to adjust many of the same settings more easily.
Adjusting Settings Through the ACS Monitor Accessing the Avid ACS Monitor Tool The ACS Monitor is not enabled by default and must be manually started to interact with the tool and adjust system settings. To adjust the Media Index configuration through the ACS Monitor: 1. Log in to the MCS server as the Linux ‘root’ user. If you have a clustered configuration, log into the master node. 2. Start the avid-acs-monitor service: service avid-acs-monitor start 3.
Adjusting Settings Through the ACS Monitor If you click on any of the examples in the “setConfigurationByField” section, you will notice that they all share the same basic structure. Line 7 is the only line that is different between them. Rather than entering all the text manually, you can alter the contents of this example to update the maximum number of markers. 5.
Adjusting Settings Through the ACS Monitor 3. Locate the setConfigurationByField section and click the link for Search: highlighting. The following text appears in the right pane: 4. Adjust the acs_disable_higlight value to true, and click the Query button in the right pane to save the change. When adjusting the value, make sure that you maintain a space between the colon and the new value and that you end the line with a comma. 5.
Adjusting Settings Through the ACS Monitor If you click on any of the examples in the “setConfigurationByField” section, you will notice that they all share the same basic structure. Line 7 is the only line that is different between them. Rather than entering all the text manually, you can alter the contents of this example to alter the Media Index scalability options. 5.
Adjusting Settings Through the ACS Monitor 4. Locate the setBoostConfiguration item in the list of service options and click the link for setBoostConfiguration. The following text appears in the right pane: The paramSet lists three different categories or “entities”: asset (base), common, and tbmd (time based metadata). Each category is associated with various fields and boost values.
Adjusting Settings Through the ACS Monitor tbmd n category created creator duration id modified modifier startTC text thumbnailURL type Avid supports making changes to the asset, common, and tbmd categories. Creating additional categories is not supported. 5. Add or adjust the boost values as desired. In the following example, the “created” parameter has been added and assigned a boost values of 3; giving it a higher value than “category” and “creator”.
Adjusting Settings Through the ACS Monitor Disabling Search Suggestions When entering text in indexed search fields, search suggestions are displayed below the text box to assist users in finding the data they are looking for faster. While this is a useful feature, Media Index configurations that use large indexes might see a slow down in search performance when search suggestions are enabled. If desired, this feature can be disabled through the ACS Monitor tool. To disable the search suggestion data: 1.
Adjusting Settings Through the ACS Monitor To configure the search criteria behavior: 1. Follow the instructions for “Accessing the Avid ACS Monitor Tool” on page 98. 2. Enter “avid.acs.media.index.configuration” (without the quotes) in the search field at the top of the ACS Monitor page to find the configuration service. 3. Expand the configuration service options by clicking the Service Operations List icon in the upper-left corner of the configuration service field. 4.
Adjusting Settings Through the ACS Monitor Adjusting Full Text Search Parameters When initiating an Indexed search, the following default rules are used to return results for all text fields: • Non-text characters such as hyphens and underscores are often dropped, dividing the search into sub-components. For example, if a user searches for an asset that is named “Running_MY_test”, Media Index divides the search into three subcomponents — “Running”, “MY”, and “test”.
Adjusting Settings Through the ACS Monitor The following text appears in the right pane: The “Request to: Local” pane on the right, line 7 lists the two default fields for the substring_search_ids parameter. 5. Add any desired fields to the list of substring_search_ids. The following is a list of fields that can be added: - systemType - systemID - type - common.description._description - common.globalAssetID - common.creator._creator - common.modifier._modifier - common.categories.
Adjusting Settings Through the ACS Monitor 7. Verify your change by selecting the getConfiguration item in the list of service options and click the Query button on the right to poll the service. The updated value appears under the "resultSet" list. 8.
Adjusting Settings Through the ACS Monitor When entering text, make sure to include the two leading spaces and the quotes as shown in the example above. Configuring this value as false disables the simple search. 6. Click the Query button in the right pane to save the changes. If the query is accepted, a Response value (in milliseconds) will be returned. If you receive any errors, verify that the information you entered does not contain any incorrect characters. 7.
Adjusting Settings Through the ACS Monitor If you click on any of the examples in the “setConfigurationByField” section, you will notice that they all share the same basic structure. Line 7 is the only line that is different between them. Rather than entering all the text manually, you can alter the contents of this example to update the search criteria. 5.
Adjusting Settings Through the ACS Monitor n The following process eliminates both the Interplay Production and iNEWS asset types from the menu. If you are using Media Index with Interplay MAM and either of these other databases, do not complete this process. This configuration option is applicable to MAM-only environments. To remove Interplay Production and iNEWS Asset Types from the interface: 1.
Adjusting Settings Through the ACS Monitor 7. Click the Query button in the right pane to save the changes. If the query is accepted, a Response value (in milliseconds) will be returned. If you receive any errors, verify that the information you entered does not contain any incorrect characters. 8. Select the deleteItemById link again. 9.
Setting Elasticsearch Memory Usage Setting Elasticsearch Memory Usage The default installation of Media | Index configures the Elasticsearch heap size to a value of 4GB. While this value might be appropriate for many installations, Avid has completed additional testing to determine alternate values which can help to optimize system performance.
Setting Elasticsearch Memory Usage Number Of Number Heap Size Assets Of Nodes Per Node 10,000,000 n 1 57.22 Recommendation In this case the calculated heap size is 57GB which far exceeds the recommended maximum of 32GB. Because this is a single-server configuration, the formula has revealed that the configuration is severely under-provisioned. Avid recommends expanding the configuration to a two-node cluster (at minimum).
Configuring the PEBCo for Custom Configurations Configuring the PEBCo for Custom Configurations For standard installations, you do not need to modify the properties file for the PEBCo. For custom configurations, you can edit the PAM agent services properties file. To configure the PEBCo: 1. Navigate to the directory containing the PAM agent services properties file: /opt/avid/etc/pam-agent-service 2. Using the Linux text editor, vi, open the pamagent-service.
Changing the Default Search Type Changing the Default Search Type By default, the Search bar and the Search pane in MediaCentral UX use the federated search. You can specify the default search type in MediaCentral UX by editing a properties file in your MediaCentral configuration. If you work with a cluster configuration, you must edit the application properties file on the master and slave nodes. Editing the file on load-balancing nodes is not required.
Changing Replica Counts Changing Replica Counts You can change the number of replicas for all indexes, for indexes at the specified alias, or for a specific index. You can view index names and any index alias names by using the Elasticsearch head plug-in Web page, which you can view by opening a browser and navigating to http://:9200/_plugin/head.
Changing the Index Refresh Interval Changing the Index Refresh Interval When you use Media Index to index an asset, you must wait for the next refresh for that asset to appear in a search. Refreshing the index requires some system resources, so by default the index refreshes after regular intervals instead of after each indexing operation. The interval is defined by the index.refresh_interval setting, which you can modify in either the Elasticsearch configuration file (/etc/elasticsearch/elasticsearch.
D Updating Media | Index Property Schemas Each Media Index index has a schema specific to its source system. You configure the schema in the Production Engine Bus Connector (PEBCo — for more information, see “Configuring and Using the PEBCo Service” on page 41). If the list of fields in the PEBCo Canonical Data tab that you configure for indexing changes, then you must migrate the index to this new schema before the system can index new assets.
Property Changes in Production and Schema Updates Changing Properties Without Immediately Updating All Assets This approach adds properties to the index, but only new or updated assets carry the new properties. It performs a re-index so there is no interruption for the user. Index updates are blocked for all indexes until the re-indexing completes. After pressing Start, you might receive a timeout error after 10 seconds. This is expected, as the re-index operation needs more than 10 seconds to complete.
Property Changes in Production and Schema Updates Changing Properties and Restarting With an Empty Index You can use this method if you can accept that search remains incomplete until the initial indexing finishes. It saves the time needed for re-indexing, but depending on the number of assets in the database it can take quite some time for the initial indexing to finish.
Monitor the Re-index Process Monitor the Re-index Process You can monitor the re-index process in the Cluster State pane of the Media Index System Settings. During the re-index, the process creates a new, temporary index which gradually fills up with copies of the existing assets. Once the re-index is complete, Media Index deletes the original index and makes the new index active. By monitoring the number of documents in the existing and the new index, you can monitor the progress of the re-index process.
Monitor the Re-index Process When re-index starts, there is one additional column with the same basic name as the original index and a different ID at the end. The new index does not have the color bars that mark it as an active index. While the re-index progresses, the docs count slowly increases until it reaches the same number as the original index. Once it reaches the same number, the re-index is complete and the original index disappears with the new index getting the color bars of the original index.
Monitor the Re-index Process Use the Refresh button at the top of the pane to refresh the information and update the number of assets currently indexed. The duration of the re-indexing process depends on a number of variables, such as the number of markers in the assets.
Avid Technical Support (USA) Product Information 75 Network Drive Burlington, MA 01803-2756 USA Visit the Online Support Center at www.avid.com/support For company and product information, visit us on the web at www.avid.