TIBCO Mashery®

(1)

Installation and Configuration Guide for Docker

Software Release 4.3 March 2018

Two-Second Advantage^®

(2)

Important Information

SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE.

USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE) OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE “LICENSE” FILE(S) OF THE

SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME.

This document contains confidential information that is subject to U.S. and international copyright laws and treaties. No part of this document may be reproduced in any form without the written

authorization of TIBCO Software Inc.

TIBCO and TIBCO Mashery are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries.

All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.

THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT ALL OPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED AT THE SAME TIME. SEE THE README FILE FOR THE AVAILABILITY OF THIS SOFTWARE VERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM.

THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF

MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT.

THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME.

THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR INDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE,

INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES.

TIBCO Software Inc. Confidential Information

(3)

TIBCO Documentation and Support Services

Documentation for this and other TIBCO products is available on the TIBCO Documentation site. This site is updated more frequently than any documentation that might be included with the product. To ensure that you are accessing the latest available help topics, visit:

https://docs.tibco.com

Product-Specific Documentation

The following document for this product can be found on the TIBCO Documentation site:

● TIBCO Mashery^® Local Installation and Configuration Guide

For information on TIBCO Cloud Integration with Mashery, refer to Integrating with Mashery.

TIBCO Mashery Professional customers will not have access to all of the features documented here. The following is a list of capabilities that are not available and as such will not be visible within the API Control Center for these customers:

● Distributed API Management (managing Organizations)

● Enriched Call Log Export

● HTTPS Client Profiles

● Mashery Local (Deploy)

● Event Triggers

Additionally, TIBCO Mashery Professional customers will not have access to the Mashery V2 API and as such will be able to use only the OAuth2 Accelerator feature.

Additionally, TIBCO Mashery Professional includes 8M QPM (Queries per month) and all traffic purchased is limited to a max of 100 QPS (Queries per second). TIBCO Mashery Professional customers can create a max of 25 APIs and 25 packages.

How to Contact TIBCO Support

For comments or problems with this manual or the software it addresses, contact TIBCO Support:

● For an overview of TIBCO Support, and information about getting started with TIBCO Support, visit this site:

http://www.tibco.com/services/support

● If you already have a valid maintenance or support contract, visit this site:

https://support.tibco.com

Entry to this site requires a user name and password. If you do not have a user name, you can request one.

How to Join TIBCO Community

TIBCO Community is an online destination for TIBCO customers, partners, and resident experts. It is a place to share and access the collective experience of the TIBCO community. TIBCO Community offers forums, blogs, and access to a variety of resources. To register, go to the following web address:

https://community.tibco.com

(8)

Introduction

This guide provides an overview of the installation, requirements and configuration for Mashery^® Local for Docker.

Mashery Local for Docker is a set of Docker images for running Mashery Local. These images can be customized for custom configurations. Mashery Local for Docker allows customers to perform hybrid traffic management on premise to run the API traffic inside data-centers. Mashery Local securely interacts with the Mashery Cloud hosted Developer Portal, Administration Dashboard and API Reporting and Analytics modules.

Mashery Local includes commercial support for Project Mashling, an open-source, event-driven microgateway. You can publish an endpoint exposed by Mashling to Mashery for access to broader API management functions. For more information about Mashling, please see https://www.mashling.io/

home or https://community.tibco.com/products/project-mashling.

Assumptions

This guide assumes that you are using Docker version 1.12 or later. If you have an internal cloud, established best practices will be applied (for example disk alignment). If you are using different servers and clients, the underlying concepts implied by the installation and configuration steps still apply.

Conventions

This guide uses the following conventions:

● Keys you press simultaneously appear with a plus (+) sign between them (for example, Ctrl+P means press the Ctrl key first, and while holding it down, press the P key).

● Field, list, folder, window, and dialog box names have initial caps (for example, City, State).

● Tab names are bold and have initial caps (for example, People tab).

● Names of buttons and keys that you press on your keyboard are in bold and have initial caps (for example, Cancel, OK, Enter, Y).

(9)

Deployment Topology

The following diagram depicts a typical deployment topology for Mashery Local.

Cluster Requirements

Mashery strongly recommends the following configuration for a Mashery Local Cluster (Master + Slaves):

● 1 Master

● 2 Slaves (at least)

Overview of Installation and Configuration Process

This section provides a roadmap of the installation process for Mashery Local.

Procedure

1. Download the Mashery Local Docker artifact from TIBCO^® eDelivery and create the Mashery Local Docker Image set as described in Installing and Configuring Mashery Local for Docker.

2. Configure a Mashery Local Master as described in Configuring a Mashery Local Master.

3. Configure slaves to the Mashery Local Master as described in Configuring Slaves to the Local Master. It is best practice to set up production with no less than 2 slaves per master.

4. Configure the load balancer as described in Configuring the Load Balancer.

5. Perform advanced configuration such as enabling notifications, as described in Advanced Configuration.

(10)

Mashery Local Failover Strategy Recommendations

Many of TIBCO customers rely on TIBCO Mashery Local to manage and distribute their revenue- generating and business-critical API traffic. The nature of the usage warrants that Mashery Local is always on without any downtime. Proper planning for redundancy and failover is recommended when high availability is expected of a mission-critical system.

The current Mashery Local architecture relies on four entities:

1. Mashery Cloud - This is where you make all your service configuration changes, through the Mashery Control Center API dashboard.

2. Mashery On-Prem Manager (MoM) - This is your Mashery Local's gateway to the Mashery Cloud.

You must have received a secured key and secret, which provides each of your clusters its unique identity. You should always have a separate MoM key for each cluster, even if they are connecting to the same Mashery area.

3. A Mashery Master node synching with Cloud for API keys, OAuth Tokens, Service Configuration, User details, etc.. The time taken for synchronization of your configuration and token data is a function of the amount of data. Each customer implementation is unique and each network topology is different, so there isn't any formula that can correctly project the amount of time taken.

4. Slaves within the cluster that replicate the API Key, OAuth Token and Service Configuration data from Master. This happens within the cluster and in your environment, so the replication speed is slightly faster than Cloud Sync, but yet depends on various other environmental and data

components.

TIBCO Recommendations to Achieve High Availability for TIBCO Mashery Local Deployment Failover and redundancy can be achieved at many levels and should be considered while building your high availability strategy. Customers should also maintain updated runbooks so that their environment specific nuances are captured for their internal teams for faster deployment and recovery of systems.

Failover systems should be tested and monitored in periodic intervals to ensure that they are in sync with production. Failure to do so will result in loss of traffic at the time of need. The following are some recommendations for redundancy – redundancy within a cluster and cross datacenter redundancy.

Redundancy within a Cluster:

Each Cluster has two type of Nodes – a Master Node that syncs the cluster with Cloud and many Slave Nodes that replicate from the Master. Though both type of Nodes are capable of serving traffic, TIBCO's recommendation is to keep the Master out of rotation in high traffic, high OAuth type implementations.

Keep one Slave extra than what is needed for optimum capacity to achieve within cluster redundancy.

If Master runs into problem due to disk, VM, or Network issues, then you can easily promote the spare Slaves to a Master and point the rest of the Slaves to the new Master. This can be achieved in minutes and will have a very low impact on the traffic. Except for newly synched up OAuth tokens, Slaves should be able to successfully service traffic during the promotion and pointing to the new Master. Fix the old Master Node and you can bring it back as a Slave into the cluster after re-imaging the VM.

If a Slave runs into problems, then take that Slave out of rotation from load balancer level. That way, you will not experience any traffic loss. Fix the issue and bring the slave back into rotation.

Cross Datacenter Redundancy:

If there is an issue with the datacenter, or if the whole cluster is having problems, having an Active- Active or Active-Passive cluster strategy is very beneficial in this scenario:

● Active-Active Strategy: Both Clusters with their unique Mashery On Prem Manager (MoM) key would connect to the same Mashery area and continue to sync. Nodes in both clusters can be used to serve traffic but both would have enough capacity (Disk Space, Caching configuration, etc.) to

(11)

● Active-Passive Strategy: Both Clusters with their unique Mashery On Prem Manager (MoM) key would connect to the same Mashery area and will sync. Nodes from only one cluster would serve traffic. If needed traffic can be routed to the other non-traffic serving cluster without any blip in service. Both clusters should be identical in their configuration and capacity (Disk Space, Caching configuration, etc.) to serve total traffic.

Please note that TIBCO Mashery's license policy in cluster-based, so please discuss this with your Sales or Mashery Customer Success team. Having cluster redundancy is absolutely essential to avoid any traffic loss. Master sync and Slave replication takes time when done from scratch, and without cluster failover, you will encounter traffic loss.

(12)

Cluster Management in Mashery Local

The following sections describe how to set up and manage a Mashery Local Cluster:

● Setting up a new Mashery Local Cluster

● Adding a Slave to a Running Mashery Local Cluster

● Changing the Master in a Mashery Local Cluster

Setting up a New Mashery Local Cluster

The following section describes how to set up a new Mashery Local Cluster.

Procedure

Prepare mysqldump from Existing Mashery Local Cluster. (Optional)

1. For Mashery Local prior to 4.2.0, root can generate mysqldump in the Master Instance under the / mnt/ folder. Because proxy service needs to be stopped when mysqldump is being generated, this Mashery Local cluster cannot handle traffic.

If you are using Docker-Machine, make sure you are talking to the right one. Execute the command

docker-machine ls to find which one is currently active. It is also recommended to always redo the command:

eval "$(docker-machine env <docker machine name>)"

a) docker exec -it ml-tm nohup service javaproxy stop b) 1. docker exec -it ml-db /bin/bash

2. mysqldump -u masherybackup -p'password_for_masherybackup'--opt --master-data --single- transaction onprem > /mnt/onprem.sql

3. md5sum /mnt/onprem.sql 4. Exit the ml-db container

Below step copies the mysqldump from container to host's file system in present working directory. You can specify other location by replacing '.' at the end.

5. docker cp ml-db:/mnt/onprem.sql

c) docker exec -it ml-tm nohup service javaproxy start

For Mashery Local 4.2.0 or later, root can generate mysqldump in the Slave or Master Instance under /mnt/dump/ folder.

Because proxy service needs to be stopped when mysqldump is being generated, this Master/Slave Instance cannot handle traffic.

a) docker exec -it ml-tm nohup service javaproxy stop b) 1. docker exec -it ml-db /bin/bash

2. mysqldump -u masherybackup -p'password_for_masherybackup'--opt --dump-slave -- single-transaction onprem > /mnt/dump/onprem.sql

3. md5sum /mnt/dump/onprem.sql 4. Exit the ml-db container

Below step copies the mysqldump from container to host's file system in present working directory. You can specify other location by replacing '.' at the end

(13)

c) docker exec -it ml-tm nohup service javaproxy start Set up Mashery Local Master Instance

2. Perform the following steps:

a) Create and configure a Mashery Local Master instance as described in the topic "Configuring a Mashery Local Master", in this Guide.

If special tuning is needed on MySQL, for example, expanding buffer pool size in "/etc/my.cnf"

(login into ml-db container first):

innodb_buffer_pool_size = 512M

Mashery Local customers should consult TIBCO Support and request assistance in tuning MySQL. After tuning of MySQL, the MySQL service should be restarted by the Administrator:

service mysqld restart

When disk expansion is needed, the Administrator should follow the instructions in the topic

"Expanding the Disk Space of a Mashery Local Instance" in this Guide.

b) Import mysqldump from a previous Mashery Local Cluster. (Optional)

Importing mysqldump from an existing Mashery Local Cluster can minimize the amount data to synchronize from Cloud, greatly reducing the amount of time required to setup the Mashery Local Instance.

For example, suppose the remote Mashery Local instance is "remote_host".

For Mashery Local prior to 4.2.0, root can copy mysqldump from remote host.

If you are using Docker-Machine, make sure your are talking to the right one. Execute the command docker-machine ls to find which one is currently active. It is also recommended to always redo the command:

1. First transfer the mysqldump from remote host to the current host. scp root@remote_host:

2. Now copy the mysqldump file from host machine to DB container's file system docker cp

<LOCAL DEST DIR>/onprem.sql ml-db:/mnt/

For Mashery Local 4.2.0 or later, root can copy mysqldump from remote host:

1. First transfer the mysqldump from remote host to the current host. scp

root@remote_host:<Absolute path of onprem.sql on remote host> <LOCAL DEST DIR>

Verify the checksum of mysqldump file (Login into ml-db container:

md5sum /mnt/dump/onprem.sql

Stop "proxy" service:

docker exec -it ml-tm nohup service javaproxy stop

Import mysqldump:

mysql -u masheryonprem -p'password_for_masheryonprem' onprem < /mnt/dump/

onprem.sql

After importing is done, restart MySQL:

Start "proxy" service:

docker exec -it ml-tm nohup service javaproxy start

c) Register the Mashery Local Instance as Master.

Follow the instructions in the topic "Configuring a Local Mashery Master" in this guide to register the Mashery Local Instance as Master, to finish the settings for the Master in Cluster Manager.

(14)

When synchronizing the Master for the first time, allow the Master to finish the synchronization. This ensures the Master Instance is set up properly and synchronization with the Cloud is normal.

d) Stop "proxy" Service (Optional).

After "proxy" service is stopped, there will be no activity in MySQL. This enables Mashery Local slaves to replicate faster. To stop "proxy" service, run the following command:

Set up the Mashery Local Slave Instance 3. Perform the following steps:

a) Create and configure a Mashery Local Slave instance as described in the topic "Configuring a Mashery Local Slave", in this Guide.

If special tuning is needed on MySQL, for example, expanding buffer pool size in "/etc/my.cnf":

innodb_buffer_pool_size = 512M

Mashery Local customers should consult TIBCO Support and request assistance in tuning MySQL. After tuning of MySQL, the MySQL service should be restarted:

When disk expansion is needed, the Administrator should follow the instructions in the topic

"Expanding the Disk Space of a Mashery Local Instance" in this Guide.

b) Register the Mashery Local Instance as a Slave.

Follow the instructions in the topic "Configuring a Local Mashery Slave" in this guide to register the Mashery Local Instance as Slave, to finish the settings for the Slave in Cluster Manager.

c) Import mysqldump from a previous Mashery Local Cluster. (Optional)

Importing mysqldump from an existing Mashery Local Cluster can minimize the amount data to synchronize from Cloud, greatly reducing the amount of time required to setup the Mashery Local Instance.

For example, suppose the remote Mashery Local instance is "remote_host".

For Mashery Local prior to 4.2.0, the user can copy mysqldump from remote host.

If you are using Docker-Machine, make sure you are talking to the right one. Execute the command docker-machine ls to find which one is currently active. It is also recommended to always redo the command:

For Mashery Local 4.2.0 or later, root can copy mysqldump from remote host:

Verify the checksum of mysqldump file (Login into ml-db container) :

Stop "proxy" service:

(15)

Import mysqldump:

onprem.sql

After importing is done, restart MySQL:

Start "proxy" service:

Ensure MySQL Slave replicates well from MySQL Master:

mysql -u masheryonprem -p'password_for_masheryonprem' onprem show slave status\G

d) Start "proxy" Service in Master after all Slaves are set. (Optional) Run the following command:

Adding a Slave to a Running Mashery Local Cluster

The following section describes how to add a Slave to a running Mashery Local Cluster.

Procedure

1. Register the Mashery Local Instance as new Slave.

Follow the instructions in the topic "Configuring a Local Mashery Slave" in this guide to register the Mashery Local Instance as Slave, to finish the settings for the Slave in Cluster Manager.

2. Exclude existing Slave or Master Instance from taking traffic.

In order to prepare mysqldump, it is recommended that the user exclude an existing Slave Instance from Load Balancer, so that the Master Instance and other Slave Instances can keep taking traffic. In the case of high availability setup, the user can also switch traffic from the primary cluster to a backup cluster, then stop proxy service in the Master Instance of the primary cluster.

3. Stop "proxy" Service in existing Slave or Master Instance.

Use the following command:

4. Initialize New Slave Database.

a) Initialize New Slave Database by importing mysqldump file.

The user should prepare mysqldump in the existing Slave Instance.

1. The user generates mysqldump in existing Slave instance:

mysqldump -u masherybackup -p'password_for_masherybackup' --opt --dump- slave --master-data --single-transaction onprem > /mnt/dump/onprem.sql

2. Or, the user generates mysqldump in existing Master instance:

mysqldump -u masherybackup -p'password_for_masherybackup' --opt --master- data --single-transaction onprem > /mnt/dump/onprem.sql

b) Import mysqldump to new Slave instance.

Copy mysqldump to new Slave instance:

1. First transfer the mysqldump from remote host to the current host.scp

(16)

2. Now copy the mysqldump file from host machine to DB container's file systemdocker cp

<LOCAL DEST DIR>/onprem.sql ml-db:/mnt/dump/

3. md5sum /mnt/dump/onprem.sql Stop MySQL Slave:

mysql -u masheryonprem -p'password_for_masheryonprem' onprem stop slave

Import mysqldump to new Slave Instance:

onprem.sql

c) Initialize New Slave Database by streaming mysqldump.

1. Stop Slave in new Slave Instance:

mysql -u masheryonprem -p'password_for_masheryonprem' onprem stop slave

2. Import mysqldump to new Slave Instance via Stream:

● Stream from existing Slave Instance:

mysqldump -h remote_host_internal_ip -u mashonpremrepl - p'password_for_mashonpremrepl' \

--opt --dump-slave --single-transaction onprem \

| mysql -u masheryonprem -p'password_for_masheryonprem' onprem

● Stream from existing Master Instance:

mysqldump -h remote_host_internal_ip -u mashonpremrepl - p'password_for_mashonpremrepl' \

--opt --master-data --single-transaction onprem \

| mysql -u masheryonprem -p'password_for_masheryonprem' onprem

5. Restart MySQL Service:

6. Start "proxy" Service:

Changing the Master in a Mashery Local Cluster

The following section describes how to change the Master in a Mashery Local Cluster.

Procedure

1. Shut down the old Master in the Mashery Local Cluster.

Follow the steps in the topic Shutting down a Master.

2. Promote one Slave to be the new Master.

Follow the steps in the topic Promoting a Slave to Master.

3. Repoint other Slaves to the new Master.

Follow the steps in the topic Repointing Other Slaves to a New Master.

(17)

Installing and Configuring Mashery Local for Docker

The following sections describe how to install and configure some basic environments complete with a master, one or more slaves, and load balancing.

Mashery Local for Docker includes a script that will download and install third-party software from third-party websites, including but not necessarily limited to CentOS and EPEL repositories located here:

● https://hub.docker.com/_/centos/

● http://vault.centos.org/

● https://dl.fedoraproject.org/pub/epel/

Such third-party software is subject to third-party software licenses that may be available on such third- party websites. For more information on CentOS repositories and EPEL, see:

● https://wiki.centos.org/AdditionalResources/Repositories

● https://fedoraproject.org/wiki/EPEL

Required Docker Images

Three images are needed to install Mashery Local for Docker:

1. On-premise database: ml-db 2. Memcache: ml-mem

3. TIBCO Mashery Local Core - Traffic Manager plus Cluster Manager UI: ml-core

Supported Cloud Platform Configurations

Mashery Local 4.3.0 is deployable in the following cloud platform configurations:

● Docker on AWS

● Docker on Azure

● Kubernetes on AWS

● Kubernetes on Azure

● Kubernetes on GCP

● OpenShift on Azure

Installing Mashery Local for Docker

To install Mashery Local for Docker:

Procedure

1. Install Docker Engine, Docker Machine (optional), and docker-compose (optional, and not needed if on Kubernetes) on your operating system.

Refer to the Docker documentation for the operating system of your choice:

● https://docs.docker.com/engine/

● https://docs.docker.com/machine/

● https://docs.docker.com/compose/

(18)

For Mac OS installations, it's recommended to install Docker Toolbox so that multiple docker hosts can be run on the same box.

2. TIBCO Mashery Local for Docker is available as a TIB_mash-local**.tar.gz file. Download this file from TIBCO eDelivery and extract the file contents.

3. Create the TIBCO Mashery Local Docker Image set:

a) Drop in custom configurations:

- Modify examples/set-user-variables.sh and drop it in the resource/addons directory

- (Optional) To use a custom https server PEM file for ML Cluster Manager, drop the PEM file to the resource/addons/certs directory and name it server.pem.

If planning to run on Kubernetes, additional customization may be required. Please see the section Installing and Running Mashery Local for Docker with Kubernetes.

b) Navigate to the root folder of the extracted contents and run the following command to build the Mashery Local image set (comprising three images): ./build-docker.sh 2>&1 |tee /tmp/

build-docker.log

This will increment the image tag revision number. You can use the command "docker images" to check it out. You will need to modify the docker-compose.yml file to use the new tag if you build more than once in the same directory. However, if you would like to keep the same revision number, you will need to remove the file

BUILD_NUMBER.txt in the current directory before starting the next build.

c) Verify three images are created: ml-db.tar.gz, ml-mem.tar.gz, ml-core.tar.gz. (The image sizes are about 400MB, 120MB, and 850MB, respectively.)

If the size of any image is significantly less than the numbers above, then the image build might have some problems. Check the build-docker.log generated from the previous step. If you see several errors, such as:

Could not retrieve mirrorlist http://mirrorlist.centos.org/?... error was14: PYCURL ERROR 22 - "The requested URL returned error: 503 Service Unavailable"

then you probably have some firewall issues on your network. Switch to a network without firewall restriction to do the build.

4. (If planning to run on Kubernetes, the remaining steps do not apply. Please go to the section Installing and Running Mashery Local for Docker with Kubernetes to continue installation.) Navigate to the examples folder and copy the docker-compose.yml and the three image .gz files to the target Docker host machine.

The docker-compose.yml may need additional edits, depending on what ports need to be exposed or for other customization. For example, to add "extra hosts" if there are any extra host names and IP mapping that need to be added for a container.

Note: The indents and dash in the docker-compose yml file are important.

Run the following commands:

● docker load -i <each of the three image files, one by one>

● docker-compose up -d

5. Verify that four Docker containers are up:

docker ps to make sure the four containers are running.

6. Repeat Steps 4-5 for each Docker host that will run a Mashery Local instance.

7. Go to the instance in a browser:

https://<docker host-IP>:5480.

8. Complete Master registration to TIBCO MOM (Mashery On-Prem Manager) or complete Slave

(19)

Additional Installation Tips

● Installation Steps with Docker Toolbox

● Working with Amazon EC2

Installing with Docker Toolbox

Docker Toolbox is a tool that lets you manage Docker engines on multiple virtual instances, and is used with Docker Machine. If you need to setup slaves for the cluster on different virtual instances, images built in the previous set of instructions (Step 3 of Installating Mashery Local for Docker) can be reused below.

1. Install Docker Toolbox from https://www.docker.com/products/docker-toolbox.

2. Use docker-machine create command to create Docker engines on virtual instances.

Drivers are available for various cloud provider platforms. Refer to https://

docs.docker.com/machine/ for the latest information. Also refer to individual cloud provider documentation for more details on authentication details and other parameters you can use to customize your Docker Machine.

Some example commands are below:

a. To create a Docker Machine on a VirtualBox setup on your machine (prerequisite: VirtualBox 5+

ideal):

docker-machine create --driver virtualbox <docker machine name>

b. To create a Docker Machine on a VMware Fusion setup on your machine:

docker-machine create --driver vmwarefusion <docker machine name>

c. To create a Docker Machine on AWS (prerequisite: AWS signup, create an IAM administrator user and a key pair: AWS access key, AWS secret key):

docker-machine create --driver amazonec2 --amazonec2-access-key <your aws access key> --amazonec2-secret-key <your aws secret key> <name for your new AWS instance>

d. To create a Docker Machine on Microsoft Azure (prerequisite: Microsoft Azure signup):

docker-machine create --driver azure --azure-subscription-id <your subscription id> <name for your new azure instance>

e. To create a Docker Machine on Google Cloud (prerequisite: Google Cloud signup, recommend installing and configuring gcloud tools locally to manage authentication. Refer to GCE

documentation.):

docker-machine create --driver google --google-project <google project id> - google-zone "us-west1-a" <name for your new google instance>

3. List all your available machines and make sure the one you just created shows up:

docker-machine ls

4. Connect your shell to a machine:

eval $(docker-machine env <docker machine name>) docker-machine ls

(confirm the machine you are connecting to has an * to it to show that it's active) 5. You can use the three images you created via running the build-docker.sh script above:

a. Copy or move the images to the Amazon instance:

b. Run load command to load the images to the docker host.

(20)

c. Run docker compose up -d.

Working with Amazon EC2 Instances

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud.

Procedure

1. Install Docker Engine: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker- basics.html

2. Install Docker Compose: https://docs.docker.com/compose/install

3. Install Docker Machine: https://docs.docker.com/machine/install-machine

4. TIBCO Mashery Local for Docker is available as a TIB_mash-local**.tar.gz file. Download this file from TIBCO eDelivery and extract the file contents.

5. Navigate to the root folder of the extracted contents and run the following command to build the Mashery Local image set (comprising three images):

a) ./build-docker.sh

This will increment the image tag revision number. You can use the command "docker images" to check it out. You will need to modify the docker-compose.yml file to use the new tag if you build more than once in the same directory.

b) Verify three images are created: ml_db.tar.gz, ml-mem.tar.gz, ml_core.tar.gz.

6. Navigate to the examples folder and copy the docker-compose.yml and the three image .gz files previously built to a target directory.

The docker-compose.yml may need additional edits, depending on what ports need to be exposed or for other customization. For example, to add "extra hosts" if there are any extra host names and IP mapping that need to be added for a container.

Note: The indents and dash in the docker-compose.yml file are important.

Run the following command: docker load -i <each of the three image files, one by one>

At this point, this AWS instance can be saved as an AM1 that can be re-used for any Mashery Local instance.

7. Run the following command: docker-compose up -d

Verify that four Docker containers are up:

docker ps to make sure the four containers are running.

8. Go to the instance in a browser:

https://<docker host-IP>:5480.

9. Complete Master registration to TIBCO MOM (Mashery On-Prem Manager) or complete Slave registration to Master.

If you are using docker-machine to create the docker container, then use the following command:

docker-machine create --driver amazonec2 --amazonec2-access-key <Your AWS Key> --amazonec2-secret-key <Your AWS Key Secret> --amazonec2-region

You will need to open the following ports under the security group for registering Slave machine with Master: 22, 443, 2200, 2376 (TCP), 3306 , 5480, 11212.

Setup Examples

(21)

Example Cloud Deployments with CLI

The following examples demonstrate Mashery Local for Docker deployment to cloud environments (Azure and AWS for now) using Command Line Interface (CLI). The advantage with CLI is that the deployment process can be done repeatedly with scripts and an environment can be easily replicated reliably.

These examples are for illustrating the concepts, they are not production-grade deployments. The scripts used in these examples are in the examples directory.

In each example, two Mashery Local instances are created - one for Mashery Local master and one for Mashery Local slave. Each Mashery Local instance is deployed to a cloud virtual machine (VM) instance (each VM instance corresponding to a docker host). Then, both the master and slave are connected to Mashery On-Prem Manager and are ready to handle traffic.

Cloud Deployment Prerequisites

● Docker, docker-machine, and docker-compose installed locally (for example, on a Mac).

● AWS or Azure accounts have been set up and CLI installed.

● Verify the command ^aws or ^az is on your path and you can do some simple CLI commands, for example:

— az group list

— aws ec2 describe-vpcs

Deploying Mashery Local to Cloud Environment

1. Follow the instructions to build Mashery Local for Docker images locally with docker-machine.

Verify the images are correct (no errors during the build and the images can be seen with the command docker images.

2. Copy all three image gz files, the docker-compose.yml file, and init-ml.sh files inside the extracted folder /examples/aws/docker for deployment into AWS, or inside the folder /examples/azure/

docker for deployment in Azure.

export MOM_KEY="<MOM key>" export MOM_SECRET="<MOM secret>"

chmod a+x *.sh

3. Set the following environmental variables for Azure or AWS:

For Azure:

export AZURE_USER="<azure user>"

export AZURE_PWD="<azure password>"

export AZURE_SUBSCRIPTION_ID="<your azure subscription id>"

export AZURE_IMAGE="<azure image for the location>"

(Default value "canonical:UbuntuServer:16.04.0-LTS:latest" for location "westus" if not specified.) export ML_LOC_NAME="<azure location for ML>"

(Default value "westus" if not specified)

export ML_RESOURCE_GROUP="<azure resource group name for ML>"

(Default value "mlResourceGroup" if not specified) export ML_DNS_NAME="<public IP DNS name>"

(22)

(Default value "testml" if not specified) For AWS:

export AWS_ACCESS_KEY_ID="<AWS_ACCESS_KEY_ID>"

export AWS_SECRET_ACCESS_KEY="<AWS_SECRET_ACCESS_KEY>"

export AWS_DEFAULT_REGION="<default AWS region>"

(Default value "us-east-1" if not specified) export AWS_ZONE="<AWS zone>"

(Default value "e" if not specified)

export AWS_AVAILABILITY_ZONE="<AWS availability zone>"

(Default value "us-east-1e" if not specified) export AWS_AMI="<AWS AMI id>"

(Default value "ami-efe09bf8" if not specified)

Where the AWS AMI has to be an Ubuntu 16.04.* image available on the region

For eu-central-1, do NOT use ami-9346bcfc because there's some extra software running inside that would cause performance degradation. Any public AWI needs be carefully examined before being used.)

And the AWS availability zone must exist for that region.

Verify that the output format is json in your default configuration (check the file <home>/.aws/config or use the command "aws configure to verify).

4. Run the following scripts:

For Azure, run the script azure-ml.sh from the location /examples/azure/docker.

For AWS, run the script aws-ml.sh from the location /examples/aws/docker.

It takes some time for the scripts to run (about 45 minutes for Azure and 100 minutes for AWS) because images must be loaded to the cloud twice - once each for the master and slave.

It's better to save the output to a file for later examination, for example:

./aws-ml.sh 2>&1 |tee /tmp/aws-ml.out

If the script hangs in docker loading images, try the following:

1. Ctrl-C out of the script.

2. Execute the following commands (they take more than an hour):

execute the following commands (they take more than an hour)

docker-machine scp ml_db.tar.gz aws-ml-master:/tmp

docker-machine ssh aws-ml-master docker load -i /tmp/ml_db.tar.gz docker-machine scp ml-mem.tar.gz aws-ml-master:/tmp

docker-machine ssh aws-ml-master docker load -i /tmp/ml-mem.tar.gz docker-machine scp ml_core.tar.gz aws-ml-master:/tmp

docker-machine ssh aws-ml-master docker load -i /tmp/ml_core.tar.gz docker-machine scp ml_db.tar.gz aws-ml-slave1:/tmp

docker-machine ssh aws-ml-slave1 docker load -i /tmp/ml_db.tar.gz docker-machine scp ml-mem.tar.gz aws-ml-slave1:/tmp

docker-machine ssh aws-ml-slave1 docker load -i /tmp/ml-mem.tar.gz docker-machine scp ml_core.tar.gz aws-ml-slave1:/tmp

docker-machine ssh aws-ml-slave1 docker load -i /tmp/ml_core.tar.gz

(23)

If something goes wrong with AWS and you need to rerun the aws-ml.sh, perform the following cleanup first:

docker-machine rm aws-ml-master aws-ml-slave1 docker-machine rm -f aws-ml-master aws-ml-slave1 aws ec2 delete-key-pair --key-name aws-ml-master aws ec2 delete-key-pair --key-name aws-ml-slave1

aws elb delete-load-balancer --load-balancer-name MLLB aws ec2 delete-vpc --vpc-id <vpc id value>

If you just cleaned up the previous install and the AWS instances were "Terminated" but still not completely removed yet, you may get the following error when you rerun aws-ml.sh: "An error occurred (InvalidInstance) when calling the RegisterInstancesWithLoadBalancer operation: EC2 instance i-xxxx... is not in running state."

As a workaround if you don't want to wait till those AWS instances are completely removed, you can modify the aws-ml.sh and init-ml.sh to replace to words ML-Master and ML-Slave1 to something else, before executing the aws-ml.sh again.

Handling Errors on docker-machine create

Update your docker-machine version to 0.12.2+ if you encounter either of the following errors when aws-ml.sh or azure-ml.sh is doing "docker-machine create":

● Error creating machine: Error running provisioning: Unable to verify the Docker daemon is listening: Maximum number of retries (10) exceeded

● Error creating machine: Error running provisioning: ssh command error:command : systemctl -f start dockererr : exit status 1output : Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

See https://github.com/docker/machine/releases for the latest release and installation instructions.

Before you retry aws-ml.sh or azure-ml.sh, make sure to remove the old broken docker-machines first:

docker-machine rm -f <docker machine name>

where the docker-machine name could be aws-ml-master, aws-ml-slave1, azure-ml-master, or azure- ml-slave1.

Also, clean up the entities (AWS VPC or AZURE ResourceGroup ... ) created on AWS or AZURE.

You may need to update your aws cli or azure cli to the latest version.

For reference, see:

● https://docs.aws.amazon.com/cli/latest/userguide/installing.html

● https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest The following are the versions that work on Mac:

● aws --version returns

aws-cli/1.11.124 Python/3.5.1 Darwin/13.0.2 botocore/1.5.87

● az --version returns

azure-cli (2.0.10)

...

Python (Darwin) 3.5.1 (default, Jan 22 2016, 17:08:33)

(24)

[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]

Python location '/usr/local/opt/python3/bin/python3.5'

Recreating Master and Slave

If docker-machines (AWS or Azure instances) are fine and Docker images are loaded, and you just want to have a clean new master and slave, perform the following steps.

● For AWS: ./init-ml.sh aws 2>&1 |tee /tmp/init-ml.out

● For Azure: ./init-ml.sh azure 2>&1 |tee /tmp/init-ml.out

A load balancer is created for both Azure and AWS cases, and traffic would go through the load balancer.

For Azure, to access the UI, you can use the docker host IP or load balancer IP with port 5480 for master and port 5481 for slave1. For AWS, to access the UI, use the instance's public IP. You can find the public IP from AWS UI or use the command for Master:

aws ec2 describe-instances --filters 'Name=tag:Name,Values=ML-Master'|grep PublicIpAddres |cut -d '"' -f 4

For Slave, (Slave1, for example), use the command:

aws ec2 describe-instances --filters 'Name=tag:Name,Values=ML-Slave1'|grep PublicIpAddres |cut -d '"' -f 4

There are other enhancements that could be made, for example, modify the security rules to restrict internal traffics from within the virtual network only, or create docker-machine without public IP address (all required access from outside could go through load balancer).

Example Setup to Run Mashery Local Master and Slave on a Local Machine Prerequisites

Docker, docker-machine, and docker-compose installed locally (for example, on a Mac).

Procedure

1. Follow the instructions to build Mashery Local for Docker images locally with docker-machine.

Prepare a directory that has the three image gz files, the modified (optional) docker-compose.yml file and init-ml.sh.

The docker-compose.yml and init-ml.sh can be found in the examples directory.

export MOM_HOST="<MOM host>"

export MOM_KEY="<MOM key>"

export MOM_SECRET="<MOM secret>"

chmod a+x *.sh

2. Create two docker machines: xxx-ml-master and xxx-ml-slave1, where ^xxx is replaced with your own machine names, then start both docker-machines.

The docker machine names must be xxx-ml-master and xxx-ml-slave1 for the init-ml.sh to work without modification.

Both docker-machines must be running (Use "docker-machine ls" to check their status.) 3. Execute the command:

init-ml.sh xxx load

(for the first time); or:

(25)

This cleanly restarts the master and slave re-using the already loaded images (with all previous data removed).

Configuring the Network Proxy

To configure the Network Proxy:

Procedure

1. Login to ml-cm or ml-tm using the following command:

docker exec -it ml-cm /bin/bash

or:

docker exec -it ml-tm /bin/bash

2. Edit the content in /etc/environment, as shown in the following example:

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/vmware/

bin"

http_proxy=http://squid.example.com:3128 http_proxy=https://squid.example.com:3128

no_proxy=localhost,127.0.0.1,127.0.1.1,ml-db,ml-mem,ml-tm,ml-cm

3. Logout of ml-cm or ml-tm using the following command:

exit

Installation Troubleshooting Tips

Use the following tips in this section to troubleshoot your installation.

Changing the Traffic Manager Port

To change the Traffic Manager port in Mashery Local for Docker, modify the docker-compose.yml file to change the ^80:80 under services:/ml-tm:/ports: to <host port>:<container port>, where the container port is the port you configured for the proxy.

Note that the host port could be different from the container port. The host port is the port that would be used to access the proxy from outside. After changing the ports in the docker-compose.yml, you will need to do docker-compose down and up to take them into effect. If you know the ports you are planning to switch in the future, you can add them in advance. Then, later when you decide to switch the port, you can simply change it from the UI (under Instance Management > Instance Settings >

HTTP/HTTPS port).

There could be two scenarios for changing the proxy port:

Scenario 1

● Add the new port mapping to docker-compose.yml

● Execute the command below if the Mashery Local Docker instance is running:

docker-compose down

● Execute

docker-compose up -d

● Change port from UI

● Check whether port is in effect:

docker exec -it ml-tm netstat -nlp |grep LISTEN|grep tcp

● If the new port is not being listened, execute the command:

docker exec -it ml-tm nohup service javaproxy restart

(26)

Scenario 2

● Change port from UI

● Add the new port mapping to docker-compose.yml

● Execute

docker-compose down

● Execute

How to Enable Additional Features That Require a New Port

To enable features, such as HTTPS, that requires a port other than 443, the port must be mapped in the docker-compose.yml file. If not, add it to the .yml file. Normally, it would be associated with Traffic Manager. So add it under the services:/ml-tm:/ports. Then, you access from outside through the Docker host IP address.

The example docker-compose.yml file already has most needed ports mapped. However, to change the ports to be used (for example HTTP/HTTPS ports), it would be better to make the changes in the docker-compose.yml file before starting the containers so that the mapping are in place. Then later, you can modify the UI to change the ports. However, if new port was not in effect after the UI change, try restarting the javaproxy. This can be done with command docker exec -it ml-tm nohup service javaproxy restart.

How to Telnet Memcache Port

Currently, only port 11212 is exposed to the outside for the memcache. You can telnet to the memcache port 11212 with the command:

telnet <docker host IP> 11212

The docker host IP can be found from the command echo $DOCKER_HOST if docker-machine is used.

Otherwise, it's the machine/vm IP.

If you need to telnet to other memcache ports, you need to add the port mapping in the docker- compose.yml file and restart the docker-compose. Then, use the command:

telnet <docker host IP> <port number>

Alternatively, you could get in the memcache container with the command:

docker exec -it ml-mem /bin/bash

Install the telnet there and then use the command:

telnet localhost <port number>

How to Troubleshoot 596 Error Caused by Memcache

The 596 Service not found error may be caused by memcache and you may see the following errors in proxy log:

[2017-03-21T19:16:42+00:00] WARN [Memcached IO over {MemcachedConnection to ml-mem/

172.19.0.2:11214}] n.spy.memcached.MemcachedConnection - Closing, and reopening {QA sa=ml-mem/172.19.0.2:11214, #Rops=0, #Wops=2, #iq=0, topRop=null, topWop=Cmd:

get Keys: ENDPOINTS_digital-api.biogen.comExp: 0, toWrite=0, interested=0}, attempt 2.

[2017-03-21T19:16:44+00:00] WARN [proxy-server-71] c.m.p.i.m.MemcachedClientImpl - Operation timed out; retrying...

net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: ml-mem/172.19.0.2:11214

(27)

First check if whether the memcached is running:

docker exec -it ml-mem ps -ef

Then, look for the memcached process. If not running, get in the ml-mem container to start it and see whether there's any error:

docker exec -it ml-mem /bin/bash then

service memcached start

If it failed to start because of running out of file limit, following the instructions in the section How to Change Ulimits on Containers to fix it.

How to Change Ulimits for a Container

To override the fault ulimits for a container, you can either specify a single limit as an integer or soft/

hard limits as a mapping. For example:

ulimits:

nproc: 65535 nofile:

soft: 65535 hard: 65535

How to Use NFS for Verbose Log To use NFS for verbose log:

1. Mount the NFS to a host directory, for example, ^/mnt/nfs.

2. Add the volume mapping in the docker-compose.yml file under the services:/ml-tm:/volumes, for example:

- /mnt/nfs:/var/log/tm_verbose_log

Use the same indent as the existing entry - mldata:/mnt. 3. Execute

docker-compose down

4. Execute

5. Modify the UI to set the Verbose Logs Location to /var/log/tm_verbose_log but leave the flag Use NFS unchecked.

6. Enable the verbose log.

7. Execute

docker exec -it ml-tm nohup service javaproxy restart

Creating a Larger Memory for Memory Allocation

The Memory Allocation factor setting (in the Management Options of Instance Management) resizes the memory of the instance. It will not resize memory to less than 1024MB, as it is the minimum required memory for Docker.

On some platforms, for example Mac OS, if Docker host is created by Docker Machine, Docker creates an instance with 1024MB by default. For the Memory Allocation factor to have an effect, the Docker Machine should be created with a larger memory than 1024MB.

For example, to create a Docker Machine with 2GB of memory, use the following command:

$docker-machine create -d virtualbox --virtualbox-memory 2048 <docker-machine-name>

(28)

For example, to create a Docker Machine with 4GB of memory, use the following command:

$docker-machine create -d virtualbox --virtualbox-memory 4096 <docker-machine-name>

How to Monitor the Health of Docker Containers

To check the container logs, use the following example command:

docker logs ml-tm

To check the container status, use the following example commands:

docker stats ml-tm docker top ml-tm

How to Increase the CPU Share and Memory of a Container

To increase the Docker CPU share and the memory of a container, use the following example command:

docker update --cpu-shares 5120 -m 3000M ml-tm

See Docker CPU share constraints and memory constraints for more information.

How to Do a Clean Restart of a Docker Instance

If you are using Docker-Machine, make sure your are talking to the right one. Execute the command

docker-machine ls to find which one is currently active. It is also recommended to always redo the command:

Deleting volumes will wipe out their data. Back up any data that you need before deleting a container.

Procedure

1. Stop the container(s) using the following command:

docker-compose down

2. Delete all containers using the following command:

docker rm -f $(docker ps -a -q)

3. Delete all volumes using the following command:

docker volume rm $(docker volume ls -q)

4. Restart the containers using the following command:

How to Register Master or Slave with Commands

You can register the master or slave with the following example commands:

● On Docker host for master:

docker exec -it ml-cm /etc/ml.sh register_master '{ "api_key": "chainsproxykey",

"api_secret": "DVUVwqjXqQ", "node_name": "ML_Master", "master_address":

"192.168.99.100", "ntp": "false", "ntp_address": ""}'

● On Docker host for slave1:

docker exec -it ml-cm /etc/ml.sh register_slave '{ "api_key": "chainsproxykey",

"api_secret": "DVUVwqjXqQ", "node_name": "ML_Slave1", "master_address":

"192.168.99.100", "ntp": "false", "ntp_address": "", "slave_address":

"192.168.99.101"}'

If NTP needs to be configured (recommended) during registration, use the following commands:

TIBCO Mashery®

Installation and Configuration Guide for Docker

Contents

TIBCO Documentation and Support Services

Introduction

Assumptions

Conventions

Deployment Topology

Cluster Requirements

Overview of Installation and Configuration Process

Mashery Local Failover Strategy Recommendations

Cluster Management in Mashery Local

Setting up a New Mashery Local Cluster

Adding a Slave to a Running Mashery Local Cluster

Changing the Master in a Mashery Local Cluster

Installing and Configuring Mashery Local for Docker

Required Docker Images

Supported Cloud Platform Configurations

Installing Mashery Local for Docker

Additional Installation Tips

Installation Troubleshooting Tips