TIBCO Mashery®

(1)

Installation and Configuration Guide for Docker

Software Release 4.1.1 July 2017

(2)

Important Information

SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE.

USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE) OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE “LICENSE” FILE(S) OF THE

SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME.

This document contains confidential information that is subject to U.S. and international copyright laws and treaties. No part of this document may be reproduced in any form without the written

authorization of TIBCO Software Inc.

TIBCO and TIBCO Mashery are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries.

All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.

THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT ALL OPERATING SYSTEM PLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED AT THE SAME TIME. SEE THE README FILE FOR THE AVAILABILITY OF THIS SOFTWARE VERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM.

THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF

MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT.

THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME.

THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR INDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE,

INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES.

TIBCO Software Inc. Confidential Information

2

(3)

TIBCO Documentation and Support Services

Documentation for this and other TIBCO products is available on the TIBCO Documentation site. This site is updated more frequently than any documentation that might be included with the product. To ensure that you are accessing the latest available help topics, visit:

https://docs.tibco.com

Product-Specific Documentation

The following document for this product can be found on the TIBCO Documentation site:

● TIBCO Mashery^® Local Installation and Configuration Guide

TIBCO Mashery Professional customers will not have access to all of the features documented here. The following is a list of capabilities that are not available and as such will not be visible within the API Control Center for these customers:

● Distributed API Management (managing Organizations)

● Enriched Call Log Export

● HTTPS Client Profiles

● Mashery Local (Deploy)

● Event Triggers

Additionally, TIBCO Mashery Professional customers will not have access to the Mashery V2 API and as such will be able to use only the OAuth2 Accelerator feature.

Additionally, TIBCO Mashery Professional includes 8M QPM (Queries per month) and all traffic purchased is limited to a max of 100 QPS (Queries per second). TIBCO Mashery Professional customers can create a max of 25 APIs and 25 packages.

How to Contact TIBCO Support

For comments or problems with this manual or the software it addresses, contact TIBCO Support:

● For an overview of TIBCO Support, and information about getting started with TIBCO Support, visit this site:

http://www.tibco.com/services/support

● If you already have a valid maintenance or support contract, visit this site:

https://support.tibco.com

Entry to this site requires a user name and password. If you do not have a user name, you can request one.

How to Join TIBCO Community

TIBCO Community is an online destination for TIBCO customers, partners, and resident experts. It is a place to share and access the collective experience of the TIBCO community. TIBCO Community offers forums, blogs, and access to a variety of resources. To register, go to the following web address:

https://community.tibco.com

6

(7)

Introduction

This guide provides an overview of the installation, requirements and configuration for Mashery^® Local for Docker.

Mashery Local for Docker is a set of Docker images for running Mashery Local. These images can be customized for custom configurations. Mashery Local for Docker allows customers to perform hybrid traffic management on premise to run the API traffic inside data-centers. Mashery Local securely interacts with the Mashery Cloud hosted Developer Portal, Administration Dashboard and API Reporting and Analytics modules.

Assumptions

This guide assumes that you are using Docker version 1.12 or later. If you have an internal cloud, established best practices will be applied (for example disk alignment). If you are using different servers and clients, the underlying concepts implied by the installation and configuration steps still apply.

Conventions

This guide uses the following conventions:

● Keys you press simultaneously appear with a plus (+) sign between them (for example, Ctrl+P means press the Ctrl key first, and while holding it down, press the P key).

● Field, list, folder, window, and dialog box names have initial caps (for example, City, State).

● Tab names are bold and have initial caps (for example, People tab).

● Names of buttons and keys that you press on your keyboard are in bold and have initial caps (for example, Cancel, OK, Enter, Y).

Deployment Topology

The following diagram depicts a typical deployment topology for Mashery Local.

(8)

Overview of Installation and Configuration Process

This section provides a roadmap of the installation process for Mashery Local.

Procedure

1. Download the Mashery Local Docker artifact from TIBCO^® eDelivery and create the Mashery Local Docker Image set as described in Installing and Configuring Mashery Local for Docker.

2. Configure a Mashery Local Master as described in Configuring a Mashery Local Master.

3. Configure slaves to the Mashery Local Master as described in Configuring Slaves to the Local Master. It is best practice to set up production with no less than 2 slaves per master.

4. Configure the load balancer as described in Configuring the Load Balancer.

5. Perform advanced configuration such as enabling notifications, as described in Advanced Configuration.

8

(9)

Installing and Configuring Mashery Local for Docker

The following sections describe how to install and configure some basic environments complete with a master, one or more slaves, and load balancing.

Mashery Local for Docker includes a script that will download and install third-party software from third-party websites, including but not necessarily limited to CentOS and EPEL repositories located here:

● https://hub.docker.com/_/centos/

● http://vault.centos.org/

● https://dl.fedoraproject.org/pub/epel/

Such third-party software is subject to third-party software licenses that may be available on such third- party websites. For more information on CentOS repositories and EPEL, see:

● https://wiki.centos.org/AdditionalResources/Repositories

● https://fedoraproject.org/wiki/EPEL

Required Docker Images

Three images are needed to install Mashery Local for Docker:

1. On-premise database: ml-db 2. Memcache: ml-mem

3. TIBCO Mashery Local Core - Traffic Manager plus Cluster Manager UI: ml-core

Installing Mashery Local for Docker

To install Mashery Local for Docker:

Procedure

1. Install Docker Engine, Docker Machine (optional), and docker-compose (optional, and not needed if on Kubernetes) on your operating system.

Refer to the Docker documentation for the operating system of your choice:

● https://docs.docker.com/engine/

● https://docs.docker.com/machine/

● https://docs.docker.com/compose/

For Mac OS installations, it's recommended to install Docker Toolbox so that multiple docker hosts can be run on the same box.

2. TIBCO Mashery Local for Docker is available as a TIB_mash-local**.tar.gz file. Download this file from TIBCO eDelivery and extract the file contents.

3. Create the TIBCO Mashery Local Docker Image set:

a) Drop in custom configurations:

- Modify examples/set-user-variables.sh and drop it in the resource/addons directory

- (Optional) To use a custom https server PEM file for ML Cluster Manager, drop the PEM file to the resource/addons/certs directory and name it server.pem.

If planning to run on Kubernetes, additional customization may be required. Please see the section Installing and Running Mashery Local for Docker with Kubernetes.

(10)

b) Navigate to the root folder of the extracted contents and run the following command to build the Mashery Local image set (comprising three images): ./build-docker.sh 2>&1 |tee /tmp/

build-docker.log

This will increment the image tag revision number. You can use the command "docker images" to check it out. You will need to modify the docker-compose.yml file to use the new tag if you build more than once in the same directory. However, if you would like to keep the same revision number, you will need to remove the file

BUILD_NUMBER.txt in the current directory before starting the next build.

c) Verify three images are created: ml-db.tar.gz, ml-mem.tar.gz, ml-core.tar.gz. (The image sizes are about 140MB, 120MB, and 850MB, respectively.)

If the size of any image is significantly less than the numbers above, then the image build might have some problems. Check the build-docker.log generated from the previous step. If you see several errors, such as:

Could not retrieve mirrorlist http://mirrorlist.centos.org/?... error was14: PYCURL ERROR 22 - "The requested URL returned error: 503 Service Unavailable"

then you probably have some firewall issues on your network. Switch to a network without firewall restriction to do the build.

4. (If planning to run on Kubernetes, the remaining steps do not apply. Please go to the section Installing and Running Mashery Local for Docker with Kubernetes to continue installation.) Navigate to the examples folder and copy the docker-compose.yml and the three image .gz files to the target Docker host machine.

The docker-compose.yml may need additional edits, depending on what ports need to be exposed or for other customization. For example, to add "extra hosts" if there are any extra host names and IP mapping that need to be added for a container.

Note: The indents and dash in the docker-compose yml file are important.

Run the following commands:

● docker load -i <each of the three image files, one by one>

● docker-compose up -d

5. Verify that four Docker containers are up:

docker ps to make sure the four containers are running.

6. Repeat Steps 4-5 for each Docker host that will run a Mashery Local instance.

7. Go to the instance in a browser:

https://<docker host-IP>:5480.

8. Complete Master registration to TIBCO MOM (Mashery On-Prem Manager) or complete Slave registration to Master.

Additional Installation Tips

● Installation Steps with Docker Toolbox

● Working with Amazon EC2

Installing with Docker Toolbox

Docker Toolbox is a tool that lets you manage Docker engines on multiple virtual instances, and is used with Docker Machine. If you need to setup slaves for the cluster on different virtual instances, images built in the previous set of instructions (Step 3 of Installating Mashery Local for Docker) can be reused below.

10

(11)

1. Install Docker Toolbox from https://www.docker.com/products/docker-toolbox.

2. Use docker-machine create command to create Docker engines on virtual instances.

Drivers are available for various cloud provider platforms. Refer to https://

docs.docker.com/machine/ for the latest information. Also refer to individual cloud provider documentation for more details on authentication details and other parameters you can use to customize your Docker Machine.

Some example commands are below:

a. To create a Docker Machine on a VirtualBox setup on your machine (prerequisite: VirtualBox 5+

ideal):

docker-machine create --driver virtualbox <docker machine name>

b. To create a Docker Machine on a VMware Fusion setup on your machine:

docker-machine create --driver vmwarefusion <docker machine name>

c. To create a Docker Machine on AWS (prerequisite: AWS signup, create an IAM administrator user and a key pair: AWS access key, AWS secret key):

docker-machine create --driver amazonec2 --amazonec2-access-key <your aws access key> --amazonec2-secret-key <your aws secret key> <name for your new AWS instance>

d. To create a Docker Machine on Microsoft Azure (prerequisite: Microsoft Azure signup):

docker-machine create --driver azure --azure-subscription-id <your subscription id> <name for your new azure instance>

e. To create a Docker Machine on Google Cloud (prerequisite: Google Cloud signup, recommend installing and configuring gcloud tools locally to manage authentication. Refer to GCE

documentation.):

docker-machine create --driver google --google-project <google project id> - google-zone "us-west1-a" <name for your new google instance>

3. List all your available machines and make sure the one you just created shows up:

docker-machine ls

4. Connect your shell to a machine:

eval $(docker-machine env <docker machine name>) docker-machine ls

(confirm the machine you are connecting to has an * to it to show that it's active) 5. You can use the three images you created via running the build-docker.sh script above:

a. Copy or move the images to the Amazon instance:

b. Run load command to load the images to the docker host.

c. Run docker compose up -d.

Working with Amazon EC2 Instances

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud.

Procedure

1. Install Docker Engine: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker- basics.html

2. Install Docker Compose: https://docs.docker.com/compose/install

(12)

3. Install Docker Machine: https://docs.docker.com/machine/install-machine

4. TIBCO Mashery Local for Docker is available as a TIB_mash-local**.tar.gz file. Download this file from TIBCO eDelivery and extract the file contents.

5. Navigate to the root folder of the extracted contents and run the following command to build the Mashery Local image set (comprising three images):

a) ./build-docker.sh

This will increment the image tag revision number. You can use the command "docker images" to check it out. You will need to modify the docker-compose.yml file to use the new tag if you build more than once in the same directory.

b) Verify three images are created: ml_db.tar.gz, ml_mem.tar.gz, ml_core.tar.gz.

6. Navigate to the examples folder and copy the docker-compose.yml and the three image .gz files previously built to a target directory.

The docker-compose.yml may need additional edits, depending on what ports need to be exposed or for other customization. For example, to add "extra hosts" if there are any extra host names and IP mapping that need to be added for a container.

Note: The indents and dash in the docker-compose.yml file are important.

Run the following command: docker load -i <each of the three image files, one by one>

At this point, this AWS instance can be saved as an AM1 that can be re-used for any Mashery Local instance.

7. Run the following command: docker-compose up -d

Verify that four Docker containers are up:

docker ps to make sure the four containers are running.

8. Go to the instance in a browser:

https://<docker host-IP>:5480.

9. Complete Master registration to TIBCO MOM (Mashery On-Prem Manager) or complete Slave registration to Master.

If you are using docker-machine to create the docker container, then use the following command:

docker-machine create --driver amazonec2 --amazonec2-access-key <Your AWS Key> --amazonec2-secret-key <Your AWS Key Secret> --amazonec2-region

You will need to open the following ports under the security group for registering Slave machine with Master: 22, 443, 2200, 2376 (TCP), 3306 , 5480, 11212.

Installation Troubleshooting Tips

Use the following tips in this section to troubleshoot your installation.

Changing the Traffic Manager Port

To change the Traffic Manager port in Mashery Local for Docker, modify the docker-compose.yml file to change the ^80:80 under services:/ml-tm:/ports: to <host port>:<container port>, where the container port is the port you configured for the proxy.

Note that the host port could be different from the container port. The host port is the port that would be used to access the proxy from outside. After changing the ports in the docker-compose.yml, you will need to do docker-compose down and up to take them into effect. If you know the ports you are planning to switch in the future, you can add them in advance. Then, later when you decide to switch the port, you can simply change it from the UI (under Instance Management > Instance Settings >

HTTP/HTTPS port).

12

(13)

There could be two scenarios for changing the proxy port:

Scenario 1

● Add the new port mapping to docker-compose.yml

● Execute the command below if the Mashery Local Docker instance is running:

docker-compose down

● Execute

docker-compose up -d

● Change port from UI

● Check whether port is in effect:

docker exec -it ml-tm netstat -nlp |grep LISTEN|grep tcp

● If the new port is not being listened, execute the command:

docker exec -it ml-tm nohup service javaproxy restart

Scenario 2

● Change port from UI

● Add the new port mapping to docker-compose.yml

● Execute

docker-compose down

● Execute

How to Enable Additional Features That Require a New Port

To enable features, such as HTTPS, that requires a port other than 443, the port must be mapped in the docker-compose.yml file. If not, add it to the .yml file. Normally, it would be associated with Traffic Manager. So add it under the services:/ml-tm:/ports. Then, you access from outside through the Docker host IP address.

The example docker-compose.yml file already has most needed ports mapped. However, to change the ports to be used (for example HTTP/HTTPS ports), it would be better to make the changes in the docker-compose.yml file before starting the containers so that the mapping are in place. Then later, you can modify the UI to change the ports. However, if new port was not in effect after the UI change, try restarting the javaproxy. This can be done with command docker exec -it ml-tm nohup service javaproxy restart.

How to Telnet Memcache Port

Currently, only port 11212 is exposed to the outside for the memcache. You can telnet to the memcache port 11212 with the command:

telnet <docker host IP> 11212

The docker host IP can be found from the command echo $DOCKER_HOST if docker-machine is used.

Otherwise, it's the machine/vm IP.

If you need to telnet to other memcache ports, you need to add the port mapping in the docker- compose.yml file and restart the docker-compose. Then, use the command:

telnet <docker host IP> <port number>

Alternatively, you could get in the memcache container with the command:

docker exec -it ml_mem /bin/bash

(14)

Install the telnet there and then use the command:

telnet localhost <port number>

How to Troubleshoot 596 Error Caused by Memcache

The 596 Service not found error may be caused by memcache and you may see the following errors in proxy log:

[2017-03-21T19:16:42+00:00] WARN [Memcached IO over {MemcachedConnection to ml_mem/

172.19.0.2:11214}] n.spy.memcached.MemcachedConnection - Closing, and reopening {QA sa=ml_mem/172.19.0.2:11214, #Rops=0, #Wops=2, #iq=0, topRop=null, topWop=Cmd:

get Keys: ENDPOINTS_digital-api.biogen.comExp: 0, toWrite=0, interested=0}, attempt 2.

[2017-03-21T19:16:44+00:00] WARN [proxy-server-71] c.m.p.i.m.MemcachedClientImpl - Operation timed out; retrying...

net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: ml_mem/172.19.0.2:11214

First check if whether the memcached is running:

docker exec -it ml_mem ps -ef

Then, look for the memcached process. If not running, get in the ml_mem container to start it and see whether there's any error:

docker exec -it ml_mem /bin/bash then

service memcached start

If it failed to start because of running out of file limit, following the instructions in the section How to Change Ulimits on Containers to fix it.

How to Change Ulimits for a Container

To override the fault ulimits for a container, you can either specify a single limit as an integer or soft/

hard limits as a mapping. For example:

ulimits:

nproc: 65535 nofile:

soft: 65535 hard: 65535

How to Use NFS for Verbose Log To use NFS for verbose log:

1. Mount the NFS to a host directory, for example, ^/mnt/nfs.

2. Add the volume mapping in the docker-compose.yml file under the services:/ml-tm:/volumes, for example:

- /mnt/nfs:/var/log/tm_verbose_log

Use the same indent as the existing entry - mldata:/mnt. 3. Execute

docker-compose down

4. Execute

5. Modify the UI to set the Verbose Logs Location to /var/log/tm_verbose_log but leave the flag Use NFS unchecked.

14

(15)

6. Enable the verbose log.

7. Execute

docker exec -it ml-tm nohup service javaproxy restart

Creating a Larger Memory for Memory Allocation

The Memory Allocation factor setting (in the Management Options of Instance Management) resizes the memory of the instance. It will not resize memory to less than 1024MB, as it is the minimum required memory for Docker.

On some platforms, for example Mac OS, if Docker host is created by Docker Machine, Docker creates an instance with 1024MB by default. For the Memory Allocation factor to have an effect, the Docker Machine should be created with a larger memory than 1024MB.

For example, to create a Docker Machine with 2GB of memory, use the following command:

$docker-machine create -d virtualbox --virtualbox-memory 2048 <docker-machine-name>

For example, to create a Docker Machine with 4GB of memory, use the following command:

$docker-machine create -d virtualbox --virtualbox-memory 4096 <docker-machine-name>

How to Monitor the Health of Docker Containers

To check the container logs, use the following example command:

docker logs ml-tm

To check the container status, use the following example commands:

docker stats ml-tm docker top ml-tm

How to Increase the CPU Share and Memory of a Container

To increase the Docker CPU share and the memory of a container, use the following example command:

docker update --cpu-shares 5120 -m 3000M ml-tm

See Docker CPU share constraints and memory constraints for more information.

How to Do a Clean Restart of a Docker Instance

If you are using Docker-Machine, make sure your are talking to the right one. Execute the command

docker-machine ls to find which one is currently active. It is also recommended to always redo the command:

eval "$(docker-machine env <docker machine name>)"

Deleting volumes will wipe out their data. Back up any data that you need before deleting a container.

Procedure

1. Stop the container(s) using the following command:

docker-compose down

2. Delete all containers using the following command:

docker rm -f $(docker ps -a -q)

3. Delete all volumes using the following command:

docker volume rm $(docker volume ls -q)

(16)

4. Restart the containers using the following command:

How to Register Master or Slave with Commands

You can register the master or slave with the following example commands:

● On Docker host for master:

docker exec -it ml-cm /etc/ml.sh register_master '{ "api_key": "chainsproxykey",

"api_secret": "DVUVwqjXqQ", "node_name": "ML_Master", "master_address":

"192.168.99.100", "ntp": "false", "ntp_address": ""}'

● On Docker host for slave1:

docker exec -it ml-cm /etc/ml.sh register_slave '{ "api_key": "chainsproxykey",

"api_secret": "DVUVwqjXqQ", "node_name": "ML_Slave1", "master_address":

"192.168.99.100", "ntp": "false", "ntp_address": "", "slave_address":

"192.168.99.101"}'

How to Promote a Slave to Master with CLI

To promote a Slave to Master without using Mashery Local Cluster Manager administrator, you can use Command Line Interface (CLI).

Copy the following code to a temporary file (for example, /tmp/promote_to_master.py):

import sys;

sys.path.append('/var/www/htdocs/service/mashery/cgi');

import Mashery;

Mashery.make_master()

Mashery.backup_mysql_server()

Then, execute the following command:

docker cp /tmp/promote_to_master.py ml-cm:/tmp

docker exec -it ml-cm python/tmp/promote_to_master.py

How to Change Master to Slave with CLI

To change a Master to a Slave without using Mashery Local Cluster Manager administrator, you can use Command Line Interface (CLI).

Create a temporary file (for example, /tmp/change_master.py) with the following code (fill in the <old master IP> and <new master IP> fields):

import sys;

sys.path.append('/var/www/htdocs/service/mashery/cgi');

import debug;

debug.change_master_to('<old master IP>','<new master IP>')

Then, execute the following command:

docker cp /tmp/change_master.py ml-cm:/tmp

docker exec -it ml-cm python/tmp/change_master.py

Managing Docker Containers

Use the following commands to manage the Docker containers:

Action Command

Pause docker-compose pause

Unpause docker-compose unpause

Restart docker-compose restart

16

(17)

Action Command

Shut down docker-compose down

Complete Cleanup

(remove persistent data) docker volume rm $(docker volume ls -q)

This will clean up all the database content and configurations.

Then, you will need to redo and register the master and slave after re- running Mashery Local for Docker.

This command removes all volumes for a docker host. If you have other volumes besides those used by Mashery Local for Docker, you must remove the volumes for Mashery Local for Docker individually.

Installing and Running Mashery Local for Docker with Kubernetes

To install and run Mashery Local for Docker with Kubernetes on Amazon Web Services (AWS) cloud, ensure your configuration meets the proper pre-requisites, then follow the steps below.

Prerequisites

● Mac OS or Linux local working environment

● AWS account with full access to the AWS APIs

● AWS Command Line Interface (CLI) installed and configured

— AWS configurations set up, such as default region, access key, and secret key

— Verify the command "aws" is on your path and that you can do some simple AWS CLI commands, for example:

aws ec2 describe-vpcs

● Local Docker environment ready (either connect to a docker-machine that is up and running, or run docker host on the machine).

You need this to upload docker images even if you are using pre-built images from S3.

● Mashery Local for Docker images built locally (For instructions on building Mashery Local for Docker images, see Installing Mashery Local for Docker.

If you have custom adapters, see the Customizing for Kubernetes section.

● Docker images verified (no critical errors during the build and the images can be seen with the command "docker images".

Procedure

1. Install Kubernetes and set up the cluster.

Since Kubernetes clusters could be set up in many different ways and they can be shared by many applications, it's the users responsibility to have the clusters set up and be ready for deployment.

Verify that there are enough m3.large (or larger) nodes for all planned Mashery Local instances (each Mashery Local instance requires a node).

The following steps are an example of how to set up a Kubernetes cluster on AWS, based on instructions from Kubernetes Getting Started Guide for AWS.

(18)

This setup only works with Kubernetes 1.5 releases and does not work for 1.6+ releases.

a) Deploy the Kubernetes cluster on AWS. Follow the instructions in the Kubernetes Getting Started Guide for AWS.

If you have already installed minikube or kubetctl before, it's better to remove those to do a clean install (remove any trace of previous kubernetes installation):

● remove/rename the kubectl

● remove/rename ~/.kube

● remove/rename ~/.ssh kube*

● install on a clean directory

b) You can override some Kubernetes default settings by setting some environment variables.

Export those variables before the install. Typically, you would need to set the following:

export KUBERNETES_RELEASE=v1.5.3

You must set KUBERNETES_RELEASE to v1.5.3 if using the installing steps from the Kubernetes Getting Started Guide for AWS.

There are other ways to install Kubernetes, such are using Kubernetes Operations (kops), that is outside the scope of this step.

export AWS_ACCESS_KEY_ID=<your AWS_ACCESS_KEY_ID>

export AWS_SECRET_ACCESS_KEY=<your AWS_SECRET_ACCESS_KEY>

export AWS_DEFAULT_REGION=us-east-1

export KUBERNETES_PROVIDER=aws export NUM_NODES=3

export MASTER_SIZE=m3.medium export NODE_SIZE=m3.large export AWS_S3_REGION=us-east-1 export KUBE_AWS_ZONE=us-east-1e

export AWS_S3_BUCKET=masheryml-kubernetes-artifacts export KUBE_AWS_INSTANCE_PREFIX=k8s

Check the AWS zone availability first (from AWS EC2 dashboard > Service Health >

Availability Zone Status) . You must choose a zone that is available. Otherwise you may get the following error:"An error occurred

(InternetGatewayLimitExceeded) when calling the CreateInternetGateway operation: The maximum number of internet gateways has been reached."

For the Kubernetes worker nodes, the NODE_SIZE needs be set to m3.large or larger to run MLCE instance, unless you restrict the resources each instance can allocate in the deployment configuration.

For the Kubernetes master node, use m3.medium for clusters of less than 5 nodes, use m3.large for 6-10 nodes, and use m3.xlarge for more than 10 nodes.

c) For a first-time installation, go to the desired directory to install Kubernetes. Execute either:

#Using wget

wget -q -O - https://get.k8s.io | bash

or

#Using cURL

curl -sS https://get.k8s.io | bash

d) Add the appropriate binary folder to your shell PATH to access kubectl:

export PATH=<path to kubernetes-directory>/client/bin:.:$PATH

e) Verify the cluster setup with the following command:

kubectl config view

You should see something similar to the following:

apiVersion: v1 clusters:

18

(19)

- cluster:

certificate-authority-data: REDACTED server: https://34.205.42.112

name: aws_k8s ...

contexts:

- context:

cluster: aws_k8s user: aws_k8s name: aws_k8s ...

current-context: aws_k8s kind: Config

preferences: {}

users:

- name: aws_k8s user:

client-certificate-data: REDACTED client-key-data: REDACTED

token: lL5WiMPIjxBfdl1e9OkoCx23zM1Mwep8 - name: aws_k8s-basic-auth

user:

password: TNYNPNf6LgCCEfQW username: admin

...

If it didn't create and start the cluster, use the command kubernetes/cluster/kube-up.sh to create and start the cluster.

f) You can access the Kubernetes console UI with the following URL: <cluster server url>/ui

In the previous example, this is: https://34.205.42.112/ui

g) Login to the console with the aws_k8s-basic-auth user "admin" and its password.

For your convenience, you can use the following command to find the server URL directly:

kubectl config view -o=json|jq '.clusters[] | select(.name=="aws_'$

{KUBE_AWS_INSTANCE_PREFIX}'") |.cluster.server' |sed -e 's/"//g'

Use the following command to find the password for the admin user:

kubectl config view -o=json|jq '.users[] | select(.name=="aws_'$

{KUBE_AWS_INSTANCE_PREFIX}'-basic-auth") |.user.password' |sed -e 's/"//g'

2. Create a private Amazon EC2 Container Registry (ECR) for Mashery Local for Docker, for example:

aws ecr create-repository --repository-name <registry name>

for example

aws ecr create-repository --repository-name tibco/mlce

If you have never used AWS ECS before, you will need to go to the AWS ECS dashboard and follow the "Getting Started" step.

3. Go to the directory examples/kubernetes extracted from the Mashery Local for Docker 4.1 release, modify the aws-env.sh with the planned configuration, and set the environment variables with the command:

. aws-env.sh

(20)

The ML_REGISTRY_NAME is the registry name used in step 2.

The ML_REGISTRY_HOST can be found with the following command:

aws ecr get-login --registry-ids `aws ecr describe-repositories -- repository-names "$ML_REGISTRY_NAME" |grep registryId |cut -d ":" -f 2|

tr -d ' ",'`|awk -F'https://' '{print $2}'

Or, from the AWS ECS dashboard, go to Repositories > Repository URI. For example, with repository URI "12345603243.dkr.ecr.us-east-1.amazonaws.com/tibco/mlce", the ML_REGISTRY_NAME is ^tibco/mlce, and the ML_REGISTRY_HOST is

12345603243.dkr.ecr.us-east-1.amazonaws.com.

4. Add or set login credentials in <home>/.docker/config.json using the command:

aws ecr get-login | sh -

5. Load Docker images.

a) Verify Mashery Local for Docker images with the correct tag are in the current docker host with the command:

docker images

The tag should match the env. variable ML_IMAGE_TAG.

b) Execute the following script to load images to the ECR Docker registry:

upload-images.sh

6. Execute the following script to store the Docker registry key as Kubernetes "Secret":

set-registry-key.sh

7. Execute the following script to store MOM host and key as Kubernetes "Secret":

set-mom-secret.sh create <MOM key> <MOM secret> [<MOM host>]

If you want to enable HTTPS or OAuth, see the section Customizing for Kubernetes for additional configuration steps.

8. Create storage classes for Mashery Local for Docker persistent stores:

set-storage-classes.sh

9. Create Mashery Local Traffic Manager service and Mashery Local Master service:

set-ml-services.sh

You can check the services with the following commands:

kubectl describe service ml-traffic-manager kubectl describe service ml-master

The ml-traffic-manager is configured with load balancer. You can find the load balancer DNS name with the following command:

kubectl describe service ml-traffic-manager|grep Ingress|awk -F' ' '{print $3}'

The load balancer can also be found on the AWS EC2 dashboard Load Balancers list.

10. Deploy Mashery Local master instance:

deploy-master.sh

You can check the ML instance pods with the command:

kubectl get pods

The ML master pod has a name like ml-master-.... When it's fully up, you should see 4/4 under the READY column with STATUS "Running" for the master instance pod.

You can check the startup init instance log with the following command:

kubectl exec -ti `kubectl get pods |grep ml-master |cut -d " " -f 1` -c ml-cm -- cat /var/log/mashery/init-instance.log

20

(21)

When it's fully ready to serve traffic, you should see something like the following:

....

Register status: Content-Type: application/json Status: 200 {"results":

[{"results": [{"address": "10.0.22.98"}], "error": null}, {"results":

[{"area_name": "Roger"}], "error": null}, {"results": [{"credentials_updated":

true}], "error": null}, {"results": [{"name": "ml-master-4209822619-sxq40",

"id": 0}], "error": null}, {"results": [{"is_master": true}], "error": null}],

"error": null}

**** 04/06 05:27:38 Register instance succeeded Load service result:

Load service result:

Load service result: 70a0b42e-2b9a-4f60-a4d6-8c5503894043 [SERVICES] 04/06/17 05:27:45 - 04/06/17 05:27:47: 254 records (Success) 70a0b42e-2b9a-4f60-

a4d6-8c5503894043 [KEYS] 04/06/17 05:27:47 - 04/06/17 05:27:55: 10963 records (Success) 70a0b42e-2b9a-4f60-a4d6-8c5503894043 [APPS] 04/06/17 05:27:55 - 04/06/17 05:28:23: 6884 records (Success) 70a0b42e-2b9a-4f60-a4d6-8c5503894043 [CLASSES] 04/06/17 05:28:23 - 04/06/17 05:28:23: 0 records (Success)

70a0b42e-2b9a-4f60-a4d6-8c5503894043 [PACKAGES] 04/06/17 05:28:23 - 04/06/17 05:29:54: 28824 records (Success) 70a0b42e-2b9a-4f60-a4d6-8c5503894043 [PACKAGEKEYS] 04/06/17 05:29:54 - 04/06/17 05:30:17: 5553 records (Success)

**** 04/06 05:30:17 Service info loaded

Load cache output first ten lines: - Trying to load mapi data for spkey:

m8hxx3wxy5wjyjhfzc328wqh key: MAPI_m8hxx3wxy5wjyjhfzc328wqh::2011w25DeveloperJay key: MAPI_m8hxx3wxy5wjyjhfzc328wqh::2011w25DeveloperRoger key:

MAPI_m8hxx3wxy5wjyjhfzc328wqh::3skjegt4ddpam6a5r8sfgpkz key:

MAPI_m8hxx3wxy5wjyjhfzc328wqh::4q5t7z4gduy388z9nk5tmptm key:

MAPI_m8hxx3wxy5wjyjhfzc328wqh::4tzw5p5h5mx8gr8ez6m34wak key:

MAPI_m8hxx3wxy5wjyjhfzc328wqh::5s8ds7dcyj7cjz4h9h5tv7ev key:

MAPI_m8hxx3wxy5wjyjhfzc328wqh::5yy6dkjbq7sr922j4wt6u2hc key:

MAPI_m8hxx3wxy5wjyjhfzc328wqh::6mbcz48nabrz682xn2hdmhzn key:

MAPI_m8hxx3wxy5wjyjhfzc328wqh::8tng6tk5bzhpfqexn525cqnj

**** 04/06 05:31:01 Cache Loaded

**** 04/06 05:31:01 Ping Traffic Manager succeeded

**** 04/06 05:31:01 Setting status ready

When the ML master instance containers are up, you can find the ML master instance node public IP with the following command:

kubectl describe node `kubectl get pods -o wide |grep ml-master |awk -F' ' '{print $7}'` |grep Addresses |cut -d "," -f 3

If you need to access the Mashery Local instance Cluster Manager UI, you need to open the port 5480 for UI access. For convenience, you can open the port for all minion nodes in the cluster with the following command:

aws ec2 authorize-security-group-ingress --group-id `aws ec2 describe-securitygroups --filters "Name=group-name,Values=kubernetes-minion-$

{KUBE_AWS_INSTANCE_PREFIX}" |jq -r '.SecurityGroups[0].GroupId'` --protocol tcp --port 5480 --cidr 0.0.0.0/0

Or you can open the port individually as needed with additional security group through AWS UI or CLI.

Then you can login to the ML master instance Cluster Manager UI with https://< ML master instance node ip>:5480.

You can get into any ML master instance container with the following command:

kubectl exec -ti `kubectl get pods |grep ml-master |cut -d " " -f 1` -c

<container name> -- /bin/bash

The container names are: ml-db, ml-mem, ml-tm, ml-cm.

(22)

You can also execute some simple remote command on a container directly:

kubectl exec -ti `kubectl get pods |grep ml-master |cut -d " " -f 1` -c

for example:

kubectl exec -ti `kubectl get pods |grep ml-master |cut -d " " -f 1` -c ml-tm -- ls -l /var/log/trafficmgr/access

At any time, you could also get in the Kubernetes dashboard UI to check the progress, such as checking the deployment, replica sets, services, pods, containers and their logs.

11. Deploy Mashery Local slave instances:

deploy-slaves.sh

You can check the Mashery Local instance pods with the command:

kubectl get pods

The Mashery Local slaves instance pods are named with ml-slave-0, ml-slave-1, ml-slave-2.

When it's fully up, you should see 4/4 under the READY column with STATUS "Running" for the slave instance pod.

You can check the startup init instance log with the following command:

kubectl exec -ti `kubectl get pods |grep <slave pod name> |cut -d " " -f 1` -c ml-cm -- cat /var/log/mashery/init-instance.log

for example:

kubectl exec -ti `kubectl get pods |grep ml-slave-0 |cut -d " " -f 1` -c ml-cm -- cat /var/log/mashery/init-instance.log

You can find the Mashery Local slave instance node IP with the following command:

kubectl describe node `kubectl get pods -o wide |grep <slave pod name> |awk -F' ' '{print $7}'` |grep Addresses |cut -d "," -f 3

Then, login to the ML slave instance Cluster Manager UI with https://<ML slave instance node ip>:

5480

If you didn't open the port 5480 for all nodes in the previous step, you need to open the port for each ML slave instance individually with additional security group through AWS UI or CLI.

You can get into any ML slave instance container with the following command:

kubectl exec -ti `kubectl get pods |grep <slave pod name> |cut -d " " -f 1` -c

<container name> -- /bin/bash

The container names are: ml-db, ml-mem, ml-tm, ml-cm.

You can also execute some simple remote command on a container directly:

kubectl exec -ti `kubectl get pods |grep <slave pod name> |cut -d " " -f 1` -c

for example:

kubectl exec -ti `kubectl get pods |grep ml-slave-0 |cut -d " " -f 1` -c ml-tm -- ls -l /var/log/trafficmgr/access

At any time, you could also get into the Kubernetes dashboard UI to check the progress, such as checking the stateful sets, services, pods, and containers and their logs.

By default, it's configured to run two slave instances.

You can use the following command to increase or reduce the number of slaves:

kubectl patch statefulset ml-slave --type='json' -p='[{"op": "replace", "path":

"/spec/replicas", "value":<the desired replica number>}]'

However, you must have enough worker nodes to run all the slave instances.

22

(23)

12. Test the traffic, using the following example commands:

export LB=`kubectl describe service ml-traffic-manager|grep Ingress|awk -F' ' '{print $3}'` && echo $LB

curl -H 'Host: roger.api.perfmom.mashspud.com' http://$LB/testep?

api_key=funjsgx8m5bsew2jngpdanxf

13. Cleanup or undeploy Mashery Local instances.

To undeploy Mashery Local slave instances:

deploy-slaves.sh delete

To undeploy Mashery Local master instances:

deploy-master.sh delete

14. Shut down Kubernetes cluster using the following command (if the example steps in Step 1 were used):

kubernetes/cluster/kube-down.sh

Customizing for Kubernetes

When Kubernetes does autoscaling or auto-repairing, you are not able to get into the Mashery Local Cluster Manager UI to change the configuration, so all customizations should be set up at image build time.

Disabling HTTP or using a different HTTP Port

To disable HTTP or to use a different HTTP port, modify the following variables in the examples/set- user-variables.sh and drop it in the resource/addons directory before building docker images:

export HTTP_ENABLED=true export HTTP_PORT=80

Also, make the corresponding changes in the aws-env.sh file.

Adding a Custom Adapter

To add a custom adapter, put the adapter zip file in the resources/addons directory and modify the following variable in the examples/set-user-variables.sh and drop it in the resource/addons directory before building docker images:

export CUSTOM_ADAPTER_ZIP_FILE_NAME=<custom adapter zip file name>

Enabling OAuth

To enable OAuth, in the aws-env.sh file, set:

export OAUTH_ENABLED=true

Then, execute the following command as an additional step in Step 7 of Installing and Running Mashery Local for Docker with Kubernetes:

set-oauth-secret.sh <create|replace|delete> <OAuth authorization user> <OAuth authorization user password>

For the first argument in the above command, use "create" for the first time, then use "replace" for subsequent changes.

Enabling HTTPS

To enable OAuth, in the aws-env.sh file, set:

export HTTPS_ENABLED=true export HTTPS_PORT=<port number>

Next, put the server certificate file in the resources/addons/certs directory before building the images.

(24)

Then, execute the following command as an additional step in Step 7 of Installing and Running Mashery Local for Docker with Kubernetes:

set-https-secret.sh <create|replace|delete> <server certificate file name> <server certificate password>

For the first argument in the above command, use "create" for the first time, then use "replace" for subsequent changes.

Configuring the Mashery Local Cluster

Mashery Local may run configured in a cluster of one master and multiple slaves.

To configure the Mashery Local cluster, you need to:

● Configure a Mashery local master

● Configure slave(s) to the local master

If you run Mashery Local with Kubernetes, the Master and Slave configuration are done automatically.

Configuring a Mashery Local Master

To configure a Mashery Local master:

Procedure

1. Browse to the Mashery Local Cluster Manager of the master by using the Docker Host IP address of the instance:

https://<IP_address_of_instance>:5480

2. Login with username administrator and the password configured in set-user-variables.sh.

Click Master.

The Configure Master window appears.

Enter an instance name (this name will eventually display in the Mashery Admin Dashboard) that is meaningful to your operation, the Mashery Cloud Key and shared secret provided by TIBCO Mashery, and the NTP server address, if used.

24

(25)

If you have multiple clusters, the Mashery Cloud Key and shared secret provided by TIBCO Mashery should be unique to each of your clusters. Mashery Local clusters should not share keys.

3. Click Commence Initiation Sequence.

After the Master initializes with the Mashery cloud service, a completion page appears.

4. Click Continue.

5. Navigate to the Cloud Sync page and perform manual syncs for API Settings and Developers by clicking the adjacent icons:

(26)

6. Test the instance as described in Testing a New Instance.

7. See the instructions in Advanced Configuration for how to enable notifications, if desired.

Configuring Slaves to the Local Master

Mashery Local may run configured in a cluster of one master and multiple slaves.

To configure slaves to the master:

Procedure

1. Browse to the Mashery Local Cluster Manager of the slave by using the Docker Host IP address of the instance:

https://<IP_address_of_instance>:5480

2. Login with username administrator and the password provided by TIBCO Mashery.

3. Click Slave.

26

(27)

4. Enter an instance name (this name will eventually display in the Mashery Admin Dashboard) that is meaningful to your operation, the Mashery Cloud Key and shared secret provided by TIBCO Mashery, and the NTP server address, if used.

(28)

5. Click Register with Mashery and Master.

6. Click Continue.

7. Test the instances as described in Testing a New Instance.

8. See the instructions in Advanced Configuration for how to enable notifications, and API and JMX reporting access, if desired.

Configuring the Load Balancer

TIBCO Mashery recommends using a Load Balancer to best utilize the cluster, although this is not required because you may route your API traffic directly to each instance.

Each instance hosts a service called /mashping. Configure the Load Balancer to access the following address, without the host header:

http://<IP_address_of_instance>/mashping

If the Load Balancer and the cluster is working correctly, /mashping returns the following response:

HTTP/1.1 200 OK Server: Mashery Proxy

Content-Type: application/json; charset=UTF-8 Transfer-Encoding: chunked

{"status":200,"time":1315510300,"message":"success"}

If /mashping returns any other response, then the load balancer should remove the instance from the cluster and either retry after a period of time or alert operations to investigate.

Mashery Local has two instance types: Master and Slave. Should the Load Balancer pull the Master out of the cluster pool, an Operations engineer should immediately investigate whether it can be recovered,

28

(29)

and, if not, promote a Slave to Master. Taking offending Slaves out of rotation through the Load Balancer can mitigate any traffic impact. If no Master exists in the pool, data synchronization with the Mashery Cloud Service will not occur with the exception of API event activity. Access Tokens, Keys, Applications, Classes and Services will not be synchronized.

For steps on how to promote a Slave to Master, see Promoting a Slave to Master.

Configuring the Instance

The Instance Management tab allows you to configure additional settings for that particular instance. You can edit the instance name, configure instance settings, and update software and custom adapters. Additional system-level parameters can be tuned here such as application memory allocation, configuration cache size, maximum concurrent connections, and connection pool size for the database.

To configure an instance:

Procedure

1. On the Mashery Cluster Manager tab, click Instance Management.

2. Click the Management Options for which you want to configure the settings.

A text box is displayed for the selected Management Options.

3. Enter the details for the following fields to configure the instance.

(30)

Field Description Use NTP

(recommended) NTP server address.

Memory Allocation Specify application memory size as a fraction of the available memory.

Concurrent

Connections Sets the maximum number of concurrent connections to the service instance.

Database Connector Sets the maximum number of concurrent connections the instance will make to its database.

Configuration Cache Specify the memory (in MB) to use for configuration cache.

Disable IPv6 Select this option to disable IPv6 if IPv6 traffic should not be allowed to the backend. By default, Mashery Local supports both IPv4 and IPv6.

4. Select the appropriate HTTP Server Security Level:

30

(31)

● Enable HTTP only: If selected, the default HTTP Port for HTTP Server Security Settings is ⁸⁰.

● Enable HTTPS only: If selected, enter the details for the following fields:

Field Description

HTTPS Port Specify the HTTPS port. The default is 443.

(32)

Field Description Certificate Common

Name (display only) Automatically displays the name of the selected certificate.

Certificate # (display

only) Automatically displays the number of the selected certificate.

New SSL Certificate Select from:

● Create new certificate: If selected, enter a Certificate Common name in the Create SSL Certificate window, then click Create.

● Upload new certificate: If selected, in the Upload SSL Certificate window, browse to the SSL certificate using the Click here to select file link, enter the Password for Certificate, then click Upload.

Download SSL

Certificate Select from:

● Download certificate in PEM: downloads the current certificate in PEM format.

● Download certificate in DER: downloads the current certificate in DER format.

32

(33)

● Enable HTTP and HTTPS: If selected, enter the details for the following fields:

HTTP Port Specify the HTTP port. The default is ⁸⁰. HTTPS Port Specify the HTTPS port. The default is ⁴⁴³. Certificate Common

Name (display only) Displays the name of the selected certificate.

Certificate # (display

only) Displays the number of the selected certificate.

(34)

New SSL Certificate Select from:

● Create new certificate: If selected, enter a Certificate Common name in the Create SSL Certificate window, then click Create.

● Upload new certificate: If selected, in the Upload SSL Certificate window, browse to the SSL certificate using the Click here to select file link, enter the Password for Certificate, then click Upload.

Download SSL

Certificate Select from:

● Download certificate in PEM: downloads the current certificate in PEM format.

● Download certificate in DER: downloads the current certificate in DER format.

5. Click Save.

You may be reminded that Mashery Local needs to restart proxy service.

The instance is configured for the specified settings.

What to do next

For detailed steps on setting up HTTPS Server, please refer to the following sections in the Appendix:

● Setting up HTTPS Server using Self-Signed Certificate

● Setting up HTTPS Server using Customer-Provided Certificate

34

(35)

Promoting a Slave to Master

Promoting a Slave to Master is important from within a cluster, and having multiple clusters (using unique MoM keys) connecting to the same area is High Availability. Taking offending Slaves out of rotation through the Load Balancer can also mitigate any traffic impact.

To promote a Slave to Master:

Procedure

1. Log into the Slave instance.

2. On the Mashery Cluster Manager tab, click Instance Management.

3. In the Promote to Master section, click Promote to Master.

(36)

4. Log into the other Slaves, go to Instance Settings in Instance Management, and in the change the Master Instance IP address to the new Master's IP address.

36

(37)

5. (Optional) Delete the old Master or shut down that Virtual Machine.

TIBCO Mashery®

Installation and Configuration Guide for Docker

Contents

TIBCO Documentation and Support Services

Introduction

Assumptions

Conventions

Deployment Topology

Overview of Installation and Configuration Process

Installing and Configuring Mashery Local for Docker

Required Docker Images

Installing Mashery Local for Docker

Additional Installation Tips

Installation Troubleshooting Tips

Managing Docker Containers

Installing and Running Mashery Local for Docker with Kubernetes

Customizing for Kubernetes

Configuring the Mashery Local Cluster

Configuring a Mashery Local Master

Configuring Slaves to the Local Master

Configuring the Load Balancer

Configuring the Instance

Promoting a Slave to Master