I'm always looking for ways to make deployment and management of Tableau Server easier, so I was super excited when the devs at Tableau announced the release of Tableau Server in a Container. Since then, I've spent a lot of time figuring out the right way to deploy Tableau Server using Kubernetes. My end goal is to deploy a highly available Tableau Server environment on any mainstream implementation of Kubernetes, using spot instance pricing — without significant application downtime during node failures. It's a tall order, but I'm getting there! Read on to find out what I've learned along the way.
Building Tableau Server Container Images
Building Tableau Server Container Images
One of the first things you'll notice when looking at Tableau Server in a Container is that it requires you to build your own container images. It seems odd at first, but it does make sense from a few perspectives:
- Customers require different drivers for their environments
- Many drivers are licensed such that Tableau can't package them
- Customers may be using technologies that don't offer other means of injecting drivers into containers; for instance, a Docker container doesn't allow for InitContainers.
Still, it would be nice if Tableau provided a base image with which customers could add any necessary drivers, scripts, and other customizations. I'm likely to implement this model myself in the meantime.
Tableau Server Container Image Size
The next thing you'll notice immediately after building your first image is how insanely large it is! For the most recent version or two of Tableau Server, my images are nearly 6GB. Large images can be a problem when dealing with spot instances and auto-scaling because it increases the time for new nodes to take over the work from old nodes. Still, it's not too bad if the images are close to the Kubernetes cluster. In my case, I've opted to store my Tableau Server images on Amazon ECR rather than Docker Hub. That download takes just over a minute on new nodes.
I haven't spent much time trying to reduce the image footprint to date, though I'd like to try a couple of things. If that ever makes it to the top of my to-do list, I'll share my findings.
Upgrading Tableau Server in a Container
So far, Tableau provides a couple of upgrade options in the documentation:
- Create a special "upgrade" image, using both your current and desired versions of Tableau Server... yuck. And,
- Take a backup of your current environment, and restore that to an environment using the new images. Also yuck.
I can think of a few ways to improve this process using Kubernetes functionality, which is my next project. I'll write an article on the process once I've got something worth sharing.
Tableau Server Kubernetes Manifests
Tableau was kind enough to provide a GitHub repository with manifest files for running Tableau Server on Kubernetes. I appreciated the resource, and I used it as a starting point for my manifests. Here are some of the things I've added/changed so far:
- Combined all Tableau nodes pods into a single StatefulSet – it fits the use case almost perfectly.
- Use volumeClaimTemplates for the data directory volumes instead of making the PVCs manually.
- Rewrote the startup and initialization logic to account for clusters of varying sizes. There's plenty more to do here, but it's a good start.
- Rewrote the readinessProbe check—it's almost right, now. Different pods will have different processes, and I want to account for those differences dynamically, whatever they might be.
- Added podAntiAffinity to ensure pods are scheduled on separate Kubernetes nodes. We're trying to tolerate pod failures gracefully in order to use spot instance pricing, so we need to minimize the impact of such failures.
- I wanted to avoid needing a ReadWriteMany PVC, so I rewrote the bootstrap.json file process for multi-node environments. More on this below.
- Added a preStop lifecycle hook to failover the repository if it's running on a pod that's terminating. More on this below, too.
- I implemented this with kustomize so I can more easily deploy and manage different environments from the same base manifests.
Using the custom manifests, I tested deploying several 3-node HA clusters to GKE and EKS and a couple of 7-node clusters on EKS as well. I didn't need to make any changes between GKE/EKS. The only differences between the 3-node and 7-node clusters were the config.json configmap and the number of replicas. The Vizstack team is building a tool to help create a valid config.json file, which will make this process easier!
Tableau Server Cluster Initialization
Initializing a Tableau Server cluster is pretty straightforward after the image is built and the configuration + manifests are ready. Once the manifests are applied, a handful of things happen:
- The initial node (pod 0 of the StatefulSet) is deployed and goes through an initialization script. If your configuration file's topology includes appzookeeper, that's filtered out at this stage because we always need to start with exactly one instance of it.
- After pod 0 initializes, the script determines how many nodes your config.json file specifies and waits for those nodes to register. All other pods in the StatefulSet will need to register at this time, which is why my manifests don't yet support OrderedReady pod management – all pods start in parallel.
- The other pods in the StatefulSet will be waiting for a "bootstrap.json" file that will allow them to register with the initial node pod. Tableau's manifests implied using a ReadWriteMany volume to distribute this file, but it's easier to get directly with the tsm command-line utility. This method requires you to build your image with TSM_REMOTE_UID and TSM_REMOTE_USERNAME variables set. But it makes the whole thing way easier to deploy, in my opinion. I shared this feedback directly with the dev team at Tableau.
- Once the other pods register, the initial pod continues with the cluster configuration. It configures and deploys all the services specified in your config.json file, except for the coordination service.
- After the configuration is applied, the initialization script will review your config.json file for your desired coordination service config. If you've specified a 3- or 5-node coordination service ensemble, it will deploy properly using the tsm topology deploy-coordination-service command, as is tradition.
- Finally, it starts the services and configures the initial user.
There are a few aspects of this workflow that I'll be improving upon in my manifests. Right now, the initial node handles all configuration tasks, so we have to provide the complete configuration at the very beginning. It has no mechanism for detecting and implementing changes to the configuration. Since I've already verified that remote tsm works, I'd like to implement the initialization such that each node is responsible for the changes it introduces to the cluster. That way, if I want to scale up or down, I can do it declaratively.
Tolerating Pod Failure
Since one of my main goals is using spot instance pricing for Tableau Server environments, I need to address the elephant in the room. Spot instance pricing saves a significant amount of money, with one very notable drawback: cloud providers have the option of reclaiming instances with barely any notice.
- Amazon EC2 Spot Instances will give you a two-minute warning before reclaiming compute capacity.
- Google Cloud Spot VMs will give you a 30-second warning before preemption.
- Azure Spot Virtual Machines will also give you a 30-second warning before eviction.
- It's not totally accurate, but I'm just going to refer to this event as "node failure" because cloud providers seem to avoid using the same terminology.
Since Tableau Server's startup time is longer than any of those timeframes, we can't just start a new pod when receiving that warning. To benefit from spot pricing, we need to make Tableau Server capable of tolerating some amount of failure. We can configure Tableau Server for high availability by ensuring we have three or more pods in a cluster and configuring instances of each process across multiple nodes. Even configured for HA, there's still some impact if the Active Repository process goes down — the application can become unavailable for five minutes before signaling the Passive Repository to take over. We can do better than that since we have a bit of advance notice!
Here's how I've done it so far:
- Implement a preStop lifecycle hook that determines whether the pod is responsible for Tableau Server's active repository and issues a failover command if so. This process takes seconds, not minutes.
- Implement a podAntiAffinity rule to ensure that no two Tableau Server pods get scheduled on the same Kubernetes node simultaneously.
- Use the cloud provider's auto-scaling functionality to request additional spot instances whenever we need more.
- Configure the auto-scaling functionality to select from many different compatible instance types, ensuring we don't run into issues finding a machine when we need one. I do this on AWS... I'm not sure how it works for other cloud providers yet.
I've tested this, and I've been pleased with how well it works. I want to do more tests, though. At worst, recovery times are still better than traditional Tableau Server HA failover recovery... but they need to be better since node failures are basically guaranteed when using spot instances.
Oh. There's one other thing, and this one's a doozy...
Tableau. Still. Requires. Static. IP. Addresses.
There. I said it. It doesn't feel good to acknowledge this problem, but we need to face it head-on to succeed with Tableau Server on Kubernetes.
If a pod terminates on one node and starts running on another, it'll get a different IP address. That's a problem for Tableau Server, and if not addressed appropriately, it would result in the pod being unable to take on much of its work. There are a couple of unique ways to get past this, each with its pros and cons:
Use static IP addresses like Tableau tells us to:
- It's not easy to implement static IPs in Kubernetes, but it can be done.
- We must use a Container Network Interface (CNI) and IP Address Management (IPAM) plugin that supports static IP addresses. I've tested Calico CNI + Calico IPAM on AWS with success.
- It requires a more significant setup before deploying Tableau Server – one does not simply change a CNI and IPAM plugin.
- This model keeps us from using StatefulSets because we need to annotate each pod with a static IP address.
- Because the static IPs follow the pods, this model is compatible with spot instances and more frequent pod termination.
Use dynamic IP addresses and instruct Tableau Server to get over it:
- Again, it's not easy to do, but it can be done.
- Any time a pod changes IP addresses, we need to reconfigure Tableau Server.
- We don't need a more significant setup before deploying Tableau Server.
- That reconfiguration is done with "tsm pending-changes apply" – even if there are no pending changes. This isn't documented anywhere, but it works.
- It's the "tail wagging the dog," so to speak. One pod changes IP addresses, and services on every other pod need to be reconfigured because of it.
- Since "tsm pending-changes apply" requires a server restart, there will be application downtime.
- Because of that downtime, it's not ideal for using spot instances or frequent pod termination.
My Thoughts, In Summary
It took quite a bit of engineering and customization, but I can now consistently and successfully deploy Tableau Server on Kubernetes. I've got some more testing regarding spot instances and node failures, but it looks like that approach will work. There's a lot of room to improve the whole experience, but I'm excited about the opportunities that Tableau Server on Kubernetes will bring!