An alternative is Rancher. It can provision clusters with the major cloud providers' managed Kubernetes services as well as with virtually any infrastructure. I wrote a post on setting up a cluster with code using Terraform, Ansible and Rancher https://vitobotta.com/2019/10/14/kubernetes-hetzner-cloud-te... - with this approach you can have more resources for less money compared to managed services. Having said that, I am now using Digital Ocean's managed Kubernetes service because managing a cluster can be quite a bit of work for a single person like me. By the way there's also K3s (also from Rancher) which makes it easier to spin up a cluster on your own servers.
Thanks for sharing your project (as well as Hetzner cloud which are indeed great).
Our target was to use official (vanilla) k8s, the official installer (kubeadm) and the official helm charts for all things we deploy.
While it would also work on a Raspberry Pi, indeed k3s would be more appropriate. We wanted to have something with the same functionality and HA, and tools like in a cloud k8s: GKE/EKS - but for people that like to own it, and use on-prem (or something like your cloud).
Getting (aka renting) a k8s from a cloud vendor is probably the most common case, but does not cover everyone's scenarios => hence this project.
Along the years we actively looked for best tools and practices and incorporated them in this project. We try to bring it closer to a common/initial generic k8s based platform. (it does not plan to compete with RH's OKD, but only it's basic features: out of the box network, ingress controller, monitoring, HighAvailability, etc).
Authentication, better security hardening, logging are in future plans.
Do you find it useful now, when one can simply pay jump to gke/eks/aks/pks ?
If so, what should be the next steps to make it a successful project (measured by users and community around it)
Sugarkube [1] is an open source alternative to using Ansible for working with Kubernetes clusters. Because it's focussed squarely on Kubernetes, it runs much faster (it's written in Go), but still supports templating & hierarchical variables. It can optionally create clusters (using Minikube, Kops or EKS) as well as install and delete your applications.
When Sugarkube installs your apps, it'll create all the necessary cloud infrastructure you define. It also respects dependencies, so e.g. your Wordpress sites will only be installed once there is a database to back them. You can also choose to only install a subset of your applications (i.e. so you don't need to bother installing the monitoring stack if you just want to work on Jenkins).
Sugarkube can also tear everything down again leaving you with the same clean slate that you started with, so you can use it to spin up and tear down clusters either locally or remotely to your heart's content.
Check out the sample project [2] and accompanying tutorials [3] which demonstrate running a web cluster for a couple of Wordpress sites, and an Ops cluster containing Jenkins, monitoring and Keycloak. Sugarkube supports several advanced use cases, including creating a Kops cluster, installing Keycloak into it, then reconfiguring the Kops cluster to authenticate against the Keycloak instance running inside the cluster. That's all with one command (after you've created a workspace).
I'm happy to answer any questions since I wrote it :-)
Nice project!
Please correct me, but it looks sugarkube is mainly focused minikube or for cloud (e.g. kops/etc), right?
We chose Ansible as our primary focus is on prem, and setting up linux vms parameters, for all machines in a cluster, and Ansible sounds like a fit.
For on prem "consumable" clusters is what we also do (that's why the project accepts any size, from 1 vm cluster to thousands vms clusters).
We should look into sugarkube for AWS cluster activities - Thanks for sharing!
Sugarkube orchestrates other tools. In this case where you have to actually SSH into machines to install kubeadm then Ansible may be a better choice for actually creating the clusters (since there's probabaly no single binary that does that).
However, Sugarkube can also be used as a ready-made release pipeline for applications. In general, using Sugarkube to release applications gives you a standardised way of being able to install apps onto different clusters (local/on-prem/cloud), keeping your options open for the future. So you could use it to promote applications through various dev/testing/staging clusters to production, some of which may be on-prem, some on AWS.
If you're looking to create a common set of applications in your project (monitoring, CI/CD, ingresses, etc.), using Sugarkube would allow users to install them into clusters regardless of how they were created (either by your project via kubeadm, or by Kops/EKS with or without Sugarkube, etc), so your applications may be more generally useful to more people.
This is awesome! I've finished constructing a small repository which runs the following in order to setup a cluster: Bazel, Terraform, Packer, Ansible (which'll run `kubeadm init`), Docker, Kubernetes, Openssl, Jsonnet.
However, it is simply in terms of getting this running. No best practices, no plugins, nothing extra. I think one great thing my little project achieved is that it uses each tool for its appropriate task correctly, without creating hard coupling between tools. I made that repository so others could learn (I pushed to github today under a private repo).
I'm glad a more mature project is out there for this.
> I've finished constructing a small repository which runs the following in order to setup a cluster: Bazel, Terraform, Packer, Ansible (which'll run `kubeadm init`), Docker, Kubernetes, Openssl, Jsonnet.
It's pretty sad that in 2019, to setup a web app that can properly scale you need to understand and use so many technologies. Developing a scalable website in 2013 was a lot easier than in 2019. Back then PaaS was all the rage (Heroku, Google App Engine, Amazon Elastic Beanstalk). Since then, things have gotten much harder.
I suspect that this development has less to do with technical challenges and more to do with revenue. Cloud providers now provide all of these services, and they make little money getting small folks up and running with something that "just works". The real money is going after enterprise clients that need to tweak every detail and have the resources to devote to all of these technologies.
There have been so many platforms and libraries that have come out over the past 10 years, the industry is more than ready for a period of consolidation.
Except you need approximately zero of those things (especially Bazel. Jesus, who would pull in Bazel for this?) To run a website in 2019, you need:
1. Server
2. nginx
3. A certificate
4. DNS
The vast majority of people don’t actually need to scale at all. I got HN hugged once and my little Digital Ocean droplet with nginx performed just fine. Granted, it was a static site. But scalability is one of those things that everyone says they want, but probably don’t actually need (or haven’t really profiled to see if scaling horizontally would help).
I’ve been burned too much by devs using Docker and Kubernetes and all of this other nonsense just to host basic web backends, because I’m always the one who has to clean up the mess when it shits the bed. Web hosting needn’t be difficult, but we keep making it difficult.
> To run a website in 2019, you need: 1. Server 2. nginx 3. A certificate 4. DNS
I wasn't talking about building a static website, I was talking about building a scalable web app. You can start a project as a hobby, but if you every hope to dream of doing something larger, you should at very least consider a path to expansion. This shouldn't mean jumping into Docker and Kubernetes, which are pretty complex. There should be platforms that handle database and instance scaling automatically. When you start off, you only need one or two instances, and as you grow, new instances get automatically added.
Back in the old days of 2013, this was super easy, no Dockerfile or Kubernetes yaml files were needed. Now, you need to configure every single tiny variable for every operation. This is great for large enterprises, but miserable for small companies that want to get started.
>It's pretty sad that in 2019, to setup a web app that can properly scale you need to understand and use so many technologies. Developing a scalable website in 2013 was a lot easier than in 2019. Back then PaaS was all the rage (Heroku, Google App Engine, Amazon Elastic Beanstalk). Since then, things have gotten much harder.
Is there anything stopping you from using these? Because they're well maintained and even better than before. In fact, many companies and individuals are still using them. And if you have different needs, it's good that there are more flexible alternatives, no?
I get your point, but your looking at it backwards now, we're past the days of PaaS because we now know were it all leads: lock in -- and I say this as an early adopter of GAE and watched it turn from "something that does all you need out of the box to run an app" to something that nickel and dimes you at every turn and then presents you with a nice fat $bill if you do anything serious on the platform (just look at the bandwidth charges ffs!).
If by lock-in that means higher margins for these major cloud providers then the only remedy is to have a free market, and the only way to achieve that is for us to have the ability (or at least the threat) to take our stuff and run.
In short, the word is out now that there is no "free cloud lunch" they're making $billions off of us for what we once called hosting (clever trick!). Yes, we want the promise of "Cloud" but not all of us are SnapChat and/or willing to pay the high premium for it.
> I get your point, but your looking at it backwards now, we're past the days of PaaS because we now know were it all leads: lock in ... I say this as an early adopter of GAE and watched it turn from "something that does all you need out of the box to run an app" to something that nickel and dimes you
Lock-in is fine when you're small, it only becomes problematic when you're big. PaaS platforms are not well suited for big organizations for precisely this reason. However, they're great for trying out something new quickly and having it scale without thinking about it. But Google (and others), realized the real money is not in startups wanting a PaaS platform, but in large enterprises, which are far more sensitive to lock-in.
In the ideal world, they'd provide systems like GAE and make it easy to later convert each service (e.g., database, memcache, tasks, email) to a finely tuned system.
> I’ve found the serverless framework with Lambda to be quite straight forward.
Lambdas are great, and they do borrow much from the ideals of PaaS, including the lack of load balancing or HTTP servers. However, there are so many other extremely common services that Lambdas don't cover, for example: email, background tasks, memcache, databases, search engines, security, and so much more. Nearly every web app needs all of these services, but no one is building a single platform that does all of these things decently well out of the box. Instead, you're pushed towards using a different tool for each one. These custom tools certainly offer more power and control, but having to go chase and evaluate each one is a huge road-block for smaller projects that just want to get off the ground.
We also think the future directions are Serverless (Knative based) and Istio. They can be added using their helm charts either post install or via the addons.yaml
The idea of this project is to prepare the k8s base for deploying either of them quickly, building on top.
I’ve been trying hard to find time to setup and play with a Kubernetes cluster for learning purposes.
Considering it is built with Go, why is such a aggregation playbook needed?
Shouldn't this be the default documentation of Kubernetes?
Do the following to use Kubernetes for what it is meant for!
If you want to simply run containers in 1-5 machines - may be look at Docker/Podman + Ansible instead?
HA for controller,
Software defined networking,
Autoscaling pods,
Deployment policies,
Auto discovering DNS
Disaster recovery etc.
If these are not what you _need_, then looking at Kubernetes is probably not smart right?
This assumes infrastructure as code, production like environments, ephemeral compute, etc. It's one of the easiest ways to achieve this and it's available anywhere.
You also don't want something like software defined networking. You want the ability for services to talk to each other, and firewalling at the service level, and when you throw multiple services on shared infrastructure together, the SDN helps you do this.
When you add it altogether, pretty much everyone wants all this stuff if they're going to deliver services at a professional level, but its obviously not the only way to do it, and most people don't want to be working at the kubernetes abstraction level.
what you are looking for is called kubeadm, which is an official CNCF project, in go.
BUT:
1. kubeadm won't change your kernel params, or docker or anything on your hosts. It expects someone took care of it (and according to their docs). And that's how we started.
2. your list of features: it's not one size fits all. Not everyone wants the overhead of _all_ of these, and not all developed under CNCF.
Therefore one needs the post cluster deploy steps, with these things.
Also, things like DNS, HA does depend on your infrastructure (it's where things get hard). E.g. We tried to solve HA on prem by using keepalived.
An issue I've been having with my on-prem cluster is it seems as though all the articles, libraries, and examples are written for clusters running in the cloud. Does anyone have any resources for on-premise clusters?
Exactly to the point: this project is mainly focused to on-prem: configures kernel params, k8s HA is with keepalived (or your HW LB should you have one), etc.
Should you see something important missings, I would be happy to answer/add it.
In theory its the same as just running a data center on-prem and then running the Kubernetes cluster on top of it.
I think you would generally have the same issues as running a general data center on-prem. I have not seen any resources yet for running one specifically for k8s. I managed a beowulf cluster for a while for running one. It was a complete pain.
Hi,
exactly to the point, these are the ones included Out Of the Box: Prometheus (operator, with Grafana, alertmanager,etc), Nginx Ingress, Heapster (next will be metrics server once helm chart will be stable), Kured, etc.
Adding any service mesh on top is simply adding the helm chart and its params in the addons.yaml
Thanks for your question, very good one.
We are currently looking to find the right balance of what perms should be OutOfBox and what should be custom setup.
E.g. Running things like telepresence CNCF need sshd. And running sshd needs root (and RunAsAny).
We are working closely with them and just proposed a work version which should be fine with less perms (currently for Openshift), and then we can tighten such things also in k8s.
