Demos used for "CrashLoopBackoff, Pending, FailedMount and Friends" talk at KubeCon 2017

You can download a ZIP file of all the demos for easy local use; otherwise you will have to download all the files individually and recreate the directory structure. The ZIP file link should always point to the server copy, while other links in this document are relative so they should work if browsing this index locally.

(Note, these are not exactly the same as the files used in the KubeCon 2017 demo -- I cleaned them up a bit for greater clarity.)

Each of these demos has two directories: "broken" which you can use to deploy the initial broken configuration, and "fixed" which when applied should result in a working deployment. The install is as easy as:

kubectl apply -f [demo directory, e.g. crashloopbackoff]/broken

You can then investigate the failure with various kubectl commands. To fix it:

kubectl apply -f [demo directory]/fixed

To clean up:

kubectl delete -f [demo directory]/fixed -f [demo directory]/broken

You can also (and I encourage you to) try to fix the broken components yourself, then compare to my fixed versions.

Prerequisites: a working Kubernetes cluster in a relatively untouched state (with RBAC enabled if you want to try the RBAC demo) that you aren't doing anything important or sensitive on (some of these demos spawn busybox containers, display data gathered from the API server, etc. -- don't do these demos in production!)

Things that may interfere with the demos functioning as desired:

• NetworkPolicy rules
• RBAC rules
• Tight resource restrictions

Demos:

• CrashLoopBackoff: the broken deployment has a busybox container that fails to start and just crashes. The fixed deployment has a container that sleeps for 3600 seconds.
• Discovery failure: the broken deployment has a Weave Scope agent DaemonSet that fails to report to the frontend because it is trying to contact it on the wrong service hostname. You can port-forward a browser connection to observe the effect on the frontend (when the DaemonSet is broken, it reports nothing):

  kubectl port-forward $(kubectl get po -o jsonpath={.items..metadata.name} -l app=discoveryfailure,component=frontend) 4040

  After starting the port forward, browse to http://localhost:4040/ and you should see the Weave Scope UI.

• FailedMount: the broken deployment tries to mount a host directory that does not exist.
• ImagePullError: the broken deployment is trying to pull an image tag that doesn't exist.
• Pending Pod: the broken deployment tries to create a pod with a memory request that most hosts cannot provide. (If your host has 80 TB of spare memory, this "broken" deployment may succeed for you!) The fixed deployment does not request a specific amount of memory.
• RBAC Issue: the broken deployment's DaemonSet is not actually broken, but it cannot gather info on the cluster through the API server because its ServiceAccount has no permissions. The fixed deployment simply adds the necessary ClusterRole and ClusterRoleBinding. You can observe the effects by port-forwarding to the Scope UI:

  kubectl port-forward $(kubectl get po -o jsonpath={.items..metadata.name} -l app=rbacissue,component=frontend) 4040

  Note that in the "broken" status you will see items gathered from the hosts directly, but not things that require access to the API server like pod and service listings (until you fix the RBAC issue). Once you fix RBAC, the agent pods may still be in a backed-off status (they will not restart since you did not change the DaemonSet itself); either wait, or simply delete them and let them be recreated.

Happy debugging, and if you have questions about these demos, feel free to contact me on Kubernetes Slack (@kensey), Twitter (@caffeinepresent), or through e-mail (kc2017@orion-com.com).

Joe Thompson