After changing my company laptop to a MacBook Pro with Apple silicon, I had to overcome difficulties setting up a working Minikube environment. A combination of two factors created the challenges:

1. Supported drivers: Once you remove the drivers, which do not have support for Apple Silicon or require a separate license for commercial use, only QEMU and SSH remain as possible options.

2. Cisco AnyConnect VPN with Umbrella: Cisco AnyConnect installs a local dns proxy listening on port 53. Unfortunately, QEMU has issues with the Cisco AnyConnect setup. (see QEMU stops working with minikube. The suggested workaround to install socket_vmnet is marked as experimental, and I couldn’t get it to work on my machine.

As an act of desperation, I wanted to try out Minikube with podman as its driver. Unfortunately, Minikube does not list podman as a supported driver for macOS, but I had nothing to lose.

And it works like a charm 🎉! Follow these steps to setup minikube with the podman driver on your Apple computer. (It works on Intel Macs as well.)

• podman
  • client version: 4.3.1
  • server version: 4.3.0
• Minikube
  • version: v1.28.0
  • commit: 986b1ebd987211ed16f8cc10aed7d2c42fc8392f

Setup podman

1. Install podman with brew: brew install podman
2. Initialize podman and prefer rootful container execution : podman machine init --rootful --cpus 4 --memory 8192
3. Start podman: podman machine start

Note:

• rootful resolves various network problems, when a pod needs to access external services.
• Minikube minimal requirements: 2 CPUs or more, 2GB of free memory 20GB of free disk space

Setup Minikube

1. Install Minikube with brew: brew install minikube
2. Start minikube: minikube start --driver=podman --cpus=max

Note:

• If you run into strange issues during minikube start, clean-up with minikube delete --all --purge and try again.
• Set podman as the default driver: minikube config set driver podman

Verify Setup

1. Create a new pod: kubectl run test --image=docker.io/alpine:3 -- sleep 86400
2. Check Internet connectivity with wget: kubectl exec test -- wget -S --spider https://www.google.com
3. Remove test pod: kubectl delete pod test

I just came across the amazing Windows Subsystem for Linux 2 (WSL2) feature which allows you to create a consistent Linux environment.

If you have not came across WSL2 yet, Microsoft advertises it as:

The Windows Subsystem for Linux lets developers run a GNU/Linux environment – including most command-line tools, utilities, and applications – directly on Windows, unmodified, without the overhead of a traditional virtual machine or dualboot setup.

Although, Microsoft provides some Linux distributions through its Windows Store, this leaves the question open what you should do if your favorite distro is not available over this channel, and how to achieve a consistent setup.

The great news is that you can import an exported Container image in WSL2 as your own distro. Microsoft’s documentation Import any Linux distribution to use with WSL shows this with vanilla CentOS 8. But let’s take this a little bit further and add customizations.

We are going to use CentOS Stream 8 as our base distribution. Then we install some additional tooling including zsh and oh my zsh. And of course we do this in a single Dockerfile.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

FROM quay.io/centos/centos:stream8
ARG USERNAME
ARG PASSWORD
RUN dnf update -y \
        && \
    dnf install -y \
        epel-release \
        && \
    dnf install -y \
        git \
        man \
        passwd \
        powerline-fonts \
        sudo \
        util-linux-user \
        zsh \
        && \
    dnf clean all -y \
        && \
    adduser -G wheel ${USERNAME} && \
        echo -e "[user]\ndefault=${USERNAME}" >> /etc/wsl.conf && \
        echo ${PASSWORD} | passwd --stdin ${USERNAME}
USER ${USERNAME}
RUN curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh | sh - 

1. We build this with: docker build -t wsl-centos:stream8 --build-arg USERNAME=veit --build-arg PASSWORD=password .

2. Now we need a container instance which we can export afterwards: Create a container instance: docker run -t wsl-centos:stream8 bash ls /

3. List all container instances docker ps -a.

   

4. Copy the container ID to the clipboard.

5. Export the container to a tar file: docker export --output="centos-stream8.tar" [CONTAINER_ID]. Paste the container ID from the previous step into this command.

Now let’s configure the WSL2 environment on Windows:

6. Windows needs a storage location for your running WSL. The storage location should be fast and have enough space for the VM which will be created. I will put this in my home directory: md C:\Users\veit\wsldistrostorage\centos-stream8

7. We can import the tar file into WSL: wsl --import [DISTRO NAME] [STORAGE LOCATION] [FILE NAME].. e.g. wsl --import CentOS-Stream8 C:\Users\veit\wsldistrostorage\centos-stream8 centos-stream8.tar

8. Finally, lets start the new WSL distro: wsl -d CentOS-Stream8

Of course this works with any Linux distribution for which you can find a Docker base image. And the nice part is, that you can create a repeatable setup with the Dockerfile.

In my post Friends don’t let friends run containers as root, I took a simplified view that Kubernetes does not add any security policies. I went for this simplified statement because, when I wrote the post, Kubernetes 1.22 was moving away from Pod Security Policies, and there had not been any replacement in a released version.

Beginning with Kubernetes 1.23, Pod Security Standards are introduced to replace Pod Security Policies. Kubernetes administrators can choose between the policy profiles privileged, baseline, and restricted. The policies are cumulative, which means that baseline contains all the rules from privileged, and restricted has all rules from privileged and from baseline.

See Kubernetes - Pod Security Standards for more details.

Note: On the Kubernetes website, the words policy profile and level are used interchangeably. The term policy profile is used for a set of policies. Whereas the term level is used when Pod Security Standards are enabled. I am going to follow this pattern in this post as well.

Select the Security Profile to Prevent Pods Running as Root

To prevent any Pod from running as root, we have to select the profile restricted. Restricted requires the fields

• spec.securityContext.runAsUser
• spec.containers[*].securityContext.runAsUser
• spec.initContainers[*].securityContext.runAsUser
• spec.ephemeralContainers[*].securityContext.runAsUser

to be either null or set to a non-zero value. This is precisely what we want. Restricted has more rules you need to comply with. Please review the list of policies included in the restricted profile to get a better understanding of the other rules.

As you can see, even with the profile restricted, the requirement is that you cannot run the container with user ID 0. This means that you could run the pod with user IDs 1, 500, or 1827632. And, if you set the runAsUser to null, then the user ID defined in the image will be used. If that is the root user, you will get an error message like “Error: container has runAsNonRoot and image will run as root”.

Enabling Pod Security Standards

Pod Security Standards are a somewhat optional feature. You need to configure them per namespace by applying a label to the namespace. The label defines which policy profile should be used and what Kubernetes does with a Pod if a policy violation has been detected. The mode can be one of the following:

The label you apply to the namespace takes the form pod-security.kubernetes.io/<MODE>: <LEVEL>. To require all pods to comply with baseline level, you need to apply the label pod-security.kubernetes.io/enforce: baseline. If you want to inform the users and write the violation to the audit log, you can also add labels for this. In the end, you end up with the following namespace manifest:

1
2
3
4
5
6
7
8

apiVersion: v1
kind: Namespace
metadata:
  name: default
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/audit: baseline
    pod-security.kubernetes.io/warn: baseline

Conclusion

The Pod Security Standard is an excellent next step after Kubernetes deprecated Pod Security Policies in Kubernetes v1.21. Currently, the feature is in the beta stage, and an administrator has to configure it per namespace. Once you begin using profiles baseline and restricted, you have an effective tool to make the cluster more secure.

The recommendation I give in my post Friends don’t let friends run containers as root changes only slightly for Kubernetes:

• Kubernetes:
  • For running a plain container in Kubernetes, you don’t have to do anything. Kubernetes starts the container with the user ID set by the USER instruction in the Dockerfile.
  • The Deployment manifest might need additional configurations depending on the selected Pod Security Standard profile.