Description

We recently went about adding windows nodes for some legacy dotnet stacks at Ancestry. As part of this we followed the AWS documentation to enable windows. We also use Karpenter to handle the scheduling and decided to make a new provisioner for just windows. After following the docs a new windows node came up but there was an error that prevented the pods from obtaining an ip address.

Warning FailedCreatePodSandBox 3m21s (x4555 over 19h) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a760b4d93ed0937341cb5083547b0b8a197a280a66ad3d0cb096562ab2a237a1": plugin type="vpc-bridge" name="vpc" failed (add): failed to parse Kubernetes args: failed to get pod IP address windows-test-5bbc88b5f9-vzgfc: error executing k8s connector: error executing connector binary: exit status 1 with execution error: pod windows-test-5bbc88b5f9-vzgfc does not have label vpc.amazonaws.com/PrivateIPv4Address

Solution

My suspicion was there was something in the control plane that can’t be seen from kubectl. After talking with AWS support through several rounds there is an admission webhook in the control plane that is triggered by setting the nodeSelector. The docs say that needs to be set, but I thought it was just to set nodeaffinity and not the other field.

nodeSelector:
        kubernetes.io/os: windows
        kubernetes.io/arch: amd64

Once we added this to the pod spec section of the deployment yaml the pods were able to successfully be triggered by the mutating webhook to add the right annotations vpc.amazonaws.com/PrivateIPv4Address to the pod and it was able to get an ip address succesffully.