When using Coder, you may encounter the following error:
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:370: starting container process caused:
process_linux.go:459: container init caused: join session keyring:
create session key: disk quota exceeded: unknown.
Why this happens
The kernel allocates a system key for each container created. When lots of developers are sharing the same instance, you may run into limits on the number and size of keys each user can have.
Resolution
To fix this error, you can increase maxkeys and maxbytes. These are global
settings that apply to all users sharing the same system. You can modify this
by adding the following to the sysctl configuration file:
sudo sysctl -w kernel.keys.maxkeys=20000
sudo sysctl -w kernel.keys.maxbytes=400000
Alternatively, you can use a DaemonSet with kubectl apply to make changes to
sysctl:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: increase-limits
namespace: kube-system
labels:
app: increase-limits
k8s-app: increase-limits
spec:
selector:
matchLabels:
k8s-app: increase-limits
template:
metadata:
labels:
name: increase-limits
k8s-app: increase-limits
annotations:
seccomp.security.alpha.kubernetes.io/defaultProfileName: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
nodeSelector:
kubernetes.io/os: linux
initContainers:
- name: sysctl
image: alpine:3
command:
- sysctl
- -w
- kernel.keys.maxkeys=20000
- kernel.keys.maxbytes=400000
resources:
requests:
cpu: 10m
memory: 1Mi
limits:
cpu: 100m
memory: 5Mi
securityContext:
# We need to run as root in a privileged container to modify
# /proc/sys on the host (for sysctl)
runAsUser: 0
privileged: true
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
containers:
- name: pause
image: k8s.gcr.io/pause:3.5
command:
- /pause
resources:
requests:
cpu: 10m
memory: 1Mi
limits:
cpu: 100m
memory: 5Mi
securityContext:
runAsNonRoot: true
runAsUser: 65535
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
terminationGracePeriodSeconds: 5
At a later point, you can delete the DaemonSet by running:
$ kubectl delete --namespace=kube-system daemonset increase-limits
daemonset.apps "increase-limits" deleted
However, note that the setting will persist until the node restarts or another
program sets the kernel.keys.maxkeys and kernel.keys.maxkeys settings.
If this doesn't resolve the issue, please contact us for further support.