The POD fails to start, and referrers to the image layer not known. The “layer not known” issue may affect one or more cluster nodes. In effect, there is a corrupt docker image on the local disk cache. The layer not known problem still exists, even after a node or cluster restart.

Image layer not known

The CRI-O container runtime v1.18.2 causes this problem. Here, the developer optimized the code for more performance. Unfortunately, this optimization, in some cases, does not flush file operations. As a result, image metadata is missing or corrupt. Accordingly, the underlying docker image becomes unusable. Openshift OKD 4.5 and event 4.6 nightlies become affected.

CRI-O v1.18.2 image issue

Failed to create pod sandbox: rpc error: code = Unknown desc = error creating pod sandbox with name "XXXX": layer not known

Layer not known affected CRI-O versions

The GitHub issue 4285 references this problem. As a result, the CRI-O v1.18.4 solves this problem. Also, OKD 4.6 should include a fixed v1.19. Therefore, you should think of upgrading.

Manually fixing corrupted images.

A suitable workaround to fix the problem is to delete images from the node. After that, the container platform can load the missing docker images from the docker registry. In effect, the pods find valid docker images and start.

delete all image from cluster node

systemctl stop kubelet
systemctl stop crio
rm -rf /var/lib/containers/
systemctl start crio
systemctl start kubelet

Conclusion

For more details, refer to the Redhat Bugzilla entry. In my case, the deletion of all images solved all problems. Think of upgrading to the next OKD release. On a productive system, the occurrence of the error is annoying. But it is at least solvable. Another article describes a similar problem. But this problem is not so easily solvable and, together with other problems, can lead to a total failure.

 

Interested in more postings?

New: OKD Docker Image is stuck – Operation not possible

Openshift and also OKD Docker image is stuck when loading. A severe bug in the CRI-O engine causes stuck the OKD Docker images in an invalid and unusable state. There are discussions about timeouts while loading the images from the docker registry or too long...

Windows Subsystem for Linux and Minikube

The Windows Subsystem for Linux is the seamless integration of Linux into Windows. Use Windows natively and quickly issue a Linux command. Apply a Linux command to the Windows file system without having to start a virtual machine. As a result, Linux is always...

The Docker daemon configuration files

Where are the Docker daemon configuration files located? How to restart the Docker daemon after applying changes to the configuration? How to change and activate the Docker configuration? These are frequently asked questions. But changes to the Docker configuration...

Docker Content Trust

Docker Content Trust feature enables your environment to run only with signed images. In this way, Docker Content Trust ensures that the docker pulls only signed containers from the docker registry. Once enabled, Docker Content Trust is active for all docker pull...

Docker, networks, subnets and IP address pools

Docker uses default address pools to create subnets. For most use cases, the shipped defaults fit. But sometimes they cause conflicts with existing networks or subnets. Overlapping networks may conflict with existing systems. Or a large number of docker networks...

Docker networks and subnets

Docker uses default address pools to create subnets. For most use cases, the shipped docker subnet defaults fit. But sometimes the docker subnets cause conflicts with existing networks or subnets. Overlapping networks may conflict with existing systems. Or a large...

Software containerization with docker reviewed

Docker software containerization reviewed Putting Software into containers seems to be state of the art. But what are the benefits? Are there any drawbacks? Most people have heard about docker technology. And not less have used docker. But we are looking towards more...

Docker process virtualization

Docker is a lightweight framework for virtualizing application processes. Instead of emulating a computer hardware that still needs an operating system to run applications, Docker takes a different approach. Docker is able to pretend an operating system environment to...