Clone job fails with ImagePull error

Cloned-restore fails on Openshift clusters with ImagePull error

Description: Cloned-restore Job failed to be executed

In Openshift 4.x, pods can pull images with the use of ImageStreams. ImageStreamTag represents an image that is retrieved by tag name from an ImageStream. If these are hard-coded in a service account, pods in the target namespace will fail to come up with an ImagePullError.  

e.g. 
Namespace = testify-ps 
SA = default
Target namespace = testify-ps-restore

Error: Failed to pull image

Cloned-restore jobs time-out and eventually fail due to an ImagePullError. Upon further investigation, the root cause is because the default SA (in the target namespace) does not have ImageStreams for the pod to pull an image from a private registry.

e.g. Failed to pull image "image-registry.openshift-image-registry.svc:5000/testify-ps/runner@sha256:9317d61bb7943fb2b3f0d62c37aab233a07a3e0ccc67b104a7e3e69cb778dea0": rpc error: code = Unknown desc = Error reading manifest sha256:9317d61bb7943fb2b3f0d62c37aab233a07a3e0ccc67b104a7e3e69cb778dea0 in image-registry.openshift-image-registry.svc:5000/testify-ps/runner: unauthorized: authentication required

Resolution

There are two ways to overcome such failures:

1. Identify missing ImageStreamTag(s), then go through steps 2, 3 or 4:

   e.g. Source Namespace
   $ oc get istag -n testify-ps
   
   NAME             IMAGE REFERENCE                                                                                                                            UPDATED 
   mgmt:develop     image-registry.openshift-image-registry.svc:5000/testify-ps/mgmt@sha256:092577fb9c72f7631b08c1f96658b34a5f928953f59ff8e1d38e91a5a64ef918   6 days ago 
   mongo:4.2        mongo@sha256:8c48baa1571469d7f5ae6d603b92b8027ada5eb39826c009cb33a13b46864908                                                              6 days ago 
   runner:develop   image-registry.openshift-image-registry.svc:5000/testify-ps/runner@
   
   Target Namespace 
   $ oc get istage -n testify-ps-restore
   
   NAME             IMAGE REFERENCE                                                                                                                            UPDATED 
   mongo:4.2        mongo@sha256:8c48baa1571469d7f5ae6d603b92b8027ada5eb39826c009cb33a13b46864908                                                              2 days ago

    

2. Manually add them to the default SA in the target namespace and restart pods.

   e.g 
   $ oc project testify-ps-restore
   $ oc edit serviceaccount default 
   
   <Add the missing mongo:4.2 secret>
   
   $ oc delete pod <pod name>

3. Run the script below to pull ImageStreams from one namespace to another.

   $ oc policy add-role-to-user \ 
   system:image-puller system:serviceaccuont:testify-ps:default \ 
   -namespace=testify-ps-restore

   More details can be found here
   
   
4. Alternatively, the K10 transforms feature can be used to add missing secrets to the default SA as part of the restore process.