Import
      Back to home
      1. Help Center
      2. Debugging
      3. Import

      Import fails with the message "chacha20poly1305: message authentication failed"

      If import failures have been observed and if the error message says "authentication failed", then this article will help you resolve it.

       

      Authentication Failure Error

      When an Import Policy in K10 runs as per it's schedule, an ImportAction gets created. Once the action has completed, if you notice that is has failed, click on the action on the dashboard and take a look at the error message in the "Phases" section of the action card. If you observe an "authentication failed" message like the one below, then please continue reading further to fix this issue.

      cause:

        cause:

          cause:

            message: "chacha20poly1305: message authentication failed"

          function: kasten.io/k10/kio/crypto.(*easyAE).Open

          linenumber: 116

          message: Could not decrypt cipher text

        function: kasten.io/k10/kio/collections.decryptData

        linenumber: 348

        message: Could not Open data during decryption

      message: Job failed to be executed

      fields: []

       

      Upgrading K10

      You will have to upgrade K10 to version 4.0.6 or greater on both the source and destination cluster.  Refer to instructions here for upgrading K10 - https://docs.kasten.io/latest/install/upgrade.html

       

      Get the latest migration string from source cluster

      After upgrading K10 on the source cluster, the migration string in the export policy will get updated.  

      Click on the policies card on K10's dashboard and then click on the policy that you are interested in. 

       

      At the bottom of the policy card , there is a button called "Show import details..." . Click on this button to obtain the migration string.

       

      In this example the migration string is

      bIzAPpoanmEy/XZNtiYovcBXfUARNVCWijCW22EeZK4LF6Mfplm7RoCUX/eAkq4bBDW3zKuNGUWv8itd1g6xizAEvxAOZLCjN9SS2ual95zWrYN2zXwLYh31v4H0TZVoeYYC5UAagclEqcOFgsbg6g2wEIyegn0Dp3THNluMwd6Ig9gHSU+Akm6Hr17ONglqsOtrJ4WpqnxEhwYePyDF8MqoWW7mXCW9smRPuIvXZYpe+4eJdiGeYocJu7qUjdn5L8PHcvnMNpM+CnX8v10p+GDS9PmUOr2Bj1TQUComCSkxWXLNvmYmx2C0otr27NlohhRiG4kKzTQal94i2GTcE7EeubUQOR9Ws5JwxZGt5v+61Ryf0JWO6yl/jVSk1C09+MWkbZhXybGgGDtvzJmgN0Smh7kABbWat1qjw4lZfztWgfZaM9ONYS54GhCD3/v6Mgyn8uYrOaOrIERQ7x5iRCuyz98197N2bkuOVU1bvogColBalgqREgcDb3pOH42vw7apcBzOjjVPuN1eh02h0m9bv/woXvCAPXwpdfiRPFGoxOePMHLx9Dj8bBM

       

      Update the import policy on the destination cluster

      On the destination cluster, we will use kubectl to edit the import policy. 

      First verify that the policy you are interested in exists on the destination cluster

      # kubectl get policy test1-import-2 -n kasten-io

      NAME             STATUS

      test1-import-2   Success

      Next lets edit this policy:

      In the screen that is used to edit the policy, you will see a section at the bottom like the one below:

      spec:

        actions:

        - action: import

          importParameters:

            profile:

              name: june24-2021-1

              namespace: kasten-io

            receiveString: bIzAPpoanmEy/XZNtiYovcBXfUARNVCWijCW22EeZK4LF6Mfplm7RoCUX/eAkq4bBDW3zKuNGUWv8itd1g6xizAEvxAOZLCjN9SS2ual95zWrYN2zXwLYh31v4H0TZVoeYYC5UAagclEqcOFgsbg6g2wEIyegn0Dp3THNluMwd6Ig9gHSU+Akm6Hr17ONglqsOtrJ4WpqnxEhwYePyDF8MqoWW7mXCW9smRPuIvXZYpe+4eJdiGeYocJu7qUjdn5L8PHcvnMNpM+CnX8v10p+GDS9PmUOr2Bj1TQUComCSkxWXLNvmYmx2C0otr27NlohhRiG4kKzTQal94i2GTcE7EeubUQOR9Ws5JwxZGt5v+61Ryf0JWO6yl/jVSk1C09+MWkbZhXybGgGDtvzJmgN0Smh7kABbWat1qjw4lZfztWgfZaM9ONYS54GhCD3/v6Mgyn8uYrOaOrIERQ7x5iRCuyz98197N2bkuOVU1bvogColBalgqREgcDb3pOH42vw7apcBzOjjVPuN1eh02h0m9bv/woXvCAPXwpdfiRPFGoxOePMHLx9Dj8bBM

        - action: restore

          restoreParameters: {}

        frequency: '@yearly'

        selector: {}

      status:

        hash: 2250204670

        specModifiedTime: "2021-06-29T20:36:26Z"

        validation: Success

      Delete the value corresponding to the existing receiveString under the importParameters. Replace it with the new migration string copied from the source cluster. Save and close this window to save the changes you made to the import policy.

      Use kubectl to get the policy again to verify that the Status of the policy says Success. 

      # kubectl get policy test1-import-2 -n kasten-io

      NAME             STATUS

      test1-import-2   Success

       

      Re running imports

      You can now either wait for the import policy to be triggered as per its schedule, or click on "Run Once" on the policy card on K10's dashboard on the destination cluster. The import should succeed this time.

       

      Additional errors caused by reinstallation of K10

       

      After following the steps in this article, there is a possibility that you might still see the "authentication failed" messages. This happens if you reinstalled K10 on the source cluster without following K10's DR(Disaster Recovery) procedure documented here - https://docs.kasten.io/latest/operating/dr.html

       

      Since K10 was reinstalled, the primary key would have been updated. This means that the newly installed K10 does not have access to the older primary key that was used to encrypt collections stored in the object/file store. As a result of this, the new migration strings only points to the newer key.  Since the new migration string has been copied over to the import policy on the destination cluster, imports fail with the "authentication failed" message since K10 on the destination cluster is unable to decrypt older collections. 

       

      There are two options at this stage:

      a) Reinstall K10 on the source cluster as per K10's DR procedure documented in the link above. This will work only if you satisfy the pre requisites for DR . You need the clusterID of the cluster where K10 was previously installed, the DR passphrase used when K10 DR was previously enabled and the credentials for the object/file store where the K10 DR restore point has been stored. After recovering K10, you will have to follow the steps described in this article regarding copying the new migration string from the source cluster's policy to the destination cluster's import policy before retrying imports.

       

      b) Delete collections in the object/file store that cannot be decrypted by K10.  You can get the list of collections that cannot be decrypted by following the steps below:

       

      • Enable debug logging in K10 by editing the k10-config config map in the kasten-io namespace. Set the field named loglevel to debug instead of the default value of info.
      • Click on "Run Once" on the import policy card on K10's dashboard on the destination cluster. 
      • Get the list of K10's pod
      • kubectl get pod -n kasten-io
      • Fetch the logs from the logging service by running a command like the one below. You will have to replace the logging service pod name with the name of the pod on your cluster. 
      • kubectl logs logging-svc-56f8c7cc49-d75cd  -n kasten-io 

      • If you find a log with the messages "Error decrypting collection with latest key" and "Error decrypting collection with previous key" for the same collection handle, then this is one of the collections that have to be deleted from the object/file store. Compile a list of such collections and delete them.

      • Click on "Run Once" on the import policy card to re trigger an import. The import should succeed as long as K10 does not find a collection that cannot be decrypted.