How to deal with 401 - Unauthorized or x509 errors while accessing K10's dashboard

Follow this guide to install K10 with the correct CA certificates for accessing K10's dashboard

Description

If  a 401 - Unauthorized error was observed while accessing K10's dashboard despite using the `cacertconfigmap.name` helm value, then this post might help resolve that issue. 

If errors related to x509 such as "x509: certificate signed by unknown authority" were found in the logs generated by K10's gateway pod or auth-svc pod, this post might help with such issues too. 

 

Debugging

You can download the latest version of the debug tool from here 

If the connection to K10 succeeds without the need for a certificate, then you will see the output as seen below.

./k10tools debug auth

Dex:
OIDC Provider URL: https://onkar-1.dev.do.kasten.io
Release name: k10
Dex well known URL:https://onkar-1.dev.do.kasten.io/k10/dex/.well-known/openid-configuration
Trying to connect to Dex without TLS (insecureSkipVerify=false)
Connection succeeded - OK

 

If the connection to K10 does require a certificate, then you may see an output like this one below:

Dex:
OIDC Provider URL: https://example.com/k10/dex
Release name: k10
Dex well known URL:https://example.com/k10/dex/.well-known/openid-configuration
Trying to connect to Dex without TLS (insecureSkipVerify=false)
Connection failed, testing other options
Trying to connect to Dex with TLS but verification disabled (insecureSkipVerify=true)
Connection succeeded
Trying to connect to Dex with TLS verification enabled and using a CA certificate
Connection failed ({"message":"HTTP Get for Dex's well known endpoint failed","function":"kasten.io/k10/kio/tools/k10primer/k10debugger.(*oidcOperate).testDexConnectivity","linenumber":29,"fields":[{"name":"statusCode","value":null}],"cause":{"Op":"Get","URL":"https://example.com/k10/dex/.well-known/openid-configuration","Err":{"Cert":{"Raw":") - Error

This means that the tool was able to connect to K10 with SSL verification disabled. But when it tried to connect with SSL verification enabled, the verification failed. This indicates that the certificate that you have installed with K10 might not be the right one for accessing K10's dashboard. 

 

Please also run this command to debug the CA certificate:

./k10tools debug ca-certificate
CA Certificate Checker:
Fetching configmap which contains CA Certificate information : custom-ca-bundle-store
Certificate exists in configmap - OK
Found container : aggregatedapis-svc to extract certificate
Certificate exists in container at /etc/ssl/certs/custom-ca-bundle.pem
Certificates matched successfully - OK

Solution

Use the openssl command below with the -host option as the domain name of K10's dashboard .

The output will show the certificate chain involved while accessing this endpoint using SSL.

You might see Root CA and Intermediate CA certificates in the chain. Please copy all the CA certificates into a file named custom-ca-bundle.pem and use that file while installing K10 using the instructions here 

openssl s_client -host k10-dashboard-example.com -port 443 -prexit -showcerts

 

An example of running the openssl command against a real endpoint: 

# openssl s_client -host kasten.io -port 443 -prexit -showcerts

CONNECTED(00000005)

depth=2 C = US, O = Amazon, CN = Amazon Root CA 1

verify return:1

depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

verify return:1

depth=0 CN = kasten.io

verify return:1

---

Certificate chain

0 s:CN = kasten.io

   i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

-----BEGIN CERTIFICATE-----

MIIFajCCBFKgAwIBAgIQBUbD8IjXx0CLiviXYl8mxTANBgkqhkiG9w0BAQsFADBG

MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRUwEwYDVQQLEwxTZXJ2ZXIg

Q0EgMUIxDzANBgNVBAMTBkFtYXpvbjAeFw0yMDA4MzEwMDAwMDBaFw0yMTEwMDEx

MjAwMDBaMBQxEjAQBgNVBAMTCWthc3Rlbi5pbzCCASIwDQYJKoZIhvcNAQEBBQAD

ggEPADCCAQoCggEBAOdyWIn1mZQ8U1N6tUv7p/+F2pfQ9Zz2+ZBKCXRh3mUUZ2PJ

DdjLP1Sf4j4EdIEQVaFOqvhSQW+krLfQa0rCLOy7gJCApvas4Z5trS3hbxyqV2u4

hzmNfhQYIdVKfshm6EMi14DQhy1YfdF39xbCiO1F6r8ush9R8duChUlL9CNRbbFu

cZi9CjaNqwffa5EXFk1oDx6pGu/fn+8Br0Ghb/0bFVoyPV9P3Tbb4RGSZN/UR72U

+LfQDx1fE4CbE2vmvH3MReJglkOBPKLbAjiARaaTmBNuTlQ870vYz5nlsYKlImM0

CkTvKR+uafDe5NY33B0ifAYcj4oRswpVbU4uC68CAwEAAaOCAoQwggKAMB8GA1Ud

IwQYMBaAFFmkZgZSoHuVkjyjlAcnlnRb+T3QMB0GA1UdDgQWBBRvQSymjcGE13hJ

BNUT9A100gviHjAjBgNVHREEHDAagglrYXN0ZW4uaW+CDXd3dy5rYXN0ZW4uaW8w

DgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjA7

BgNVHR8ENDAyMDCgLqAshipodHRwOi8vY3JsLnNjYTFiLmFtYXpvbnRydXN0LmNv

bS9zY2ExYi5jcmwwIAYDVR0gBBkwFzALBglghkgBhv1sAQIwCAYGZ4EMAQIBMHUG

CCsGAQUFBwEBBGkwZzAtBggrBgEFBQcwAYYhaHR0cDovL29jc3Auc2NhMWIuYW1h

em9udHJ1c3QuY29tMDYGCCsGAQUFBzAChipodHRwOi8vY3J0LnNjYTFiLmFtYXpv

bnRydXN0LmNvbS9zY2ExYi5jcnQwDAYDVR0TAQH/BAIwADCCAQQGCisGAQQB1nkC

BAIEgfUEgfIA8AB2APZclC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAAB

dEWdC3cAAAQDAEcwRQIhAIMe1mQS0Nz5ErXIrHuitRc1vZREAo5hxpyrR5mEW7Xy

AiBzeVq4lZ7sS1rON8yghuQWtaEVuwNLYIHRttg4feSxSQB2AFzcQ5L+5qtFRLFe

mtRW5hA3+9X6R9yhc5SyXub2xw7KAAABdEWdC6sAAAQDAEcwRQIgCSYmbXgVyfLs

V3znTN6p8dYbiK/LH1tRj3NY9HWc/uICIQCMoU2ImWxle8DJ+7cGxS6NpoujWtWo

QUE1jDyem3QN1jANBgkqhkiG9w0BAQsFAAOCAQEARb/oG6v7R0LVbIvdq9PHtrli

EZgPHGtxFRrgQzAZs9NQBZQ/GOMNffV4YD9jxFi05o9jUQM8d0xZUQbVsDzsQXjW

rWr7iVgmdt+efW60ZEfyqRJeoJIvynrg92mY0mhHtPwzG7hiGKDEHhzyTd9kHo3e

iZvEZq9BIe8FsMJ2iQw1q16fTE+SUdxOtYZQMq1w1VckZPVSTkkd8BhGORBoOfSa

xZqxbBZ96lzamMUB6hjbe4AuAgty/7yJwesTwRMyf/h/abtNIEQPvvfb5fM1Qr30

oI9Q4hgkcj+C75v70PYP4vcYMzxX91fuiEzfbddhcH5TPmd2aerNdkxOeqsGtg==

-----END CERTIFICATE-----

1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

   i:C = US, O = Amazon, CN = Amazon Root CA 1

-----BEGIN CERTIFICATE-----

MIIESTCCAzGgAwIBAgITBn+UV4WH6Kx33rJTMlu8mYtWDTANBgkqhkiG9w0BAQsF

ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6

b24gUm9vdCBDQSAxMB4XDTE1MTAyMjAwMDAwMFoXDTI1MTAxOTAwMDAwMFowRjEL

MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEVMBMGA1UECxMMU2VydmVyIENB

IDFCMQ8wDQYDVQQDEwZBbWF6b24wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK

AoIBAQDCThZn3c68asg3Wuw6MLAd5tES6BIoSMzoKcG5blPVo+sDORrMd4f2AbnZ

cMzPa43j4wNxhplty6aUKk4T1qe9BOwKFjwK6zmxxLVYo7bHViXsPlJ6qOMpFge5

blDP+18x+B26A0piiQOuPkfyDyeR4xQghfj66Yo19V+emU3nazfvpFA+ROz6WoVm

B5x+F2pV8xeKNR7u6azDdU5YVX1TawprmxRC1+WsAYmz6qP+z8ArDITC2FMVy2fw

0IjKOtEXc/VfmtTFch5+AfGYMGMqqvJ6LcXiAhqG5TI+Dr0RtM88k+8XUBCeQ8IG

KuANaL7TiItKZYxK1MMuTJtV9IblAgMBAAGjggE7MIIBNzASBgNVHRMBAf8ECDAG

AQH/AgEAMA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUWaRmBlKge5WSPKOUByeW

dFv5PdAwHwYDVR0jBBgwFoAUhBjMhTTsvAyUlC4IWZzHshBOCggwewYIKwYBBQUH

AQEEbzBtMC8GCCsGAQUFBzABhiNodHRwOi8vb2NzcC5yb290Y2ExLmFtYXpvbnRy

dXN0LmNvbTA6BggrBgEFBQcwAoYuaHR0cDovL2NydC5yb290Y2ExLmFtYXpvbnRy

dXN0LmNvbS9yb290Y2ExLmNlcjA/BgNVHR8EODA2MDSgMqAwhi5odHRwOi8vY3Js

LnJvb3RjYTEuYW1hem9udHJ1c3QuY29tL3Jvb3RjYTEuY3JsMBMGA1UdIAQMMAow

CAYGZ4EMAQIBMA0GCSqGSIb3DQEBCwUAA4IBAQCFkr41u3nPo4FCHOTjY3NTOVI1

59Gt/a6ZiqyJEi+752+a1U5y6iAwYfmXss2lJwJFqMp2PphKg5625kXg8kP2CN5t

6G7bMQcT8C8xDZNtYTd7WPD8UZiRKAJPBXa30/AbwuZe0GaFEQ8ugcYQgSn+IGBI

8/LwhBNTZTUVEWuCUUBVV18YtbAiPq3yXqMB48Oz+ctBWuZSkbvkNodPLamkB2g1

upRyzQ7qDn1X8nn8N8V7YJ6y68AtkHcNSRAnpTitxBKjtKPISLMVCx7i4hncxHZS

yLyKQXhw2W2Xs0qLeC1etA+jTGDK4UfLeC0SF7FSi8o5LL21L8IzApar2pR/

-----END CERTIFICATE-----

2 s:C = US, O = Amazon, CN = Amazon Root CA 1

   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2

-----BEGIN CERTIFICATE-----

MIIEkjCCA3qgAwIBAgITBn+USionzfP6wq4rAfkI7rnExjANBgkqhkiG9w0BAQsF

ADCBmDELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNj

b3R0c2RhbGUxJTAjBgNVBAoTHFN0YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4x

OzA5BgNVBAMTMlN0YXJmaWVsZCBTZXJ2aWNlcyBSb290IENlcnRpZmljYXRlIEF1

dGhvcml0eSAtIEcyMB4XDTE1MDUyNTEyMDAwMFoXDTM3MTIzMTAxMDAwMFowOTEL

MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv

b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj

ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM

9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw

IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6

VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L

93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm

jgSubJrIqg0CAwEAAaOCATEwggEtMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/

BAQDAgGGMB0GA1UdDgQWBBSEGMyFNOy8DJSULghZnMeyEE4KCDAfBgNVHSMEGDAW

gBScXwDfqgHXMCs4iKK4bUqc8hGRgzB4BggrBgEFBQcBAQRsMGowLgYIKwYBBQUH

MAGGImh0dHA6Ly9vY3NwLnJvb3RnMi5hbWF6b250cnVzdC5jb20wOAYIKwYBBQUH

MAKGLGh0dHA6Ly9jcnQucm9vdGcyLmFtYXpvbnRydXN0LmNvbS9yb290ZzIuY2Vy

MD0GA1UdHwQ2MDQwMqAwoC6GLGh0dHA6Ly9jcmwucm9vdGcyLmFtYXpvbnRydXN0

LmNvbS9yb290ZzIuY3JsMBEGA1UdIAQKMAgwBgYEVR0gADANBgkqhkiG9w0BAQsF

AAOCAQEAYjdCXLwQtT6LLOkMm2xF4gcAevnFWAu5CIw+7bMlPLVvUOTNNWqnkzSW

MiGpSESrnO09tKpzbeR/FoCJbM8oAxiDR3mjEH4wW6w7sGDgd9QIpuEdfF7Au/ma

eyKdpwAJfqxGF4PcnCZXmTA5YpaP7dreqsXMGz7KQ2hsVxa81Q4gLv7/wmpdLqBK

bRRYh5TmOTFffHPLkIhqhBGWJ6bt2YFGpn6jcgAKUj6DiAdjd4lpFw85hdKrCEVN

0FE6/V1dN2RMfjCyVSRCnTawXZwXgWHxyvkQAiSr6w10kY17RSlQOYiypok1JR4U

akcjMS9cmvqtmg5iUaQqqcT5NJ0hGA==

-----END CERTIFICATE-----

3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2

   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority

-----BEGIN CERTIFICATE-----

MIIEdTCCA12gAwIBAgIJAKcOSkw0grd/MA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV

BAYTAlVTMSUwIwYDVQQKExxTdGFyZmllbGQgVGVjaG5vbG9naWVzLCBJbmMuMTIw

MAYDVQQLEylTdGFyZmllbGQgQ2xhc3MgMiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0

eTAeFw0wOTA5MDIwMDAwMDBaFw0zNDA2MjgxNzM5MTZaMIGYMQswCQYDVQQGEwJV

UzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMKU2NvdHRzZGFsZTElMCMGA1UE

ChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjE7MDkGA1UEAxMyU3RhcmZp

ZWxkIFNlcnZpY2VzIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IC0gRzIwggEi

MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVDDrEKvlO4vW+GZdfjohTsR8/

y8+fIBNtKTrID30892t2OGPZNmCom15cAICyL1l/9of5JUOG52kbUpqQ4XHj2C0N

Tm/2yEnZtvMaVq4rtnQU68/7JuMauh2WLmo7WJSJR1b/JaCTcFOD2oR0FMNnngRo

Ot+OQFodSk7PQ5E751bWAHDLUu57fa4657wx+UX2wmDPE1kCK4DMNEffud6QZW0C

zyyRpqbn3oUYSXxmTqM6bam17jQuug0DuDPfR+uxa40l2ZvOgdFFRjKWcIfeAg5J

Q4W2bHO7ZOphQazJ1FTfhy/HIrImzJ9ZVGif/L4qL8RVHHVAYBeFAlU5i38FAgMB

AAGjgfAwge0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0O

BBYEFJxfAN+qAdcwKziIorhtSpzyEZGDMB8GA1UdIwQYMBaAFL9ft9HO3R+G9FtV

rNzXEMIOqYjnME8GCCsGAQUFBwEBBEMwQTAcBggrBgEFBQcwAYYQaHR0cDovL28u

c3MyLnVzLzAhBggrBgEFBQcwAoYVaHR0cDovL3guc3MyLnVzL3guY2VyMCYGA1Ud

HwQfMB0wG6AZoBeGFWh0dHA6Ly9zLnNzMi51cy9yLmNybDARBgNVHSAECjAIMAYG

BFUdIAAwDQYJKoZIhvcNAQELBQADggEBACMd44pXyn3pF3lM8R5V/cxTbj5HD9/G

VfKyBDbtgB9TxF00KGu+x1X8Z+rLP3+QsjPNG1gQggL4+C/1E2DUBc7xgQjB3ad1

l08YuW3e95ORCLp+QCztweq7dp4zBncdDQh/U90bZKuCJ/Fp1U1ervShw3WnWEQt

8jxwmKy6abaVd38PMV4s/KCHOkdp8Hlf9BRUpJVeEXgSYCfOn8J3/yNTd126/+pZ

59vPr5KW7ySaNRB6nJHGDn2Z9j8Z3/VyVOEVqQdZe4O/Ui5GjLIAZHYcSNPYeehu

VsyuLAOQ1xk4meTKCRlb/weWsKh/NEnfVqn3sF/tM+2MR7cwA130A4w=

-----END CERTIFICATE-----

---

Server certificate

subject=CN = kasten.io




issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon




---

No client certificate CA names sent

Peer signing digest: SHA256

Peer signature type: RSA-PSS

Server Temp Key: X25519, 253 bits

---

SSL handshake has read 5369 bytes and written 375 bytes

Verification: OK

---

New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256

Server public key is 2048 bit

Secure Renegotiation IS NOT supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

Early data was not sent

Verify return code: 0 (ok)

In this output, there are 4 certificates belonging to the following Certificate Authorities:

  • The Amazon Root CA 1  
  • The Server CA 1B 
  • Starfield Services Root Certificate Authority
  • Starfield Class 2 Certification Authority