Follow this guide to install K10 with the correct CA certificates for accessing K10's dashboard
Description
If a 401 - Unauthorized error was observed while accessing K10's dashboard despite using the `cacertconfigmap.name` helm value, then this post might help resolve that issue.
If errors related to x509 such as "x509: certificate signed by unknown authority" were found in the logs generated by K10's gateway pod or auth-svc pod, this post might help with such issues too.
Debugging
You can download the latest version of the debug tool from here
If the connection to K10 succeeds without the need for a certificate, then you will see the output as seen below.
./k10tools debug auth
Dex:
OIDC Provider URL: https://onkar-1.dev.do.kasten.io
Release name: k10
Dex well known URL:https://onkar-1.dev.do.kasten.io/k10/dex/.well-known/openid-configuration
Trying to connect to Dex without TLS (insecureSkipVerify=false)
Connection succeeded - OK
If the connection to K10 does require a certificate, then you may see an output like this one below:
Dex:
OIDC Provider URL: https://example.com/k10/dex
Release name: k10
Dex well known URL:https://example.com/k10/dex/.well-known/openid-configuration
Trying to connect to Dex without TLS (insecureSkipVerify=false)
Connection failed, testing other options
Trying to connect to Dex with TLS but verification disabled (insecureSkipVerify=true)
Connection succeeded
Trying to connect to Dex with TLS verification enabled and using a CA certificate
Connection failed ({"message":"HTTP Get for Dex's well known endpoint failed","function":"kasten.io/k10/kio/tools/k10primer/k10debugger.(*oidcOperate).testDexConnectivity","linenumber":29,"fields":[{"name":"statusCode","value":null}],"cause":{"Op":"Get","URL":"https://example.com/k10/dex/.well-known/openid-configuration","Err":{"Cert":{"Raw":") - Error
This means that the tool was able to connect to K10 with SSL verification disabled. But when it tried to connect with SSL verification enabled, the verification failed. This indicates that the certificate that you have installed with K10 might not be the right one for accessing K10's dashboard.
Please also run this command to debug the CA certificate:
./k10tools debug ca-certificate
CA Certificate Checker:
Fetching configmap which contains CA Certificate information : custom-ca-bundle-store
Certificate exists in configmap - OK
Found container : aggregatedapis-svc to extract certificate
Certificate exists in container at /etc/ssl/certs/custom-ca-bundle.pem
Certificates matched successfully - OK
Solution
Use the openssl command below with the -host option as the domain name of K10's dashboard .
The output will show the certificate chain involved while accessing this endpoint using SSL.
You might see Root CA and Intermediate CA certificates in the chain. Please copy all the CA certificates into a file named custom-ca-bundle.pem and use that file while installing K10 using the instructions here
openssl s_client -host k10-dashboard-example.com -port 443 -prexit -showcerts
An example of running the openssl command against a real endpoint:
# openssl s_client -host kasten.io -port 443 -prexit -showcerts
CONNECTED(00000005)
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = kasten.io
verify return:1
---
Certificate chain
0 s:CN = kasten.io
i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
-----BEGIN CERTIFICATE-----
MIIFajCCBFKgAwIBAgIQBUbD8IjXx0CLiviXYl8mxTANBgkqhkiG9w0BAQsFADBG
MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRUwEwYDVQQLEwxTZXJ2ZXIg
Q0EgMUIxDzANBgNVBAMTBkFtYXpvbjAeFw0yMDA4MzEwMDAwMDBaFw0yMTEwMDEx
MjAwMDBaMBQxEjAQBgNVBAMTCWthc3Rlbi5pbzCCASIwDQYJKoZIhvcNAQEBBQAD
ggEPADCCAQoCggEBAOdyWIn1mZQ8U1N6tUv7p/+F2pfQ9Zz2+ZBKCXRh3mUUZ2PJ
DdjLP1Sf4j4EdIEQVaFOqvhSQW+krLfQa0rCLOy7gJCApvas4Z5trS3hbxyqV2u4
hzmNfhQYIdVKfshm6EMi14DQhy1YfdF39xbCiO1F6r8ush9R8duChUlL9CNRbbFu
cZi9CjaNqwffa5EXFk1oDx6pGu/fn+8Br0Ghb/0bFVoyPV9P3Tbb4RGSZN/UR72U
+LfQDx1fE4CbE2vmvH3MReJglkOBPKLbAjiARaaTmBNuTlQ870vYz5nlsYKlImM0
CkTvKR+uafDe5NY33B0ifAYcj4oRswpVbU4uC68CAwEAAaOCAoQwggKAMB8GA1Ud
IwQYMBaAFFmkZgZSoHuVkjyjlAcnlnRb+T3QMB0GA1UdDgQWBBRvQSymjcGE13hJ
BNUT9A100gviHjAjBgNVHREEHDAagglrYXN0ZW4uaW+CDXd3dy5rYXN0ZW4uaW8w
DgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjA7
BgNVHR8ENDAyMDCgLqAshipodHRwOi8vY3JsLnNjYTFiLmFtYXpvbnRydXN0LmNv
bS9zY2ExYi5jcmwwIAYDVR0gBBkwFzALBglghkgBhv1sAQIwCAYGZ4EMAQIBMHUG
CCsGAQUFBwEBBGkwZzAtBggrBgEFBQcwAYYhaHR0cDovL29jc3Auc2NhMWIuYW1h
em9udHJ1c3QuY29tMDYGCCsGAQUFBzAChipodHRwOi8vY3J0LnNjYTFiLmFtYXpv
bnRydXN0LmNvbS9zY2ExYi5jcnQwDAYDVR0TAQH/BAIwADCCAQQGCisGAQQB1nkC
BAIEgfUEgfIA8AB2APZclC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAAB
dEWdC3cAAAQDAEcwRQIhAIMe1mQS0Nz5ErXIrHuitRc1vZREAo5hxpyrR5mEW7Xy
AiBzeVq4lZ7sS1rON8yghuQWtaEVuwNLYIHRttg4feSxSQB2AFzcQ5L+5qtFRLFe
mtRW5hA3+9X6R9yhc5SyXub2xw7KAAABdEWdC6sAAAQDAEcwRQIgCSYmbXgVyfLs
V3znTN6p8dYbiK/LH1tRj3NY9HWc/uICIQCMoU2ImWxle8DJ+7cGxS6NpoujWtWo
QUE1jDyem3QN1jANBgkqhkiG9w0BAQsFAAOCAQEARb/oG6v7R0LVbIvdq9PHtrli
EZgPHGtxFRrgQzAZs9NQBZQ/GOMNffV4YD9jxFi05o9jUQM8d0xZUQbVsDzsQXjW
rWr7iVgmdt+efW60ZEfyqRJeoJIvynrg92mY0mhHtPwzG7hiGKDEHhzyTd9kHo3e
iZvEZq9BIe8FsMJ2iQw1q16fTE+SUdxOtYZQMq1w1VckZPVSTkkd8BhGORBoOfSa
xZqxbBZ96lzamMUB6hjbe4AuAgty/7yJwesTwRMyf/h/abtNIEQPvvfb5fM1Qr30
oI9Q4hgkcj+C75v70PYP4vcYMzxX91fuiEzfbddhcH5TPmd2aerNdkxOeqsGtg==
-----END CERTIFICATE-----
1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
i:C = US, O = Amazon, CN = Amazon Root CA 1
-----BEGIN CERTIFICATE-----
MIIESTCCAzGgAwIBAgITBn+UV4WH6Kx33rJTMlu8mYtWDTANBgkqhkiG9w0BAQsF
ADA5MQswCQYDVQQGEwJVUzEPMA0GA1UEChMGQW1hem9uMRkwFwYDVQQDExBBbWF6
b24gUm9vdCBDQSAxMB4XDTE1MTAyMjAwMDAwMFoXDTI1MTAxOTAwMDAwMFowRjEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEVMBMGA1UECxMMU2VydmVyIENB
IDFCMQ8wDQYDVQQDEwZBbWF6b24wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQDCThZn3c68asg3Wuw6MLAd5tES6BIoSMzoKcG5blPVo+sDORrMd4f2AbnZ
cMzPa43j4wNxhplty6aUKk4T1qe9BOwKFjwK6zmxxLVYo7bHViXsPlJ6qOMpFge5
blDP+18x+B26A0piiQOuPkfyDyeR4xQghfj66Yo19V+emU3nazfvpFA+ROz6WoVm
B5x+F2pV8xeKNR7u6azDdU5YVX1TawprmxRC1+WsAYmz6qP+z8ArDITC2FMVy2fw
0IjKOtEXc/VfmtTFch5+AfGYMGMqqvJ6LcXiAhqG5TI+Dr0RtM88k+8XUBCeQ8IG
KuANaL7TiItKZYxK1MMuTJtV9IblAgMBAAGjggE7MIIBNzASBgNVHRMBAf8ECDAG
AQH/AgEAMA4GA1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUWaRmBlKge5WSPKOUByeW
dFv5PdAwHwYDVR0jBBgwFoAUhBjMhTTsvAyUlC4IWZzHshBOCggwewYIKwYBBQUH
AQEEbzBtMC8GCCsGAQUFBzABhiNodHRwOi8vb2NzcC5yb290Y2ExLmFtYXpvbnRy
dXN0LmNvbTA6BggrBgEFBQcwAoYuaHR0cDovL2NydC5yb290Y2ExLmFtYXpvbnRy
dXN0LmNvbS9yb290Y2ExLmNlcjA/BgNVHR8EODA2MDSgMqAwhi5odHRwOi8vY3Js
LnJvb3RjYTEuYW1hem9udHJ1c3QuY29tL3Jvb3RjYTEuY3JsMBMGA1UdIAQMMAow
CAYGZ4EMAQIBMA0GCSqGSIb3DQEBCwUAA4IBAQCFkr41u3nPo4FCHOTjY3NTOVI1
59Gt/a6ZiqyJEi+752+a1U5y6iAwYfmXss2lJwJFqMp2PphKg5625kXg8kP2CN5t
6G7bMQcT8C8xDZNtYTd7WPD8UZiRKAJPBXa30/AbwuZe0GaFEQ8ugcYQgSn+IGBI
8/LwhBNTZTUVEWuCUUBVV18YtbAiPq3yXqMB48Oz+ctBWuZSkbvkNodPLamkB2g1
upRyzQ7qDn1X8nn8N8V7YJ6y68AtkHcNSRAnpTitxBKjtKPISLMVCx7i4hncxHZS
yLyKQXhw2W2Xs0qLeC1etA+jTGDK4UfLeC0SF7FSi8o5LL21L8IzApar2pR/
-----END CERTIFICATE-----
2 s:C = US, O = Amazon, CN = Amazon Root CA 1
i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgITBn+USionzfP6wq4rAfkI7rnExjANBgkqhkiG9w0BAQsF
ADCBmDELMAkGA1UEBhMCVVMxEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNj
b3R0c2RhbGUxJTAjBgNVBAoTHFN0YXJmaWVsZCBUZWNobm9sb2dpZXMsIEluYy4x
OzA5BgNVBAMTMlN0YXJmaWVsZCBTZXJ2aWNlcyBSb290IENlcnRpZmljYXRlIEF1
dGhvcml0eSAtIEcyMB4XDTE1MDUyNTEyMDAwMFoXDTM3MTIzMTAxMDAwMFowOTEL
MAkGA1UEBhMCVVMxDzANBgNVBAoTBkFtYXpvbjEZMBcGA1UEAxMQQW1hem9uIFJv
b3QgQ0EgMTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALJ4gHHKeNXj
ca9HgFB0fW7Y14h29Jlo91ghYPl0hAEvrAIthtOgQ3pOsqTQNroBvo3bSMgHFzZM
9O6II8c+6zf1tRn4SWiw3te5djgdYZ6k/oI2peVKVuRF4fn9tBb6dNqcmzU5L/qw
IFAGbHrQgLKm+a/sRxmPUDgH3KKHOVj4utWp+UhnMJbulHheb4mjUcAwhmahRWa6
VOujw5H5SNz/0egwLX0tdHA114gk957EWW67c4cX8jJGKLhD+rcdqsq08p8kDi1L
93FcXmn/6pUCyziKrlA4b9v7LWIbxcceVOF34GfID5yHI9Y/QCB/IIDEgEw+OyQm
jgSubJrIqg0CAwEAAaOCATEwggEtMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/
BAQDAgGGMB0GA1UdDgQWBBSEGMyFNOy8DJSULghZnMeyEE4KCDAfBgNVHSMEGDAW
gBScXwDfqgHXMCs4iKK4bUqc8hGRgzB4BggrBgEFBQcBAQRsMGowLgYIKwYBBQUH
MAGGImh0dHA6Ly9vY3NwLnJvb3RnMi5hbWF6b250cnVzdC5jb20wOAYIKwYBBQUH
MAKGLGh0dHA6Ly9jcnQucm9vdGcyLmFtYXpvbnRydXN0LmNvbS9yb290ZzIuY2Vy
MD0GA1UdHwQ2MDQwMqAwoC6GLGh0dHA6Ly9jcmwucm9vdGcyLmFtYXpvbnRydXN0
LmNvbS9yb290ZzIuY3JsMBEGA1UdIAQKMAgwBgYEVR0gADANBgkqhkiG9w0BAQsF
AAOCAQEAYjdCXLwQtT6LLOkMm2xF4gcAevnFWAu5CIw+7bMlPLVvUOTNNWqnkzSW
MiGpSESrnO09tKpzbeR/FoCJbM8oAxiDR3mjEH4wW6w7sGDgd9QIpuEdfF7Au/ma
eyKdpwAJfqxGF4PcnCZXmTA5YpaP7dreqsXMGz7KQ2hsVxa81Q4gLv7/wmpdLqBK
bRRYh5TmOTFffHPLkIhqhBGWJ6bt2YFGpn6jcgAKUj6DiAdjd4lpFw85hdKrCEVN
0FE6/V1dN2RMfjCyVSRCnTawXZwXgWHxyvkQAiSr6w10kY17RSlQOYiypok1JR4U
akcjMS9cmvqtmg5iUaQqqcT5NJ0hGA==
-----END CERTIFICATE-----
3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
-----BEGIN CERTIFICATE-----
MIIEdTCCA12gAwIBAgIJAKcOSkw0grd/MA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
BAYTAlVTMSUwIwYDVQQKExxTdGFyZmllbGQgVGVjaG5vbG9naWVzLCBJbmMuMTIw
MAYDVQQLEylTdGFyZmllbGQgQ2xhc3MgMiBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eTAeFw0wOTA5MDIwMDAwMDBaFw0zNDA2MjgxNzM5MTZaMIGYMQswCQYDVQQGEwJV
UzEQMA4GA1UECBMHQXJpem9uYTETMBEGA1UEBxMKU2NvdHRzZGFsZTElMCMGA1UE
ChMcU3RhcmZpZWxkIFRlY2hub2xvZ2llcywgSW5jLjE7MDkGA1UEAxMyU3RhcmZp
ZWxkIFNlcnZpY2VzIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5IC0gRzIwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDVDDrEKvlO4vW+GZdfjohTsR8/
y8+fIBNtKTrID30892t2OGPZNmCom15cAICyL1l/9of5JUOG52kbUpqQ4XHj2C0N
Tm/2yEnZtvMaVq4rtnQU68/7JuMauh2WLmo7WJSJR1b/JaCTcFOD2oR0FMNnngRo
Ot+OQFodSk7PQ5E751bWAHDLUu57fa4657wx+UX2wmDPE1kCK4DMNEffud6QZW0C
zyyRpqbn3oUYSXxmTqM6bam17jQuug0DuDPfR+uxa40l2ZvOgdFFRjKWcIfeAg5J
Q4W2bHO7ZOphQazJ1FTfhy/HIrImzJ9ZVGif/L4qL8RVHHVAYBeFAlU5i38FAgMB
AAGjgfAwge0wDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAYYwHQYDVR0O
BBYEFJxfAN+qAdcwKziIorhtSpzyEZGDMB8GA1UdIwQYMBaAFL9ft9HO3R+G9FtV
rNzXEMIOqYjnME8GCCsGAQUFBwEBBEMwQTAcBggrBgEFBQcwAYYQaHR0cDovL28u
c3MyLnVzLzAhBggrBgEFBQcwAoYVaHR0cDovL3guc3MyLnVzL3guY2VyMCYGA1Ud
HwQfMB0wG6AZoBeGFWh0dHA6Ly9zLnNzMi51cy9yLmNybDARBgNVHSAECjAIMAYG
BFUdIAAwDQYJKoZIhvcNAQELBQADggEBACMd44pXyn3pF3lM8R5V/cxTbj5HD9/G
VfKyBDbtgB9TxF00KGu+x1X8Z+rLP3+QsjPNG1gQggL4+C/1E2DUBc7xgQjB3ad1
l08YuW3e95ORCLp+QCztweq7dp4zBncdDQh/U90bZKuCJ/Fp1U1ervShw3WnWEQt
8jxwmKy6abaVd38PMV4s/KCHOkdp8Hlf9BRUpJVeEXgSYCfOn8J3/yNTd126/+pZ
59vPr5KW7ySaNRB6nJHGDn2Z9j8Z3/VyVOEVqQdZe4O/Ui5GjLIAZHYcSNPYeehu
VsyuLAOQ1xk4meTKCRlb/weWsKh/NEnfVqn3sF/tM+2MR7cwA130A4w=
-----END CERTIFICATE-----
---
Server certificate
subject=CN = kasten.io
issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5369 bytes and written 375 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
In this output, there are 4 certificates belonging to the following Certificate Authorities:
- The Amazon Root CA 1
- The Server CA 1B
- Starfield Services Root Certificate Authority
- Starfield Class 2 Certification Authority