Chef 12: Fix Untrusted Self Sign Certs
Scenario: You’ve started up a brand new Chef Server using version 12, and you have installed Chef 12 on your local system. You log into the Management Console to create a user and organization (or do this with the command-line chef-server-ctl
commands), and you’re ready to rock with this knife.rb:
node_name 'jtimberman'
client_key 'jtimberman.pem'
validation_client_name 'tester-validator'
validation_key 'tester-validator.pem'
chef_server_url 'https://chef-server.example.com/organizations/tester'
However, when you try to check things out with knife:
% knife client list
ERROR: SSL Validation failure connecting to host: chef-server.example.com - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
ERROR: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
This is because Chef client 12 has SSL verification enabled by default for all requests. Since the certificate generated by the Chef Server 12 installation is self-signed, there isn’t a signing CA that can be verified, and this fails. Never fear intrepid user, for you can get the SSL certificate from the server and store it as a “trusted” certificate. To find out how, use knife ssl check
.
Connecting to host chef-server.example.com:443
ERROR: The SSL certificate of chef-server.example.com could not be verified
Certificate issuer data: /C=US/ST=WA/L=Seattle/O=YouCorp/OU=Operations/CN=chef-server.example.com/emailAddress=you@example.com
Configuration Info:
OpenSSL Configuration:
* Version: OpenSSL 1.0.1j 15 Oct 2014
* Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs"
TO FIX THIS ERROR:
If the server you are connecting to uses a self-signed certificate, you must
configure chef to trust that server's certificate.
By default, the certificate is stored in the following location on the host
where your chef-server runs:
/var/opt/chef-server/nginx/ca/SERVER_HOSTNAME.crt
Copy that file to your trusted_certs_dir (currently: /Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs)
using SSH/SCP or some other secure method, then re-run this command to confirm
that the server's certificate is now trusted.
(note, this chef-server location is incorrect, it’s /var/opt/opscode
)
There is a fetch
plugin for knife
too. Let’s download the certificate to the automatically preconfigured trusted certificate location mentioned in the output above.
% knife ssl fetch
WARNING: Certificates from chef-server.example.com will be fetched and placed in your trusted_cert
directory (/Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs).
Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.
Adding certificate for chef-server.example.com in /Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs/chef-server.example.com.crt
The certificate should be verified that what was downloaded is in fact the same as the certificate on the Chef Server. For example, I compared SHA256 checksums:
% ssh ubuntu@chef-server.example.com sudo sha256sum /var/opt/opscode/nginx/ca/chef-server.example.com.crt
043728b55144861ed43a426c67addca357a5889158886aee50685cf1422b5ebf /var/opt/opscode/nginx/ca/chef-server.example.com.crt
% gsha256sum .chef/trusted_certs/chef-server.example.com.crt
043728b55144861ed43a426c67addca357a5889158886aee50685cf1422b5ebf .chef/trusted_certs/chef-server.example.com.crt
Now check knife client list again.
% knife client list
tester-validator
Victory!
Now, we need to get the ceritficate out to every node in the infrastructure in its trusted_certs_dir
- by default this is /etc/chef/trusted_certs
. The most simple way to do this is to use knife ssh
to run knife on the target nodes.
% knife ssh 'name:*' 'sudo knife ssl fetch -c /etc/chef/client.rb'
node-output.example.com WARNING: Certificates from chef-server-example.com will be fetched and placed in your trusted_cert
node-output.example.com directory (/etc/chef/trusted_certs).
node-output.example.com
node-output.example.com Knife has no means to verify these are the correct certificates. You should
node-output.example.com verify the authenticity of these certificates after downloading.
node-output.example.com
node-output.example.com Adding certificate for chef-server.example.com in /etc/chef/trusted_certs/chef-server.example.com.crt
The output will be interleaved for all the nodes returned by knife ssh
. Of course, we should verify the SHA256 checksums like before, which can be done again with knife ssh
.