Containerized Testing with Kerberos and SSH

Kerberos authentication is widely used in today’s client/server applications; however getting started with Kerberos may be a daunting task if you don’t have prior experience. Information on setting up Kerberos with an SSH server and client on the web is fragmented and hasn’t been presented in a comprehensive end-to-end way on a simple local setup.

At Confluent, several of our connectors for Apache Kafka^® support Kerberos-based authentication. For development and testing of these connectors, we often leverage containers due to their fast, iterative benefits. This tutorial aims to provide a simple setup for a Kerberos test environment with SSH for a passwordless authentication that uses Kerberos tickets. You may use this as a guide for testing the Kerberos functionality of SSH-based client-server applications in a local environment or as a hands-on tutorial if you’re new to Kerberos. To understand the basics of Kerberos before diving into this tutorial, you may find this video helpful. Additionally, if you are looking for a non-SSH-based setup, the setup below for the KDC server container may also be useful.

The setup will consist of three parts:

1. Client: a client SSH application that authenticates via Kerberos to the SSH server
2. KDC server: a local KDC server in a container
3. Server: an SSH server with a user

The software packages used in this tutorial are OpenSSH Server and MIT Kerberos. The configuration may be different for other package distributions. The following steps are performed on a Mac, but a few tweaks may be needed for other platforms.

We will explore two ways for the client to be set up: (1) as a client Docker container and (2) as a local client on your host machine. The latter is useful for conveniently testing applications from a local run environment, such as running from an IDE without having to configure a client Docker container.

The container images for this tutorial have been pre-configured with the required package dependencies, but Kerberos needs to be set up manually. This tutorial explains how to configure the containers in the following way:

1. KDC server: We will configure the hosts file for a correct Kerberos lookup in a Docker environment, as well as adding the user and service principals
2. SSH server: We will review how to configure the SSH server to support Kerberos authentication and add a user and the server Kerberos sshserver.keytab file
3. SSH client: We will review how to configure the client to use Kerberos tickets as part of the SSH authentication and how to configure the Kerberos client to point to the correct KDC in the client krb5.conf file

Approach 1: Triple container setup with the container client

As a first step, start all three required containers. To start the KDC server, use the provided kdc-server image with docker-compose up from the subdirectory. The KDC server container also maps the localhost ports 749 and 88 in case you would like to perform testing or operations on the KDC from your local machine. To start the SSH client and server, simply call docker-compose up again from the ssh-container subdirectory. A successful start of the containers should resemble the below output on docker ps.

CONTAINER ID   IMAGE                      COMMAND                  CREATED          STATUS         PORTS                                      NAMES
8845aa0f05cf   kdc-server_kdc-server      "/bin/sh -c /tmp/ini…"   3 seconds ago    Up 3 seconds   0.0.0.0:88->88/udp, 0.0.0.0:749->749/tcp   kdc-server_kdc-server_1
8fb55d49237a   ssh-container_ssh-server   "/usr/sbin/sshd -D"      4 seconds ago    Up 4 seconds   0.0.0.0:2222->22/tcp                       ssh-container_ssh-server_1
e154b902e3de   ssh-container_ssh-client   "/bin/bash"              14 minutes ago   Up 4 seconds   22/tcp                                     ssh-container_ssh-client_1

KDC

Once the KDC container is running, switch into the container’s bash session.

docker exec -it kdc-server_kdc-server_1 /bin/bash

From within the container, start the kadmin shell using the command kadmin.local.

If you list the principals, you can see the defaults as below:

kadmin.local:  listprincs
K/M@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/kdc-server@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
noPermissions@EXAMPLE.COM

Here, the kadmin/admin@EXAMPLE.COM and noPermissions@EXAMPLE.COM were added by the init script of the container and may be used for KDC admin testing purposes. kadmin/kdc-server@EXAMPLE.COM is the principal for the Docker container and is added automatically. You can see that the host of the principal is the same as the host name of the container. The rest of the principals are created by default and are not used in the rest of this tutorial.

Using kadmin, we will now add the principals for the SSH server and client.

In order to add the principal for the SSH server, we first need to identify the host name of the SSH server container. In our case, this will be ssh-server. If you are using your own container, it may be the same as the container ID.

kadmin.local:  addprinc -randkey host/ssh-server@EXAMPLE.COM
WARNING: no policy specified for host/ssh-server@EXAMPLE.COM; defaulting to no policy
Principal "host/ssh-server@EXAMPLE.COM" created.

Next, we will create the keytab file for the service principal that will need to be placed on the SSH server container.

kadmin.local:  ktadd -k /sshserver.keytab host/ssh-server@EXAMPLE.COM
Entry for principal host/ssh-server@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/sshserver.keytab.

The command above will save the keytab file as sshserver.keytab in the root of the KDC server container.

Now we will add the principal for the (future) user on the SSH server. The username here will need to be identical to the username on the SSH server. Here, we will simply use sshuser as the username.

Method 1: Keytab

This method will save the Kerberos key of the user principal in the same way as above for the host and will need to be transferred to the SSH client.

kadmin.local:  addprinc -randkey sshuser
WARNING: no policy specified for sshuser@EXAMPLE.COM; defaulting to no policy
Principal "sshuser@EXAMPLE.COM" created.
kadmin.local:  ktadd -k /sshuser.keytab sshuser@EXAMPLE.COM
Entry for principal sshuser@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/sshuser.keytab.

Method 2: Password

This method will assign a password to the Kerberos user principal and will require that you specify the password instead of a keytab file on the SSH client when initializing using kinit.

kadmin.local:  addprinc sshuser
WARNING: no policy specified for sshuser@EXAMPLE.COM; defaulting to no policy
Enter password for principal "sshuser@EXAMPLE.COM":
Re-enter password for principal "sshuser@EXAMPLE.COM":
Principal "sshuser@EXAMPLE.COM" created.

After adding the principals, the results of the listprincs command should look like what you see below:

kadmin.local:  listprincs
K/M@EXAMPLE.COM
host/ssh-server@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/kdc-server@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
noPermissions@EXAMPLE.COM
sshuser@EXAMPLE.COM

Once the service and user principals are created, copy the keytab file(s) from the KDC container to your local machine. The sshserver.keytab will need to be copied to the SSH server, and if you are using the keytab Method 1 above, the sshuser.keytab to the SSH client.

docker cp kdc-server_kdc-server_1:/sshserver.keytab ./sshserver.keytab
docker cp kdc-server_kdc-server_1:/sshuser.keytab ./sshuser.keytab

Final step

To make sure the KDC is able to look up the SSH server container, we will need to create an entry in the /etc/hosts file on the KDC container. We will need to find the IP address of the SSH server container, which can be done from the SSH container or using the handy command below:

docker inspect -f '{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $(docker ps -aq)

/kdc-server_kdc-server_1 - 172.21.0.2
/ssh-container_ssh-server_1 - 172.31.0.3
/ssh-container_ssh-client_1 - 172.31.0.2

We see that our SSH server has the IP 172.31.0.3.

Now all we need to do is add the entry that maps the SSH server hostname to its IP address in /ets/hosts on the KDC server. Note that if you are using your own container, the host name may be different, but the default case is the same as the container ID. The KDC container has Vim installed for convenience. After the changes, the /etc/hosts file should look similar to the one below, where the last entry is the SSH container:

127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.21.0.2      kdc-server
172.31.0.3      ssh-server

SSH server

The setup for the SSH server is simpler than the KDC server. Keep in mind that the file sshd_config is responsible for SSH server configurations (note the extra letter d), while ssh_config contains the configurations for the SSH client.

In order to enable the SSH server to accept Kerberos tickets as part of the authentication, we must enable the following configurations in /etc/ssh/sshd_config.

GSSAPIAuthentication yes
GSSAPICleanupCredentials yes

These configurations are automatically turned on in the ssh-server container based on the sshd_config in the ssh-container subdirectory, but on other systems, they may need to be added or uncommented manually. Keep in mind that for changes in sshd_config to take effect, you will need to reload the SSH server. The relevant configurations for ticket-based authentication will be the ones starting with GSSAPI*. For more on the configurations, refer to the manual. Note that the configuration #KerberosAuthentication no remains off as this would require the re-entry of our Kerberos password and would not allow using only existing tickets, which would impede our passwordless authentication goal.

Now we will configure the /etc/krb5.conf file on the SSH server container to point to the correct KDC and allow for forwardable tickets. If you are using the container from this tutorial, you will need to overwrite the existing krb5.conf file that was created by the krb5-user package. You can switch into the container using the command below:

docker exec -it ssh-container_ssh-server_1 /bin/bash

In a real-world scenario, the IP address of the KDC server would be used instead of the host.docker.internal address. We’ll use the host.docker.internal address because the KDC container’s ports are exposed on the host machine.

[libdefaults]
    default_realm = EXAMPLE.COM
    forwardable = TRUE
[realms]
    EXAMPLE.COM = {
            kdc_ports = 88
            kadmind_port = 749
            kdc = host.docker.internal
            admin_server = host.docker.internal
    }
[domain_realm]
        host.docker.internal = EXAMPLE.COM

The host.docker.internal host name is specific to Docker on Mac and Windows. If you are using Linux, you will need to configure your /etc/hosts file or apply an alternate workaround to get the setup working.

Next, we will copy the sshserver.keytab that we created earlier into the SSH server container under the path /etc/krb5.keytab. Note the name change of the file—this is so that the SSH server can find the Kerberos host keytab on the default path that it checks for.

docker cp ./sshserver.keytab ssh-container_ssh-server_1:/etc/krb5.keytab

Lastly, we will create the user sshuser on the SSH server:

root@ssh-server:/# adduser sshuser
Adding user `sshuser' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for sshuser
Enter the new value, or press ENTER for the default
	Full Name []: test
	Room Number []:
	Work Phone []:
	Home Phone []:
	Other []:
Is the information correct? [Y/n] Y

As a sanity check, we can try to SSH into the new user account with the password that was created above. This is also the same password that is required to SSH from another client when Kerberos ticket authentication is not used.

root@ssh-server:/# ssh sshuser@localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is ad:48:a6:ca:20:b4:97:12:40:78:2f:52:15:47:12:d8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
sshuser@localhost's password:
...
sshuser@ssh-server:~$

The SSH server is now ready to accept a ticket-based login for the user ssshuser. If for any reason you would like to test SSH from your host machine, you may use ssh sshuser@localhost -p 2222, as the container maps the port 2222 by default to the host machine.

SSH client

Let’s bring it all together. Now that we have the KDC server and the SSH server setup, the only remaining element is the client. First, let’s switch into the SSH client container:

docker exec -it ssh-container_ssh-client_1 /bin/bash

Configure /etc/hosts by adding the mapping for the SSH server container’s host name and IP.

172.31.0.3      ssh-server

As a check, we will attempt to SSH directly to the server with the username and password. For this, we will use the host name of the server container, in our case, simply ssh-server:

root@ssh-client:/# ssh sshuser@ssh-server
sshuser@ssh-server's password:
...
sshuser@ssh-server:~$

This confirms that the user was set up correctly and that our SSH connection to the server works properly. We can exit from the session back to the container shell.

The important configurations for our use case in the SSH client config /etc/ssh/ssh_config start with GSSAPI*. For our tutorial, we will need minimal configuration.

GSSAPIAuthentication yes

The configuration above is configured for the ssh-client container automatically during the build. For the other GSSAPI* configurations applicable in the SSH client, please refer to the documentation.

We will now configure the /etc/krb5.conf file used by the client to establish communication with the KDC server. If you are using the container from the tutorial, you will (again) need to overwrite the existing krb5.conf file.

[libdefaults]
    default_realm = EXAMPLE.COM
    forwardable = TRUE
[realms]
    EXAMPLE.COM = {
            kdc_ports = 88
            kadmind_port = 749
            kdc = host.docker.internal
            admin_server = host.docker.internal
    }
[domain_realm]
        host.docker.internal = EXAMPLE.COM 

If you used Method 1 for the sshuser principal, you may now copy the sshuser.keytab to the SSH client container:

docker cp ./sshuser.keytab ssh-container_ssh-client_1:/sshuser.keytab

We can now initialize our Kerberos ticket using kinit.

If you used a keytab for the user principal, the command will look like this:

root@ssh-client:/# kinit sshuser -k -t sshuser.keytab

If you used a password (Method 2), the command will prompt you for the Kerberos password that you have configured on the KDC. Note that this password is different from the user password on the SSH server.

root@ssh-client:/# kinit sshuser
Password for sshuser@EXAMPLE.COM:

If the above kinit command returns without connection errors, this means that the client was able to reach the KDC successfully based on the krb5.conf settings. After initializing the TGT ticket for the user principal, you can verify if the ticket was cached correctly using klist.

root@ssh-client:/# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: sshuser@EXAMPLE.COM

Valid starting     Expires            Service principal
01/03/21 05:21:10  01/04/21 05:21:06  krbtgt/EXAMPLE.COM@EXAMPLE.COM

Finally, we can now connect to our SSH server using Kerberos tickets and without needing a password.

root@ssh-client:/# ssh sshuser@ssh-server

The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Sun Jan 3 05:20:33 2021 from ssh-container_ssh-client_1.ssh-container_default sshuser@ssh-server:~$

Approach 2: Single-container setup with a local client

This container setup combines the KDC and SSH server into one container. It enables you to use your host machine as the SSH client and test applications from your local environment. To start the container, simply use docker-compose up from the kdc-ssh-server directory.

CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS         PORTS                                                                    NAMES
42dbfa15c4b4   kdc-ssh-server_kdc-ssh-server   "/bin/sh -c /tmp/ini…"   53 seconds ago   Up 5 seconds   88/tcp, 0.0.0.0:88->88/udp, 0.0.0.0:749->749/tcp, 0.0.0.0:2222->22/tcp   kdc-ssh-server_kdc-ssh-server_1

KDC in the combined container

This KDC setup is similar to part 1 of the tutorial but differs in some of the key steps. To switch into the container, use the following command:

docker exec -it kdc-ssh-server_kdc-ssh-server_1 /bin/bash

Initially, the KDC principals should be the ones listed below:

kadmin.local:  listprincs
K/M@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/kdc-ssh-server@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
noPermissions@EXAMPLE.COM

We need to add a service principal for ssh-server, along with a keytab, which will be saved in the default /etc/krb5.keytab location. Note that this step differs from part 1 of the tutorial in the keytab location. Since the SSH server will be running on the same container, the keytab can be saved directly to the default location.

kadmin.local:  addprinc -randkey host/ssh-server@EXAMPLE.COM
WARNING: no policy specified for host/ssh-server@EXAMPLE.COM; defaulting to no policy
Principal "host/ssh-server@EXAMPLE.COM" created.
kadmin.local:  ktadd -k /etc/krb5.keytab host/ssh-server@EXAMPLE.COM
Entry for principal host/ssh-server@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/krb5.keytab.

You can create the sshuser principal in the same way that we did in part 1. After creating the principals, the list should look like this:

kadmin.local:  listprincs
K/M@EXAMPLE.COM
host/ssh-server@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/kdc-ssh-server@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
noPermissions@EXAMPLE.COM
sshuser@EXAMPLE.COM

Next, we need to edit the /etc/hosts file. On the first line, add a mapping for the container’s IP to the ssh-server host name (the same IP as the last mapping). The SSH command from the host machine will use the ssh-server host name, and this mapping will enable the KDC to look up and find the SSH server on the same container.

172.22.0.2      ssh-server
127.0.0.1      localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.22.0.2      kdc-ssh-server

SSH server in the combined container

In this version of the container, the build and init scripts take care of the krb5.conf configuration and the sshd_config. We only need to create the user and make sure the host keytab that we created in the KDC step is under /etc/krb5.keytab. We can create the user the same way that we did in part 1.

Local SSH client

This part may require some system-specific configuration. Make sure that your system has a Kerberos client and SSH client installed. Your setup should be ready if the commands kinit and ssh work.

Add an entry to your local /etc/hosts file. The entry will map the localhost’s IP address 127.0.0.1 to the ssh-server host name.

127.0.0.1 ssh-server

To test if the user was created successfully earlier and that the container’s SSH connection is open, you can try to SSH from your host machine into the container.

The exposed container port for the SSH server is 2222, so the command will look like this:

ssh sshuser@ssh-server -p 2222
sshuser@localhost's password:
...
sshuser@kdc-ssh-server:~$

Let’s configure the SSH client to attempt ticket-based authentication. This may be configured in either /etc/ssh/ssh_config globally or in ~/.ssh/config for a specific user. The key line to add is the one below, which enables the client to use the tickets. There may be additional GSSAPI* options that you may want to configure depending on your preferences. For our example, we only use the minimum requirement.

GSSAPIAuthentication yes

Now we need to configure the /etc/krb5.conf to point to the KDC at the localhost. This configuration will be similar to that in part 1, but the IP address used will be for the host machine’s localhost.

[libdefaults]
    default_realm = EXAMPLE.COM
    forwardable = TRUE
[realms]
    EXAMPLE.COM = {
            kdc_ports = 88
            kadmind_port = 749
            kdc = 127.0.0.1
            admin_server = 127.0.0.1
    }
[domain_realm]
        127.0.0.1 = EXAMPLE.COM

Initialize the TGT with kinit. Depending on whether you created a keytab or password for the user principal, you will need to use the correct method for kinit, which is explained in part 1 under the SSH client section.

kinit sshuser
sshuser@EXAMPLE.COM's password:

A klist command should provide the result below:

Credentials cache: API:CC2F710D-E71E-4F56-A1B0-75144E3D0015
        Principal: sshuser@EXAMPLE.COM

  Issued                Expires               Principal
Jan  4 01:59:42 2021  Jan  4 11:59:37 2021  krbtgt/EXAMPLE.COM@EXAMPLE.COM

Finally, we can SSH to the server using the Kerberos ticket and without a password.

ssh sshuser@localhost -p 2222

The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Mon Jan 4 01:20:15 2021 from 172.22.0.1 sshuser@kdc-ssh-server:~$

Helpful notes for troubleshooting

If you currently have a cached ticket and it’s interfering with the Kerberos authentication, or if you made a mistake earlier, you may remove it using the command below at your own risk:

kdestroy -A

If you’re having issues at the SSH stage or would like to see the process of ticket authentication, you can get a verbose output from the SSH client command:

ssh sshuser@ssh-server -vvv

A successful Kerberos authentication will resemble the following:

debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 60
debug3: send packet: type 61
debug3: receive packet: type 61
debug3: send packet: type 66
debug3: receive packet: type 52
debug1: Authentication succeeded (gssapi-with-mic).

A failed authentication will look similar to the one below—in some cases, stating the cause of the error, though in others, the issue may be less obvious. In general, issues such as the one below arise from a misstep during configuration. In this case, the SSH server’s IP was mapped incorrectly on the KDC server in /etc/hosts.

debug1: Next authentication method: gssapi-with-mic
debug3: send packet: type 50
debug2: we sent a gssapi-with-mic packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1:  An invalid name was supplied
unknown mech-code 0 for mech 1 2 752 43 14 2
debug1:  Miscellaneous failure (see text)
unknown mech-code 0 for mech 1 3 6 1 5 5 14
debug1:  Miscellaneous failure (see text)
unknown mech-code 2 for mech 1 3 6 1 4 1 311 2 2 10

For a verbose output from kinit, you may use the command below. This may be helpful when you have trouble reaching the KDC or looking up the user principal.

kinit sshuser -k -t sshuser.keytab -V

If you encounter the unable to reach KDC error below with tried 1 KDC, the issue may be a misconfigured KDC container. In this case, you should verify the /etc/hosts configuration. If nothing helps, you can try to stop and remove the container completely and rebuild it from scratch.

kinit sshuser
sshuser@EXAMPLE.COM's password:
kinit: krb5_get_init_creds: unable to reach any KDC in realm EXAMPLE.COM, tried 1 KDC

Next steps

Now that you have a Kerberos setup and an understanding of the basic concepts, we encourage you to try our Elasticsearch, SFTP, and Cassandra connectors, all with support for Kerberos authentication.

Try Now

Special thanks to Sajana Weerawardhena for providing resources and help during my Kerberos ramp up.