Question: How to create an encrypted XFS filesystem that’s automatically unlocked at boot using clevis (client) and tang (server)?

Red Hat has included disk encryption for years with Linux Unified Key Setup-on-disk-format (LUKS). This solution is easy to implement and configure for your encryption needs, but the management and practicality of its key management is horrible for servers. It requires a passphrase at boot or mount time which has to be entered manually. This makes the solution a headache for system administrators.

Starting in RHEL 7.4, with complete support in RHEL 7.5, Red Hat has implemented an additional component that can be leveraged to enable LUKS disks remotely. This is called Network Bound Disk Encryption (NBDE).

Network Bound Disk Encryption (NBDE) is:

  • Linux Unified Key Setup (LUKS) is a disk encryption standard.
  • Cryptsetup configures disk based encryption and includes support for LUKS
  • Tang is a network service that provides cryptographic services over HTTP
  • Clevis is an encryption framework. Clevis can use keys provided by Tang as a passphrase to unlock LUKS volumes
  • The client, clevis, has to be CentOS/RHEL 8, as clevis on CentOS/RHEL 7 has limited functionality and requires a different set of commands which are not covered in this post.
  • The server, tang, can be ran on CentOS/RHEL 7 or 8

Setup Tang Server

1. Install the RPMs:

2. Allow the required port through the firewall:

# firewall-cmd --add-service=http --permanent
# firewall-cmd --reload

3. Enable the service:

# systemctl enable --now tangd.socket

Setup Clevis Client

Create an encrypted filesystem

1. Install necessary packages on the client:

# dnf install -y cryptsetup clevis-systemd clevis-luks

2. Creating an crypted disk on /dev/xvdc:

Note: Ensure xvdc is an empty drive as all data on it will be deleted.

# cryptsetup luksFormat --type luks2 --cipher aes-xts-plain64 --key-size 512 --hash sha256 --use-random /dev/xvdc
WARNING!
========
This will overwrite data on /dev/xvdc irrevocably.
Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/xvdc:
Verify passphrase:

3. Unlock the block device:

# cryptsetup --verbose luksOpen /dev/xvdc demodisk
Enter passphrase for /dev/xvdc:
Key slot 0 unlocked.
Command successful.

4. Create a filesystem on the encrypted disk:

# mkfs.xfs /dev/mapper/demodisk
meta-data=/dev/mapper/demodisk isize=512 agcount=4, agsize=326656 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=1306624, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0

5. Identify the UUID of the new filesytem:

# blkid -s UUID /dev/mapper/demodisk
/dev/mapper/demodisk: UUID="24f9bfe9-1f6d-491d-8fa2-eab946464166"

6. Create a fstab entry for the filesystem:

# echo "UUID=24f9bfe9-1f6d-491d-8fa2-eab946464166 /encrypted xfs defaults 0 0" | sudo tee -a /etc/fstab
UUID=24f9bfe9-1f6d-491d-8fa2-eab946464166 /encrypted xfs defaults 0 0

7. Mount the filesystem:

# mkdir /encrypted
# mount /encrypted

Add a remote key to the encrypted device

1. View the keys:

# cryptsetup luksDump /dev/xvdc
LUKS header information
Version: 2
Epoch: 3
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: 47e0d1c3-d74b-41f0-9bc7-5ea367e9cb26
Label: (no label)
Subsystem: (no subsystem)
Flags: (no flags)
Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 512 [bytes]

Keyslots:
0: luks2
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: argon2i
Time cost: 4
Memory: 399588
Threads: 4
Salt: 0d 75 99 50 22 e2 2a 93 82 51 fc 49 6f 54 61 fe
10 b6 24 62 21 2e 07 2b 04 0a 56 c8 03 23 6f 8f
AF stripes: 4000
AF hash: sha256
Area offset:32768 [bytes]
Area length:258048 [bytes]
Digest ID: 0
Tokens:
Digests:
0: pbkdf2
Hash: sha256
Iterations: 30284
Salt: de a0 3f 92 e1 c7 1d 5e 0e 52 65 37 19 84 2b 3c
46 2a ba 56 77 25 78 c3 27 fa 5b 07 e1 9a 25 24
Digest: 5b 95 93 af 0e a9 8d 24 cb 35 1e 77 e5 9f 15 85
eb d1 53 85 5b e4 10 2a 68 d0 11 6b 9b 71 f4 05

2. Add remote key:

# clevis luks bind -d /dev/xvdc tang '{"url":"http://"}'
The advertisement contains the following signing keys:

KlbbdbNpdMrVwrk6hZ1wCCeabOY

Do you wish to trust these keys? [ynYN] Y
Enter existing LUKS password:

3. Show there is a new keyslot in use by Clevis in slot 1:

# cryptsetup luksDump /dev/xvdc
LUKS header information
Version: 2
Epoch: 5
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: 47e0d1c3-d74b-41f0-9bc7-5ea367e9cb26
Label: (no label)
Subsystem: (no subsystem)
Flags: (no flags)

Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 512 [bytes]

Keyslots:
0: luks2
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: argon2i
Time cost: 4
Memory: 399588
Threads: 4
Salt: 0d 75 99 50 22 e2 2a 93 82 51 fc 49 6f 54 61 fe
10 b6 24 62 21 2e 07 2b 04 0a 56 c8 03 23 6f 8f
AF stripes: 4000
AF hash: sha256
Area offset:32768 [bytes]
Area length:258048 [bytes]
Digest ID: 0
1: luks2
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: argon2i
Time cost: 4
Memory: 508554
Threads: 4
Salt: 21 c8 91 58 22 9f 50 83 77 6f fe 12 0b 3a 66 3c
d2 47 70 88 45 70 5e f1 c1 1e d0 e1 8f 96 97 ce
AF stripes: 4000
AF hash: sha256
Area offset:290816 [bytes]
Area length:258048 [bytes]
Digest ID: 0
Tokens:
0: clevis
Keyslot: 1
Digests:
0: pbkdf2
Hash: sha256
Iterations: 30284
Salt: de a0 3f 92 e1 c7 1d 5e 0e 52 65 37 19 84 2b 3c
46 2a ba 56 77 25 78 c3 27 fa 5b 07 e1 9a 25 24
Digest: 5b 95 93 af 0e a9 8d 24 cb 35 1e 77 e5 9f 15 85
eb d1 53 85 5b e4 10 2a 68 d0 11 6b 9b 71 f4 05

Mount the encrypted filesystem on boot

1. Identify the block device UUID for later use:

# blkid -s UUID /dev/xvdc
/dev/xvdc: UUID="47e0d1c3-d74b-41f0-9bc7-5ea367e9cb26"

2. Enable the service using the UUID discovered for ‘/dev/xvdc’:

# systemctl enable [email protected]
Created symlink /etc/systemd/system/basic.target.wants/[email protected] → /usr/lib/systemd/system/[email protected]

3. Have the block device unlocked during boot:

# echo "encrypteddisk UUID=47e0d1c3-d74b-41f0-9bc7-5ea367e9cb26 - _netdev" | sudo tee -a /etc/crypttab
encrypteddisk UUID=47e0d1c3-d74b-41f0-9bc7-5ea367e9cb26 - _netdev

4. Mount the filesystem later in the boot:

change the enty:

UUID=24f9bfe9-1f6d-491d-8fa2-eab946464166 /encrypted xfs defaults 0 0

to:

/dev/mapper/encrypteddisk /encrypted xfs _netdev 0 0

5. Restart the server and verify functionality:

# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 8.5M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/mapper/ol_dhcp-root 22G 1.6G 20G 8% /
/dev/xvda1 1014M 221M 794M 22% /boot
tmpfs 365M 0 365M 0% /run/user/0
/dev/mapper/encrypteddisk 5.0G 68M 5.0G 2% /encrypted

Optional: Remove known passphrase

You can remove the known passphrase from slot 0 and mandate use of the tang server. If the tang server is unreachable the disk cannot be unlocked and filesystem inaccessible.

1. Remove the passphrase:

# cryptsetup --verbose luksRemoveKey /dev/xvdc
Enter passphrase to be deleted:
Key slot 0 unlocked.
Keyslot 0 is selected for deletion.
Key slot 0 removed.
Command successful.

2. Show that there are now only Clevis keyslots:

# cryptsetup --verbose luksRemoveKey /dev/xvdc
Enter passphrase to be deleted:
Key slot 0 unlocked.
Keyslot 0 is selected for deletion.
Key slot 0 removed.
Command successful.
# cryptsetup luksDump /dev/xvdc
LUKS header information
Version: 2
Epoch: 6
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: 47e0d1c3-d74b-41f0-9bc7-5ea367e9cb26
Label: (no label)
Subsystem: (no subsystem)
Flags: (no flags)

Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 512 [bytes]

Keyslots:
1: luks2 
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: argon2i
Time cost: 4
Memory: 508554
Threads: 4
Salt: 21 c8 91 58 22 9f 50 83 77 6f fe 12 0b 3a 66 3c
d2 47 70 88 45 70 5e f1 c1 1e d0 e1 8f 96 97 ce
AF stripes: 4000
AF hash: sha256
Area offset:290816 [bytes]
Area length:258048 [bytes]
Digest ID: 0
Tokens:
0: clevis
Keyslot: 1
Digests:
0: pbkdf2
Hash: sha256
Iterations: 30284
Salt: de a0 3f 92 e1 c7 1d 5e 0e 52 65 37 19 84 2b 3c
46 2a ba 56 77 25 78 c3 27 fa 5b 07 e1 9a 25 24
Digest: 5b 95 93 af 0e a9 8d 24 cb 35 1e 77 e5 9f 15 85
eb d1 53 85 5b e4 10 2a 68 d0 11 6b 9b 71 f4 05

3. Reboot and verify functionality:

# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs 1.8G 8.5M 1.8G 1% /run
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup
/dev/mapper/ol_dhcp-root 22G 1.6G 20G 8% /
/dev/xvda1 1014M 221M 794M 22% /boot
/dev/mapper/encrypteddisk 5.0G 68M 5.0G 2% /encrypted
tmpfs 365M 0 365M 0% /run/user/0