How I Secured My Ubuntu 24.04 Server with SSH Key Authentication

If you're running a home server or any Linux machine that's accessible over the internet, password authentication is a ticking time bomb. Automated bots are constantly scanning for SSH servers and attempting brute-force attacks. The solution? SSH key-based authentication.

In this guide, I'll walk you through exactly how I set up SSH key authentication on my Ubuntu 24.04 LTS server, including a frustrating quirk that had me scratching my head until I figured out what was going on.

---

What We're Going to Accomplish

By the end of this guide, you'll have:

• A secure SSH key pair generated on your local machine

• Key-based authentication working on your server

• Password authentication completely disabled

• A backup strategy so you don't lock yourself out

My Setup:

• Server: Ubuntu 24.04 LTS

• Local Machine: Linux (commands are similar for macOS; Windows users can use PowerShell or WSL)

• SSH Port: Custom port (not the default 22)

---

Prerequisites

Before we start, make sure you have:

• SSH access to your Ubuntu server (currently with password authentication)

• A local Linux or macOS machine (or Windows with WSL/PowerShell)

• Basic familiarity with the terminal

---

Step 1: Generate Your SSH Key Pair

The first step happens on your local machine — not the server.

Open your terminal and run:

ssh-keygen -t ed25519 -C "yourname@your-server"

What does this do?

You'll be prompted for:

1. File location — Press Enter to accept the default (~/.ssh/id_ed25519), or specify a custom path like ~/.ssh/myserver/key

2. Passphrase — I strongly recommend setting one. This encrypts your private key, so even if someone steals the file, they can't use it without the passphrase.

Pro tip: I like to organize my keys by creating subdirectories. For example:

mkdir -p ~/.ssh/homelab
ssh-keygen -t ed25519 -C "homelab-server" -f ~/.ssh/homelab/homelab

This creates:

• ~/.ssh/homelab/homelab — your private key (never share this!)

• ~/.ssh/homelab/homelab.pub — your public key (this goes on the server)

Verify your keys were created:

ls -la ~/.ssh/
# Or if you used a custom directory:
ls -la ~/.ssh/homelab/

You should see two files. The private key should have permissions 600 (readable only by you):

-rw------- 1 user user 464 Jan 19 21:20 homelab
-rw-r--r-- 1 user user 101 Jan 19 21:20 homelab.pub

---

Step 2: Copy Your Public Key to the Server

Now we need to get your public key onto the server. The easiest way is using ssh-copy-id.

Run this on your local machine:

ssh-copy-id -i ~/.ssh/homelab/homelab.pub -p 22 youruser@your-server-ip

Adjust the command for your setup:

• -i — path to your public key (note the .pub extension)

• -p — your SSH port (22 is default, but use your custom port if you changed it)

• youruser@your-server-ip — your username and server address

You'll be asked for your server password one last time. After that, the tool will:

1. Connect to your server

2. Create the ~/.ssh directory if it doesn't exist

3. Append your public key to ~/.ssh/authorized_keys

4. Set the correct permissions

You should see output like:

Number of key(s) added: 1

---

Step 3: Test Key-Based Authentication

Before we disable password authentication, let's make sure key-based login actually works.

Run this on your local machine:

ssh -i ~/.ssh/homelab/homelab -p 22 youruser@your-server-ip

What should happen:

• If you set a passphrase on your key, you'll be prompted for it

• You should NOT be asked for your server account password

• You should be logged in!

Test from different networks too. If you're setting this up for remote access (like through a DDNS domain), make sure to test that connection as well:

ssh -i ~/.ssh/homelab/homelab -p 22 youruser@your-ddns-domain.com

---

Step 4: Disable Password Authentication

This is where things get interesting — and where Ubuntu 24.04 threw me a curveball.

⚠️ Important: Keep your current SSH session open throughout this process. We'll test changes from a new terminal window. If something goes wrong, your existing session stays connected so you can fix it.

Check the Current Configuration

On the server, let's see what the current settings look like:

sudo grep -E "^#?PasswordAuthentication|^#?KbdInteractiveAuthentication" /etc/ssh/sshd_config

You might see something like:

#PasswordAuthentication yes
KbdInteractiveAuthentication no

The # means the line is commented out, so it's using the default value (which is yes for PasswordAuthentication).

Edit the Main Config File

Let's change PasswordAuthentication to no:

sudo sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config

Verify the change:

sudo grep -E "^PasswordAuthentication" /etc/ssh/sshd_config

You should see:

PasswordAuthentication no

Restart SSH

sudo systemctl restart ssh

Test... Wait, Why Is It Still Working?

Here's where I got confused. I opened a new terminal and tried to connect with password-only authentication:

ssh -p 22 -o IdentitiesOnly=yes -o PreferredAuthentications=password youruser@your-server-ip

This command forces SSH to only try password authentication (no keys). It should fail with Permission denied (publickey).

But for me? It still asked for my password. And let me in.

---

The Ubuntu 24.04 Gotcha: Override Files

After some investigation, I found the culprit. Ubuntu 24.04 uses drop-in configuration files that can override your main sshd_config settings.

Check for override files:

grep -r "PasswordAuthentication" /etc/ssh/sshd_config.d/

And there it was:

/etc/ssh/sshd_config.d/50-cloud-init.conf:PasswordAuthentication yes

The 50-cloud-init.conf file was setting PasswordAuthentication yes, and because files in sshd_config.d/ are loaded after the main config, it was overriding my change.

Fix the Override

sudo sed -i 's/^PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config.d/50-cloud-init.conf

Verify:

cat /etc/ssh/sshd_config.d/50-cloud-init.conf

You should see PasswordAuthentication no.

Restart SSH Again

sudo systemctl restart ssh

Now Test Again

From a new terminal on your local machine:

ssh -p 22 -o IdentitiesOnly=yes -o PreferredAuthentications=password youruser@your-server-ip

This time you should see:

youruser@your-server-ip: Permission denied (publickey).

Password authentication is now disabled.

---

Step 5: Verify Key-Based Access Still Works

Just to be safe, let's confirm you can still log in with your key:

ssh -i ~/.ssh/homelab/homelab -p 22 youruser@your-server-ip

You should get in without any issues. If you set a passphrase, you'll be asked for that (which is your key passphrase, not your server password).

---

Backup Strategies: Don't Lock Yourself Out

With password authentication disabled, your private key is the only way in. Here's how to protect yourself:

Option 1: Backup Your Key Files

Copy your private key to secure storage:

• A USB drive kept in a safe place

• A password manager that supports file attachments (like Bitwarden or 1Password)

• Encrypted cloud storage

# Create a backup copy
cp -r ~/.ssh/homelab ~/.ssh/homelab-backup

# Or copy to a USB drive
cp -r ~/.ssh/homelab /media/usb/ssh-backup/

Option 2: Generate Keys on Multiple Devices

Instead of copying your private key around (which increases risk if one device is compromised), you can generate separate key pairs on each device:

1. Generate a new key pair on your laptop

2. Generate another on your desktop

3. Generate one on your phone (if using an SSH app like Termius)

Then add all public keys to your server's ~/.ssh/authorized_keys file. This way, if you lose one device, you still have access from others.

Option 3: Physical Access Fallback

If you have physical access to your server (like a home server), you can always plug in a monitor and keyboard to recover if you're completely locked out. This is my ultimate fallback — it's why I feel comfortable being aggressive with security on my home server.

---

Quick Reference: Connecting to Your Server

Once everything is set up, here are the commands you'll use:

# Basic connection
ssh -i ~/.ssh/homelab/homelab -p 22 youruser@your-server-ip

# Or via DDNS domain
ssh -i ~/.ssh/homelab/homelab -p 22 youruser@your-domain.com

Simplify with SSH Config (Optional)

Create or edit ~/.ssh/config on your local machine:

Host myserver
    HostName your-server-ip
    User youruser
    Port 22
    IdentityFile ~/.ssh/homelab/homelab

Now you can simply type:

ssh myserver

---

Key Takeaways

1. Ed25519 is the modern choice — It's more secure and faster than RSA, with shorter keys.

2. Always test before disabling password auth — Keep an existing SSH session open while making changes.

3. Ubuntu 24.04 uses override files — Check /etc/ssh/sshd_config.d/ if your changes don't seem to take effect. The 50-cloud-init.conf file often sets PasswordAuthentication yes by default.

4. Backup your private key — Without it, you're locked out. Store it securely in multiple locations.

5. Consider separate keys per device — More secure than copying one key everywhere.

6. Use a passphrase — Encrypts your private key, adding another layer of protection.

---

Summary Table

---

Final Thoughts

Setting up SSH key authentication felt like a straightforward task until Ubuntu 24.04's configuration override tripped me up. It's a good reminder to always verify that changes actually took effect, especially on newer OS versions that might handle things differently than expected.

With key-based authentication enabled and password access disabled, your server is now significantly more secure against brute-force attacks. Automated bots can scan and probe all they want — without your private key, they're not getting in.

Happy securing! 🔐

---

Found this helpful? Have questions? Drop a comment below or connect with me on [your social platform]. I'm always happy to chat about homelab setups and DevOps adventures.