Securing Server Connections with TLS / SSL

If you have a public server with general services such as SMTP, POP3, and Web Services, you need to design security for your server. One way to secure the server is by restricting access to it. Essentially, this method filters connections going to or coming from the server. Connections are limited to those that are explicitly allowed, while others are denied.

This is necessary because connections may not be secure; for instance, ongoing connections can be intercepted using network sniffing tools by third parties to obtain valuable information such as usernames, emails, or passwords.

This can occur because data packets or connections are sent in plain text. Therefore, encryption technology is used to secure the connection. Encryption technology is used by the server to encrypt data packets before they are sent to the client. Here, we will discuss how to use TLS/SSL to secure server connections. TLS/SSL can be used to secure all server connections that support TLS/SSL, such as email servers, web servers, and LDAP servers.

In this tutorial, we will use OpenSSL. OpenSSL is an application that implements SSL (Secure Socket Layer) and TLS (Transport Layer Security) protocols. It is used to create the certificates needed by the server and the server applications to secure their connections.

Step 1: Installation

Update the repository and install OpenSSL

$ sudo apt update
$ sudo apt install openssl

Step 2: Create a Self-Signed Certificate

There are two ways to create a self-signed certificate: either directly without creating a CA or by first creating a CA which will then be used to sign all certificates for each host or application that requires it.

First, we create the CA's private key.

$ openssl genrsa -des3 -out ca-example.web.id.key 4096
Generating RSA private key, 4096 bit long modulus
......................++
.......................................++
e is 65537 (0x010001)
Enter pass phrase for ca-example.web.id.key:
Verifying - Enter pass phrase for ca-example.web.id.key:

The command above will call the genrsa application to generate a CA private key using the DES3 cipher and a length of 4096 bits. This private key will be saved to the file ca-example.web.id.key.

You must securely store the private key you have. Do not let others even read your server's private key. For security reasons, change the file permissions of your private key to 600 and change the file ownership to root.

$ sudo chmod 600 ca.example.web.id.key
$ sudo chown root ca.example.web.id.key

Next, we create the CA public certificate

$ openssl req -new -x509 -days 365 -key ca-example.web.id.key -out ca-example.web.id.crt
Enter pass phrase for ca-example.web.id.key: ******
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:ID
State or Province Name (full name) [Some-State]:Central Java
Locality Name (eg, city) []:Surakarta
Organization Name (eg, company) [Internet Widgits Pty Ltd]:PT Example
Organizational Unit Name (eg, section) []:IT Division - PT Example
Common Name (e.g. server FQDN or YOUR name) []:example.web.id
Email Address []:[email protected]

Next, we create the CA public certificate.

The command above will call the req application to create a new public certificate ca-example.web.id.crt using the CA private key ca-example.web.id.key that we created earlier. The public certificate we create is valid for 365 days or one year.

The command will use the CA private key, so it will ask for the password for the private key. It will also request the necessary information to include the common name in the certificate. Use your domain name as the common name for the CA certificate.

For the Common Name question, you should answer with your domain name, which in this case is example.web.id.

If you plan to use a commercial CA service to sign your certificate request, you simply need to submit the certificate signing request (CSR) file. For the domain example.web.id, you can send the CA public certificate example.web.id created without the -x509 option.

Create a certificate signing request (CSR) that will later be signed by the CA

$ openssl req -new -key example.web.id.key -out example.web.id.csr
Enter pass phrase for ca-example.web.id.key: ******
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:ID
State or Province Name (full name) [Some-State]:Central Java
Locality Name (eg, city) []:Surakarta
Organization Name (eg, company) [Internet Widgits Pty Ltd]:PT Example
Organizational Unit Name (eg, section) []:IT - PT Example
Common Name (e.g. server FQDN or YOUR name) []:example.web.id
Email Address []:[email protected]

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

For the last two extra questions, you can simply press enter without filling them in.

A brief explanation:

genrsa: This application is used to create a private key using the RSA algorithm.

req: This application processes certificates in PKCS#10 format and can also be used to create self-signed certificates.

x509: This application can be used to display certificate information, convert, sign, and edit certificates.

ca: This application can be used to sign certificate requests.

Conclusion

Self-signed certificates are often considered questionable in terms of authenticity. Why? Think about it logically. Who would trust a certificate that was created and signed by itself? No one. Therefore, commercial CAs take advantage of this by providing services to validate the authenticity of the certificates we create.

So, who validates the authenticity of these commercial CAs? The answer is no one. Their certificates are trusted because their public certificates are included in client applications like browsers. This way, the client applications assume the certificates are trustworthy. Thus, all certificates signed by the commercial CA are considered trustworthy.

That's it, see ya~