Quick draft on why your server might be compromised!

Photo by Massimo Botturi on Unsplash

Why your server might be compromised

• Improper firewall configuration

• Lack of web application firewall 

• Server hardening is a must 

• File integrity monitoring should always be there 

• Vulnerability assessment 

• Patch management 

• Always scan code with a web application scanner

• Monitor for sudden open ports and logs

Incident response: 

In my previous environment, our ec2 was compromised, and they are several steps that I considered as best practices when such a thing happened. Which I suggested and we performed, such as 

1. locking the instance down 

2. Take the ebs snapshot

3. Perform a memory dump 

4. Perform a forensic analysis 

5. Terminate the instance

Locking the instance down is performed just in case we have any brute force, or it is attacking some other services in my environment. The purpose of the action is to prevent my instance from being a malware spreader and the potential of it bringing my other services down with it … this could be extremely dangerous.

This instance should not contact the outside world. It should only have access to the forensic server for examination. 

With this aspect, automation is critical. This was done with a gui script. So the instance and the script will be posted inside the gui. The gui will remove all the security groups and place a forensic group. And this security group would allow the connection from one of our forensic machines, so nothing else was allowed. SO as you can see, automation is critical.

Any instance that is compromised, you just put the instance id and put them inside, and the script handles the rest.

EBS snapshot and memory dump

This step is crucial. EBS snapshot will help ensure that all the files in the server will be snapshotted - which would later be used in the forensic. Although more than this step is needed because a malicious script is likely already running.

Then the next step is to perform a memory dump to ensure that all the processes running in memory are being dumped.

The next few steps, or should the next last steps to perform forensics and terminate the instance.

Having the ebs snapshot and the memory dump should be a good base for performing forensics.

Once forensics are done and RCA ( root cause analysis) is released, we can terminate the ec2 instance.

Applying RCA

Applying the RCA process helps you understand the causes of a failure and how to prevent it from happening again.

Effective RCA requires focusing on eliminating the problem rather than just understanding it. It would be best if you were wary of cognitive biases that make you concentrate on recent occurrences (vs all) or look for data that confirms your assumptions.

General process:

• Define problem

• Collect Data

• Identify Root Causes

  • Use Five whys and Ishikawa diagrams

• Implement Countermeasures

• Evaluate & Follow-up

5 Whys approach

Five whys is helpful as it provides a consistent approach to understanding causes. It helps overcome assumptions that we might have. The process is easy to learn and use, either individually or with a whole team. Five whys should be applied blame-free, focusing on finding the "why" rather than blaming the "who".

How

• Identify the problem

• Ask why the problem happened, and record the reason.

• Decide if the reason is the root cause

  • Could the reason have been prevented?

  • Could the reason have been detected before it happened?

  • If the reason is human error, why was it possible?

• Repeat the process using the reason as the problem. Stop when you are confident that you have found the root causes.

The Ishikawa diagram 

Guide to creating a snapshot 

Steps by Step Guide to Create a Snapshot

1. Sign in to the AWS console and select Amazon EC2 service

2. In the navigation pane, click Snapshots under Elastic Block Store

3. Click on Create Snapshot

4. Choose Volume as the resource type

5. Select the volume for Volume ID

6. Provide the appropriate name and a description for the snapshot. (Optional)

7. Add tag. Provide a tag key and a tag value for each tag. (Optional)

Click on Create snapshot.

How to copy an Amazon EBS Snapshot?

You can create point-in-time snapshots of volumes with Amazon EBS, which we will store for you in Amazon S3. You can copy a snapshot from one AWS Region to another or within the same Region when it has finished copying to Amazon S3 (when the snapshot status is completed). The ID assigned to the snapshot copy differs from the ID assigned to the original snapshot.

Steps to copy a Snapshot

1. In the navigation pane, choose Snapshots

2. Select the snapshot to be copied, and then choose Copy from the Actions list

3. Provide the appropriate description for the snapshot

4. In Destination Region, select the region where you want the snapshot to be copied

5. If your account is enabled for encryption by default, or if the original snapshot is encrypted, the snapshot copy is encrypted by default, and you cannot change the encryption state. If the source snapshot is unencrypted and your account is not enabled for encryption by default, encryption is optional. To encrypt the snapshot copy, checkbox the Encrypt this snapshot option. Then, for the KMS key, select the appropriate KMS key.

6. Choose Copy snapshot