Another Debugging Lesson

The other day, I was solving another problem from the 100 Days DevOps Engineer by KodeKloud, and this is what I encountered.

The requirement was clear:

• Only the load balancer (LBR) at 172.16.238.14 should be able to hit port 5000 on all of the app servers.

• Everyone else → blocked.

• And of course, it had to survive a reboot.

Here’s how it played out.

🔎 The Problem

Apache was running fine on all three app servers (stapp01, stapp02, stapp03) — but literally any host on the network could curl port 5000 and get in. No firewall, no restrictions. Just wide open.

That was the gap we had to close.

---

🛠 First Steps — Installing iptables

The Lab i was assigned with was using CentOS so i used these commands:

sudo yum install -y iptables iptables-services

---

⚡ Where Everything Became Chaos

This is where things got interesting. I added a rule to allow port 5000 from the load balancer… but it wasn’t working.

Turns out I had placed my ACCEPT rule below a general REJECT all. Rookie mistake. In iptables, rules are evaluated top to bottom. Once traffic hits a REJECT or DROP, that’s it — it never even sees the rules below.

Lesson learned.

Here’s what the final working rules looked like:

# Allow established/related connections
sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow ICMP (ping) from load balancer
sudo iptables -A INPUT -p icmp -s 172.16.238.14 -j ACCEPT

# Allow Apache port 5000 only from LBR (insert before REJECT)
sudo iptables -I INPUT 5 -p tcp --dport 5000 -s 172.16.238.14 -j ACCEPT

# Reject everything else
sudo iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

# Extra safeguard: Drop all other traffic to port 5000
sudo iptables -A INPUT -p tcp --dport 5000 -j DROP

---

💾 Making It Stick

Of course, rules in memory don’t survive reboots. So:

RHEL/CentOS:

sudo service iptables save
sudo systemctl enable iptables

---

✅ Testing the Fix

• From the load balancer (stlb01):

    curl -v http://172.16.238.10:5000
  

  I was able to get the html page.

• From another host(It was jumphost in mycase):

    curl -v http://172.16.238.10:5000
  

  ❌ Blocked with No route to host.

That’s exactly what we wanted.

---

🚦 Key Takeaways From This

1. Rule order matters — top to bottom evaluation will burn you if you’re not careful.

2. Confirm listening ports before you go rule-crazy:

    sudo ss -tlnp | grep 5000
   

3. Test both allowed and blocked sources — don’t assume.

4. Persistence is key — otherwise you’ll be redoing this on the next reboot.

5. Logging helps — adding a quick log rule before DROP/REJECT can save you a lot of guessing.

---

📚 What Helped Me Learn

• Red Hat iptables docs

• Ubuntu iptables how-to

• Googling and Little ChatGPT

---

At the end of it, I locked down Apache’s port 5000 so only the load balancer could talk to it, and in the process I walked away with a much better understanding of just how picky iptables can be about rule order.

Sometimes the fix isn’t about the tool itself, but about how you structure the rules.