Nginx upstream health checks and passive circuit breaker configuration

In the previous three chapters, we completed the architecture design, studied the principles, and developed the health check service. Now it is finally time to truly get Nginx running.

This chapter will guide you step by step through configuring Nginx's upstream health check and Passive circuit breaker A mechanism that enables Nginx to automatically identify and remove faulty nodes.

1. The role of Nginx in this architecture

Let's review the overall architecture:

Nginx assumes three key responsibilities at this layer:

Responsibilities	Description
Reverse proxy	Receive all requests and forward them to the backend Master cluster.
Health check	Periodically probe whether the Master is alive and automatically remove faulty nodes.
Load balancing	Distribute requests among multiple Master nodes (optional)

💡 In our architecture, Nginx and the backend Master are Same-machine deployment of (Nginx and Master run on the same physical machine). This means:

• What Nginx proxies is 127.0.0.1:8080, rather than the IPs of other machines

• The health check targets the local Master service

II. Environment Preparation: Installing Nginx

Before starting configuration, you need to, on three machines (192.168.0.201,192.168.0.202,192.168.0.203) respectively complete the installation of Nginx thereon.

2.1 Install Nginx

# Update software package index
sudo apt update

# Install Nginx
sudo apt install -y nginx

# Check the version to confirm the installation was successful.
nginx -v
# Output: nginx version: nginx/1.18.0 (Ubuntu)

2.2 Verify whether Nginx is running properly

# Check Nginx service status
sudo systemctl status nginx

# Test Nginx default page
curl http://localhost
# You should be able to see the Nginx welcome page.

2.3 Ensure Nginx starts automatically on boot

sudo systemctl enable nginx

2.4 Confirm the installation path and configuration file location

# View Nginx installation path
whereis nginx
# nginx: /usr/sbin/nginx /etc/nginx /usr/share/nginx

# View the main configuration file
ls -la /etc/nginx/
# Key focus: nginx.conf, sites-available/, sites-enabled/

💡Section objectives: Nginx installation has been completed on all three machines, preparing for the subsequent upstream configuration.

III. Two Modes of Nginx Health Checks

Nginx provides two health check mechanisms:

Mode	principle	Advantages	缺点
Passive inspection	Determine whether a node is healthy based on the request's response status (timeout, refusal, 5xx)	No additional configuration required; takes effect automatically.	A faulty node will be removed only after it has been requested at least once.
Proactive inspection	Nginx periodically proactively sends probe requests (such as /health)	Faults are discovered in advance and will not affect business requests.	Requires Nginx Plus or third-party modules

⚠️Important Notice: Nginx Open Source Edition Active health checks are not supported.(Actively send probe requests). Active health checks are a feature of the paid version of NGINX Plus.

But we can through Passive check + Keepalived health check script combination to achieve the same effect. This is also the choice in most production environments. We will cover Keepalived configuration in full in Part 5.

IV. Passive Health Check Configuration

Passive inspection relies on several core parameters:

Parameter	effect	Recommended value
max_fails	The maximum number of allowed failures; if exceeded, mark as down	2
fail_timeout	The time window for failure counting, which is also the marker down recovery attempt interval after	10s
proxy_connect_timeout	Backend connection timeout	3s
proxy_read_timeout	Read backend response timeout	30s

4.1 Basic upstream configuration

Rewrite the default nginx site configuration file on three machines:

sudo vim /etc/nginx/sites-available/default

However, specifically on each server,/etc/nginx/sites-available/default The configuration file will have slight differences, mainly in the upstream configuration!

nginx

# /etc/nginx/sites-available/default
upstream master_backend {
    # 同机部署，代理到本机的 8080 端口
    server 127.0.0.1:8080 max_fails=2 fail_timeout=10s;

    # Other Master nodes
    server 192.168.0.202:8080 max_fails=2 fail_timeout=10s;  
    server 192.168.0.203:8080 max_fails=2 fail_timeout=10s backup;  

    # Maintain persistent connections to improve performance.
    keepalive 32;
}

server {
    listen 80;
    server_name _;

    # Client request body size limit
    client_max_body_size 10M;

    location / {
        proxy_pass http://master_backend;

        # Request header passthrough
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout configuration (affects the evaluation of passive checks)
        proxy_connect_timeout 3s;
        proxy_read_timeout 30s;
        proxy_send_timeout 30s;

        # Long connection configuration
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Failure retry configuration
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 10s;
    }

    # Independent health check endpoint (optional, for external probing)
    location /nginx-health {
        access_log off;
        return 200 "nginx ok\n";
        add_header Content-Type text/plain;
    }
}

nginx

# /etc/nginx/sites-available/default
upstream master_backend {
    # 同机部署，代理到本机的 8080 端口
    server 127.0.0.1:8080 max_fails=2 fail_timeout=10s;

    # Other Master nodes
    server 192.168.0.201:8080 max_fails=2 fail_timeout=10s;  
    server 192.168.0.203:8080 max_fails=2 fail_timeout=10s backup;  

    # Maintain persistent connections to improve performance.
    keepalive 32;
}

server {
    listen 80;
    server_name _;

    # Client request body size limit
    client_max_body_size 10M;

    location / {
        proxy_pass http://master_backend;

        # Request header passthrough
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout configuration (affects the evaluation of passive checks)
        proxy_connect_timeout 3s;
        proxy_read_timeout 30s;
        proxy_send_timeout 30s;

        # Long connection configuration
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Failure retry configuration
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 10s;
    }

    # Independent health check endpoint (optional, for external probing)
    location /nginx-health {
        access_log off;
        return 200 "nginx ok\n";
        add_header Content-Type text/plain;
    }
}

nginx

# /etc/nginx/sites-available/default
upstream master_backend {
    # 同机部署，代理到本机的 8080 端口
    server 127.0.0.1:8080 max_fails=2 fail_timeout=10s;

    # Other Master nodes
    server 192.168.0.201:8080 max_fails=2 fail_timeout=10s;  
    server 192.168.0.202:8080 max_fails=2 fail_timeout=10s backup;  

    # Maintain persistent connections to improve performance.
    keepalive 32;
}

server {
    listen 80;
    server_name _;

    # Client request body size limit
    client_max_body_size 10M;

    location / {
        proxy_pass http://master_backend;

        # Request header passthrough
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout configuration (affects the evaluation of passive checks)
        proxy_connect_timeout 3s;
        proxy_read_timeout 30s;
        proxy_send_timeout 30s;

        # Long connection configuration
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Failure retry configuration
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;
        proxy_next_upstream_tries 2;
        proxy_next_upstream_timeout 10s;
    }

    # Independent health check endpoint (optional, for external probing)
    location /nginx-health {
        access_log off;
        return 200 "nginx ok\n";
        add_header Content-Type text/plain;
    }
}

4.2 Application Configuration

Run the following command to test whether the configuration is correct:

nginx -t
# Output:
# nginx: configuration file /etc/nginx/nginx.conf test is successful

Execute the following command,Smooth hot loading(Zero-downtime update configuration):

nginx -s reload

# or use systemctl
sudo systemctl reload nginx

4.3 Parameter Details

Working principle of passive inspection:

proxy_next_upstream Configuration interpretation:

proxy_next_upstream error timeout invalid_header http_500 http_502 http_503;

Condition	Explanation	Whether to trigger retry
error	An error occurred while connecting to the backend.	✅
timeout	Connection, read, or send timeout	✅
invalid_header	The backend returned an invalid response header.	✅
http_500	Backend returns 500 Internal Server Error	✅
http_502	The backend returns 502 Bad Gateway	✅
http_503	Backend returns 503 Service Unavailable	✅
http_404	Backend returns 404 (not triggered by default)	❌

V. Verify Nginx passive checks

Before deploying Keepalived, let's first independently verify whether Nginx's passive health check mechanism works properly.

5.1 Ensure the Master service is running properly

# Confirm on each of the three machines that the gateway service is running.
sudo systemctl status gateway

# If it is not running, start it.
sudo systemctl start gateway

5.2 Simulating a Master Service Failure

at node1（192.168.0.201） Stop the Master service on:

sudo systemctl stop gateway

5.3 Sending a Request Triggers Passive Inspection

# Test locally on node1 (192.168.0.201)
# Because Nginx proxies to 127.0.0.1:8080, you can just access the local Nginx directly.

for i in {1..5}; do
    curl -X POST http://127.0.0.1/api/v1/task \
         -H "Content-Type: application/json" \
         -d '{"test": "failover"}' \
         -w "\nHTTP Status: %{http_code}\n"
    sleep 1
done

5.4 Observe Nginx Error Logs

# View the Nginx error log on node1
sudo tail -f /var/log/nginx/error.log

# Expected to see similar logs:
# [error] upstream timed out (110: Connection timed out) while connecting to upstream
# [error] no live upstreams while connecting to upstream

5.5 Restore the Master service and verify

# Restore Master service
sudo systemctl start gateway

# Send the request again and observe whether the node becomes available again.
curl -X POST http://127.0.0.1/api/v1/task \
     -H "Content-Type: application/json" \
     -d '{"test": "recovery"}'

# Expected: request successful, returns 200

5.6 Verification Conclusion

Test steps	Expected results	Whether passed
Stop Master service	Service unavailable	Through
Send request (1st–2nd time)	Nginx records timeouts/errors	Through
Send request (from the 3rd time onward)	Nginx returns an error directly and no longer attempts to forward	Through
Restore the Master service	Wait fail_timeout available again after	Through

💡Description: The verification in this chapter uses 127.0.0.1(local Nginx), without relying on Keepalived. VIP verification will be performed after Keepalived is deployed in Part 5.

VI. Common Problems and Solutions

Q1: Does the Nginx open-source version really not have active health checks?

Indeed, there isn't. The Nginx open source version only supports passive health checks. If active checks are needed, consider:

• Buy Nginx Plus

• Use OpenResty (an enhanced version based on Nginx)

• Use third-party modules nginx-upstream-check-module(Need to recompile Nginx)

• Use HAProxy instead (native support for active checks)

Q2：max_fails and fail_timeout How should it be set?

It depends on your business tolerance:

Scene	max_fails	fail_timeout	Description
High availability requirements are strict.	1	5s	Quick removal, but possible misjudgment.
General production environment	2	10s	Recommended configuration, balancing speed and stability
The network is unstable.	3	30s	Avoid frequent switching

Q3: How can Nginx avoid faulty nodes when retrying?

proxy_next_upstream Cooperate max_fails will be automatically implemented. When the failure count of a node reaches max_fails, Nginx will fail_timeout Within, requests will no longer be forwarded to it.

Q4: What is the difference between health checks in Nginx and Keepalived?

component	Inspection object	Actions after discovering a problem
Nginx	Backend Master Service	Temporarily remove the node, but the VIP is still on the local machine.
Keepalived	Nginx + Master service	VIP drifts to another node

The two work together to form a complete fault protection chain. Keepalived configuration will be fully covered in Part 5.

Q5: Do all three machines need to have Nginx configured?

Yes. Because the VIP may drift to any machine, all three machines need to have Nginx installed and maintain consistent configurations. This way, no matter which machine the VIP drifts to, Nginx can work properly.

Q6: How can we ensure that the Nginx configurations on the three machines are consistent?

It is recommended to use a configuration management tool (such as Ansible) to synchronize the configuration, or manually perform the same configuration steps on the three machines. We will provide a one-click deployment script in later chapters of this series.

VII. Summary

This chapter completes the Nginx upstream health check and circuit breaker configuration. Key points:

• Environment Setup: Install Nginx on three machines and confirm that the service is running normally.

• Passive inspection: Through max_fails+fail_timeout Implement automatic removal of faulty nodes

• Timeout configuration:proxy_connect_timeout and proxy_read_timeout Affects fault determination

• Retry strategy:proxy_next_upstream Define which errors require retrying

• Independent verification: No need for Keepalived; verify Nginx passive health check mechanism independently

In the next chapter, we will formally deploy Keepalived, configure VIP failover, and make the high-availability entry point truly operational.

💡 This article is the 4th installment of the "Distributed High-Availability Entry Architecture in Practice Series"

• Click to view all articles

• Previous:Implement a Web service with health checks using Golang + Gin

• Next article:Keepalived Deployment and VIP Failover Configuration Guide