amazon web services - NGINX inside ECS suddenly pointing to other hostname - Stack Overflow

i have deployment using container at ECS that contain static web and nginx for reverse proxy. I'm using nginx:alpine as base image

Use Nginx to serve a static website and as a reverse proxy for another service inside ECS, which is behind a VPC.

I have an AWS load balancer in front of ECS.

/etc/nginx/nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    resolver 8.8.8.8 valid=60s;
    resolver_timeout 5s;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    upstream backend {
        server api:443;
        keepalive 32;
    }
#    proxy_read_timeout 10s;
#    proxy_connect_timeout 10s;
#    proxy_send_timeout 10s;
#    send_timeout 10s;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/default.conf

server {
    listen 3000;
    server_name web;

    location / {
        root /usr/share/nginx/html;
        index index.html;
        try_files $uri /index.html;
    }

    location /intools/ {
        client_max_body_size 50M;
        proxy_pass ;
        proxy_set_header Host api;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization $http_authorization;

        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_connect_timeout 10s;
        proxy_send_timeout 10s;
        proxy_read_timeout 10s;
        send_timeout 10s;

        access_log /dev/stdout;
        error_log /dev/stderr debug;
    }

    location /api/ {
        proxy_pass ;
        proxy_set_header Host api;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization $http_authorization;

        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_connect_timeout 10s;
        proxy_send_timeout 10s;
        proxy_read_timeout 10s;
        send_timeout 10s;

        access_log /dev/stdout;
        error_log /dev/stderr debug;
    }
}

My problem is that the first deployment works fine, but after a few days, Nginx starts behaving strangely. The routing, which is supposed to point to api, unexpectedly redirects to random unknown hostnames. This issue happens frequently. Every time I restart the ECS container, it works fine for a while, but then it starts pointing to another random hostname again.

Has anyone ever experience this?

multiple response that i got first

{
  "Message": "User: anonymous is not authorized to perform: es:ESHttpGet because no resource-based policy allows the es:ESHttpGet action"
}

second

404 Not Found
nginx

third

i have deployment using container at ECS that contain static web and nginx for reverse proxy. I'm using nginx:alpine as base image

Use Nginx to serve a static website and as a reverse proxy for another service inside ECS, which is behind a VPC.

I have an AWS load balancer in front of ECS.

/etc/nginx/nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    resolver 8.8.8.8 valid=60s;
    resolver_timeout 5s;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    upstream backend {
        server api.com:443;
        keepalive 32;
    }
#    proxy_read_timeout 10s;
#    proxy_connect_timeout 10s;
#    proxy_send_timeout 10s;
#    send_timeout 10s;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

/etc/nginx/conf.d/default.conf

server {
    listen 3000;
    server_name web.com;

    location / {
        root /usr/share/nginx/html;
        index index.html;
        try_files $uri /index.html;
    }

    location /intools/ {
        client_max_body_size 50M;
        proxy_pass https://api.com;
        proxy_set_header Host api.com;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization $http_authorization;

        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_connect_timeout 10s;
        proxy_send_timeout 10s;
        proxy_read_timeout 10s;
        send_timeout 10s;

        access_log /dev/stdout;
        error_log /dev/stderr debug;
    }

    location /api/ {
        proxy_pass https://api.com;
        proxy_set_header Host api.com;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Authorization $http_authorization;

        proxy_http_version 1.1;
        proxy_set_header Connection "";

        proxy_connect_timeout 10s;
        proxy_send_timeout 10s;
        proxy_read_timeout 10s;
        send_timeout 10s;

        access_log /dev/stdout;
        error_log /dev/stderr debug;
    }
}

My problem is that the first deployment works fine, but after a few days, Nginx starts behaving strangely. The routing, which is supposed to point to api.com, unexpectedly redirects to random unknown hostnames. This issue happens frequently. Every time I restart the ECS container, it works fine for a while, but then it starts pointing to another random hostname again.

Has anyone ever experience this?

multiple response that i got first

{
  "Message": "User: anonymous is not authorized to perform: es:ESHttpGet because no resource-based policy allows the es:ESHttpGet action"
}

second

404 Not Found
nginx

third

• amazon-web-services
• nginx
• amazon-ecs

Share Improve this question Follow edited Jan 30 at 3:55 BIlly asked Jan 30 at 3:02 BIllyBIlly 43711 gold badge66 silver badges2121 bronze badges 2

• That is certainly not normal behavior. If nginx is suddenly redirecting to some domain name you have never heard of, it sounds like your nginx server may possibly have been hacked in some way. Have you tried connecting to the ECS instance using ECS Exec when this is happening, to check if the config files on the instance have been changed? docs.aws.amazon.com/AmazonECS/latest/developerguide/… – Mark B Commented Jan 30 at 12:53
• @MarkB I have already verified the config file multiple times before posting here, so it's certainly not a case of being hacked or having configuration file changes. Another piece of information: when I reload the Nginx config (using nginx -s reload inside the container), it returns to normal. – BIlly Commented Jan 31 at 5:34

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Turns out I needed to add a DNS resolver to AWS DNS under the server block:

server {
    resolver 169.254.169.253 valid=10s;
}

After that, the issue never happened again.