nginx: [emerg] host not found in upstream

countfak

Hi,

I tried IndigoStack several times in the last months but I never managed to make it work.
I always get this error:

The process /Users/ROOTDIR/.indigo/stacks/system-stack/nginx_reverse_proxy/bin/nginx ran with PID 5613 and returned exit code 1 but one of the following was expected: "0". The stdErr contained: nginx: [emerg] host not found in upstream "welcome.test" in /Users/ROOTDIR/.indigo/stacks/system-stack/nginx_reverse_proxy/conf/sites-enabled/rp.welcome.test.conf:36

I'm on a MacBook Air M1 with macOS 13.3

Thanks

Indigo

Sorry to hear this! And thanks for trying it again 🙂

That error implies the System stack's nginx (which is there for reverse-proxy duties) cannot find the server its supposed to be proxying. In your case that's the Welcome stack's nginx.

Could you click on the Welcome stack, and right-click the nginx service. Choose "Show compiled in Finder...". You should see a logs directory there; can you tell me if any logs are in there, and if they contain anything pertinent?

Also, does the Welcome stack have a green light beside it indicating it has started up, or is it shut down by the time this error is shown?

countfak

Hi,

all logs inside the log folder are empty.

The welcome stack has a green light when the error is shown, then I hit retry and it becomes orange (if I click quit the app closes), then the light goes out after closing the dialog.

I'm attaching screenshots.

Indigo

Hmm, this is very strange and frustrating. It feels rather like https://forums.indigostack.app/d/67-reverse-proxy-not-working-with-mamp-stack/2 which at this point is still unresolved, primarily because I simply cannot replicate it, and because after extensive research neither @crystalenka nor I could ascertain anything unusual at all about her environment.

Your screenshots are most helpful, thanks! They indicate that the stack has indeed started up and thus should be hosting the welcome site on a randomised port. The reverse proxy is then failing to start due to not finding the welcome.test domain.

Are you able to launch Indigo without it attempting to auto-start "My First Stack"? If so, could you try creating a new stack with only nginx and one site? Give it hard-coded ports (eg 5080 and 5443 or whatever), and see if it starts up. If the stack starts but the RProxy still throws an error, leave the error dialog unanswered, and see if you can load your site in the browser by using its ports eg https://mysite.test:5443. If that doesn't work, try https://127.0.0.1:5443

One other thought: do you happen to have access to a second Mac, or Mac virtual machine (eg with UTM)? It would be very interesting if you could try it there also.

I would really love to get to the bottom of this issue. My theory at this point is that some Macs are not able to resolve indigo's domains (eg "mysite.test"). The only thoughts I can come up with regarding that are perhaps due to their DNS configuration is interfering (perhaps due to ISP or router settings?) or there's some left-over environmental thing in their Mac from something previously installed (eg /etc/hosts entries). However, in @crystalenka's case, none of that seems the case...

countfak

You gave me a great hint mentioning DNS configuration. I'm using NextDNS on my computer, and if I disable it, everything works just fine. So the issue is with custom DNS configuration.
I was thinking that probably Local by Flywheel, which I have installed, was the issue here, but it's not.
In case this is helpful: I'm using NextDNS with the profile configuration, not with their app.
I hope this is something that is fixable somehow, or maybe we can find a workaround that doesn't involve disabling NextDNS.
Thanks!

Indigo

Wow, this is great progress! Well done on finding the cause!

The next step is to work out why NextDNS causes issues with Indigo. Indigo uses Dnsmasq to provision domains as needed. There seems to be some discussion around the use of both NextDNS and Dnsmasq at the same time (eg here) so I will do some research down that path.

TBH I am beginning to wonder if I simply have the Nginx reverse proxy configuration a bit too specific or something. I will dig into that also.

Indigo

As a followup to the nginx reverse-proxy config idea: if you have the time to try it, I'd like to see if some changes to the rp config would provide a fix for the DNS issue.

Background info

The template for new rp entries is at ~/.indigo/stacks/system-stack/nginx_reverse_proxy/templates/reverseproxy.http.conf

Every time you start a stack this template is compiled into appropriate entries inside system-stack/nginx_reverse_proxy/conf/sites-enabled.

The idea to try

Could you try editing the template, as follows?

1) remove the line containing resolver 127.0.0.1;

...and/or...

2) replace the following line:

proxy_pass  {{backend_protocol}}://{{backend_domain}}:{{backend_port}};

with:

proxy_pass  {{backend_protocol}}://127.0.0.1:{{backend_port}};

In each case you will need to stop the stack and re-start it. You can confirm your changes have taken effect by inspecting the compiled files in sites-enabled at the path mentioned above.

countfak

Hi,

step number 2 seems to do the trick. if I replace that line, the stack starts even without disabling NextDNS.
step 1 doesn't seem to be useful.

anyway, the site is unreachable when I open it in the browser. I get the "this site can't be reached" error.

Indigo

Had my hopes up for a minute there! 🤣

So I think with step 2 in place, the Nginx RP is just avoiding the need for a DNS lookup, and thus we are bypassing the System Stack startup error. However it sounds like the underlying site is still not accessible even at a url such as https://127.0.0.1:50443. Could you test that for me? You'd need to get the port number from the Welcome stack's nginx site config. If necessary choose a port, so it's not on "auto":

If that works, then try accessing the site using its domain together with its port eg https://mysite.test:50443.

Finally, could you run scutil --dns on the command line and let me know its contents? Skim through it before posting it publicly here — if you'd rather, feel free to send it to me at hello@indigostack.app.

Indigo

Also, @crystalenka if you have time to try out the suggestions in my earlier comment re: editing the nginx reverse-proxy template, I would be very interested in the outcome.

countfak

Ok so,

accessing the site like this is working: http://127.0.0.1:50080
also https://127.0.0.1:50443 is working but it's triggering the browser security alert because of https

however this is not working: https://mysite.test:50443

I'll send you an email with the results of scutil --dns

Indigo

Thanks for trying that, and for emailing through the scutil output.

The scutil output looks perfect to me, although I had expected to see a NextDNS entry in there; in its absence I am not entirely sure how it intercepts your DNS entries.

Re: the SSL security alert you mentioned — this is not unexpected as the cert is registered for the domain name.

So overall, with the change from 2) above, essentially it looks like everything is working perfectly except for the domain name resolution. Which is kind of a big problem ha.

(I'm rolling the 2) change into the next Indigo release.)

You could most likely work around the resolution issue for now if you'd like to, by hand-editing your /etc/hosts adding this to the end:

127.0.0.1   mysite.test

I'm curious to know if that fixes it for you.

countfak

Yes, manually editing the hosts file fixes it.
It's not uncommon for local development softwares to automatically add entries to the hosts file whenever you configure a new site. So this could be a nice functionality to add. I don't know if it goes against how Indigo is intended to work.

Indigo

Prior to building Indigo I always hand-edited my /etc/hosts file and it worked fine. Then when considering my options for domain resolution in Indigo, Dnsmasq seemed like the right choice. However, for a bunch of reasons (not least yours above) I'm now wondering whether the simpler "hosts file entries" mechanism might be more pragmatic.

Sadly, we would lose Dnsmasq's ability to resolve wildcard domains, but we don't currently leverage that, and overall DNS has been a recurring issue during the beta.

Therefore I've spent the past few days working on an Indigo build which manages hosts file entries rather than running Dnsmasq. I finished an initial version of that just now so will begin testing (and using) this build and all being well will release it in a week or so.