A few thoughts

za0

Indigo
Sure thing, thanks Jeff - and thanks for the thoughtful replies on all points!

Re adding sites, certainly wouldn't say poor - very much user error! But yes, a big "add" button on the stack overview page would make a lot of sense for dolts like me, perhaps with a warning if no webserver rack has been added to a stack that it won't actually work until one is?

Re dnsmasq, I can see that using the hosts file certainly would keep things simpler in some ways. One caveat is that I'm already using a menubar app to toggle different sets of hosts files so would need to be able to turn off Indigostack's automatic additions if that is the way you end up going, as I'd revert to handling it all manually (which is not a problem at all, and is what I used to do with Docker.)

Re dock vs menubar, completely understand! Somehow conceptually I prefer ongoing/background things to be in the menubar as they feel more integrated into the system that way.

Cheers!

Indigo

Thanks for the followup. I'd never heard of such a hosts file management app, so thanks for the heads-up. I see now there are several — may I ask which one you use? Could you give me a brief rundown on how they work, or are they all different?

Would it work for Indigo to always write to /etc/hosts* but also to write only its entries to another file, say ~/.indigo/hosts? Or would I need to add a preference setting to turn off modifications to /etc/hosts? I'd rather have it transparent if possible (ie no UI), as I think the scenario is a bit specialised.

[*] my hosts file implementation cleans up after itself when stopping stacks & quitting Indigo, unlike the current dnsmasq implementation

za0

Indigo
No problem! Yes I am trialling a mini-app within Raycast called iHosts to manage my hosts files at the moment but will shortly revert to a simple app called "Gas Mask" which I used previously. As far as I know they all work pretty much the same in that you give them a bunch of hosts entries (or they retrieve them from a remote location, eg a list of adblocking entries) and depending on which you activate at any one time they combine entries into one file which they write to /etc/hosts

This would mean that yes you would unfortunately need a toggle switch as to whether Indigostack would also make modifications to it, because otherwise it would potentially conflict.

It would definitely be good to have Indigostack always write the active sites' entries (both IPv4 and IPv6!) to eg ~/.indigo/hosts as that could then be added as a source to the hosts management app.

Indigo

Thanks, this is extremely timely! I'll be working on this tomorrow.

My plan is to add two things to the build I'm working on:

add a checkbox in the advanced prefs titled "Self-manage domain resolution" with subtext "When enabled, Indigo will not update your /etc/hosts file. ~/.indigo/hosts will always be updated."
have Indigo always also write its host file entries to ~/.indigo/hosts in addition to /etc/hosts (depending on the above setting).

Please let me know if you have any thoughts.

za0

Indigo Sounds great to me! Thanks for being so responsive on this.

za0

One more thing which just struck me - even if you don't go down the route of having a menu bar option, is it possible/desirable that closing the window doesn't close the app? Would let things tick away in the dock. Obviously can already achieve this by choosing to 'Hide' the window, but too often I find myself automatically clicking the red close button then kicking myself as my stack stops and the app fully quits.

Indigo

It's important to me that Indigo conceptually "is the thing" that makes stuff run, not "the thing that controls the other things" that make stuff run. So there needs to be a strong connection between it visibly running and your sites working. I would certainly consider an "Are you sure?" dialog prior to closing the window though.

The hosts file functionality as above is now written. I'll test it further then hopefully release it in RC.3 very shortly.

I do have a followup question for you, regarding your earlier comment about pointing IPv6 to the Indigo domains (ie adding ::1 domain.test entries into /etc/hosts).

I'm unsure under which circumstances the lack of IPv6 is a problem?
is it even possible to tell Safari which to use?

I assume the Nginx reverse proxy does not currently respond to IPv6 requests, since we only have entries such as listen 127.0.0.1:8443 in the reverse proxy site configurations. That certainly seems typical of Indigo's peers also. If you can shed more light on this it would be great!

za0

Indigo
Hi Jeff,

Re the window visibility, yes I understand your thinking on this. For me, it's enough that the dock indicator is lit (and/or in future perhaps the menu icon is there?) to show that Indigo is powering things; it's running basically 24/7 as I work to build sites, but seeing the whole stack is something I need only when checking or changing something. It's the same for my email client or Spotify etc - still listening to music but don't need the full window visible until I want to change something.

On the IPv6 front I'm not sure I can shed much light! I presume it's because my router supports IPv6 (can see this from the mac's wifi settings for the current network.) All I know is that if I only allocate IPv4 localhost entries in my /etc/hosts file then sometimes (not always, annoyingly!) Safari resolves a domain to its IPv6 address which won't be localhost. I don't choose this, and don't think I can choose for it not to happen - it's just what happens.

Re the reverse proxy, is there a reason to include the IPv4 localhost address as well as the port, rather than just the port (eg "listen 8443")? I'm not familiar enough with networking stuff to know the details I'm afraid..!

Indigo

We've just released RC3, which includes the hosts file management. Give it a try! 🙂

Just heads-up: in RC3, we're only resolving IPv4 to 127.0.0.1; I've not seen any example elsewhere of a local development nginx config responding on IPv6 so I'm unsure that's the right direction at this point.

za0 [Sometimes] Safari resolves a domain to its IPv6 address which won't be localhost

Could you tell me where it did resolve to?

Are you using domain TLDs during development that might conflict with actual domains registered in the global DNS system (ie not those specified in rfc 2606)?

If no to that, I'm unsure why your router's DNS server would respond with a result (regardless of IPv4 or 6) for your development domains.

za0 Re the reverse proxy, is there a reason to include the IPv4 localhost address as well as the port, rather than just the port (eg "listen 8443")?

My understanding is that when not specifying the IP, nginx would respond on external requests also, which is a security issue for a local development environment. 127.0.0.1 locks it down.

za0

Indigo

Hi Jeff, I've upgraded to RC3 - afraid it's not been smooth sailing! It seems that even with the 'Automatically manage hosts' option unticked, Indigo still 'touches'/resaves the hosts file (and maybe changes its ownership/privileges?) when it's started up and when it's closed - this triggers a change notification from my hosts manager each time.

Then separately, it seems that the Nginx reverse proxy only works if the relevant hosts entry exists when it's brought up - which means I can't dynamically switch hosts entries without stopping the stack first, then starting it after the relevant hosts entries are in place - in a way this defeats the purpose of using a hosts manager.

(Unfortunately the plan of using ~/.indigo/hosts to help manage also doesn't work, but that's on the host manager side - it doesn't observe for changes on source files, so it doesn't see changes that Indigo makes when a stack is activated/deactivated.)

In terms of domains in use during development, yes some of these are localhost overrides to real domain names - we need to do this instead of having separate (eg .test) domains for local development due to dependency on external auth which whitelists only one possible callback.

Sorry it's not better news but I hope this is helpful - I appreciate your continuing work to improve all of this!

Indigo

za0 Indigo still 'touches'/resaves the hosts file

Wow, it should definitely not be doing that. Thanks for noticing it.

za0 it doesn't observe for changes on source files, so it doesn't see changes that Indigo makes when a stack is activated/deactivated

I'm a bit unsure how you would typically use Gas Mask during development (eg previously with Docker). Would you not have a hand-built hosts file somewhere containing your development domains, and then tell Gas Mask to include that file in your /etc/hosts, perhaps with its "combined files" feature? I guess that's a similar process to how you could do it now, either with Indigo's generated ~/.indigo/hosts file or with one you hand-build, containing your requisite IPv6 entries?

za0 using ~/.indigo/hosts to help manage also doesn't work

Let me know if there's definitely no possible way for it to help you. I lean towards removing that functionality if it's useless.

za0 yes some of these are localhost overrides to real domain names

I won't pretend to understand the complexities you're dealing with here (external auth stuff) 😮 However it sounds like a very specific use-case, so not something Indigo should be doing directly, but definitely something it would ideally enable (my own usage of Indigo is full of similar oddities.) It sounds like you need two things:

you need IPv6 hosts file entries for each domain
nginx reverse proxy site definitions should also listen on IPv6 (or just on the port as you said earlier).

The latter should be possible with overrides.

For the IPv6 hosts entries, my hope is you can hard-code a hosts file somewhere and turn it on or off (ie include it or not) using Gas Mask? Another option might be to add a stack which simply serves your custom hosts file over http, for Gas Mask to access.

Indigo

za0 Indigo still 'touches'/resaves the hosts file (and maybe changes its ownership/privileges?)

I've fixed this bug and will release RC3b (goodness me!) shortly.

Incidentally, I'm writing changes atomically, which means the file is first written to a temporary location then the existing file is replaced, rather than writing to it directly. This is intended to provide a level of safety by ensuring that the original hosts file is not lost in the event of an interruption or failure during the write operation. It does however mean that the file is recreated each time Indigo writes to it. I'm unsure of the impact that might have on permissions; I know it does change the creation date each time.

On my Mac the permissions, even after Indigo writes to it, are 644 and owned by root. Anything else would be a security issue, I think.

Indigo

Thanks for your helpful feedback. I should have mentioned I had to re-release RC3 as the first attempt didn't correctly run the version upgrade process. The outcome of that being that the reverse proxy still required domains to resolve before it would start. RC3a fixes that, so please make sure you're on RC3a.

I'll respond more fully tomorrow.

za0

Indigo Thanks, yes I've updated to RC3a and that fixes the second point!

za0

Indigo Thanks for all this Jeff!

Indigo Wow, it should definitely not be doing that. Thanks for noticing it.
Indigo I've fixed this bug and will release RC3b (goodness me!) shortly.

Great, all works as expected now!

Indigo Would you not have a hand-built hosts file somewhere containing your development domains, and then tell Gas Mask to include that file in your /etc/hosts, perhaps with its "combined files" feature?

Yes that's exactly it - and all works fine this way now: I have a 'local' and a 'global' set (amongst others) which I manually toggle. I was thinking that it would be nice to have an 'auto' set which would add the Indigo file from ~/.indigo/hosts but as said the hosts manager doesn't monitor source files for changes so it doesn't re-aggregate all the rules when Indigo is activated and its hosts file changes, meaning 'auto' is off the table for now.

Indigo The latter should be possible with overrides.

Yes I've been experimenting with this and got it working fine - in the site-specific .conf file within nginx_reverse_proxy/conf/sites-enabled there just needs to be two new 'listen' lines for IPv6 (listen [::]:80; and listen [::]:443 ssl;) - I tried using only the port but for some reason then Indigo served content from one of my other configured sites, not quite sure why. I understand you don't want to add IPv6 by default but as far as I can tell this is all working fine, even when Safari resolves and requests over IPv6.

Indigo For the IPv6 hosts entries, my hope is you can hard-code a hosts file somewhere

Yes these are already in my 'local' hosts file and work well.

Thanks again for your responsiveness on all this, and I'm looking forward to seeing where you take IndigoStack!