Day 84 of DevOps, Day 34 of Azure. Both tasks today turned on a path that is not where it looks like it is.

An Ansible task with two paths in it, where one is on the machine running the playbook and the other is on the machine being configured, and nothing in the task says so. And an Azure VM that resolved DNS perfectly while being unable to open a single outbound connection, because that lookup travels down a channel the rule I was hunting for cannot touch.

One Ansible task, one Azure task. Copy a file from the jumphost to three servers, then fix a VM that cannot install packages. The tasks come from the KodeKloud Engineer platform.

Task 1: One file, three servers

The task: copy /usr/src/data/index.html from the jumphost to /opt/data on all three app servers.

Step 1: Check empty first

[tony@stapp01 ~]$ ls -la /opt/data total 12 drwxr-xr-x 2 root root 4096 May 23 12:11 . 

On each of the three. Same discipline as yesterday, and it is what makes changed: [stapp01] mean "the file arrived" rather than "something about the file is different now".

Step 2: The playbook, and the direction hiding in it

--- - hosts: app_servers become: yes tasks: - name: Copy file from the jumphost server to the remote app servers. copy: src: /usr/src/data/index.html dest: /opt/data 

Two paths, two machines, and the parameter names do not tell you which is which.

src is documented as "Local path to a file to copy to the remote server", meaning the control node. dest is "Remote absolute path where the file should be copied to", meaning the managed host. So the task straddles two filesystems and the YAML reads as though it is operating on one.

The switch that reverses it is remote_src. At its default of false the module "searches for src on the controller node"; set to true it "searches for src on the managed (remote) node", which turns the same four lines into a copy within the target rather than a transfer to it.

One boolean, two entirely different jobs, and no visible difference in the rest of the task.

One thing the documentation does not cover: dest: /opt/data is an existing directory, and the file landed inside it as /opt/data/index.html. The page covers src being a directory, and dest being a non-existent path, and it warns that for a file-to-file copy "the parent directory of dest is not created and the task fails if it does not already exist". The directory-as-destination case is not described, so I am recording it as observed rather than quoted.

Step 3: The output arrives out of order

TASK [Gathering Facts] *************** ok: [stapp01] ok: [stapp03] ok: [stapp02] TASK [Copy file...] *************** changed: [stapp02] changed: [stapp03] changed: [stapp01] 

stapp01, stapp03, stapp02, then stapp02, stapp03, stapp01. Neither is inventory order, and the tool is working correctly.

Ansible runs each task against several hosts in parallel, so the order they report in is the order they finished. Which means the task output is not a list you can read down to check every host did the thing. The recap is:

stapp01 : ok=2 changed=1 stapp02 : ok=2 changed=1 stapp03 : ok=2 changed=1 

What the docs recommend

This playbook is idempotent and the previous one was not, and the difference is the module rather than anything I did.

Run this again and it reports changed=0, because copy compares source and destination content and does nothing when they match. Day 83's package and service tasks are the same, because they describe a desired state. Day 85's file with state: touch is not, because it defaults both timestamps to now.

So "is Ansible idempotent" turns out to have no single answer. Each module decides, and the only reliable way to know is to run a playbook twice and read the number.

Two other things worth knowing before this pattern grows.

The scaling note in the documentation: "The ansible.builtin.copy module recursively copy facility does not scale to lots (>hundreds) of files." The alternative it points at is ansible.posix.synchronize, "A wrapper around rsync to make common tasks in your playbooks quick and easy."

Which is the same shape as the Azure half of Day 82, where az storage blob copy start does one blob and azcopy sync is the tool for an actual sync. One file is a different problem from many files, and in both ecosystems the answer is a purpose-built transfer tool rather than the convenient one.

And the file came out root-owned, because become: yes means root writes it. Fine here, since mode 644 means a web server can read it. Not fine if anything is supposed to write it. Tomorrow is the day that sets ownership on purpose.

Task 2: A VM that cannot install packages

The task: a VM where apt-get does not work. Find out why.

Step 1: The symptom made no sense

TCP to 1.1.1.1:443 -> TCP_FAIL DNS lookup -> resolved fine SSH into the VM -> worked throughout 

An outbound connection to a literal IP fails, DNS resolves, and SSH is unaffected. Three facts that look mutually exclusive.

The fault was one rule:

Name Dir Access Pri Port Proto Block-All-Outbound Outbound Deny 200 * * 

Every NSG carries three default outbound rules at fixed priorities: AllowVnetOutBound at 65000, AllowInternetOutBound at 65001, and DenyAllOutBound at 65500.

A custom Deny at 200 is evaluated long before the Allow at 65001, because "Rules are processed in priority order, with lower numbers processed before higher numbers", and "Once traffic matches a rule, processing stops."

The bit worth internalising is why this is so easy to do. The defaults cannot be removed, only outranked: "You can't remove the default rules, but you can override them by creating rules with higher priorities." Azure has deliberately put its permissive defaults where any custom rule beats them. So one rule at a low number silently replaces the platform's behaviour, and nothing warns you.

This is the ordering hazard AWS security groups cannot produce. AWS rules are allow-only and unordered, so no rule can contradict another. An Azure NSG can.

Step 2: Why SSH still worked

NSG rules are stateful, and the documentation states it in the mirror direction: "A flow record is created for existing connections... If you specify an outbound security rule to any address over port 80, for example, it's not necessary to specify an inbound security rule for the response to the outbound traffic. You only need to specify an inbound security rule if communication is initiated externally."

So the inbound SSH rule kept SSH entirely working. The connection is inbound, and its replies belong to an established flow rather than being new outbound traffic. The Deny only touched connections the VM started, which is precisely the reported symptom: you can log in, and the machine cannot fetch anything.

Worth setting beside Day 28, where the fault was a route rather than a rule:

Day 28, route to None Day 34, outbound Deny
Mechanism Routing, not stateful NSG, stateful
Breaks VM-initiated traffic Yes Yes
Breaks replies to inbound traffic Yes No
Symptom Nothing works at all Inbound fine, outbound dead

A broken egress route takes the replies with it, because replies are egress too. A directional NSG Deny does not. Same apparent problem, opposite diagnostic signature.

Step 3: Why DNS still worked, and a correction

My notes said traffic to Azure's platform address 168.63.129.16 "is not subject to NSG rules", full stop. That is too broad, and the real picture is more useful because the exemptions differ by function:

Function NSG treatment
Agent ports 80 and 32526 "The communication on these ports with 168.63.129.16 isn't subject to the configured network security groups."
DNS, port 53 "By default DNS communication isn't subject to the configured network security groups unless targeted using the AzurePlatformDNS service tag."
Load balancer health probes Not exempt. They work because "The default network security group configuration has a rule that allows this communication."

So DNS survived not because the address is untouchable, but because a generic Deny does not reach it. A rule that named the AzurePlatformDNS service tag as its destination would have killed DNS too, and the documentation spells out how.

Which sharpens the diagnostic rather than weakening it. On Azure, "DNS works" tells you the platform channel is intact and nothing whatsoever about internet connectivity.

One more fact from the same page, which explains something about Day 28 I had not connected: "168.63.129.16 is a virtual IP of the host node and as such it isn't subject to user-defined routes." So Day 28's route blackholing everything never cut the platform's own path to that VM, which is why it stayed manageable while being unable to reach anything at all.

What the docs recommend

The test that found this in one command:

timeout 6 bash -c "</dev/tcp/1.1.1.1/443" && echo TCP_OK || echo TCP_FAIL 

A literal IP, no name resolution. That separates routing and filtering from DNS, and it returned TCP_FAIL while DNS succeeded, which located the fault immediately.

Testing with apt-get update or a curl to a hostname would have failed too and told me nothing, because both need DNS and egress, so a failure implicates either. When two subsystems are both required, test them separately before testing them together. And /dev/tcp is a bash built-in, so it works with no nc or telnet installed.

Then on the fix: delete the rule rather than flipping it to Allow.

Day 28 taught the opposite, that repointing a broken route beats deleting it, so the difference matters. There, deleting left an empty route table still attached to a subnet: configuration that functions and makes no sense. Here, an NSG with no custom outbound rules is the normal state, because Azure's defaults already allow internet outbound and that is what a fresh NSG looks like.

So the rule from Day 28 holds and resolves differently. Minimise what you change, then check the end state is a configuration someone would deliberately choose. An empty route table fails that test; an NSG with only inbound rules passes it. Flipping it to Allow would also have left a rule named Block-All-Outbound that permits everything, which is Day 28's other mistake.

And verify the sentence in the requirement. "Restore package installation capabilities" is a claim about apt-get install, so the closing check installs a package:

TCP_OK APT_UPDATE_OK INSTALL_OK tree v2.0.2 

A TCP connection proves egress. apt-get update proves the repositories resolve and respond. Only the install proves the thing asked for, because it also exercises package resolution, download and dpkg.

What I learned

Both halves were about a path that is not where it looks like it is.

The Ansible task has two paths in it, and the YAML gives no hint that they live on different machines. One boolean reverses which machine src means, and the task text is otherwise identical. If you read that playbook cold, there is nothing to tell you the file is crossing a network.

"DNS resolves" looks like "the network is fine". On Azure it means the platform's own channel is intact, which it almost always is, because that traffic is exempt from the rule you are hunting for. The only test that said anything useful was a TCP connection to a bare IP address.

And the thing that caught me out in my own notes was the blanket claim. I had written that the platform address bypasses NSGs. It bypasses them on two ports, bypasses them for DNS unless you aim at it specifically, and does not bypass them for health probes at all. Three answers where I had recorded one.

So here is the Day 84 question. Which of your working signals travels on a path that the thing you are testing cannot affect?

Day 84 down. Sixteen to go.