Bind #Docker container ports to virtual interfaces to simulate cluster nodes

My setup implies that I have 3 almost identical application nodes (running as containers) that are part of a cluster and they need to be mapped to different external IPs (not part of the docker ingress network) because they are added in a cluster configuration external to docker.

The default behavior of port binding is to bind a port to the localhost more specific to 0.0.0.0
This can be done very easy using docker-compose:

...
       ports:
         - "9101:9101"
         - "8080:8080"
         - "8443:8443"
...

This is not OK for my setup because I need to map the same exact ports for all the nodes.

For this I have to simulate somehow 3 actual physical machines on my test environment.

Virtual interfaces to the rescue:

The first step was to define 3 virtual network interfaces defined over my real lan interface enp2s0:

# ip link add virtual0 link enp2s0 type macvlan mode bridge
# ip link add virtual1 link enp2s0 type macvlan mode bridge
# ip link add virtual2 link enp2s0 type macvlan mode bridge
# ip address add 40.0.0.101/24 broadcast 40.0.0.255 dev virtual0
# ip address add 40.0.0.102/24 broadcast 40.0.0.255 dev virtual1
# ip address add 40.0.0.103/24 broadcast 40.0.0.255 dev virtual2
# ip link set virtual0 up
# ip link set virtual1 up
# ip link set virtual2 up

The result of the above can be seen with ifconfig as:

...
virtual0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 40.0.0.101  netmask 255.255.255.0  broadcast 40.0.0.255
        inet6 fe80::f820:f6ff:fee8:b5ad  prefixlen 64  scopeid 0x20<link>
        ether fa:20:f6:e8:b5:ad  txqueuelen 1000  (Ethernet)
        RX packets 364126  bytes 32951872 (31.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 121  bytes 17951 (17.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virtual1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 40.0.0.102  netmask 255.255.255.0  broadcast 40.0.0.255
        inet6 fe80::88ed:dcff:fe8a:c2f0  prefixlen 64  scopeid 0x20<link>
        ether 8a:ed:dc:8a:c2:f0  txqueuelen 1000  (Ethernet)
        RX packets 364566  bytes 32993227 (31.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 126  bytes 18422 (17.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virtual2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 40.0.0.103  netmask 255.255.255.0  broadcast 40.0.0.255
        inet6 fe80::e07d:a1ff:fe70:b4c7  prefixlen 64  scopeid 0x20<link>
        ether e2:7d:a1:70:b4:c7  txqueuelen 1000  (Ethernet)
        RX packets 364798  bytes 33012693 (31.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 125  bytes 18338 (17.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
...

Now we can bind each specific application node to a virtual interface. This can be done by changing the ports declaration in docker-compose to:

       ports:
         - "40.0.0.101:9101:9101"
         - "40.0.0.101:8080:8080"
         - "40.0.0.101:8443:8443"

In the end the whole docker-compose looks like:

 
version: '3.1'

services:

    node1:
       image: jetty
       volumes:
         - /home/docker-volumes/node1:/opt/jetty.base/node:Z
       env_file:
         - ./node.env
       ports:
         - "40.0.0.101:9101:9101"
         - "40.0.0.101:8080:8080"
         - "40.0.0.101:8443:8443"
       working_dir: /opt/jetty.base/node/
       command: ["./start.sh"]

    node2:
       image: jetty
       volumes:
         - /home/docker-volumes/node2:/opt/jetty.base/node:Z
       env_file:
         - ./node.env
       ports:
         - "40.0.0.102:9101:9101"
         - "40.0.0.102:8080:8080"
         - "40.0.0.102:8443:8443"
       working_dir: /opt/jetty.base/node/
       command: ["./start.sh"]

    node3:
       image: jetty
       volumes:
         - /home/docker-volumes/node3:/opt/jetty.base/node:Z
       env_file:
         - ./node.env
       ports:
         - "40.0.0.103:9101:9101"
         - "40.0.0.103:8080:8080"
         - "40.0.0.103:8443:8443"
       working_dir: /opt/jetty.base/node/
       command: ["./start.sh"]

By starting the environment with the above docker-compose we end up with:

[root@localhost cluster]# docker-compose -f docker-compose-ext.yml ps
         Name                       Command               State                                        Ports                                     
-------------------------------------------------------------------------------------------------------------------------------------------------
cluster_node1_1       /docker-entrypoint.sh ./st ...   Up      40.0.0.101:8080->8080/tcp, 40.0.0.101:8443->8443/tcp, 40.0.0.101:9101->9101/tcp
cluster_node2_1       /docker-entrypoint.sh ./st ...   Up      40.0.0.102:8080->8080/tcp, 40.0.0.102:8443->8443/tcp, 40.0.0.102:9101->9101/tcp
cluster_node3_1       /docker-entrypoint.sh ./st ...   Up      40.0.0.103:8080->8080/tcp, 40.0.0.103:8443->8443/tcp, 40.0.0.103:9101->9101/tcp                                            

The above is exactly as I wanted, I can use the same cluster setup external to docker with my containers.