How to open a backdoor to your place ==================================== 1. Why on earth would one do that. Most of us have some wireline hookup at home for internet. All dandy unless something fails and does not recover due to power glitches caused by weather or network overload, resulting in a brown-out which is not enough to reset equipment, only confuse it to the point that it locks up. All not a big deal as long as one is home to pull the plug, but it becomes an issue if one requires remote access to self-hosted gear, and cannot have somebody run the reset. Think a road-warrior's third-worst nightmare. With worst being "no cellphone signal", followed by "no battery". In an earlier age some CPE (customer-premise equipment, stuff that an ISP deploys at your place to convert fiber, DSL, cable, fixed wireless to copper (ethernet, coax) or WiFi), such as an Optical Network Terminal Terminal (ONT), came with batteries built in, typically not covered under maintenance, so when that battery aged out, one had to replace it oneself. Newer gear appears to no longer contain a battery, so one would have to provide an UPS, which probably is a good idea, but still leaves an opening for a glitch. 2. How do we do that. The idea is to put in place a secondary uplink, nothing expensive, because the primary is quite likely expensive enough. The secondary has to be not only low-cost, but also must not have any dependencies on the primary. One approach is to obtain a personal hotspot, i.e. a small portable WiFi hotspot that uses LTE or 5G for an uplink, using a low-cost plan, normally typically offered for IOT. Unless there is an intention to use it on a regular basis, a low end data plan will do. 3. Options There are a few ways on how to do that. Back when dial-up was still a thing, one could hook up a modem to one's phone, configure it to pick up after a few rings, then dial into a terminal server or PC, take it from there. Downside with that approach is that phone lines are no longer as common, and, being tuned to VoIP, also lack some ability to properly sample modem noise. One would also need a second phone line to somehow call home. Another option would be a second uplink, like a cable modem in addition to fiber, which, if used just as and exclusively a fallback, is too expensive. This leaves us with a kind of fixed wireless, on a diet. Cellphone providers have been getting into the broadband business by offering internet access through devices which, towards the premise, look like a regular router, but use LTE or 5G for the uplink, similar to satellite internet, minus the dish. There is one network issue, though. Unlike a wired router (fiber, cable, ...), whose outside-facing IP address is directly reachable from the outside (unless the provider engages in creative port blocking), the outside-facing IP address of an LTE hotspot is not the same as the one seen by servers on the net. Easy to prove, connect to your hotspot, find the status page, note its WAN port IP address, then surf to one of the "what's my IP" web pages (best viewed with an ad blocker). The IP address they'll display is very likely not going to match the one on the outside of your LTE router. That is not helpful, because it means that one cannot connect directly from the outside through the provider's network to your LTE hotspot. Means a meeting needs to be arranged. Where two parties meet at a place at a time, like for a rendezvous. For which one needs a place, and a way to get there, where one of the parties meeting offers a way for the other to hitch a ride home. The meeting place ideally is some host offering login through ssh, as well as remote port forwarding. This might be real hardware, or a virtual VM somewhere within the usual suspects' clouds. The parties meeting consists of an ssh connection, originated by a process running on a backdoor gateway/server located within your own network, the one whose CPE is acting up, the other by a regular ssh session, initiated by you from some other outside location. That second part is trivial, when one finds out that one's primary connection is dead, ssh into the rendezvous host, wait for the other to arrive, then take the ride back using the forwarded port. The tricky part is to establish the connection from home through a wireless LTE router to the rendezvous host, all automated, hands-off. 3. Network setup. Typically your main router assigns IP addresses using DHCP, which includes provisioning both a DNS server and a default route, pointing back to the main router. Since the main router won't be able to connect to the provider's network, due to the CPE acting up, equipment trying to reach the rest of the world through that default route won't be able to establish any connection. Consequently, the device hosting the backdoor must not obtain a DHCP lease for your main router, but rather use a static IP address, without a default route. For Raspbian and other Linux distributions using dhcpcd, modify the default /etc/dhcpcd.conf to add these two lines to the end: interface eth0 static ip_address=192.168.1.1/24 Substitute the IP address and netmask length with what your router typically assigns (most home routers use 192.168.x.1 or 192.168.x.254, with a netmask of 255.255.255.0, which translates into /24), and an IP address outside of the DCHP address range used by the router. Many routers use a range of 100-200 (so 192.168.x.100-192.168.x.200). Above settings were set using the sample static IP configuration: # Example static IP configuration: #interface eth0 #static ip_address=192.168.0.10/24 #static ip6_address=fd51:42f8:caae:d92e::ff/64 #static routers=192.168.0.1 #static domain_name_servers=192.168.0.1 8.8.8.8 fd51:42f8:caae:d92e::1 The important bit here is that there is no "static routers=" and no "static domain_name_servers" setting. Since the backdoor server no longer has a default route, you need another device on the same network to talk to the backdoor server, because it is no longer directly reachable through the main uplink. Furthermore, the backdoor server needs to talk to another server on the internal network to offer access to gear capable of fixing the outage, such as remote power strips, reboot- and terminal-servers, ... Reboot the backdoor server to activate and test the change. 4. Proxy setup. The LTE's hotspot's data plan might be low volume, where, after all high-speed data within the billing cycle has been used, access is shut down or slowed down until the next billing cycle. To keep this from happening, larger amounts of data need to be handled using the main uplink, the one that we just avoided using by not establishing a default route. The main gourmands (gluttons) of data are browsers, updaters and streaming apps. There is no good reason to run a stream client on an access device, while a browser comes in handy if recover equipment is only accessible through an internal web page. Updaters, on the other hand, are something to maintain, because a backdoor hosting server is an outside-network-facing device, and while the hotspot, LTE router, ... includes firewall functionality, why would one run the risk, so one has to either run updates over LTE, at the expense of having to wait a little longer if one has run out of high-speed data, or configure a proxy server for updates. To do so, create, in /etc/apt/apt.conf.d, a file named 10proxy, containing this: Acquire::http::Proxy "http://192.168.1.2:3128/"; This tells apt (one of the widely used Linux package managers. Yum is another one) to not talk to its package servers directly, but send all and any requests to 192.168.1.2, port 3128. This is not a randomly chosen port, but rather squid's (a web proxy, a server process used to provide web access to devices sitting behind a proxy server on a non-routed network) default port. So yay, we need a second device. No news there, because that second device quite likely allows us to move freely within our own home network, because it may have routes to places not directly accessible from the backdoor server. The next step takes place on that other server. Download squid using your package manager, then modify the configuration by adding or uncommenting the line kinda matching your local network (in our case, the one with 192.168 in it) within this block: #acl localnet src 10.0.0.0/8 # RFC1918 possible internal network #acl localnet src 172.16.0.0/12 # RFC1918 possible internal network #acl localnet src 192.168.0.0/16 # RFC1918 possible internal network acl localnet src 192.168.0.0/16 # RFC1918 possible internal network #acl localnet src fc00::/7 # RFC 4193 local private network range #acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines Also, make sure that localnet is allowed access further down. Look for "http_access", find the one allowing localnet. Add it if not present: # Example rule allowing access from your local networks. # Adapt localnet in the ACL section to list your (internal) IP networks # from where browsing should be allowed http_access allow localnet http_access allow localhost Might want to restart squid, or reboot the proxy server host to activate the change. This concludes "the other server" setup. Back to the backdoor gateway. The software looking for network connectivity loss is written in Tcl (see https://www.tcl.tk), the door maintaining the backdoor is written in Expect, a Tcl interpreter with chat capability bolted on, see https://wiki.tcl-lang.org/page/Expect and https://core.tcl-lang.org/expect/index Apt, yum, or .... both onto your backdoor gateway. Once the scripts (see below) are in place, invoke crontab to add this line: 2,7,12,17,22,27,32,37,42,47,52,57 * * * * /usr/bin/tclsh8.6 /home/pi/Expect/ping.tcl (all that needs to be on one line!) It invokes the command after all the stars every 5 minutes. Make sure to explicitly specify paths to both executable (/usr/bin/tclsh8.6) and script (/home/pi/Expect/ping.tcl) explicitly, because cron runs jobs using a limited, not fully populated environment, which may not include a fully populated PATH. There are good reasons to do so, because one could slink code of one's own design into the path used by code running as root to add some creative functionality. The crontab augmentation needs to be the very last step within this exercise, once all other pieces have been installed and tested. Chat capability refers to the ability of being able to spin up other processes, send commands to, receive responses from these processes, then to act upon whatever comes back. Chat capabilities are nothing new, they have been around since the dial-up days, where an application intent on talking to a remote server would have to establish a dial-up connection through a modem connected to an old-fashined phone line. To do so, the application opened the serial port the modem was connected to, send configuration commands, wait for responses (an OK was good, most others were not), then issue a dial command containing a phone number, something like ATDT12345678 (dial, touch tone (DT!) 12345678). The responses to the ATDT might be something like "NO DIALTONE" or "BUSY", both reason to disconnect and try again later, "RINGING" would be kinda OK, as long as it would be followed by "CONNECT", optionally with a bit rate like 9600 or 19200, which is good, or "NO ANSWER" (bad, try again later). But, we are still not there yet, because logins have always been a thing, so the next string to wait for is "ogin:" (not the L, because that could be upper or lower case), send your login, wait for "assword:" (no P, because upper or lower case), hopefully we are in, start up your protocol. If something goes sideways, disconnect and try again. Expect makes it easy to implement these sequences, because it goes beyond just simple send/wait-for/send/wait-for sequences, but rather wraps that send-receive ability into a scripting language. The remaining parts of this exercise contain their own documentation within their code: There is ping.tcl. It figures out if: - the secondary uplink device is available (ping that thing), - the secondary uplink has an outside connection (ping something likely always up), - your stuff is up (ping your gear through the secondary device. If it answers, your primary uplink is good), and ... - if your primary stuff is not reachable, invoke expect against autodial.exp, to open an ssh connection to an outside ssh-capable host, tunneling one remote port back to port 22. In an outage situtation this is an acceptable risk, as long as account on the outside ssh-cable host has a reasonably long and unpredictable password, just as the backdoor gateway, so that an attacker has to provide two different passwords to gain access. Review both ping.tcl and autodial.exp, to modify as needed. In particular, you will need to provide your hotspot's internal (LAN) IP address, and the (set of) IP address(es) of your own outside facing gear, as well as the name or IP address of the rendezvous host. In my case, they are both found in /home/pi/Expect. Modify locations as needed. 5. Configuration Both scripts (ping.tcl and autodial.exp) require configuration: - in ping.tcl, change the value of ROUTER (look for a line starting with "set ROUTER" to your router's IP adress on the LAN (not the outside facing one). Use "netstat -an" to find that. - in ping.tcl, set pingMine to the space-separated list of IPs to monitor, in quotes. The line to look for starts with "set pingMine", like this: set pingMine "1.2.3.4 1.2.3.5 1.2.3.6 1.2.3.7 1.2.3.8 1.2.3.9 1.2.3.10" - in ping.tcl, set the address of the rendezvous host. Look for a line starting wih "set RHOST" - in autodial.exp, find a comment saying "SET THIS STUFF!". There are four settings to provide, the first three of them in quotes. They provide to the remainder of the code your remote user name (USERNAME), remote host, i.e.t the name of the rendezvous host (HOSTNAME), the IP address of the rendezvous host (RHOST) and the remote port to forward to the local ssh port (REMOTEPORT). Leave the rest alone unless the chat script fails.