This is Part 1 of a 3-part series. Check out “What Was It Like?” and “A Retrospective and The End: Part 3” for more! “What Was It Like?” has the technical info and “A Review and The End” is more personal anecdotes and funny (and not-so-funny) experiences. Thanks and enjoy!

I’m looking for free or cheap Ubiquiti equipment to explore, please reach out if you have some

I am available for IT work; if you read this and think “I need this guy”, contact me directly, or through Upwork. I also offer a service called My Genius where I will answer one question per month for you to your satisfaction, including up to one hour of combined research and discussion. Additional questions are an extra fee. You can hire me as a consultant for literally anything.

What is a WISP?

A WISP is a Wireless Internet Service Provider. They provide internet service wirelessly, as opposed to fiber, cable, or copper and are found in more rural areas, typically. The types vary from 5G/LTE to various forms of licensed or consumer wireless data technologies. This was the latter, consisting almost entirely of 802.11 devices (although we later added a 24 GHz link that is a custom, proprietary protocol designed by ex-Motorola Engineers; the thing is a beast and can run 850+Mbps real TCP traffic). We operated in the 900MHz, 2.4GHz, 3.65GHz and 5GHz (sometimes called 5.8GHz) WiFi ranges. Some people will tell you there is “only 5GHz and not 5.8GHz”, but the UNI-III band operates from 5.7-5.8GHz and we used it regularly as there was less interference, typically. There is now a 6 GHz WiFi band available, but that wasn’t around when I was working there. The 3.65GHz band is the only one that requires a license from the FCC, which we had. You can find a good list of WiFi channels and where they are allowed on WikiPedia. There is a 45GHz band available in China, but we never used that.

How big was your WISP?

We provided service to slightly less than 1,000 customers in a 900 sqmi region in the Wausau area of Wisconsin (I’d sometimes call this the “Wausau Metro Area” and get laughed at because all the cities together are barely 40,000 people). We didn’t operate within the city limits, mostly because there was no line-of-sight (LoS) to any of our Access Points (APs). All of this was fed through a redundant fiber uplink where we peered with Hurricane Electric and had two primary IP blocks.

This service started as a single T1-speed link where “the owner” (who will only be referred to as “the owner” due to privacy and personal reasons) sold business level internet from one location where it was available to a more rural location where it was otherwise completely unavailable, at any speed or price point. That’s right - “there was no internet”. We’ll take a quick trip down memory lane to what the internet was like “in my day”. Most internet was available only over phone lines, originally at 110-300 baud (although this was before my time) then eventually up to a “usable” (for terminal access) 9600 baud. There is a very good History of Telephone Internet and modems here.

Early Internet

When I was a kid (and I’m unfortunately old enough for that to no longer be a joke) internet was available at 14.4-28.8 kbps (or 0.01373291-0.02746582 MiB/s). You would use your standard telephone line and a modem (modulator-demodulator) to connect to an ISP, who would then connect you to the wider “internet”. Some businesses would have “business-level internet” at a blazing 1.544Mbps (T1 line) to the much-drooled-over DS3 line (28 T1 lines, or ‘45Mbps’). These were all based on DS0, which is a 64Kbps ‘channel’. If you were really a baller, you’d have an OC-3 line, which is fiber and part of the SONET standard and runs at a blazing 155 Mbps (the highest “OC” line is OC-768 at 39,512Mbps. These were all “leased lines” available from the phone companies, who had run the original copper and fiber lines. You could even just get a set of copper pairs and your own modems and shoot for the moon (there is non-standard equipment that will reach higher data rates on bare copper than T1 speeds). There’s a great overview here.

Getting internet in rural areas is difficult in Wisconsin for a number of reasons. The first one is that all of the phone lines are really old and poor quality, so DSL and T1 lines (the first “high speed” internet) is widely unavailable and poor quality. I’m not going to cover DSL here, but it was basically a technology to push faster speeds than T1 over a pair of copper. If your lines are crappy (meaning noisy) enough, you won’t be able to push even 1.544Mbps over them because the error rate will be too high. The next difficulty is that no one ever ran fiber optics out past the major metropolitan areas because the population density is simply too low to make it profitable. So there’s typicall no T1, DSL or SONET internet “high speed” available, and you can often only achieve 9600-15Kbps even with a 56k modem because the error rate is too high due to noise.

The other main issue is geography. A couple years back (OK maybe more than a couple) giant glaciers passed through the area that is now known as “Wisconsin” carrying gigantic rocks that “roughed-up” the surface of the earth, so the terrain is very hilly and pocked with [kettles](https://en.wikipedia.org/wiki/Kettle_(landform), or depressions in the surface of the earth. Why is this important? In order to transmit a wireless signal in most circumstances you need LoS (line-of-sight, meaning you have to see from one point to another). You actually need more than that due to what’s known as the Fresnel Zone (if your Wireless Guy doesn’t know what this is, especially if you’re doing anything outdoors fire him and hire me or, if I’m not available and he’s otherwise competent, tell him to get learnt or GG as the kids say). Due to the way waves propagate you need an almond-shaped “zone” between the two points (although I have seen things that sometimes work inexplicably). It is possible to do things like bounce signals off of the moon and the troposphere but I haven’t done that…yet. Which brings us to how do you get internet in rural areas?

What about 5G and satellite? 5G didn’t exist at the time and cell phone signals were often not available either, especially if you lived at the bottom of a kettle. Satellite was limited in total bandwidth and super expensive if you went over the limit. The latency was also hundreds of ms compared to ours, which was usually 10-20ms, we wouldn’t install anything if the latency was over 100ms unless they signed a waiver, because it feels slow AF, you can’t play games, etc. Still, some people were desperate for anything at all and would beg us for even a bad connection as their dial-up was even worse

Access Points and Backhaul

IIRC, the first AP was on a feed leg on what was Marathon Feed, the local granary. The building is actually a hidden gem - it was manufactured a very long time ago and is all wood inside. I don’t think it’s hand-hewn, but honestly it might be. It’s full of huge wooden beams and has a 150 ft feed leg. There’s actually two Marathon Feed locations - the old one in downtown Marathon City, which was not used while I worked there, and the new location south of Marathon City, which we also had equipment on and was my first climb. You could probably contact the Marathon City Heritage Center if you want to know more.

This is where my memory gets a little hazy, as my employment was discontinued suddenly in 2017 in an incident which was never fully explained to me by the owner. IIRC, we had between 20-50 ‘link radios’, which included backhaul and APs. Originally, the T1 was fed to a link from Marathon Feed to the client, I believe using Tranzeo radios. From this, the company gradually expanded to its current size. The main fiber feed was at the Office in Marathon City, and the signal branched and fed various APs, which then provided the access to the CPE (Customer Premise Equipment), which is another WiFi radio mounted to the outside of the customer’s house…or silo…or up in a tree…or in a pole in the yard…or even on a giant tower erected solely for the purpose of internet service (we’ll get into this later in the crazy installs.

How did you get into this field?

History

You can safely skip to The WISP Job if you don’t care about me as a person and my history. It’s fine. Go on, now, I can handle the incosiderate dismissal of myself as a person I’m KIDDING. Honestly, this is just a short trip down the who I am road.

I wasn’t historically trained in wireless communication, I don’t even have a Bachelor’s Degree. I’ve loved computers and technology since I was a child. When I was 6, I used to play with cords and cables instead of sports or dolls. I could see that inanimate things were made animate and pictures and sounds were somehow magically transferred over these thin lines. How did it work? How did a vacuum, which did nothing by itself, suddenly roar and suck up little bits of things when the cord connected to the outlet? How did the TV, which ordinarily only looked like gray fuzz, suddenly spring to life with pictures and sounds when this…line…was attached to it? I’m amazed I didn’t electrocute myself (I have, however shocked myself many times, both with electricity and my voracity in the sack << >> [guys/gals: I’m available, twinks to the front of the line, please, lol]). Anyways, where were we?

I got my first computer in 1994, a Gateway running at the phenominal speed of 66MHz (that’s 0.066 GHz for you youngins [see above ;-)]) with 8MiB of RAM and a 14.4Kbps modem. I fell in love. My personal and social life at the time was awful, despite a good family who has always provided for me and supported me. I was neurodivergent and queer but I didn’t know it yet. I just knew I was different. I lost almost all my friends in 1st grade when one of the popular kids decided I was weird and most of my prior friends decided to “side with them” and stopped doing things with me because I was “uncool” or whatever 7-year-old kids think. I definitely had little professor syndrome and I couldn’t understand or comprehend the complex social reasons why I was being rejected, but it seemed unfair and unkind (and, indeed, it was - but they didn’t know that at the time). Due to this, I sought refuge in things that made sense to me, like reading, math, and machines. They behaved more predictably, and this order and predictability gave me a sense of control, but more importantly stability that I needed.

I first got on the internet in 1995 (IIRC) at the age of 11 and quickly rang up a $500 bill with AOL that I denied fervently. Thankfully, they relented and bought the story about some person beige boxing our line with a modem outside or whatever it was I came up with. The Internet gave me a sense of frontier and horizon that I otherwise lacked, being a very shy and socially awkward child. You could do anything, be anyone, and learn about anything - or so it seemed at the time. Of course this led to all sorts of forbidden knowledge which sang to me like a siren song. I got into hacking (or at least the idea of it) not as a means of doing devious things (though a babysitter once called me and my brother hell children) but as a means of pushing boundaries, exploring potential, and doing things that I just didn’t have the resources for as a then-12-year-old. “You’re not old enough” meant nothing to me. If I could find a way to do it, then I was “old enough”, de facto. This, and my unacknowledeged

The WISP Job

I was trying to move to the Family Farm, in “up north”, Wisconsin (this is everywhere north of Madison, but in this case a small town called Maine or Athens outside of Wausau and I would visit for a couple weeks at a time as often as I could. One day, I was talking to the neighbors and they had a technician at their house fixing the internet and I asked about it. I was fascinated and wanted that F-ing internet, as I had to walk 1/4-mile for cell service where I lived. We had awful dial-up and a house phone - that was it. But I loved living on the farm, so I dealt with it. What was this black magic internet they had? She had a remote job, half a mile from what seemed like last century to me at home. I talked to her and she set me up an interview with the owner. I got a call at a coffee shop where I was using the internet one day and he said “can you swing by the shop and pick something up for me”. He needed a power supply. The shop was a chaotic place with all sorts of various electronics and strange devices on shelves and I immediately felt at home. It was a tinkerers workshop, and that’s what I was. I immediately found and identified the proper power bricks and met them in a field. He said something like “I hope you weren’t expecting a guy in a suit. He was wearing a t-shirt with the company logo and the tech guy was just dressed “like a normal person”. No office casual or anything like that. Awesome. I had to use the bushes during my interview and he didn’t flinch. I passed the initial meeting with the tech guy and the next step was…

Heights - we had to regularly climb 60-150’ high structures, which lots of people didn’t like. I loved it. As a small child, I used to climb pine trees and was a little monkey, happily. We climbed Marathon Feed without issue and I got the job. Prior to this, I’d installed OpenWRT on my Linksys WRT-54G, but that was all I knew about wireless. I had a good head on my shoulders, though, sufficient knowledge of electronics, and 9 years as a Network and Systems admin doing primarily first-contact Tech Support (I have no idea what “Tier” that is because we only had 3 people, I just know I fielded the calls and escalated when necessary). It started with a primer on wireless and RF from our resident electronics know-it-all and RF guy - fresnel zones, LoS, path loss, all that good stuff. Then we started with site surveys.

Site Surveys

This is when you go to a customer’s house, take into consideration the location, obstacles, nearby Access Points (we had a map) and decide whether they can get internet and what kind of speed we could offer. It entailed climbing onto the roof of their house and setting up a modified standard camera tripod with a radio and an ethernet cable (328 ft) on an electrical cable spool with a female adapter. The most common faux pas was plugging in the female adapter first and then unspooling the cable, resulting in a super twisted wire. Once it eventually got too bad, it would have to be unspooled, reterminated and replaced. We had binoculars, an extension ladder, a radio on a tripod, and most importantly permission to be on the roof and we’d show up at a random time. Occasionally a customer would forget they ordered it and come out of the house, confused and wondering why someone was on their roof with what looks like a spy device (the parabolic dish was used for testing) and binoculars. It was actually pretty fun.

We would then set up the radio and point it at the nearest AP and then climb down to check and see what we could see signal-wise on a laptop. We did experiment with having another radio serve the connection to a mobile phone or tablet, to preven the need for climbing down, but for various reasons this didn’t work as well as you’d think. Once on the laptop, we’d do a scan and, if an AP was available, connect to it, climb back up on the roof and use the LED indicators on the side of the radio and visual guesswork to try to get the best signal we could. Then, we’d climb back down and test the connection on the laptop to see what we could offer speed-wise. IIRC, we had 1/1 5/1 15/3 (downlink/uplink) and custom rates for people who wanted the absolute best internet they could get, usually remote workers (and yes, you could work remote on our network with no issues when it was functioning properly if you had a good enough connection).

Sometimes, though, the roof wasn’t the place with a clear LoS, which was the prerequisite for good internet. So, you’d set up in the yard, or climb a silo, walk around the yard, get creative. Any signal less than -70dBm was usually unusable, with signals in the 60s solid. Of course, the signal wasn’t the only factor involved, which is why we used a radio they would use and verified functionality. We would usually start with a NanoBridge M5, as they had a relatively wide reception pattern. Every antenna has areas that it receives strongly and areas that it receives weakly. Omnidirectional antennas attempt to receive signals equally in all directions, while directional antennas attempt to optimize reception in certain directions and minimize it in others, to increase SNR (signal-to-noise ratio). There’s a great guide to antenna radiation patterns here. To calculate your SNR (it’s done automagically) you take the amount of signal and divide by the amount of noise; higher is better. If you want a really in-depth discussion of SNR and how it affects MCS Rates and modulation, this is the best article I could find, it’s a couple pages but it’s worth the read.

The final part was the recommendation of the placement and the radio type. Ubiquiti at the time had a few different types of CPE that had different radiation patterns and receptivity, as well as range. The range is determined by a couple of factors: the emitted strength of the signal at the AP, and the directivity and antenna gain of the radio. The antenna gain is how much the signal is increased over baseline by the characteristics of the antenna - how well it “picks up” the signal and oscillates in sychrony. Antenna design is the real black magic of WiFi if you ask me. You can get an amazing amount of signal amplitude “for free” (ie without an amplifier, which would also increase noise along with signal) with an antenna. For instance, the Ubiquiti airMAX AC 5 GHz, 30/34 dBi RocketDish Antenna provides up to 34dBi of gain (when properly aligned, which we’ll get into later). This means that the signal strength in the antenna’s favored direction is approximately 2512 times stronger than it would be with an isotropic radiator (a theoretical antenna radiating equally in all directions). And since it’s directed, the interference that’s present in the local environment is minimized, as the gain in other directions is quite low. Learning to read radiation pattern diagrams will allow you to develop an intuitive sense of how to position a given radio and is a highly desirable skill worth developing. Read, experiment, understand.

Responsibilities

Although I started with Site Surveys, I’m a can-do kind of guy and as I proved my worth, I was given more and more responsibilities. I’m a highly adaptable person and a voracious learner. And for whatever reason, I’m fascinated by wireless signal propagation. I used to be a hardcore “cable” guy - I wired my house with gigabit ethernet because I thought WiFi was for posers (and it was 54Mb vs 1000Mb so it’s not like it was a small difference). As much as I was fascinated at the seeming magic of cords and cables as a child, though, I was even more fascinated by wireless, how it works, how to optimize it, and how best to deploy it. If cords were magic, wireless was god-tier mystery. I dove in deep.

The company was also an almost perfect match. They were very DIY, do what works, KISS, improvised and open source when they could and proprietary when they had to. The servers were primarily linux, as were most of the radios (Ubiquiti radios run linux with busybox, which is another mind-blowing feat of optimization, as the entire OS fits in a 8MiB flash). I was encouraged and allowed to learn on my own, try new things, and inherited a hodge-podge of systems developed by different people that I had to reverse-engineer. There was one guy that did a lot of the linux and Cisco stuff who’d since moved on to a different job but occasionally came back to help, the RF guru, a technician, and the owner. That was it. After 2 years I was basically running the network single-handedly and for one low-staff month me and the owner ran the whole thing - all the EU (End User) support, network maintenance, sight surveys, the whole nine yards.

Here’s some of my responsibilities:

  • Customer Installs
  • Customer Support
  • Documentation
  • Link Radios
  • Access Points
  • Hardware Design
  • Server Administration
  • Network Administration
  • Site Surveys
  • Link Budgeting
  • RF Surveys
  • Interference Mitigation
  • Network Monitoring
  • IDS/IPS
  • Locating and Eliminating Packet Storms from viruses
  • Dealing with the monthly mail server blacklisting
  • Zabbix/Suricata/SecurityOnion implementation and monitoring
  • FCC Reporting