New office – new network!

As we recently moved to the coolest office in all of Graz, we had the pleasure of designing our network setup from the ground up. What we came up with and why we are pleased with it on a daily basis you will find out in the following lines.

If Abraham Maslow had known how essential stable and performant network connectivity would become in our daily lives, he would probably have designed his hierarchy of needs in a different way. What can occur as an annoyance in a personal context (say, if your phone continuously reconnects to your WiFi network of if your Netflix stream drips into your TV bit by bit) can mean very different things for a business. In a worst-case scenario, this could mean that operations within the whole company stand still and everyone is forced to take coffee break. Since we generally try to avoid such horror scenarios (at least from a network administrator’s point of view, who, by the way, would also love to take breaks in such cases), we tried to think of a few ways to mitigate problems before they can even arise.

Furthermore, we tried to come up with a solution that would be future-proof for many years to come, while also being easily extensible and feature-packed to the maximum but which would, on the other hand, not lead the company into bankruptcy.

A big thanks goes out to our network-grandmaster, conqueror of VLANs and expert for almost anything; Stefan B., for our collaborative effort of designing, estimating, setting up and running this whole endeavour. And for the joy we get from receiving new “toys” to install in our network rack. Further his ingenious mind also created the technical marvel of combining an LTE Router and two ADSL lines into one usable WAN connection, that worked tirelessly at our old office until our recent move. He is also the author of an older blogpost about an OpenVPN server, which he successfully configured to run on our, at the time, main router, behind this jumble of load-balanced WANs. Behind the mentioned router, a 24 port switch did its job, followed by a few other “smaller” ones, cascading the network to even the most secluded places and remote workstations in our old office. If we were talking graph theory right now, I would compare the setup to a tree – a, uhm, “special”, nowhere-near-balanced and, for my liking, much too tall tree.

Trees become stars

And so, we set off fully motivated and started laying out network plugs based on the floor plan of our new office. We had the following four points in mind while doing so, with all of them somewhere in the range of must-have to nice-to-have:

  1. WAN connection via a fiber-optic cable
  2. At least one network plug per workplace leading directly to our rack supporting Gigabit speed
  3. Managed Switches and VLANs for proper segmentation
  4. Access Points (APs) capable of running on Power over Ethernet (PoE)

With an estimated number of 25 workstations spread across four rooms, one showroom, one large meeting room and three to four APs we initially felt like one 48 port managed switch was ample capacity for all of our needs. This changed drastically after we found ourselves staring at the over 80 network plugs we had naively end enthusiastically laid out. But your plan might as well be future-proof, right? And maybe, one day, the desks will be rearranged and then you are going to need all of those network plugs in those walls nobody is currently sitting next to. Mark my words, this will be the point in time when you’ll thank us for not having to run a network cable across the room – not if it can be prevented as easily as by accounting for more plugs in the initial plan. And there it was: the need for a second 48 port switch. From what once was a tree arose a beautiful star, centered right in the middle of our office – at the place where our network rack is situated and which is also my new favourite place in the entire office.

We also decided that joining multiple slow plus one LTE connection with comparably high latency should be a thing of the past and that we would instead dive into the, to us, unknown territories of fibre-optic cabling for our WAN connection. After hearing from a well-known ISP that supplying our site with fibre-optic cables was simply “not possible”, another one told us, in a call to their business hotline, that it would “of course, be possible […]”, but they would have to “[…] tear up the street leading to costs in the neighborhood of 10.000 to 20.000 Euros.” I think, I don’t have to tell you that both had thus put themselves out of contention.

Eventually, we tried our luck at Citycom Graz (which we should have from the very beginning) – and believe it or not; a short time later we had their technicians standing at our rack, splicing away. One of the two replied to my question of `what would be possible, speed-wise` that “[…] each of the fibers would be able to handle Gigabit” and that they could “[…] connect the remaining 20 of the 24 fibers, if necessary”. Keep in mind, we’re only using a single one of the installed four and with an asynchronous contract, providing us with 300Mbit down- and 50Mbit upload, at that.

Future-proofing at its finest 🔥

Hardware

Questions upon questions: Which Router? Which Switches? Which Access Points? Which vendor? CLI or GUI – why not both?

In times of such severe questions we found answers at our oracle, which spit out almost 200 relevant products when being asked about our search criteria concerning the switches (manageable, 48 port Gigabit, at least one 10Gbit port). At the time of writing this blogpost, and beforehand, while researching thoroughly, the top two ranks were occupied by devices from Uqiquiti Networks. Additionally, I still had an article in mind, written by a former sysadmin, who had created his very own (and definitely also my) dream setup using Ubiquiti devices. To paraphrase parts of his article:

“[…] I didn’t just want to set up Wi-Fi with a guest network—that’s so pedestrian. No, I wanted to emulate the things I used to have at work. I wanted multiple segments and VLANs. I wanted to sequester my IoT crap on its own little isolated chunk of space. I wanted complex packet filter rules. […] I wanted metrics, deep packet inspection, intrusion detection, charts and graphs and data everywhere. I wanted something to play with.
Source

Yes, damn it, we want that too!

Switches

We were pleased to see those two results since we had previously had only positive experiences with Access Points and Router of that brand – time for a decision:

Yet, the search for switches was only the beginning of our travels down the rabbit hole of EdgeMax and UniFi products. One would think that I’m getting paid by Ubiquiti Networks to rave about their products or that I get free samples, but that is not the case (yet, unfortunately) *looking at you Ubiquiti* The components should harmonize well (which was another one of the reasons to stick to one brand), be configurable to the smallest detail while, on the other hand, not be too complicated to do so. Products from the UniFi line always require controller software, which actually brings the various components to life, but also provides beautiful graphs and detailed looks behind the scenes of the network. Other differences can be found in the command line interface (CLI) of EdgeMax devices, which is much more versatile and feels like it was actually designed to be used. This, among the other mentioned points, is why we eventually decided to use EdgeMax devices for our switches and the router, since they all come with their own web interface for management which is used to configure them, without the need for a controller. Additionally, Ubiquiti also offers software to administer and monitor EdgeMax devices: UNMS – but more on that later. With that decision out of the way, we had killed two birds with one stone and in terms of pricing, there was no significant difference between the two devices. EdgeSwitch Lite 48 – welcome to our network.

Right, before I forget: since both switches have two SFP+ ports each, we thought it would be a shame not to use them and therefore ordered two direct-attach-copper cables and connected them in a link aggregation group (LAG). This is definitely necessary, say, if 20 different clients on one switch wanted to talk to 20 different ports on the other one. At the same time. With full Gigabit speed. Well, now they could do that.

Router

For the router, our wish was that it was rack-mountable, even if we had to use an adapter as we currently do, and that it was powerful enough for the years to come. An SFP port and PoE support would be nice to have – you never know what you might need it for – but on the other hand, a lack of these would not have been a disqualifier. Among all of the EdgeMax devices we could easily exclude the bigger EdgeRouter models, since having eight and more ports, let alone models with exclusively SFP(+) Ports like the ER-8-XG, is way too overkill for our use case. We had therefore shortlisted the ER-4 and ER-6P, both rocking a quad core CPU clocked at 1GHz and 1GB of RAM, but not excessively many ports, as mentioned. According to the buyers’ guide, both fall into the category of “maximize performance” and eventually, we crowned the ER-6P to become our router, since it also supports PoE.

Access Points

Also in this category, we found what we needed at Ubiquiti Networks, this time in their range of UniFi devices. Again, we could relatively easily narrow down the flood of offered access points, since the more high-end models like the UAP-HD or UAP-SHD are designed to handle 1000 and more clients, which is just a tad too overkill for our setup. However, we came across the, at the time, relatively new UAP-nanoHD which, in contrast to the cheaper AC Pro series, supported 4×4 MU-MIMO, was physically smaller, and did not exceed our budget. Oh yeah, we bought four of those.

Software

The following two sections both show our controller responsible for administering the UniFi and EdgeMax devices, respectively. We hope that, in the future, there will be a way to link both systems together, at least for an analysis of statistical or historical data. Unfortunately, there is currently no way of doing that. *still looking at you, Ubiquiti*

UniFi

As mentioned previously, UniFi devices need a controller for configuration and distribution of the same. For that purpose, we use a Docker Container which runs the controller software locally on a server, providing a detailed web interface for statistics such as connected clients, their transferred amount of data, current connection speed, as well as a calculated “quality number” in percent, among other metrics. The controller is also responsible for management or firmware upgrades and rolling them out to the various devices and, as one of its main features, the definition of SSIDs for the wireless networks to be created. It’s also the controller’s job to spread the networks over the free channels in a way that they interfere with one another as little as possible. Additionally, it is also recommended to perform the management of said devices on a separate VLAN, which can also be set in the web interface for each access point. On our router we have a DHCP server, that is responsible for handing out IP addresses for this network only, including the APs in this case.

In the screenshots above, you can see a small part of the statistical overview for a deeper insight into your clients. Further, we decided to exile our guest WiFi network to its own VLAN in order to prevent access to our general network, containing mostly workstations, and only allowing internet access – again, our router is doing this for us.

UNMS

As mentioned before, we use a second bit of software in our setup which, in contrast to the UniFi controller, is only responsible for monitoring EdgeMax devices. For its installation we used parts of the installation script for UNMS, that can be found on github.com with some minor changes regarding the ports we use.

Analogous to the UniFi controller, UNMS is responsible for frequently checking for new device firmware upgrades and, if configured to do so, rolling them out and installing them on the selected devices.

Both UNMS and the UniFi controller automatically generate backups of the configuration files and renew and incrementally save them, if changes to the configuration were made. These settings will be kept for a defined amount of time in which you can also easily manually download them or return back to any previous configuration, without any effort basically.

Is this finally over now?

Not quite, I didn’t really talk about our router yet, even though it is the one component that is working so hard for us. So, sit yourself back down and read along.

As mentioned above, we tried to segregate the various network segments from each other by using VLANs, which is a good idea in any case. The ports can be tagged, excluded or untagged, which can all easily be done from the web interface of both of the switches as well as the router (and surely also via the CLI, if that’s your kind of thing). Generally, there are tons of tutorials and explanations on the internet and you can really configure very impressive stuff with it – or lock yourself out unintentionally. Yet, this is still not everything.

Additionally, we still need to allow certain connections under certain conditions to be forwarded from one to another specific part of the network. Good thing that somebody came up with stateful and zone-based firewalls and their corresponding zone-policies. With those it is possible to define zones that can span over multiple real but also virtual network interfaces. In order to allow or block connections between the various zones, the mentioned policies are put in place, that make it possible to reach the WAN zone and therefore the Internet from our CF-LAN, containing most of our workstations (action accept). However, connections going in the other direction are blocked, except for incoming packets that are associated with an already existing connection (status established) or arrive as a response to a request of such an existing connection (status related). Packets with any other status (new or invalid), however, will be discarded upon arrival (action drop). Ubiquiti themselves offer a great beginners’ article about the topic and how such a zone-based firewall can be configured.

Finally, we installed a VPN server, configured all relevant policies correctly and the network is now basically functional. Any further details and adventure stories about the experiences with dnsmasq and its friends is a topic which I won’t bore you with today. If that disappoints you, maybe there will one day be another blogpost about that, so stay tuned. Until then, you can admire all the pretty colours the web interface of our router produces – our delayed contribution to pride month. 😉

Anyways, I find a lot of joy in walking past the happily blinking LEDs of our router and switches each morning. Speaking of joy, I have to return to the network rack in order pet my lovely toys. See you next time!