10Gb Home Network (P3) – Build A Switch


Welcome back my fellow 10Gb fanatics. So you made it all the way to part 3 in our
10Gb Home Area Network Series. As promised, we are going to overcome the
obstacles and expense of networking more than two systems using a custom built 10Gb switching
solution. If you missed part 1 or part 2 of our 10Gb
Home Area Network Series, make sure you go back and watch those videos first. Part 3 builds off concepts explained in the
first and second videos. Follow the embedded links now to watch those
videos first. Otherwise, it’s time to build a bad-ass 10Gb
switch. Here is a brief overview of what we are going
to cover in this video. Why build a 10Gb switch? Number one, a custom built switch allows you
to meet a specific requirement. If you are only connecting three systems together,
then why pay for an expensive 8 port vendor switch? Number two, you can achieve an excellent level
of customization. This solution employs an advanced feature
set that you can leverage, if you choose to. Lastly, for some it just comes down to cash
flow. A custom solution allows you to control your
costs better. Now, the part everyone has been waiting for. How in the world do you build a custom 10Gb
switch? Well for starters, you have to build a bridge. According to Wikipedia, a network switch is
officially known as a MAC Bridge. A switch is a multiport network bridge that
uses hardware addresses to process and forward data at the data link layer (layer 2) of the
OSI model. So to summarize, a switch is a bridge is a
switch. So, the plan is to combine multiple Network
Interface Cards (NICs) into a reasonable platform, load an optimized network operating system,
and finally, tie multiple ports together using interface bridging. So those of you that guessed bridging, you
were correct! Gold star for you! The end result is a customizable 10Gb switching,
and routing, platform that allows for media conversion. In other words, you can throw together a dual
port 10Gb SFP+ card, a quad port 1Gb copper card, and a fiber card. Bridge the ports, and you have just built
a media converter. Now, are you starting to see the possibilities? There are some pretty interesting things you
can do with this type of flexibility. I’ll cover a few scenarios in an upcoming
section. First things first! Let’s talk hardware. You can do this with a PC or server. Just like with anything else, your mileage
will vary depending on what you use. See the video description for my setup. You will need a system with at least two PCIe
x8 slots for a 10Gb – 4 Port switch. Your switch will be limited by the number
of PCIe slots that you have, and cards you use. Don’t forget about your embedded network ports
too. When it comes to the hardware, focus on a
motherboard with plenty of bus bandwidth, maximum PCIe slots (x8 for 10Gb dual cards),
and a decent multi-core processor to reduce CPU loads and increase parallel bandwidth. With the introduction of technologies like
DMA, I/OAT, DCA, and NAPI, PC hardware is achieving line rate network speeds formally
only seen in datacenters on enterprise switches. Software Defined Networking (SDN) is a prime
example of this concept already in play. For home users, you don’t need a heavy-handed
solution. On the other end of the spectrum, aiming for
maximum performance, go with a Xeon (E3 or better) and I/OAT-enabled motherboard when
planning for port density. Now for the most critical component in this
entire setup, the network operating system. And the winner is, VyOS! What is VyOS? Why VyOS? Okay, I’m getting there. VyOS was originally developed by Brocade Communications
and known as Vyatta. In 2013 they stopped development on Vyatta
Core. A group of enthusiast developers forked the
source code and created what is now VyOS. “VyOS is a Linux-based routing solution built
on the Debian Linux distribution, and currently runs on x86 and x86-64 platforms.” VyOS is an incredibly light-weight, heavily
customized, and optimized distro with the sole purpose of providing Layer 2 and Layer
3 network functionality on a physical or virtual platform. Why did I choose VyOS over pfSense (as suggested
by viewers)? The answer is performance, among other things. To be clear, I have only tested the aforementioned
distros. I’m sure there are other solutions available
though. I discuss further details about my pfSense
testing in the summary. I can personally vouch for VyOS as a routing/switching
solution in a production environment. I know others use it as an edge router in
datacenters, and it is widely used in cloud implementations like Amazon Web Services (AWS). Awesome! Now let’s dig in with some digital grease,
and get these gears moving. Follow along as I narrate the BIOS Configuration,
VyOS Install, Basic Setup, Interface Reorder, System Setup, Bridge Creation, Bandwidth Testing,
and Useful Utilities Demo. First up, we are going to start by reviewing
some important settings in the BIOS. Check your CPU section and enable: Direct
Cache Access (DCA). Next, under Chipset/Northbridge check for
IOAT and VT-d settings. Enable IOAT for sure and try “No Snoop” and
“Relaxed Ordering” if present. In addition I enabled VT-d and Interrupt Remapping. Make your way to the PCIe Configuration and
look for Above 4G Decoding, Maximum Payload, and Maximum Read Request. Maximum Payload and Maximum Read Request are
very important and you will see why later in the video. Just know that they should be set to the max
value. Save your BIOS settings, and have a bootable
copy of VyOS ready. Boot your system to the VyOS CD. It’s a live CD and allows you to run from
boot or install an image to disk or USB. The default username/password is vyos/vyos. Run the “install image” command to kick off
the install. You can get context sensitive help at any
point in the command line by typing question mark. Most of the install process is accepting the
defaults. The install is self-explanatory as you can
see. I installed to a disk drive, but you can install
to USB as you only need about 2GB of space total. Set your new VyOS user password and continue
accepting defaults. When the install completes, make sure to reboot
your system without the CD. Otherwise, any changes you make at this point
will be lost. When you boot into your newly installed image,
you are in “Operational Mode”. Take a look at your network interfaces with
the “show interfaces” command. Interface name on the left, and link status
on the right. Capital “D” signifies nothing is connected
to our switch at this point. Plug a copper Cat5 cable into one of your
ports and run “show interfaces” again. Now you can see a lower-case “u” in the right
column, signifying the port is up. A better method is to “blink” the interface
you plan to configure. Run “show interface ethernet eth0 identify”
to visually blink the interface you plan to configure. At this point you should notice the command
line is structural in nature. And remember to use the question mark to better
understand your options. We’ve been working in Operational Mode. Now we are going to enter what is known as
Configuration Mode. This allows you to modify your system configuration. We are going to set the IP address of interface
eth1 where we connected our cable. Notice the slash 24 at the end of the IP address. This is our subnet mask specified in CIDR
notation. Slash 24 is equivalent to 255.255.255.0. Now set a description on the interface. Remember to always label for easier troubleshooting. Next allow a terminal program like PuTTy to
connect to your new interface. Run the command “compare” to see a list of
queued commands ready for execution. They do not take effect until you issue a
“commit” command. Following the commit, you need to save, so
your changes are persistent across reboots. Did you notice the plus sign next to the queued
commands? This means they are a new addition to your
configuration, just as a minus sign would mean they are being removed from configuration. Exit Configure Mode and show interfaces to
see if changes have taken effect. In this optional section, I’m going to show
how you can reorder your ports however you like. VyOS may not place your ports in an order
that makes sense. So if you are particular like me, I’ll show
you how to customize it exactly how you want it. I’m going to use the Linux ifconfig command
to pull, and sort by the MAC address. Essentially, we are going to reassign the
MAC address to the interfaces in a different order. Fun fact: The first 3 octets of a MAC address
identify the vendor of the hardware. So try macvendors.com to lookup a NIC vendor. Awesome troubleshooting technique by the way. Once you sort by your MAC addresses, copy
it out to an editor for reference. In my list, the bottom 4 MACs belong to 4
embedded Intel NICs. You can tell by the matching first 5 octets,
and last octet order. So I will make these interfaces eth0 through
eth3. The remaining interfaces, all 10Gb, will be
eth4 through eth11 for a total of 12 ports on my home brew switch. Before you make this kind of change, it does
not hurt to backup your main configuration file at /config/config.boot. Enter Configure Mode and use the “hw-id” command
to reassign the physical port to the logical ethernet interface in your preferred order. Notice I’m using abbreviated versions of commands
to minimize typing. You only have to type enough of the command
to make it distinct from other commands. Here, I’m taking the lower 4 MAC IDs, and
assigning them to eth0 through eth3. Then I assign the top 8 MAC IDs to eth4 through
eth11. Since It’s sorted by MAC ID, it will have
the effect of correcting the ethernet interface order. See, how the compare command shows a greater
than symbol? This means you are replacing a configuration
item instead of adding or subtracting from the configuration. Finally, “commit”, “save”, “exit”, and “reboot”
for the re-mapping to take effect. After rebooting, you may need to relocate
your IP address so you can PuTTy back into your switch. Here, I show how to remove the configuration. If you just tried to set the IP again it would
add a secondary IP address on the interface. On the system console, I will remove the IP
from eth1 and add it back to eth0 where my link status is “up”. Now I can PuTTy back in and everything looks
good once again. Connect to your switch with PuTTy, and enter
Configure Mode. If you type “show interfaces” in config mode
it will show the current, or running, state of your configuration. This is completely different than displaying
the contents of your config.boot file. You can also show other subsections of your
running config. For example “show system”, shows the system
related aspects of your configuration. Let’s go through some basic configuration
settings to get things running smoothly, starting with your hostname. Next, set your gateway address so you can
communicate with the outside world. Then finally DNS, and time zone. Compare your changes, verify, commit and save. Ping sun.com to test your connectivity to
the outside world, and check that time is syncing correctly with the command “ntpq -p”. You can also use the command “show ntp” from
Operational Mode. This completes the section on System Setup. Time to build our bridge. Use a console, or serial connection for this
portion, as we need to reconfigure our management IP again. Login, show your interfaces, enter Configure
Mode, and remove the IP address from eth0. Notice this time around my delete command
stops at “address”. VyOS allows you to remove configuration at
various levels. Now compare, verify, and commit. Here is where we create our pseudo interface
“br0”, and assign it an IP address. This interface represents our bridge, or collection
of interfaces forwarding packets to each other at layer 2. Assign a meaningful description to your bridge
interface. Looking at our compare statement, notice how
default options are assigned to the new bridge. Commit and Save. While in Configure Mode you can “show interfaces”
by using the keyword “run”. You can now see the newly created interface
br0 with assigned IP address, subnet mask and description. Let’s identify a port where we can connect
our 1Gb uplink. Again, blink eth2 to visually identify. Okay, plugged in and showing link up on eth2. Since we have our bridge interface in place,
we will assign ethernet ports 2 through 11 to the bridge. This give us 10 bridged ports. You could assign all your interfaces to the
bridge, but it’s not required and you can change it later anyway. Remember eth0 through eth3 represent my 1Gb
copper ports, while eth4 through eth11 represent my 10Gb Direct Attach Copper ports. In the video I mistakenly started with eth3
when I should have started with eth2. Next add a description to each interface. For your 10Gb interfaces you want to set your
MTU to 9000 as I discussed in part two of the series. When finished assigning interfaces, setting
descriptions, and adjusting MTU don’t forget to compare, review, commit, and save. After I make some quick corrections to my descriptions show interfaces, and marvel at your
shinny new 10Gb switch that you lovingly crafted by hand. Isn’t she a beauty?!? I run the command “sudo ifconfig -s” to verify my 10Gb interfaces are set to
use jumbo frames. At this point save your configuration. Your 10Gb switch build is complete! Now, lets see what this baby can do! Let’s start by checking our bandwidth to the
bridge itself. Open a few terminal windows to the new switch
on Workstation #1. The top left window is running a session of
iPerf server and binding to the switch IP. The lower left window is running the “top”
command, so we can monitor CPU utilization, context switching, and the iPerf process. The right window is the jPerf client. Workstation #1 is connected directly to the
10Gb bridge via Direct Attach Copper. So at this point we are just going from Workstation
#1 to the switch. For this test we are getting around 9.5Gbs,
which is excellent! Everything looks great! For the second bandwidth test example, Workstation
#1 is connected to the VyOS switch, and Workstation #2 is connected to the VyOS switch. The top left window is running a utility (on
switch) called “bmon” (bandwidth monitor) to visualize our receive and transmit traffic,
and top is running in the bottom left window on Workstation #2. Workstation #2 is using a slower dual core
processor to demonstrate how it changes the bandwidth picture. Notice iPerf is pegged at 100% CPU with one
stream of traffic from Workstation #1. So lets try pushing two streams of traffic
and see what happens. You can see two streams of traffic
allow us to hit our 10Gb mark. So the VyOS switch is not our problem, it’s
the architecture of Workstation #2 that can’t handle a single stream of 10Gb traffic. Another good test! For the third test, consider what happens
when heavy traffic has to traverse your system bus. In other words, inbound traffic on card 1
is forwarded to card3 outbound. In my testing I discovered great speeds when
I forwarded traffic across the same card (inbound card 1 to outbound card 1). This made sense as the traffic was local to
the Network Interface Card. But when I passed traffic across cards, my
results were less than desirable and a bit shaky to boot. Digging in a bit further, I realized it was
important to adjust BIOS settings to ensure my PCIe bus was using maximum possible payload
values. This made all the difference in the world. For this test I was using a much more capable
Workstation #2 so I could avoid any processor bottlenecks. Upon making the PCIe adjustment in my BIOS
(shown in the BIOS Settings Segment) things were screaming fast once again. You can see the chart wavers slightly, but
in my testing without recording video, it was a very consistent 10Gb per second. Here is what the CPU looks like on Workstation
#1. Bear in mind, it’s consuming proc cycles while
I record video too. And here is the 10Gb ethernet adapter in Workstation
#1. Perfect! These were the results I was looking for. As promised I threw in some power goodies
for being so patient while I put the final touches on the last video in the 3-part series
of 10Gb Home Network. Since these last 3 short sections are self-explanatory,
kick back, and enjoy the music. Here are some amazingly powerful commands
I demonstrate. And make sure you catch the summary, following
the utility demo, as there are some really important tips I include. There are some other pretty amazing things
you can do with this build. We built a 10Gb switch, but you could build
a 10Gb router, a 10Gb firewall, a 10Gb media converter, or a 10Gb VPN. Keep in mind it will only be as good as the
hardware you use. Make sure your network adapters receive good
airflow from your case fans as they tend to get hot. Especially if you are stacking multiple cards
next to each other. On the topic of why I selected VyOS over pfSense,
or some other distro, I selected VyOS because it’s a purpose-built tool with a focus on
switching, routing, NAT, and firewall. On the other hand pfSense is focused on boundary
protection acting as a firewall. You can bridge interfaces, but packets are
still inspected and require a firewall rule, unless you completely disable the firewall
functionality on your pfSense system. Even at that point, performance was still
erratic during my testing. I encourage you to test for yourself if pfSense
is your preferred platform. My intent is not to start a flame war or distro
showdown. My point is, that you always want to use the
right tool for the right job, and testing is how you get there. Alright guys, I had an amazing time assembling this three
part series, and interacting with everyone throughout the process. The feedback I have been receiving is overwhelmingly
positive, and helped me structure this series in a way that benefits you even more than
I originally intended. I can’t thank you enough for all the thumbs-up
and shares. Keep the comments flowing, and subscribe if
you enjoyed this series. This is iTechStorm wishing you happy bandwidth
trails.

100 Comments

Add a Comment

Your email address will not be published. Required fields are marked *