Simulating Poor Network Performance

Where is My Data?

Application developers have to respect the boundaries of the systems their code runs in. Whether you are writing software to display a clock or analyze data, your code is restricted by the machines on which it is installed. Collaboration software has to take into account how much bandwidth exists between endpoints, and these types of full stack requirements are ever more important. Mobile developers have an interesting problem in that they cannot predict network performance between their code and their data. Simulating poor network performance allows developers to better understand how their applications react in different scenarios.

It is important for mobile developers to include testing to simulate a wide range of network conditions so code can be optimized. The biggest hurdle to making this possible is finding a tool and making it easy to use. When I was first asked to look into this, I knew we couldn’t deploy 1x, 3g and LTE cellular networks in-house. While possible, the solution would be extremely difficult to scale and would quickly become cost prohibitive. Besides that, your test network must be able to simulate broken and degraded states. It isn’t all about bandwidth, but also involves introducing latency, jitter, out of order packets, and loss. It would be much easier to provide a test wireless network, but we needed a way to allow developers and testers to make use of it easily.

ATC is the Key (Thanks Facebook!)

During my research I found ATC. Augmented Traffic Control is a relatively simple front-end to existing Linux tools. It allows developers to tune their network experience on the fly, putting the keys in the hands of developers and testers in real time.

ATC consists of a Linux server configured as a gateway router. It uses existing tools like tc and iptables, wrapping the controls for them in a django based web front-end. A wireless access point in bridging mode passes user traffic to the gateway. The gateway then imposes whatever performance criteria is set for that device and routes traffic out the other side accordingly. Each device is controlled individually, allowing multiple developers to simulate conditions in different ways at the same time. One developer tests at 2g simulated speeds while another tests as if their bandwidth is provided by a cable modem.

These types of testing capabilities go a long way towards ensuring customers get a consistent response from your application no matter where they are in the world.

My Notes on the Project

I took a few notes along the way in setting up my ATC environment and wanted to share them. I decided to build a bare metal server because we had several available. Ensuring testers wouldn’t be contending for resources while testing was important. A couple of higher rated consumer grade access points were purchased and connected via a switch to the gateway. I simply turned off all DHCP features and connected the access point to the ATC system using one of their switch ports rather than using their up-link ports. This effectively put them in “bridging” mode. This tends to work on consumer grade systems because they rarely restrict traffic between connected devices on the network.

I chose to start out my installation on Ubuntu server 14.04. I added python-dev and the base python-django package. The instructions accompanying the ATC distribution were sufficient to get things up and running with only the following general notes:

  1. the stylesheet doesn’t render in the demo ui. I edited the base.html template page and statically called the stylesheets and other objects.
  2. The configuration needs to call a couple extra includes, including “include” and “patterns”.
  3. Performance of TCP flows is impaired if you do not include the switch --atcd-dont-drop-packets. My command line currently is atcd --atcd-lan eth1 --atcd-wan eth0 --atcd-dont-drop-packets --logfile /tmp/atcd.log --daemon
  4. I renamed my NIC cards eth0 and eth1 in Linux rather than keeping them their defaults. This was mainly so my installation would follow ATC documentation more closely.

Developers Love It!

When I first turned this test network up it was a skunkworks project. No one was certain whether our developers were going to find it useful, so I spent a good bit of time monitoring it. I also took laps through development groups, asking them when I saw they were not busy and was pleasantly surprised. They had been struggling with exactly this problem and had been able to create new an interesting solutions because they had a way to test. It is truly exciting to contribute to the success of our products in such a direct way, so I highly recommend taking a look at offering the same thing to your teams.

Leave a Reply