[ale] new data - buffer testing (was: to speed up your internet connection, slow it down (buffer bloat))

Erik Mathis erik at mathists.com
Sun Jul 8 22:34:53 EDT 2012


There is a direct correlation to upload and download speeds when using
TCP. For every payload that is received, another packet is sent to
confirm the receipt of the sent payload. TCP reconfigures itself in
favor of reliability over performance on the fly based in part of that
receipt packet. So if the reply is lagged by a saturated link then TCP
congestion control kicks in.
http://en.wikipedia.org/wiki/TCP_congestion_avoidance_algorithm


Also I was thinking that my side of the test may have flawed the whole
experiment. I was thinking today that since my side dosn't have QoS
controls in place it may have poisoned the test.

Also you may want to look for for interface errors and allot of tcp
retransmits on obth sides of the test. IPTarf should show you this
info.



On Sun, Jul 8, 2012 at 11:17 AM, Ron Frazier (ALE)
<atllinuxenthinfo at techstarship.com> wrote:
> Hi all,
>
> I've been in communication with Erik. He set up a knoppix box at his
> location and allowed me to SSH into it and use it for testing. I want to
> thank Erik for his suggestions and help. (PS Erik, do some speed tests on
> your line when wired to the router, if you're not already. If you're using
> wifi, and it's set to G mode, your line may be capable of more speed that
> you think. My speed increased a bit over the last couple of years and I'm on
> the same plan. If you're on wifi, and it's in G mode, your max data transfer
> rate will be about 24 mbps. When I plugged in with a wire, I got up to 32
> Mbps. If you have N compatible adapters, you may want to set the router's
> transmitter to 145 Mbps or 300 Mbps. Mine is still at 54 Mbps since I think
> it may destabilizing the G devices.)
>
> Erik suggested that I use the iperf utility to do some testing by dumping
> data from my PC to his. I did this while simultaneously downloading data to
> mine. I'm not totally sure this is a buffer bloat problem, since the
> Netalyzr test I mentioned earlier in the thread estimates my outbound
> buffering to be 320 ms. However, I did get some interesting results which I
> don't understand. I started out on wifi, but later connected to my router
> with a wire to eliminate that variable.
>
> First, let me explain the test. The iperf utility is in the Ubuntu
> repositories, and probably built in or in the repositories of other distros.
> It can run as a client or a server. If you run it as a server, it sets
> itself up as a listener and prepares to be a data sink. If you set it up as
> a client, it prepares to be a data source and send data to the server. Here
> are some links.
>
> http://iperf.sourceforge.net/
> http://openmaniak.com/iperf.php
>
> There is also a MAN page for it.
>
> So, first I opened a terminal and SSH into Erik's box as follows. All
> addresses and user names here are fake, but you get the idea.
>
> SSH joeuser at eriksfakedomain.com
> I entered the password I was given.
> I got a command prompt.
>
> Then, I entered this command on his system to start the iperf server.
>
> iperf -s
>
> Now, it's waiting to sink data from an iperf client somewhere.
>
> I opened another terminal window on my machine and started the iperf client.
>
> In my case, I wanted to send enough data to let the link stabilize and get a
> good picture of the bandwidth. I also wanted the test to run long enough to
> be able to do other things. The following command, on my own local terminal,
> sends almost 40 MB of data to the remote server. (PS When I see MB or KB in
> Linux, or for that matter, Windows, I never know whether it's doing the
> divide by 1024 or divide by 1000 thing.)
>
> iperf -c eriksfakedomain.com -n 40000000
>
> On my line, this takes about 1 minute. When it's done, the client will end
> and report the results. Note that, every time this is done, this uses up
> some of the bandwidth limit for each of us with Comcast.
>
> I did this several times and got a fairly consistent upload speed of 5.6
> Mbps.
>
> Now, I decided to add a separate download test.
>
> I started Firefox and went to the Ga Tech Ubuntu mirror. I started
> downloading an Ubuntu ISO. According to the download window, it was
> downloading 3.2 MB / sec, which I figure is approximately 32 Mb (small b) /
> sec. I did that a few times and got pretty consistent results.
>
> Then I decided to combine the two tests.
>
> I started the ISO download again and let it get up to an estimated 32 Mbps.
> Then, I started the iperf client and increased the data count to 80 MB to
> make it last longer. Immediately, within a few seconds, after I started the
> upload, the download data rate dropped to about 12 Mbps and stayed there for
> a while. After about half a minute, it slowly inched back up to about 28
> Mbps. I tried this a few times. Most of the time, similar things happened.
> Once, the download data rate continued to decline down to below 2 Mbps and
> stayed there and was very erratic. On that occasion, even after I terminated
> the iperf client upload, the download never recovered past 5 Mbps.
>
> I also tried the same experiment a couple of times with my router set to
> limit upstream bandwidth to 90% of the normal maximum. It didn't appear to
> change things at all. I'm not sure whether throttling was on or off when the
> download became unstable.
>
> Well, that's it. I've discovered that an extreme upload can severely hurt
> and sometimes totally disrupt a download. I don't know what's causing it,
> and I don't know how to prevent it. If anyone can explain this or suggest
> further tests, please do. I don't know how much more time I and Erik can
> spend testing, but we'll see.
>
> Sincerely,
>
> Ron
>
>
>
> --
>
> Sent from my Android Acer A500 tablet with bluetooth keyboard and K-9 Mail.
> Please excuse my potential brevity.
>
> (To whom it may concern. My email address has changed. Replying to former
> messages prior to 03/31/12 with my personal address will go to the wrong
> address. Please send all personal correspondence to the new address.)
>
> (PS - If you email me and don't get a quick response, you might want to
> call on the phone. I get about 300 emails per day from alternate energy
> mailing lists and such. I don't always see new email messages very quickly.)
>
> Ron Frazier
> 770-205-9422 (O) Leave a message.
> linuxdude AT techstarship.com
>
>
> Erik Mathis <erik at mathists.com> wrote:
>>
>> Hey Ron,
>>
>> If you like, I can lend my 20x3M comcast line to you for testing. I can
>> throw up an quick knoppix box and you can test away with your own shell. Its
>> not really being used (except for me checking emails).  Its a Walking Dead
>> marathon today, so I doubt I will get any real work done.
>>
>> Hit me off list for IP and login details.
>>
>> -Erik-
>>
>>
>> On Sat, Jul 7, 2012 at 2:56 PM, Ron Frazier (ALE)
>> <atllinuxenthinfo at techstarship.com> wrote:
>>>
>>> Hi Erik,
>>>
>>> I haven't had a chance to do any testing. I'm not sure if I even can.
>>> Here machine runs Windows and is a locked down corporate computer that I
>>> have little access to. I can't install anything without their permission. I
>>> know some of her coworkers have been complaining, so I know that the problem
>>> is often on their side of the fence. I just wanted to eliminate any possible
>>> hitches on my end. My routers are basic off the shelf Netgear boxes. They
>>> have very little in the way of configuration settings. My machines are
>>> sometimes running Windows and sometimes Linux, but they don't have any
>>> visibility into her data traffic.
>>>
>>> She runs a citrix remote desktop system. So, she is viewing the video
>>> output of a remote system at all times. Presumably, this generates a fair
>>> amount of continuous time sensitive data traffic. Every mouse click she
>>> makes, every move she makes in the program she's running is sent to the
>>> remote system and the results are sent back. If the connection is disrupted
>>> for more than a second or so, her system complains about it. At this point,
>>> I cannot tell if my changes have made any substantial difference. They were
>>> more of a preemptive change, to sort of head off any local problems before
>>> they happen. If I come up with any notable results, or any viable way to
>>> test things, I'll certainly be glad to share.
>>>
>>> Here's something you can do to test the latencies on your own network.
>>>
>>> There's a tool called Netalyzr that's put out by the International
>>> Computer Science Institute at Berkeley. You will have to trust Berkeley and
>>> turn on javascript for this to work. This performs VERY extensive testing on
>>> your network and reports some findings to their study.
>>>
>>> http://netalyzr.icsi.berkeley.edu/index.html
>>>
>>> Here's something else which you could try. Turn off all QOS or traffic
>>> shaping systems on your router closest to the internet, and on any other
>>> routers or switches between your PC and the internet. Try to upload a really
>>> big file, like multiple GB to dropbox or something. While that's uploading,
>>> try to download another large file at the same time, say an ISO or
>>> something. The upload can disrupt the download because the ACK packets going
>>> back upstream may have a hard time getting through. You could also try a
>>> VOIP call or some gaming, etc. while the upload is in progress. If the
>>> testing with large downloads, VOIP, or gaming fails, it's probably because
>>> you have some large buffers somewhere and because they're getting full. If
>>> those tests don't fail, you may not have large buffers, or there may be
>>> traffic shaping going on that you don't know about, maybe in the cable
>>> modem. Again, I don't know about all the nitty gritty details of TCP/IP
>>> throttling, so I'm not an expert on the topic.
>>>
>>> Good luck. Let us know if you find out anything new for your network.
>>>
>>> Sincerely,
>>>
>>> Ron
>>>
>>>
>>>
>>> --
>>>
>>> Sent from my Android Acer A500 tablet with bluetooth keyboard and K-9
>>> Mail.
>>> Please excuse my potential brevity.
>>>
>>> (To whom it may concern. My email address has changed. Replying to former
>>> messages prior to 03/31/12 with my personal address will go to the wrong
>>> address. Please send all personal correspondence to the new address.)
>>>
>>> (PS - If you email me and don't get a quick response, you might want to
>>> call on the phone. I get about 300 emails per day from alternate energy
>>> mailing lists and such. I don't always see new email messages very
>>> quickly.)
>>>
>>> Ron Frazier
>>> 770-205-9422 (O) Leave a message.
>>> linuxdude AT techstarship.com
>>>
>>>
>>> Erik Mathis <erik at mathists.com> wrote:
>>>>
>>>> I was wondering if you had done any pre/post iperf (or something smiler)
>>>> testing? If so, can you share the results?
>>>>
>>>> -Erik-
>>>>
>>>> On Fri, Jul 6, 2012 at 5:13 PM, Ron Frazier (ALE)
>>>> <atllinuxenthinfo at techstarship.com> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I want to share some information about a phenomenon which can
>>>>> dramatically slow down your internet connection, or connections within a LAN
>>>>> some times. It's called buffer bloat. I first heard about it over on the NTP
>>>>> questions list. I don't remember why that came up, probably related to
>>>>> network latencies for NTP servers. Then, later, Steve Gibson discussed it on
>>>>> the Security Now podcast. I've provided several links below for those who
>>>>> wish to research it.
>>>>>
>>>>> For those not familiar, buffer refers to a memory queue in a router or
>>>>> other networking gear. The problem occurs when you go from a large bandwidth
>>>>> pipe to a smaller bandwidth pipe, such as the transition from your LAN to
>>>>> the internet WAN. At this point, you might go from 100 Mbps or 1 Gbps
>>>>> bandwidth to something like 3 Mbps or 20 Mbps or 50 Mbps or whatever. The
>>>>> point is, that it is a dramatic reduction in bandwidth.
>>>>>
>>>>> So, if you're trying to transmit to the internet at 100 Mbps and it can
>>>>> only take 20 Mbps, the link will become saturated. Without buffers or
>>>>> queues, about 4/5 of the packets will be dropped. The system will rapidly
>>>>> recognize that it cannot go that fast and it will scale down to something
>>>>> which the link can support.
>>>>>
>>>>> However, with large buffers, which many routers have, the problem
>>>>> becomes much worse. Let's say the router has a 4 M Byte or approximately 40
>>>>> M bit ram buffer on it's outbound transmission channel. Your computer fills
>>>>> that buffer in about 4/10 sec, but, that buffer is going to take 2 sec to
>>>>> empty out sending the data to the internet. While I don't understand all the
>>>>> technical magic that happens, I do understand that the normal automatic
>>>>> throttling systems no longer work. So, your computer might be seeing a 2 sec
>>>>> delay to get packets out on the internet while they meander through the
>>>>> buffer on a first in first out basis.
>>>>>
>>>>> There is a new intelligent packet dropping algorithm called CODEL that
>>>>> may be the solution. The bufferbloat site mentions it, and Steve did a
>>>>> podcast talking about it. It shows great promise, however, most routers
>>>>> don't implement the algorithm, and many probably never will get upgraded,
>>>>> including many home routers.
>>>>>
>>>>> So, here, as I understand it, is a way you can work around the problem.
>>>>>
>>>>> My wife works from home sometimes and uses a VPN back to work.
>>>>> Sometimes, here system locks up and says the connection is lost. I have as
>>>>> many as 7 devices sharing the same internet connection, so her system may be
>>>>> experiencing congestion. I suspect that many times, the problem is on the
>>>>> other end at her office, but just in case, I decided to tweak the router. I
>>>>> turned on a QOS (quality of service) setting and told it to prioritize her
>>>>> data traffic over mine. I also made some changes to avoid any possible
>>>>> buffer bloat problem.
>>>>>
>>>>> The buffer bloat problem only shows up when the buffer fills. By the
>>>>> way, a clogged upstream buffer can shut down downloads too, since, during
>>>>> downloads, all tcp packets have to be acknowledged, and those
>>>>> acknowledgements must go upstream. A clogged buffer can essentially make
>>>>> your Internet connection almost unusable. I think this is what happens at
>>>>> many coffee shops. If you can't run CODEL or something like it, one way to
>>>>> prevent the problem is to make sure the buffer never fills up. One way to do
>>>>> that is to limit your upstream bandwidth to something less than what it's
>>>>> possible to do. In my case, the QOS menu of the router allows me to limit
>>>>> upstream bandwidth. I used speedtest.net to test the system. I was able to
>>>>> get a peak upstream bandwidth of 5.6 Mbps. So, I set the QOS controls on the
>>>>> router to limit the upstream bandwidth to 5 Mbps. Theoretically, this should
>>>>> mean that the outbound buffer on my router never will fill up because it's
>>>>> always emptying out faster than I'm putting data in. Theoretically, that
>>>>> should prevent the buffer bloat problem on my LAN. This, combined with
>>>>> prioritization of my wife's data, will hopefully solve her data problems.
>>>>>
>>>>> If you've had experience with this problem, please share what you
>>>>> learned and what you did about it.
>>>>>
>>>>> If you need info on the down and dirty operation of TCP/IP, ask some of
>>>>> the other wizards on the list.
>>>>>
>>>>> Hope this is helpful.
>>>>>
>>>>> Sincerely,
>>>>>
>>>>> Ron
>>>>>
>>>>> links below
>>>>>
>>>>> ----------------------
>>>>>
>>>>>
>>>>> http://en.wikipedia.org/wiki/Buffer_bloat
>>>>>
>>>>> http://www.bufferbloat.net/
>>>>>
>>>>> Steve Gibson discusses buffer bloat on the Security Now podcast episode
>>>>> 345.
>>>>> He introduces a potential solution, CODEL, developed by industry
>>>>> researchers, in episode 359.
>>>>>
>>>>> http://www.grc.com/securitynow.htm - Reference episodes 345 and 359.
>>>>>
>>>>> http://twit.tv/show/security-now/345
>>>>> http://media.grc.com/sn/sn-345.mp3
>>>>>
>>>>> http://twit.tv/show/security-now/359
>>>>> http://media.grc.com/sn/sn-359.mp3
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Sent from my Android Acer A500 tablet with bluetooth keyboard and K-9
>>>>> Mail.
>>>>> Please excuse my potential brevity.
>>>>>
>>>>> (To whom it may concern. My email address has changed. Replying to
>>>>> former
>>>>> messages prior to 03/31/12 with my personal address will go to the
>>>>> wrong
>>>>> address. Please send all personal correspondence to the new address.)
>>>>>
>>>>> (PS - If you email me and don't get a quick response, you might want to
>>>>> call on the phone. I get about 300 emails per day from alternate energy
>>>>> mailing lists and such. I don't always see new email messages very
>>>>> quickly.)
>>>>>
>>>>> Ron Frazier
>>>>> 770-205-9422 (O) Leave a message.
>>>>> linuxdude AT techstarship.com
>>>>>
>>>>> _______________________________________________
>>>>> Ale mailing list
>>>>> Ale at ale.org
>>>>> http://mail.ale.org/mailman/listinfo/ale
>>>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>>>> http://mail.ale.org/mailman/listinfo
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Ale mailing list
>>> Ale at ale.org
>>> http://mail.ale.org/mailman/listinfo/ale
>>> See JOBS, ANNOUNCE and SCHOOLS lists at
>>> http://mail.ale.org/mailman/listinfo
>>>
>>
>
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>


More information about the Ale mailing list