神刀安全网

Windows 10 updates via UDP bypassing QoS restrictions

Post reply

User #102078   2383 posts

ChopsyWA You want QoS? I got QoS.

Whirlpool Forums Addict

http://whrl.pl/ReCCoM

herring | anchor

Reply to this post

posted 2016-May-15, 11:33 am

reply short code

posted 2016-May-15, 11:33 am

O.P.

Hi All

This is a very long post. I am looking for some peer input and feedback from others who may have seen what follows and found a workaround.

I have been doing QoS for many years. Inbound QoS has worked for me and many clients due to the fact that the bulk of inbound traffic is TCP and I could always successfully shape incoming TCP due to the design of the protocol. TCP uses a variable Window size and backs off from sending when it does not receive ACKs from the receiving end. By dropping packets on inbound traffic, the sending speed can be controlled quite precisely and very quickly to keep priority traffic like VOIP (usually UDP), and higher priority TCP traffic flowing smoothly. I have always known that the day will come when high volume UDP traffic will become more common and I have been hoping (really, really hoping) that ISPs would have the sense, or business savvy to start selling higher priority data with corresponding traffic marking so I could continue to build and maintain quality converged networks.

Anyway, I have come across a phenomenon over the past few month that is very worrying and it seems to come from come a common source. The first time a customer called in, it was a 20Mb fibre site which was almost unusable on the inbound. I jumped onto the router and saw that the inbound link was saturated. Looking at the queues, everything was perfectly normal and within range. It was like a DOS attack. However, when I got down to checking the interface traffic in depth, I noticed a couple of internal hosts accessing a couple of IPs 203.32.125.151 and 203.32.125.152, making many connections. As soon as I blocked these addresses the data dropped away.

A reverse DNS lookup shows that the addresses are akamai addresses and looking up the source of the addresses shows them to be Zettanet owned addresses. So I am assuming that the issue comes from an akamai distribution point on the Zettanet network.

A workaround turned out to be to put a new connection rate limiter on the outbound TCP connections. I used a rate limiter that will automatically allow x number of new connections per second to any single address.

This issue popped up on some other sites and we quickly realised it was Windows 10 updates. The rate limiter seemed to work, but this really had me worried. We can’t control many of the networks we have setup at the client level as they are shared for public use in some places, or tenanted networks in others.

The problem escalated recently when one of my techs was staging a new server in our workshop. All of a sudden our VOIP fell in a heap and the Internet was almost unusable. This was on a connection that has been hammered relentlessly for over 4 years without so much as a whimper. It was the same range of source addresses and this was with Windows server and then Office updates. What seems to be happening is that instead of the sending server reducing its window size when packets are dropped, it just keeps re-sending large windows, which are obviously being dropped at my end. The queue algorithm has no idea of this and it will be letting packets through at a rate it thinks is correct, so the flow continues even though much of the traffic is dropped. However as the traffic keeps coming, the link is totally saturated.

Our issue was only a week ago and since then our first problem site had a reoccurence of the problem and now even rate limiting did not work. I had to completely block the IPs to stop the problem. This is not a long term fix of course, so I have to find another. I am trying to contact Zetta’s network gurus to shed more light on it, but I am concerned that this could be some form of RFC breach by akamai.

I thought I would put this out there to see if anyone else who does QoS on Internet connections has come across this and whether there is a novel solution, or at least a router based workaround that will not stop client updates.

Please before anybody suggests it, putting in a second business grade connection at a business grade price is not an option. I am looking for technical input on this actual issue.

Cheers.

User #725513   325 posts

Slaziar

Participant

http://whrl.pl/ReCCyd

herring | anchor

Reply to this post

posted 2016-May-15, 12:27 pm

reply short code

posted 2016-May-15, 12:27 pm

Does your ISP allow you to perform or submit requests for QoS and Rate Limiting on their gateway?

User #102078   2383 posts

ChopsyWA You want QoS? I got QoS.

Whirlpool Forums Addict

http://whrl.pl/ReCCyM

herring | anchor

Reply to this post

posted 2016-May-15, 12:34 pm

reply short code

posted 2016-May-15, 12:34 pm

O.P.

Slaziar writes…

Does your ISP allow you to perform or submit requests for QoS and Rate Limiting on their gateway?

Nope. Show me a tier 1 or 2 who does. That would be the holy grail.

User #185591   1985 posts

EasyBB Once upon …..called PC

Whirlpool Enthusiast

http://whrl.pl/ReCCHo

herring | anchor

Reply to this post

posted 2016-May-15, 1:48 pm

reply short code

posted 2016-May-15, 1:48 pm

Hi Chops,It might be a good idea to have a local update server for reasonably big workplaces:

https://mizitechinfo.wordpress.com/2013/08/19/step-by-step-installing-configuring-wsus-in-server-2012-r2/

https://support.microsoft.com/en-us/kb/3095113

This way you may be able to schedule updates to download at night and distribute to LAN clients without affecting WAN bandwidth.

For smaller places: http://www.pcworld.com/article/2955491/windows/how-to-stop-windows-10-from-using-your-pcs-bandwidth-to-update-strangers-systems.html

User #576198   150 posts

d-m-z

Participant

http://whrl.pl/ReCCKB

herring | anchor

Reply to this post

posted

2016-May-15, 2:15 pm

edited 2016-May-15, 2:22 pm

reply short code

posted 2016-May-15, 2:15 pm (edited 2016-May-15, 2:22 pm)

this post was edited

Akamai has invested in FastTCP. That might be part of the issue.

Also, CDNs tweak their TCP parameters such as the initial congestion window, and how quickly they back off when detecting loss. Could be they’re experimenting.

User #102078   2383 posts

ChopsyWA You want QoS? I got QoS.

Whirlpool Forums Addict

http://whrl.pl/ReCDBk

herring | anchor

Reply to this post

posted 2016-May-15, 8:12 pm

reply short code

posted 2016-May-15, 8:12 pm

O.P.

d-m-z writes…

Could be they’re experimenting.

I sure hope so and they fix it soon. They are breaking the net badly.

User #102078   2383 posts

ChopsyWA You want QoS? I got QoS.

Whirlpool Forums Addict

http://whrl.pl/ReCDBL

herring | anchor

Reply to this post

posted 2016-May-15, 8:14 pm

reply short code

posted 2016-May-15, 8:14 pm

O.P.

EasyBB writes…

It might be a good idea to have a local update server for reasonably big workplaces

This is all true, but I can’t tell tenants that. Hosts should behave. This is tantamount to a dos. The result is far worse than normal congestion.

User #81627   4560 posts

Cap’n Silver Of the seven sea’s

Whirlpool Forums Addict

http://whrl.pl/ReCDLj

herring | anchor

Reply to this post

posted 2016-May-15, 9:34 pm

reply short code

posted 2016-May-15, 9:34 pm

I haven’t seen this behaviour creep up for any if my clients yet, but ill keep my eyes peeled.

User #430537   828 posts

Lord Whirly Gig

Whirlpool Enthusiast

http://whrl.pl/ReCEnn

herring | anchor

Reply to this post

posted 2016-May-16, 7:38 am

reply short code

posted 2016-May-16, 7:38 am

You also might explore whether Akamai and the path to you supports ECN (RFC3168). Or maybe the two end-points are using ECN but the path is mangling the bits?

User #23512   8734 posts

Nik G

Whirlpool Forums Addict

http://whrl.pl/ReCEoK

herring | anchor

Reply to this post

posted

2016-May-16, 8:00 am

edited 2016-May-16, 8:06 am

reply short code

posted 2016-May-16, 8:00 am (edited 2016-May-16, 8:06 am)

this post was edited

Stock standard behaviour for those implementing TCP Vegas / FastTCP where it uses delay instead of loss for congestion control.

People trying to shape Internet connectionsin the manner you are will experience more and more problems moving forward with this as the TCP stack gets further optimised. Post the packet capture online as I would be really keen to take a look at how they’re running it now (you did get a packet capture right??).

ISP’s will never sell (at least I hope not) differentiated class services over the Internet as that breaks Net Neutrality. If you need guaranteed bandwidth to a specific provider (e.g. VoIP) and its mission critical, pay for private links e.g. MPLS VPN.

User #132901   3242 posts

jb206 debug all

Whirlpool Forums Addict

http://whrl.pl/ReCEpT

herring | anchor

Reply to this post

posted 2016-May-16, 8:15 am

reply short code

posted 2016-May-16, 8:15 am

Interesting you run qos on Internet circuits. .. I’ve never tried as its really a best effort thing that can’t be garanteed but good on you for trying.

As nik said best to avoid it if you can afford it.

Another option not mentioned is per user rate limiting… just depends what products you have available to implement it on.

User #102078   2383 posts

ChopsyWA You want QoS? I got QoS.

Whirlpool Forums Addict

http://whrl.pl/ReCEMt

herring | anchor

Reply to this post

posted 2016-May-16, 10:47 am

reply short code

posted 2016-May-16, 10:47 am

O.P.

Nik G writes…

f you need guaranteed bandwidth to a specific provider (e.g. VoIP) and its mission critical, pay for private links e.g. MPLS VPN.

This is a ridiculous notion.

1. People need to be able to provide prioritisation over the Internet. It could easily be done with net neutrality, if every router in the Internet passed on and upheld the TOS/DHCP bits. That is never going to happen.

2. Net neutrality is not compromised by an ISP offering differentiated services based on customer specification. ISPs are private entities offering an end to end service. Them to the customer. It does not affect the Internet’s neutrality.

3. How impractical to suggest that a company should be getting (if it even existed) an MPLS service with every provider. Think about what you just proposed. You just took us back 50 years to the circuit switched telephone system, only using data. Providers won’t offer this, it would make data way too expensive and it would monopolise services.

User #102078   2383 posts

ChopsyWA You want QoS? I got QoS.

Whirlpool Forums Addict

http://whrl.pl/ReCEMO

herring | anchor

Reply to this post

posted 2016-May-16, 10:49 am

reply short code

posted 2016-May-16, 10:49 am

O.P.

jb206 writes…

Interesting you run qos on Internet circuits. .. I’ve never tried as its really a best effort thing that can’t be garanteed but good on you for trying.

Not just trying JB, very successfully succeeding. I am not attempting end to end QoS. I am just getting across the restriction that is the customer connection. The end to end has always been successful to date due to there being ample bandwidth in the backbone across the country. This may change in time, but FastTCP (of which I will now have to research) is not going to help with congestion if it just keeps blasting away when the connection downstream is not able to keep up.

User #430537   828 posts

Lord Whirly Gig

Whirlpool Enthusiast

http://whrl.pl/ReCENB

herring | anchor

Reply to this post

posted 2016-May-16, 10:54 am

reply short code

posted 2016-May-16, 10:54 am

ChopsyWA writes…

if it just keeps blasting away when the connection downstream is not able to keep up.

To be fair this has always been the goal of TCP. Sortof like how you tighten up a nut and bolt. Tighten it until it strips, then back it off one.

TCP blasts away until it fails then it backs off a notch. You really need to convince people that TCP should use a different service model if you hope to change that thinking.

If anything, TCP and TCP-like thinkers are trying to make the blasts bigger – albeit more responsive – as average pipe sizes get larger. Eg, Chrome and Google do a lot of QUIC which your QOS strategies are going to completely miss.

User #147567   1864 posts

tricky 

Whirlpool Enthusiast

http://whrl.pl/ReCEVy

herring | anchor

Reply to this post

posted

2016-May-16, 11:29 am

edited minutes later

reply short code

posted 2016-May-16, 11:29 am (edited)

d-m-z writes…

Akamai has invested in FastTCP. That might be part of the issue.

Wonder how widely deployed FastTCP is across akamai’s servers.

@OP.

I personally rate limit http/bulk traffic here (debian/linux) and haven’t noticed anything from windows updates doing anything weird, that been said.. I cache windows updates using nginx, whether that has anything todo with it or not is another question.

User #23512   8734 posts

Nik G

Whirlpool Forums Addict

http://whrl.pl/ReCEXR

herring | anchor

Reply to this post

posted

2016-May-16, 11:42 am

edited 2016-May-16, 11:47 am

reply short code

posted 2016-May-16, 11:42 am (edited 2016-May-16, 11:47 am)

this post was edited

ChopsyWA writes…

1. People need to be able to provide prioritisation over the Internet. It could easily be done with net neutrality, if every router in the Internet passed on and upheld the TOS/DHCP bits. That is never going to happen.

People need to be able to provide prioritisation over a medium that offers no guaranteed metrics regarding performance, packet loss, latency, and is operated as "best effort"? This is a ridiculous notion… but I will humour it for a while.

What exactly would carrying the ToS/DSCP bit end-to-end for you actually achieve? How would you receiving a packet with this value set make any difference or provide additional entropy in this scenario beyond the standard five tuples you already have access too? Sure, it’d make you’re configuration cleaner but technically it’s not going to fix anything for you without provider involvement.

We will also ignore the fact it is technically impossible given how carrier/major ISP networks are built and operated, from a logical configuration level to a hardware pipeline forwarding level.

2. Net neutrality is not compromised by an ISP offering differentiated services based on customer specification

Net neutrality (also network neutrality, Internet neutrality, or net equality) is the principle that Internet service providers and governments should treat all data on the Internet the same, not discriminating or charging differentially by user, content, site, platform, application, type of attached equipment, or mode of communication.

What you are proposing goes directly against net neutrality. The Internet is an undifferentiated medium, not my problem you’re trying to use it for (or wishing it was) something it has never and will never be.

3. How impractical to suggest that a company should be getting (if it even existed) an MPLS service with every provider. Think about what you just proposed. You just took us back 50 years to the circuit switched telephone system, only using data. Providers won’t offer this, it would make data way too expensive and it would monopolise services.

If you have applications that require differential service policies and specific metrics, you should not be operating it across a medium that is best effort and provides none of the above. While it may work 95% of the time, you can’t get angry at the system for the other 5% when the system wasn’t designed to do what you’re wanting it to do, there are other specific products for that. For example, if a business customer of mine was heavily dependent on the telephone and their business couldn’t operate without it, they would have a L2/L3VPN circuit with appropriate SLA’s to said telephony provider, I wouldn’t be running it over the Internet. Alternatively, if the business doesn’t need their phones to function and can fall back to their mobiles, I would happily run a cheap and cheerful VoIP service over the Internet for them.

Anyway, our personal opinions on that topic aside, still happy to assist if you post up the packet capture.

User #102078   2383 posts

ChopsyWA You want QoS? I got QoS.

Whirlpool Forums Addict

http://whrl.pl/ReCFbw

herring | anchor

Reply to this post

posted 2016-May-16, 1:01 pm

reply short code

posted 2016-May-16, 1:01 pm

O.P.

Nik the purpose of this post is not to get into a pissing competition about who knows most about a subject. I am not going to get caught up in a blow by blow discussion. As interesting as that is, it is for another thread.

Managing the last mile connection from an ISP to a client is all I am trying to do. I have not mentioned end to end QoS in my post. I am not trying to change the Internet. For years I have managed to make a living and help people achieve great productivity with their Internet connections, ranging from those lucky enough to get a good ADSL service, to those willing to invest in fibre services. There is no reason this cannot continue at least until the services available to us and our customers become too congested, which has not happened yet.

The problem I am experiencing now is due to a problem with one service provider . The whole tenet of TCP is that it adjusts end to end to ensure a guaranteed connection with minimum packet loss. This is not happening at present in this scenario.

User #102078   2383 posts

ChopsyWA You want QoS? I got QoS.

Whirlpool Forums Addict

http://whrl.pl/ReCFbT

herring | anchor

Reply to this post

posted 2016-May-16, 1:03 pm

reply short code

posted 2016-May-16, 1:03 pm

O.P.

Sir Whirly Gig writes…

To be fair this has always been the goal of TCP. Sortof like how you tighten up a nut and bolt. Tighten it until it strips, then back it off one.

I agree, but it should and always does find a balance. This has been proven over many years. We are talking one provider who is doing this and only over the past few months. This is not normal TCP behaviour and it is extremely inefficient use of bandwidth.

User #102078   2383 posts

ChopsyWA You want QoS? I got QoS.

Whirlpool Forums Addict

http://whrl.pl/ReCFb3

herring | anchor

Reply to this post

posted 2016-May-16, 1:04 pm

reply short code

posted 2016-May-16, 1:04 pm

O.P.

tricky writes…

I personally rate limit http/bulk traffic here (debian/linux) and haven’t noticed anything from windows updates doing anything weird, that been said

Same here. This has popped up several times over the past couple of months and always from the same addresses.

User #23512   8734 posts

Nik G

Whirlpool Forums Addict

http://whrl.pl/ReCFdR

herring | anchor

Reply to this post

posted 2016-May-16, 1:14 pm

reply short code

posted 2016-May-16, 1:14 pm

ChopsyWA writes…

Nik the purpose of this post is not to get into a pissing competition about who knows most about a subject

Sorry if that’s how my original post came across. My style is more straight to the point, e.g. you’re having problem A because service provider is doing X, you’re doing Y and the two aren’t necessarily going to play nice together. With some of the optimisations being rolled out and how you’re doing Y, you may see things get worse over time as more and more people start adopting this more "efficient" method of moving data. If that causes issues for other business critical services, you may need to look at alternates.

Anyway, I digress…

The whole tenet of TCP is that it adjusts end to end to ensure a guaranteed connection with minimum packet loss. This is not happening at present in this scenario.

I really want to help you as this stuff interests me, and I unfortunately don’t have access to a Windows environment right now to test on my own. So do you have a packet capture you can share? If not, can you replicate the issue easily and get one?

Your basing your assumptions on things "not happening" because you assume TCP congestion control should only kick in for dropped packets. In reality a lot of the new optimisations are using delay to "keep the pipes full" so it doesn’t take as long to recover from dropped packets. This is where it starts to (possibly) become incompatible with your configuration Y.

Archive Version Post reply

转载本站任何文章请注明:转载至神刀安全网,谢谢神刀安全网 » Windows 10 updates via UDP bypassing QoS restrictions

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址