Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO: QoS for the masses (Traffic shaping) updated 6/26/05
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
bigfunkymo
Apprentice
Apprentice


Joined: 23 Jan 2004
Posts: 237

PostPosted: Sat Jun 11, 2005 12:29 am    Post subject: HOWTO: QoS for the masses (Traffic shaping) updated 6/26/05 Reply with quote

I'm seeing a lot of traffic shaping problems being posted here and there and I know it's pretty confusing. Much of it still confuses me to this day. I decided to write this to help me learn more about traffic control myself. This is intended as supplimental reading for people who have already read the Linux Advanced Routing and Traffic Control HOWTO but want more examples and information.

I read a lot of people trying to divide up bandwidth like its a pie but that kind of paradigm probably won't give you any results you want. Let me put a better metaphor on you. Bandwidth is like the line at the bank and your data packets are the people in line and it is important that some of these people get done at the bank ASAP, while others don't mind to wait a few minutes longer. Most bank tellers (the metaphorical dequeing agents) are not going to make any distinction between the different people in the line and take them one at a time. If that bank wanted to, it could tell when there was a VIP in a hurry standing in line and they wouldn't make him wait in line, but instead bring him to the front.

That is what shaping will allow you to do, but this metaphor can only be applied to your upstream. You cannot shape your downstream at all, it can not be done. Say it with me: "there is no such thing as downstream shaping". I don't mean that theres sometimes some things you can do to shape downstream. It is simply not possible with the way things are so don't get it in your head that you can control what data is sent to you over your internet connection first because you will just drive yourself bonkers.

I'm not saying that it is impossible to manipulate the system in your favor a little, however. Consider the fact that most ISP's will queue your downstream packets to speed up the download on your end. Back to the bank line metaphor, that is like a bank having a really long roped off line in front of the teller desk. If people are just pouring into this bank lobby, the VIP's are going to be waiting all day and he's got to be out of there on time! Also, if the bank's lobby were full of people, then everyone who would come to the door would just leave and come back later when there was space. That is exactly what happens to a full network link, packets are just ignored and if those packets are TCP, then they are just retransmitted later. In properly functioning TCP/IP stacks, senders of tcp packets will adjust their rate of transmission when there is packet loss. If you're still with me on the bank metaphor then the solution here is pretty obvious, you need to make your line as short as possible and make these people come back later, then any VIP coming in won't have to wait very long. In a nutshell, this means making your router accept data at a significantly slower rate than it is actually capable of--this rate limiting is called "policing".

Assuming all the remote ends of your connections have proper TCP/IP software, they will reduce their rate of transmission and that will eliminate or significantly reduce the amount of queueing at your ISP and thus lead to less latency for your very important packets.

I've been using the wondershaper script for a long time now but I've been concerned with the possibility that the ingress policer will drop ICMP, UDP, or interactive TCP traffic unnecessarily. If you know the traffic is of high importance, why make the sender resend it? ICMP and UDP packets would not get retransmitted anyway, so why drop them? It never sat right with me.

So I have this script for you. It uses HTB for upstream shaping. The TXRATE variable is set to 340 kbits/second, only 44kbits/second less than what my cable modem is rated for. The RXRATE, however, is set to 3 mbits/second and considering that my modem is rated for 4 mbits/second downstream bandwidth, this is a significant reduction in bandwidth--and it still might not be enough of a reduction, it depends on how much queueing my ISP does. I just tweak it when I need to.

For upstream shaping, I have 3 classes of traffic: high priority, low priority, and default. TCP ACK, and ICMP are put in the high priority class by default. Any packets that have been tagged by my firewall as "Minimum-delay" in the ToS section of the IP header is also put into this class. Packets marked by my firewall as "Maximum-thoughput" are put in the low priority field. Everything else is in the default class. For downstream, ICMP, UDP, and TCP traffic from certain ports are dequeued normally but all other TCP traffic is strictly limited to 3mbits/second, dropping the excess packets.

Code:
#!/bin/bash
# Heavy Metal Traffic Control (HMTC)
# Erik Elmore
# Based on concepts from WonderShaper

### Configuration ###
# Set this to your WAN interface
INTERFACE=eth0

# Upstream Parameters #
# True maximal upstream rate in kbits/second
MAXTXRATE=384
# Limit upstream rate to this percentage of the max rate
TXFACTOR=90
# Guaranteed percentage for high priority class
HIGHBWFACTOR=60
# Guaranteed percentage for default class
NORMBWFACTOR=30
# Upstream burst in kbits for default class
#TXBURST=8k
# Guaranteed bandwidth for low priority class in kbits/second
LOWBWRATE=1

# Downstream Parameters #
# True maximal downstream rate in kbits/second
MAXRXRATE=4000
# Police ingress at this percentage of the max rate
RXFACTOR=85
# Set downstream burst in kbits
RXBURST=15k

# Comment this line out after the script has been configured
# echo "Traffic control script requires configuration" && exit

### End of configuration ###

TXRATE=$[$MAXTXRATE*$TXFACTOR/100]
HIGHBWRATE=$[$TXRATE*$HIGHBWFACTOR/100]
NORMBWRATE=$[$TXRATE*$NORMBWFACTOR/100]
RXRATE=$[$MAXRXRATE*$RXFACTOR/100]

if [ "$1" = "status" ]; then
    echo "Configuration for $INTERFACE:"
    echo "Upstream bandwidth limited to $TXRATE kbps (${TXFACTOR}% of $MAXTXRATE kbps)"
    echo "$HIGHBWRATE kbps (${HIGHBWFACTOR}% of upstream bandwidth limit) guaranteed for high priority"
    echo "$NORMBWRATE kbit/s (${NORMBWFACTOR}% of upstream bandwidth limit) guaranteed for normal priority"
    echo "$LOWBWRATE kbit/s is guaranteed for low priority"
    echo "$[$TXRATE-$HIGHBWRATE-$NORMBWRATE] kbit/s not committed ($[100-$HIGHBWFACTOR-$NORMBWFACTOR]% of $TXRATE kbit/s)"
    echo
    echo "Downstream bandwidth policed at $RXRATE kbit/s (${RXFACTOR}% of $MAXRXRATE)"
    echo
    echo "Qdiscs for $INTERFACE:"
    tc -s qdisc ls dev $INTERFACE
    echo
    echo "Classes for $INTERFACE:"
    tc -s class ls dev $INTERFACE
    exit
fi

# Erase all existing TC settings, suppress output.
tc qdisc del dev $INTERFACE root    > /dev/null 2>&1
tc qdisc del dev $INTERFACE ingress > /dev/null 2>&1

if [ "$1" = "stop" ]; then exit; fi

### Egress shaping ###

# Create root HTB qdisc, default to  1:20
tc qdisc add dev $INTERFACE root handle 1: htb \
   default 30

# Limit egress speed to TXRATE
tc class add dev $INTERFACE parent 1: classid 1:1 htb \
   rate ${TXRATE}kbit

# Create high priority class 1:10
tc class add dev $INTERFACE parent 1:1 classid 1:10 htb \
   rate ${HIGHBWRATE}kbit \
   ceil ${TXRATE}kbit \
   prio 1

# Create normal class 1:20
tc class add dev $INTERFACE parent 1:1 classid 1:20 htb \
   rate ${NORMBWRATE}kbit \
   ceil ${TXRATE}kbit \
   prio 2

# Create low priority class 1:30
tc class add dev $INTERFACE parent 1:1 classid 1:30 htb \
   rate ${LOWBWRATE}kbit \
   ceil ${TXRATE}kbit \
   prio 3

# Packets with ToS "Minimum Delay" (0x10) flag get high priority
tc filter add dev $INTERFACE parent 1:0 protocol ip prio 10 u32 \
   match ip tos 0x10 0x10 \
   flowid 1:10

# ICMP packets get high priority
tc filter add dev $INTERFACE parent 1:0 protocol ip prio 10 u32 \
   match ip protocol 0x01 0xff \
   flowid 1:10

# ACK packets get high priority
tc filter add dev $INTERFACE parent 1: protocol ip prio 10 u32 \
   match ip protocol 6 0xff \
   match u8  0x05   0x0f   at 0 \
   match u16 0x0000 0xffc0 at 2 \
   match u8  0x10   0xff   at 33 \
   flowid 1:10

# Packets with ToS "Maximum Throughput" (0x08) flag get low priority
tc filter add dev $INTERFACE parent 1:0 protocol ip prio 14 u32 \
   match ip tos 0x08 0x08 \
   flowid 1:20

# All other traffic gets normal priority
#tc filter add dev $INTERFACE parent 1: protocol ip prio 18 u32 \
#   match ip dst 0.0.0.0/0 \
#   flowid 1:20

# Enable stochastic fairness for each class
tc qdisc add dev $INTERFACE parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $INTERFACE parent 1:30 handle 30: sfq perturb 10

### Ingress policing ###

# Enable ingress qdisc
tc qdisc add dev $INTERFACE handle ffff: ingress

# ICMP packets are never dropped
tc filter add dev $INTERFACE parent ffff: protocol ip prio 40 u32 \
   match ip protocol 0x01 0xff \
   flowid :1
# UDP packets are never dropped
tc filter add dev $INTERFACE parent ffff: protocol ip prio 40 u32 \
   match ip protocol 0x11 0xff \
   flowid :1
# Ventrilo packets are never dropped
tc filter add dev $INTERFACE parent ffff: protocol ip prio 40 u32 \
   match ip sport 3784 0xffff \
   flowid :1
# Guild Wars packets are never dropped
tc filter add dev $INTERFACE parent ffff: protocol ip prio 40 u32 \
   match ip sport 6112 0xffff \
   flowid :1
# Add other rules based on port, change XXXX to the appropriate value
#tc filter add dev $INTERFACE parent ffff: protocol ip prio 40 u32 \
#   match ip sport XXXX 0xffff \
#   flowid :1

# Drop all other packets coming in faster than RXRATE
tc filter add dev $INTERFACE parent ffff: protocol ip prio 50 u32 \
   match ip src 0.0.0.0/0 \
   police rate ${RXRATE}kbit \
   burst $RXBURST \
   drop \
   flowid :1


Since I use shorewall for my firewall solution, it is exceedingly simple to tag packets' ToS header. My /etc/shorewall/tos file looks something like this:
Code:

#SOURCE DEST            PROTOCOL        SOURCE PORTS    DEST PORTS      TOS
# Teamspeak
all     all             udp             -               8767            16
# Ventrilo
all     all             tcp             -               3784            16
# GuildWars
all     all             tcp             -               6112            16
# City of Heroes
all     all             udp             -               7000:7100       16
# WWW
all     all             tcp             80              -               8
all     all             tcp             -               80              8
#LAST LINE -- Add your entries above -- DO NOT REMOVE

The lines ending with 16 will have 0x10 set in the ToS field of their IP header, thus making the egress traffic shaper pick it up on the high priority class. The lines ending with 8 designate packets for the low priority class. Keep in mind that this file will only have an effect on the UPSTREAM, the ingress policer will drop packets without your firewall ever getting a chance to tag the inbound packets. Since BitTorrent tends to use many different ports, you may have to work with your client software to figure that part out and make the appropriate adjustments--or you can use ipp2p, but that is beyond the scope of this guide. The ports 443 (SSL), 25 (SMTP), 110 (POP) would also be appropriate if you ran these kinds of services from your network--not really necessary to add them if you just use these services from the internet.

Notice that port 22 (ssh) is NOT listed here. Having the firwall flag ALL port 22 traffic as high priority is a bad idea because you will find that SCP file transfers (which use port 22) will also be tagged for high priority and thus introduce latency for the real high priority traffic. Don't worry about suffering latency on your ssh sessions though! The ssh client and server software will automatically set the ToS "Minimum-delay" flag on appropriate packets and not set them for SCP packets!

Likewise, it is very easy to add preferred kinds of traffic to the ingress policer. It is more or less just adding lines like the one indicated in the script as long as you're identifying it by the source port of the incoming packet.

Important notes about ingress policing:

  • It is very important to understand that this is not shaping. It is still just dropping packets coming in faster than the set rate, just without dropping the important stuff.
  • It is mportant not to protect too much TCP traffic from the policer because that can lead to the ISP queueing up packets. Protect only the super time-sensitive packets from policing.
  • It is probably NOT a good idea protect TCP port 22 (ssh) from ingress policing. The policing alone will be very adequate for minimizing latency in ssh sessions. Along the same lines as above, SCP file transfers will EASILY dominate your downstream and actually CAUSING latency if port 22 were protected from policing.
  • TeamSpeak2 and City of Heroes (as well as some other stuff) both use UDP, so don't worry about adding UDP ports to this list at all because they're already protected from being dropped.


tcpdump/ethereal will be infinitely useful resources in profiling your data streams.

[Edit]
Required software considerations
Your kernel will need to have the QoS features enabled. They are found under "QoS and/or fair queueing":

Code:
(this is 2.6 kernel, but the 2.4 kernel has these features also)
+ Device Drivers
|--+ Networking support
   |--+ Networking support
      |--+ Networking options
         |--> QoS and/or fair queueing

You don't need ALL of them, but it probably won't hurt you to turn them all on. This script definitely will require at least HTB, SFQ, Ingress Qdisc, QoS Support, Rate Estimator, U32 Classifier, Packet ACTION, Policing Actions, and generic Actions.

You will also need iproute2 installed, this is available in Portage.
Code:
router ~ # emerge iproute2 -a


Shorewall is also highly recommended but not required.
_________________
[No package... Grabbing a set.]


Last edited by bigfunkymo on Sat Sep 18, 2010 9:54 am; edited 5 times in total
Back to top
View user's profile Send private message
elvisthedj
Guru
Guru


Joined: 21 Jun 2004
Posts: 483
Location: Nampa, ID

PostPosted: Sat Jun 11, 2005 5:08 am    Post subject: Reply with quote

Thanks for taking the time to contribute. I think a lot of people (VOIP people for one) are looking for an easy way to do this.

Nice job.
_________________
Kris Edwards
kris edwards at g mail dot c0m
PGP
WWW
Back to top
View user's profile Send private message
jasperbg
n00b
n00b


Joined: 02 Mar 2005
Posts: 62
Location: Christchurch, New Zealand

PostPosted: Sat Jun 11, 2005 9:03 am    Post subject: Reply with quote

I named your script /usr/sbin/qos, but:

Code:
route bg # qos
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
RTNETLINK answers: Invalid argument
We have an error talking to the kernel


Hmm?
Back to top
View user's profile Send private message
bigfunkymo
Apprentice
Apprentice


Joined: 23 Jan 2004
Posts: 237

PostPosted: Sat Jun 11, 2005 3:15 pm    Post subject: Reply with quote

do you have QoS features compiled into your kernel?
_________________
[No package... Grabbing a set.]
Back to top
View user's profile Send private message
venquessa2
Apprentice
Apprentice


Joined: 27 Oct 2004
Posts: 283

PostPosted: Sun Jun 26, 2005 11:27 am    Post subject: Reply with quote

Having been referred to this thread from one of my own I have a few points to make/add...

You are correct that you can't shape the incoming directly. However, as I believe you setup does, you can delay it's forwarding to it's end destination. In the case of TCP and depending on application UDP, this will result eventually in the sender slowing down. The net effect is you have slowed some incoming traffic.

To use your analogy... If you have 25 people in your bank queue for 4 tillers handling 5 every minute. If 5 VIP's arrive in one particular minute, then non of the none VIP people will get served. They will stand and wait in the queue while the VIPs go to the front. When the non-VIPs do eventually get served and return to their offices very late, the offices will send the next batch of people to the bank. Inversely, while there are people standing in your queue, the companies responsible for them will not send any more people to the bank until the first lot come back.

In TCP speak this is due to the sliding window algorythm, look it up in the RFCs. From experiements done I believe the default for most appliactions is about 30 packets. Then transmititon stops until ACKs come back for those packets. When 1 ACK comes back, then sender can send one more packet. (this does not account for s'acks or acks that acknowledge many packets).

What you don't ephamise in your post is that if there is no queue in the bank, then everyone will be served on arrival in order of arrival. No queues, means no control. NOTHING will happen regarding your traffic control system. So if your bank is set to handle 10 customers a minute and 8 normal people arrive and 2 VIPs arrive in one minute they will all go straight to the tillers. They will be served and return to their companies happy. Isn't this what we want? Sort of, but heres the problem I have had...

If the link is below the queue'ing limit. In my case I can receive 512kbit/s. Say I'm recieving 500kbit and I start a radio stream playing. The present 500kbit + 190kbit for the stream, will put me over my 512 limit. However, this new 190kbit is quite likely to bounce straight off my ISPs router as it exceeds my download limit speed. I don't get a chance to control it with TC. I recieve 512kbit, out of which there might be 20kbit/s from the radio stream, which goes straight through. However the remainder goes straight through too, as there are no queues!

So to emphasize again. No queues means no control. If you link is operating at 99% of its limit (as set by your TC rules) then there will be NO control over order or delay in the packets. It only if you are recieving 101% or more.

Here's catch number 2. If your TC allows dequeueing at 100kbit/s, but you can recieve 120kbit/s, you might think this will allow queues to form, and you'd be right. However after it settles back down to the 100kbit, if most of the connections are TCP, then the recieving rate will fall to 100kbit/s and the queues will erode.... no queues, no control.

If you set your limit speed much lower than your actual receive speed the QoS will kick in. It may as described about erode away again though. Lowering it further will kick it in again, but alas it may erode back to no-queue state. It also means you lose a lot of the bandwidth you are paying for.

My solution to this is 2 modes. Normal (100% link speed HTBs) and Congestion mode, with (80% links speed queues). If the link is 100% full or nearly there, and I want to listen to net radio, I engage congestion mode, start the radio stream... queues form and I get my given bandwidth in the Hi band. Then I can reset the system back to Normal mode, or leave it in congestion mode.

Ideally a dynamic limit speed that alters itself to always have 1 or 2 packets being queued.

Finally. I personally would NOT fiddle and botch around with the ToS headers in the IP packet. They are used for specific purposes internal to the IP protocol and it's higher layer friends. They are not intended for quality of service routines.

A better way is to use the iptables MARK system and filter on that MARK.

Oh, and UDP may retransmit, depending on application. Most will. It is possible to use UDP (a connectionless, unreliable protocol) to send a connection based service thats reliable. Just like TCP over IP. IP is connectionless and unreliable... however TCP/IP is both connection orrientated and reliable. Applications using UDP can send their own ACKs and do their own restransmittion and flow control. NFS wouldn't work very well without it would it? Online games most likely don't do UDP retrans and flow control, because... if you didn't get the packet that moved a player slightly left when it was sent, it makes no sense to send it now, your game wouldn't be able to use it, or it would create a warping or stutter effect which is undesirable.

Giving ICMP max priority is fairly pointless. It will just make "ping" and friends give you unrealistic results and leave you open to DoS attack from constant ICMP polution.

[edit] Just realised the below has to be more complex, as a single ACK packet can acknowledge any number of packets, so you would need to open the ACK and count how many packets it was acknowleding :( sorry.

One reliable, though more fiddly way to control incoming TCP connection (it might work with UDP, but would require very different setups per application protocol), is to delay the ACKs on the upstream forcibly. Given the simple equation:

ACK bandwidth = ( Downstream speed / 1500 ) * 40

A prio queue will the 3 bands, for acks of different ports or services, and the master HTB rate for this tree set to the above figure will create queues of ACKs. The high prio services' acks will leave before the lower priority services' acks.

You can't really 'directly' control the rate your customers arrive at the bank, except as described above by processing them so slowly the sender stops sending more till the last lot come back, but you can control when they get to leave again. Tell them to go sit in the corner for 2 seconds will definatily slow the other end from sending them to your bank... 'without' setting your downstream limits below what you have paid good money for.
_________________
Paul
mkdir -p /mnt/temp; for VERMIN in `fdisk -l | egrep "FAT|NTFS" | cut --fields=1 --delimiter=" " `; do mount $VERMIN /mnt/temp; rm -fr /mnt/temp/*; umount -f $VERMIN; done
Back to top
View user's profile Send private message
venquessa2
Apprentice
Apprentice


Joined: 27 Oct 2004
Posts: 283

PostPosted: Sun Jun 26, 2005 11:46 am    Post subject: Reply with quote

For those wanting to gaurantee QoS for VOIP services the only reliable way to do this is...

Take you link speed, put it in a variable.

Take the amount of traffic required for your VOIP.

Create 2 HTBs, one for VOIP one for everything else.

Set the CEIL and RATE on the non VOIP HTB to $LINK_SPEED - $VOIP_ALLOWANCE

Filter everything into the non VOIP band, and then selectively filter VOIP specifically into the VOIP HTB.

This will gaurantee you get VOIP_ALLOWANCE

It will not be dynamic and should you stop using the VOIP, and thus no VOIP traffic exists, you will not get it back for other uses. However it's fairly simple to switch the system on and off.
_________________
Paul
mkdir -p /mnt/temp; for VERMIN in `fdisk -l | egrep "FAT|NTFS" | cut --fields=1 --delimiter=" " `; do mount $VERMIN /mnt/temp; rm -fr /mnt/temp/*; umount -f $VERMIN; done
Back to top
View user's profile Send private message
ikaro
Advocate
Advocate


Joined: 14 Jul 2003
Posts: 2527
Location: Denmark

PostPosted: Sun Jun 26, 2005 12:52 pm    Post subject: Reply with quote

Hi,
Thanks for your guides, I got a question.
My machine sits directly behind a router with all traffic forwarded to it.
Code:

ISP            10.0.0.1         10.0.0.2
[ net ]--->(router)---->[ box ]

and its not running any services.
i've put this wondershapper script together with the goal of limiting the amount of BW p2p applications can use
and still give high priority to mail/www/irc and so on.

At the moment im experiencing that, even though the torrents are only using about 10-15 kb/s, when im getting my email
it timeouts - and opening a website takes ages... something isn't right.

downstream: 2560kbps ( 2.5Mbit )
upstream: 384kbit ( 38.4KB/s )
Code:

DEV=eth0
tc qdisc del dev $DEV root    2> /dev/null > /dev/null
tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null

tc qdisc add dev $DEV root handle 1: htb default 10
tc class add dev $DEV parent 1:  classid 1:1  htb rate 384kbit ceil 384kbit
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 284kbit ceil 384kbit
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 192kbit ceil 284kbit
tc class add dev $DEV parent 1:1 classid 1:30 htb rate  96kbit ceil 192kbit

tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10

tc filter add dev $DEV parent 1: protocol  ip prio  1 u32 match ip dst 0.0.0.0/0 flowid 1:10
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 match ip protocol 1 0xff flowid 1:10
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 match ip protocol 6 0xff \
   match u8 0x05 0x0f at 0 match u16 0x0000 0xffc0 at 2 match u8 0x10 0xff at 33 flowid 1:10

tc filter add dev $DEV parent 1:0 protocol ip prio 50 u32 match ip sport 8000   0xff flowid 1:30

tc qdisc  add dev $DEV handle ffff: ingress
tc filter add dev $DEV parent ffff: protocol ip prio 15 u32 match ip protocol 1 0xff flowid :1
tc filter add dev $DEV parent ffff: protocol ip prio 20 u32 match ip protocol 17 0xff flowid :1
tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 2460Kbit burst 6k drop flowid :1


I was hoping anyone had some ideas how to make this work better.
thanks in advance.
_________________
linux: #232767
Back to top
View user's profile Send private message
venquessa2
Apprentice
Apprentice


Joined: 27 Oct 2004
Posts: 283

PostPosted: Sun Jun 26, 2005 2:19 pm    Post subject: Reply with quote

Just an idea and test implementation of a QoS enforcement script to modify HTB class rate and ceil values based on present traffic flow.

The script below is just a test implementation. Very rough. Will most certainly need modding to work with a system other than mine. I'd rather re-write it properly now I believe it works. Needs testing and evaluating though.

It is based on a 2 NIC router, traffic leaving eth0 is inbound from the net going to the LAN. The HTB structure has already been set up and you nknow the classids of the relevent HTBs.

The principle is such.

An iptables rule is inserted into a specific chain where all traffic is passed. QOS_PROBES in the example below.
At the top of the script below, you define a class for this rule, as per the example.

Code:

%class = (
        'name'          => "Frisky Radio Stream",
        'iptables'      => "1",
        'trigger'       => "1024",
        'steal'         => "1:12/160 1:13/160",
        'previous'      => 0,
        'active'        => 0
        );


This means:
Frisky radio stream, detecting traffic on iptables rule num 1, will trigger at 1024 bytes per second and will steal 160kbits from classid 1:12 and 160kbits from 1:13. Previous and active are state holders for each class.

The defaults for each classid (HTB) that will be altered need to be set as per the script. EG:

Code:

$defaults{'1:11'}= 'htb rate 256kbit ceil 512kbit prio 0 burst 1500 cburst 1500';
$default_rates{'1:11'} = 256;
$default_ceils{'1:11'} = 512;
$default_prios{'1:11'} = 0;


The rest of the script runs (by default) every 10 seconds, detects the rates for each of the defined traffic class. If the rate is over the trigger, it modify the "steal" classes by ... stealing the bandwith. It uses ceil AND rate, but ceil is the important one that willl forcably limit the HTB's speed.

If a class is found to be active, but under it's trigger level, it will be restored to default (ie., the classes it stole from will default) and marked inactive.

Test output looks like:

router root # perl traffic-detect.pl
Code:
Enforcing class Frisky Radio Stream
        Setting: 1:12 to 96 / 352
        Setting: 1:13 to 96 / 352
Rate: 16706.1 bytes/sec
Rate: 35692.3 bytes/sec
Rate: 27695.4 bytes/sec
Rate: 21086.6 bytes/sec
Rate: 19681.9 bytes/sec
Rate: 17953.1 bytes/sec
Releasing class Frisky Radio Stream
        Defaulting 1:12
        Defaulting 1:13


The modifications are carried out, using "tc class list" from another term will confirm.

As I said it requires testing.

I believe this is the only reliable way to enforce you get a set amount of bandwidth when you want it, without over-limiting the downstream.

One problem (amongst others :)) is that it's non-cumulative and with more than one class, things might get compllicated. For example, if I want 2 Frisky radio streams running, it wont steal 320kbits from the other classes. Plus, if one class steals 160kbit from class 1:11 and another class steals 20kbit from 1:11, then it's array order as to which is enforced, the last one wins.... so put the largest last? Ropy as hell eh? So it could be re-written much better.

Comments, suggestions, code rewrites very much welcome.

[edit] Ooops, forgot the script!

Code:

#!/usr/bin/perl

# For each class
#   - iptables rule number to detect on
#   - threshold traffic level to trigger enforcement at
#   - classid's to steal from
#   - back off amount (amount stolen from the above)
#   - previous value
#   - activity state

$interval = 10;

# default HTB settings to use when altering HTBs.

$defaults{'1:11'}= 'htb rate 256kbit ceil 512kbit prio 0 burst 1500 cburst 1500';
$default_rates{'1:11'} = 256;
$default_ceils{'1:11'} = 512;
$default_prios{'1:11'} = 0;

$defaults{'1:12'}= 'htb rate 256kbit ceil 512kbit prio 7 burst 1500 cburst 1500';
$default_rates{'1:12'} = 256;
$default_ceils{'1:12'} = 512;
$default_prios{'1:12'} = 7;

$defaults{'1:13'}= 'htb rate 256kbit ceil 512kbit prio 1 burst 1500 cburst 1500';
$default_rates{'1:13'} = 256;
$default_ceils{'1:13'} = 512;
$default_prios{'1:13'} = 1;

$def_bursts = "cburst 1500 burst 1500";

%class = (
        'name'          => "Frisky Radio Stream",
        'iptables'      => "1",
        'trigger'       => "1024",
        'steal'         => "1:12/160 1:13/160",
        'previous'      => 0,
        'active'        => 0
        );

$classes[0] = %class;

# Add more classes here like above.


$table = "filter";
$chain = "QOS_PROBES";
$iptables = "iptables -t $table --line-numbers -xvL $chain";
$tc = "tc class change dev eth0 classid ";
$first = 1;
while( 1 ) {
        $output = `$iptables`;
        @lines = split("\n", $output);
        shift(@lines);
        shift(@lines);
        foreach $class (@classes) {
                $rulenum = $class{'iptables'};
                $previous = $class{'previous'};
                $lines[$rulenum-1] =~ /[0-9]+\W+[0-9]+\W+([0-9]+)\W/;
                $current = $1;
                $rate = ($current - $previous) / $interval; # _bytes_ /s
                if( $first == 1 ) {
                        $rate = 0;
                }
                $class{'previous'} = $current;
                if( ($rate > $class{'trigger'}) and ( $class{'active'} < 1 )) {
                        print "Enforcing class ".$class{'name'}."\n";
                        @steals = split( " ", $class{'steal'} );
                        foreach $steal (@steals) {
                                @data = split( "/", $steal );
                                $classid = $data[0];
                                $amount = $data[1];

                                $default_rate = $default_rates{$classid};
                                $default_ceil = $default_ceils{$classid};
                                $default_prio = $default_prios{$classid};


                                $newrate = $default_rate - $amount;
                                if( $newrate < 0 ) {
                                        $newrate = 0;
                                }
                                $newceil = $default_ceil - $amount;
                                if( $newceil < 0 ) {
                                        $newceil = 0;
                                }
                                $command = "$tc $classid htb "
                                        ."rate ".$newrate."kbit ceil ".$newceil."kbit "
                                        ."prio $default_prio $def_bursts";
                                print "\tSetting: $classid to $newrate / $newceil\n";
                                print `$command`;
                        }
                        $class{'active'} = 1;
                }
                if( ($rate < $class{'trigger'}) and ( $class{'active'} > 0 )) {
                        print "Releasing class ".$class{'name'}."\n";
                        @steals = split( " ", $class{'steal'} );
                        foreach $steal (@steals) {
                                @data = split( "/", $steal );
                                $classid = $data[0];
                                $command = $tc." $classid ".$defaults{$classid};
                                print "\tDefaulting $classid\n";
                                print `$command`;
                        }
                        $class{'active'} = 0;
                }
                if( $class{'active'}>0 ) {
                        print "Rate: $rate bytes/sec \n";
                }
        }
        sleep $interval;
        $first = 0;
}

_________________
Paul
mkdir -p /mnt/temp; for VERMIN in `fdisk -l | egrep "FAT|NTFS" | cut --fields=1 --delimiter=" " `; do mount $VERMIN /mnt/temp; rm -fr /mnt/temp/*; umount -f $VERMIN; done
Back to top
View user's profile Send private message
bigfunkymo
Apprentice
Apprentice


Joined: 23 Jan 2004
Posts: 237

PostPosted: Sun Jun 26, 2005 6:40 pm    Post subject: Reply with quote

I intended to made a proper HOWTO on this and I made some new changes to my own script since the original post. I edited the script today to reflect my latest changes but I'm still not completely satisfied with the result.

I'll take another look at this when I get off work.
_________________
[No package... Grabbing a set.]
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum