Jumbo Frames in practice for entry-level Linux Admins

From my own experience, playing with Linux distros in your home-lab its the default entry point to the TUX world, but things get slight different when you become the Linux guy for some company or when for some external needs you must deep dive into the TUX insights. The intention of this writing its to focus on one of this things: 10Gbps Interfaces and one of the necessary tricks to be done in Linux to get the maximum performance on them: Jumbo Frames.

From wikipedia:

“Jumbo frames are Ethernet frames with more than 1500 bytes of payload. Conventionally, jumbo frames can carry up to 9000 bytes of payload, but variations exist and some care must be taken using the term.”


Now, to illustrate the usage of Jumbo Frames let’s consider the following example:

One Linux server with 10G interface, and some NFS mounts mounted from NetApp Filer.

Properties of the NIC:

 # ethtool eth0
Settings for eth0:
        Supported ports: [ FIBRE ]
        Supported link modes:   
        Supports auto-negotiation: No
        Advertised link modes:  10000baseT/Full 
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: off
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x00000005 (5)
        Link detected: yes

IP configuration on the NIC:

# ifconfig 
eth0      Link encap:Ethernet  HWaddr D8:D3:85:XX:XX:XX  
          inet addr:102.XX.XX.15  Bcast:  Mask:
          RX packets:685956 errors:0 dropped:0 overruns:0 frame:0
          TX packets:876178 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:256205828 (244.3 Mb)  TX bytes:179815892 (171.4 Mb)


With the default MTU set at default 1500 bytes, let’s see how it goes when trying to write 10G file:

# dd if=/dev/zero of=file_1GB bs=1024 count=10000000
10000000+0 records in
10000000+0 records out
10240000000 bytes (10 GB) copied, 221.047 seconds, 46.3 MB/s
And let’s compare that with a MTU of 9022 bytes or Jumbo Frame configured on the NIC:
dd if=/dev/zero of=file_1GB bs=1024 count=10000000
10000000+0 records in
10000000+0 records out
10240000000 bytes (10 GB) copied, 113.765 seconds, 90.0 MB/s

As you can see, there’s pretty much double the write speed achieved.

Well, this is just one of the performance tricks you must be aware when dealing with this interfaces, others may include TCP window sizing, buffers, etc. But the idea its to get a good example of the importance of setting Jumbo Frames.

Another important thing to consider: The same MTU size must be configured on the Switch port, and the remote peer you are getting resources from.

Before trying to set a custom MTU or Jumbo Frame in your server, first do yourself a favor a verify the switch port it’s configured and capable of dealing with the MTU size you need, and secondly the end peer also it’s configured with the same MTU size.  




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s