From my own experience, playing with Linux distros in your home-lab its the default entry point to the TUX world, but things get slight different when you become the Linux guy for some company or when for some external needs you must deep dive into the TUX insights. The intention of this writing its to focus on one of this things: 10Gbps Interfaces and one of the necessary tricks to be done in Linux to get the maximum performance on them: Jumbo Frames.
From wikipedia:“Jumbo frames are Ethernet frames with more than 1500 bytes of payload. Conventionally, jumbo frames can carry up to 9000 bytes of payload, but variations exist and some care must be taken using the term.”
Now, to illustrate the usage of Jumbo Frames let’s consider the following example:
One Linux server with 10G interface, and some NFS mounts mounted from NetApp Filer.
Properties of the NIC:
# ethtool eth0 Settings for eth0: Supported ports: [ FIBRE ] Supported link modes: Supports auto-negotiation: No Advertised link modes: 10000baseT/Full Advertised auto-negotiation: No Speed: 10000Mb/s Duplex: Full Port: FIBRE PHYAD: 0 Transceiver: external Auto-negotiation: off Supports Wake-on: g Wake-on: g Current message level: 0x00000005 (5) Link detected: yes
IP configuration on the NIC:
# ifconfig eth0 Link encap:Ethernet HWaddr D8:D3:85:XX:XX:XX inet addr:102.XX.XX.15 Bcast:126.96.36.199 Mask:255.255.254.0 UP BROADCAST NOTRAILERS RUNNING MULTICAST MTU:1500 Metric:1 RX packets:685956 errors:0 dropped:0 overruns:0 frame:0 TX packets:876178 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:256205828 (244.3 Mb) TX bytes:179815892 (171.4 Mb) Interrupt:154
With the default MTU set at default 1500 bytes, let’s see how it goes when trying to write 10G file:
# dd if=/dev/zero of=file_1GB bs=1024 count=10000000 10000000+0 records in 10000000+0 records out 10240000000 bytes (10 GB) copied, 221.047 seconds, 46.3 MB/s
dd if=/dev/zero of=file_1GB bs=1024 count=10000000 10000000+0 records in 10000000+0 records out 10240000000 bytes (10 GB) copied, 113.765 seconds, 90.0 MB/s
As you can see, there’s pretty much double the write speed achieved.
Well, this is just one of the performance tricks you must be aware when dealing with this interfaces, others may include TCP window sizing, buffers, etc. But the idea its to get a good example of the importance of setting Jumbo Frames.
Another important thing to consider: The same MTU size must be configured on the Switch port, and the remote peer you are getting resources from.
Before trying to set a custom MTU or Jumbo Frame in your server, first do yourself a favor a verify the switch port it’s configured and capable of dealing with the MTU size you need, and secondly the end peer also it’s configured with the same MTU size.