Compiling Kernel 3.8 on Debian Testing/Wheezy

linuxNOTE: It seems like series 3.8 has issues with intel (i915) graphics — it occasionally generates kworker threads that causes unresponsiveness as seen by slow mouse and keyboard response when e.g. plugging or unplugging mains power. No issues on e.g. nvidia though.

http://verahill.blogspot.com.au/2013/03/368-slow-mouse-and-keyboard-triggered.html
http://forums.gentoo.org/viewtopic-p-7278760.html
https://bbs.archlinux.org/viewtopic.php?pid=1248190

Post: Kernel 3.8 is out now. Not much to say — the compilation works well using the standard method. The compressed kernel is about 81 Mb to download.

The approach below shows how to compile the kernel on Debian. If you’re interested in a more generic approach, see this post:

http://verahill.blogspot.com.au/2013/02/344-compile-kernel-38-without-using-kpkg.html

NOTE: kernel 3.8 — in contrast to the 3.7 series — now compiles fine on AMD FX 8150.

NOTE: kernel 3.8 plays well with nvidia dkms

Here we go:

sudo apt-get install kernel-package fakeroot build-essential ncurses-dev
mkdir ~/tmp
cd ~/tmp
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.8.tar.bz2
tar xvf linux-3.8.tar.bz2
cd linux-3.8/
cat /boot/config-`uname -r`>.config
make oldconfig

You will be asked a lot of questions — how many depends on what version you upgrade from. If in doubt, pick the default answer (i.e. hit enter). If really in doubt, use google.

Then continue:

make-kpkg clean

Do

make menuconfig

if you want to make any specific changes to the kernel (e.g. add support for certain devices)

Then continue:

time fakeroot make-kpkg -j4 --initrd kernel_image kernel_headers
As usual 4 is the number of threads you wish to launch — make it equal to the number of cores that you have for optimum performance during compilation (more about that here).
The build takes around 20 minutes on a four-core intel i5-2400 with -j4, and 14 minutes on an fx-8150 with -j8 (96 minutes with -j1).
Install:
sudo dpkg -i ../linux-image-3.8.0_3.8.0-10.00.Custom_amd64.deb ../linux-headers-3.8.0_3.8.0-10.00.Custom_amd64.deb
New stuff/Questions:
Offload RCU callback processing from boot-selected CPUs (RCU_NOCB_CPU) [N/y/?] (NEW) *
Memory placement aware NUMA scheduler (NUMA_BALANCING) [N/y/?] (NEW) *
Enable to assign a node which has only movable memory (MOVABLE_NODE) [N/y/?] (NEW)
Allow for memory hot-add (MEMORY_HOTPLUG) [Y/n] y
Allow for balloon memory compaction/migration (BALLOON_COMPACTION) [Y/n/?] (NEW)
Set default setting of cpu0_hotpluggable (BOOTPARAM_HOTPLUG_CPU0) [N/y/?] (NEW
Debug CPU0 hotplug (DEBUG_HOTPLUG_CPU0) [N/y/?] (NEW)
ACPI tables can be passed via uncompressed cpio in initrd (ACPI_INITRD_TABLE_OVERRIDE) [N/y/?] (NEW)
Support multiple cpuidle drivers (CPU_IDLE_MULTIPLE_DRIVERS) [N/y/?] (NEW)
"NOTRACK" target support (DEPRECATED) (NETFILTER_XT_TARGET_NOTRACK) [N/m] (NEW

Default SCTP cookie HMAC encoding
  > 1. Enable optional MD5 hmac cookie generation (SCTP_DEFAULT_COOKIE_HMAC_MD5) (NEW)
    2. Enable optional SHA1 hmac cookie generation (SCTP_DEFAULT_COOKIE_HMAC_SHA1) (NEW)
    3. Use no hmac alg in SCTP cookie generation (SCTP_DEFAULT_COOKIE_HMAC_NONE) (NEW)
  choice[1-3?]:   Enable optional MD5 hmac cookie generation (SCTP_COOKIE_HMAC_MD5) [Y/?] (NEW) y

Enable optional SHA1 hmac cookie generation (SCTP_COOKIE_HMAC_SHA1) [N/y/?] (NEW) *
Enable optional MD5 hmac cookie generation (SCTP_COOKIE_HMAC_MD5) [Y/?] (NEW) y
Distributed ARP Table (BATMAN_ADV_DAT) [N/y/?] (NEW)
Kvaser CAN/USB interface (CAN_KVASER_USB) [N/m/?] (NEW)
LSI MPT Fusion SAS 3.0 Device Driver (SCSI_MPT3SAS) [N/m/?] (NEW)
Chelsio Communications FCoE support (SCSI_CHELSIO_FCOE) [N/m/?] (NEW) *
Marvell 88E6060 ethernet switch chip support (NET_DSA_MV88E6060) [N/m/y/?] (NEW)
Marvell 88E6085/6095/6095F/6131 ethernet switch chip support (NET_DSA_MV88E6131) [N/m/y/?] (NEW)
Marvell 88E6123/6161/6165 ethernet switch chip support (NET_DSA_MV88E6123_61_65) [N/m/y/?] (NEW) *
Cadence devices (NET_CADENCE) [Y/n/?] (NEW)
AT91RM9200 Ethernet support (ARM_AT91_ETHER) [N/m/y/?] (NEW)
Cadence MACB/GEM support (MACB) [N/m/y/?] (NEW)
Broadcom devices (NET_VENDOR_BROADCOM) [Y/?] y
Marvell MDIO interface support (MVMDIO) [N/m/y/?] (NEW)
CDC MBIM support (USB_NET_CDC_MBIM) [N/m/?] (NEW)
Atheros Wireless Cards (ATH_CARDS) [N/m/?] (NEW)
Atheros AR5523 wireless driver support (AR5523) [N/m/?] (NEW)
Wilocity 60g WiFi card wil6210 support (WIL6210) [N/m/?] (NEW) *
Realtek RTL8723AE PCIe Wireless Network Adapter (RTL8723AE) [N/m/?] (NEW)
ARC UART driver support (SERIAL_ARC) [N/m/y/?] (NEW) *
CBUS I2C driver (I2C_CBUS_GPIO) [N/m/?] (NEW)
TS-5500 DIO blocks and compatibles (GPIO_TS5500) [N/m/y/?] (NEW) 
TI BQ2415x battery charger driver (CHARGER_BQ2415X) [N/m/?] (NEW)
Board level reset or power off (POWER_RESET) [N/y/?] (NEW) *

 Default Thermal governor
  > 1. step_wise (THERMAL_DEFAULT_GOV_STEP_WISE) (NEW)
    2. fair_share (THERMAL_DEFAULT_GOV_FAIR_SHARE) (NEW)
    3. user_space (THERMAL_DEFAULT_GOV_USER_SPACE) (NEW)
  choice[1-3?]:   Fair-share thermal governor (FAIR_SHARE) [N/y/?] (NEW)

Step_wise thermal governor (STEP_WISE) [Y/?] (NEW) y
User_space thermal governor (USER_SPACE) [N/y/?] (NEW)
SSB GPIO driver (SSB_DRIVER_GPIO) [N/y/?] (NEW) *
BCMA GPIO driver (BCMA_DRIVER_GPIO) [N/y/?] (NEW)
Support for Realtek PCI-E card reader (MFD_RTSX_PCI) [N/m/y/?] (NEW)
TI ADC / Touch Screen chip support (MFD_TI_AM335X_TSCADC) [N/m/y/?] (NEW)
Support for Nano River Technologies Viperboard (MFD_VIPERBOARD) [N/m/?] (NEW)
Support for Retu multi-function device (MFD_RETU) [N/m/?] (NEW) *
Maxim MAX8973 voltage regulator  (REGULATOR_MAX8973) [N/m/?] (NEW)
TI TPS51632 Power Regulator (REGULATOR_TPS51632) [N/m/?] (NEW)
Siano SMS1xxx based MDTV receiver (SMS_USB_DRV) [N/m/?] (NEW)
Siano SMS1xxx based MDTV via SDIO interface (SMS_SDIO_DRV) [N/m/?] (NEW)   *
Stanton Control System 1 MIDI (SND_SCS1X) [N/m/?] (NEW) *
ION iCade arcade controller (HID_ICADE) [N/m/?] (NEW)
HID over I2C transport layer (I2C_HID) [N/m/?] (NEW) *
Renesas R-Car USB phy support (USB_RCAR_PHY) [N/m/?] (NEW)   *
SDHCI support for ACPI enumerated SDHCI controllers (MMC_SDHCI_ACPI) [N/m/?] (NEW)
NXP PCF8523 (RTC_DRV_PCF8523) [N/m/?] (NEW)
Philips PCF8563/Epson RTC8564 (RTC_DRV_PCF8563) [M/n/?] m
Userspace platform driver with generic irq and dynamic memory (UIO_DMEM_GENIRQ) [N/m/?] (NEW)
Microsoft Hyper-V Balloon driver (HYPERV_BALLOON) [N/m/?] (NEW) *
SystemBase PCI Multiport UART (SB105X) [N/m/y/?] (NEW)
TTY over Firewire (FIREWIRE_SERIAL) [N/m/?] (NEW) *
F2FS filesystem support (EXPERIMENTAL) (F2FS_FS) [N/m/y/?] (NEW) *
Enable CIFS debugging routines (CIFS_DEBUG) [Y/n/?] (NEW)
Simplified Mandatory Access Control Kernel Support (SECURITY_SMACK) [N/y/?] (NEW)
Camellia cipher algorithm (x86_64/AES-NI/AVX) (CRYPTO_CAMELLIA_AESNI_AVX_X86_64) [N/m/y/?] (NEW)

Compiling Linux Kernel 3.4.5 Vanilla Final — Debian way

Here is my guide on Compiling 3.4.x Vanilla Final — Ubuntu/Debian using the debian way

This article is about compiling a kernel on Ubuntu systems. It describes how to build a custom kernel using the latest unmodified kernel sources from www.kernel.org (vanilla kernel) so that you are independent from the kernels supplied by your distribution.
Install the Required packages for building it

sudo apt-get install git-core kernel-package fakeroot build-essential ncurses-dev

Then download latest kernel version

cd /usr/src
sudo wget --continue http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.4.5.tar.bz2
sudo tar jxvf linux-3.4.5.tar.bz2
cd linux-3.4.5

$ sudo cp /boot/config-`uname -r` ./.config
$ sudo make menuconfig
Disable xen
Processor Type and Features -> Paravirtualized Guest Support -> Xen

support otherwise you will get this error

sudo make-kpkg clean
sudo fakeroot make-kpkg --initrd --append-to-version=-vanillaice kernel_image kernel_headers
cd ..
sudo dpkg -i linux-image-3.4.*
sudo shutdown -r now

you can install the headers too from /usr/src/linux-headers-3.4.*-*
in my case i can show you how the packages are named

ls *.deb
linux-image-3.4.*-vanillaice_4.4.*-vanillaice-10.00.Custom_amd64.deb
linux-headers-3.4.*-vanillaice_4.4.*-vanillaice-10.00.Custom_amd64.deb

Отключение IPv6 в Linux

В ядрах до 2.6.29 (не уверен!) этот протокол собирался отдельным модулем, поэтому его выключение было возможно помещением в /etc/modprobe.d/blacklist строчку blacklist ipv6.

В ядрах 2.6.30 протокол ipv6 собирался уже на уровне ядра и отключение его подразумевало пересборку ядра без оного.

В ядрах 2.6.31 хоть протокол и размещается в ядре, но предусмотрена возможность отключить его опцией при запуске ядра:

В GRUB’е первом ищем /boot/grub/menu.lst и дописываем строчку запуска ядра (кроме обычных опций ro quiet splash) ipv6.disable=1. Перегружаем компутер и проверяем ifconfig.

В GRUB2 идём в /etc/default/grub и ищем строчку с опциями по умолчанию «quiet splash» и дописываем опцию ipv6.disable=1. Должно получиться вот так: «quiet splash ipv6.disable=1».
И запускаем:

$ sudo update-grub

Перегружаем компутер и проверяем ifconfig.

Теперь я не отключаю ipv6, а просто запрещаю его iptables’ом.

$ ip6tables -S
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT DROP
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT

Автоматическая перезагрузка Linux при kernel panic, перегрузке CPU или системном сбое

If you are a system administrator, you have probably wondered at least once ohw to configure your Linux server to automatically reboot itself if it crashes, is going through a mass CPU overload, e.g. the server load average “hits the sky”.

I just learned from a nice article found here that there is a kernel variable which when enabled takes care to automatically restart a crashed server with the terrible Kernel Panic message we all know.

The variable I’m taking about is kernel.panic for instance kernel.panic = 20 would instruct your GNU Linux kernel to automatically reboot if it experiences a kernel panic system crash within a time limit of 20 seconds.

To start using the auto-reboot linux capabilities on a kernel panic occurance just set the variable to /etc/sysctl.conf

debian-server:~# echo 'kernel.panic = 20' >> /etc/sysctl.conf

Now we will also have to enable the variable to start being use on the system, so execute:

debian-server:~# sysctl -p

There you go automatic system reboots on kernel panics is now on.
Now to further assure yourself the linux server you’re responsible of will automatically restart itself on a emergency situation like a system overload I suggest you check Watchdog

You might consider checking out this auto reboot tutorial which explains in simple words how watchdog is installed and configured.

On Debian installing and maintaining watchdog is really simple and comes to installing and enabling the watchdog system service, right afteryou made two changes in it’s configuration file/etc/watchdog.conf

To do so execute:

debian-server:~# apt-get install watchdog
debian-server:~# echo "file = /var/log/messages" >> /etc/watchdog.conf
debian-server:~# echo "watchdog-device = /dev/watchdog" >> /etc/watchdog.conf

Well that should be it, you might also need to load some kernel module to monitor your watchdog.

On my system the kernel modules related to watchdog are located in:

/lib/modules/2.6.26-2-amd64/kernel/drivers/watchdog/
If not then you should certainly try the software watchdog linux kernel module called softdog , to do so issue:

debian-server:~# /sbin/modprobe softdog

It’s best if you load the module while the softdog daemon is disabled.
If you consider auto loadig the softdog software watchdog kernel driver you should exec:

debian-server:~# echo 'softdog' >> /etc/modules

That should be all your automatic system reboots should be now on! 🙂

iptables and kernel :) (for beginners)

To quote the iptables homepage

iptables is the userspace command line program used to configure the Linux 2.4.x and 2.6.x IPv4 packet filtering ruleset. It is targeted towards system administrators.

In order to run iptables on the WARP, there are two things required. It must be enabled in the kernel and you need the user space libraries from www.netfilter.org. Both of these things must be completed to get it to run.

iptables – kernel portion

IP Tables is available in our kernel. You can enable it through the ‘Target Architecture Configuration (Custom Kernel Options)’ option in the main page of menuconfig. To invoke the ‘custom kernel’ selection menu when you run ‘make’ here is a little trick.

1) go to the /build_warp/linux
2) do a ‘ls- al’ —> you should see a ‘.configured’ file – please remove this file
3) run ‘make menuconfig’ from your main PADS checkout directory > select the second item ‘Target Arch Configuration (Custom Kernel Options)’ from the menu. Select ‘Custom Kernel Options’ from the next menu. Save this configuration.
4) Upon your next ‘make’ you should be presented with a new menu (’custom kernel’) where you can select which kernel modules you would like to add.
Of interest to you will be:
> Linux Kernel Configuration
-> Networking
–> Networking Options
—> Network packet filtering framework (Netfilter)
—-> IP: Netfilter Configuration

You will also see there is many other available kernel options available however I would recommend being selective as each of this options has the potential to have undesirable consequences.

iptables – user mode

The package for the user mode libraries is available from the PIKA extra_packages SVN repository here. In order to compile this, just check it out into your packages directory in PADS and do a make iptables in the root of your PADS directory. There are sometimes issues with this cross compiled version not accepting all commands. If you run into this issue, you can try the precompiled version below.

iptables – binary – user mode

The package for the pre-compiled version of the user mode libraries is available from the PIKA extra_packages SVN repository here. In order to install this, just check it out into your packages directory in PADS and do a make iptables-binary in the root of your PADS directory.

If you need assistance with configuring iptables, I suggest you read the man pages that come with it.

——

Here is the kernel configuration suggested to me by a colleague of mine as a starting point (thanks Sean). It provides routing and firewall capabilities. Instead of selecting options through ‘menuconfig > Linux Kernel Configuration’ you can also simply modify the ‘/package/linux/linux-config’ file in PADS and rebuild.
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NF_NAT_FTP=m
CONFIG_IP_NF_MANGLE=m

Сетевой адаптер Intel ServerAdapter 1000 ET Quad Port PCIe в CentOS 5

Купили такую железку Intel ServerAdapter 1000 ET Quad Port PCIe .

Смонтировали в сервер. Задача заставить данный адаптер работать в CentOS 5, и распределить сетевую нагрузку по нескольким очередям.

Загружаемся:

Информация в dmesg


Intel(R) Gigabit Ethernet Network Driver - version 1.3.16-k2
Copyright (c) 2007-2009 Intel Corporation.
ACPI: PCI Interrupt 0000:09:00.0[A] -> GSI 17 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:09:00.0 to 64
EDAC MC0: Giving out device to i5000_edac.c I5000: DEV 0000:00:10.0
intel_rng: FWH not detected
ACPI: PCI Interrupt 0000:00:1f.3[B] -> GSI 20 (level, low) -> IRQ 162
igb 0000:09:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:09:00.0: eth2: (PCIe:2.5Gb/s:Width x4) 00:1b:21:3e:ae:28
igb 0000:09:00.0: eth2: PBA No: e64750-002
igb 0000:09:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
ACPI: PCI Interrupt 0000:09:00.1[B] -> GSI 18 (level, low) -> IRQ 106
PCI: Setting latency timer of device 0000:09:00.1 to 64
igb 0000:09:00.1: Intel(R) Gigabit Ethernet Network Connection
igb 0000:09:00.1: eth2: (PCIe:2.5Gb/s:Width x4) 00:1b:21:3e:ae:29
igb 0000:09:00.1: eth2: PBA No: e64750-002
igb 0000:09:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
ACPI: PCI Interrupt 0000:0a:00.0[A] -> GSI 19 (level, low) -> IRQ 218
PCI: Setting latency timer of device 0000:0a:00.0 to 64
igb 0000:0a:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:0a:00.0: eth3: (PCIe:2.5Gb/s:Width x4) 00:1b:21:3e:ae:2c
igb 0000:0a:00.0: eth3: PBA No: e64750-002
igb 0000:0a:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
ACPI: PCI Interrupt 0000:0a:00.1[B] -> GSI 16 (level, low) -> IRQ 177
PCI: Setting latency timer of device 0000:0a:00.1 to 64
igb 0000:0a:00.1: Intel(R) Gigabit Ethernet Network Connection
igb 0000:0a:00.1: eth2: (PCIe:2.5Gb/s:Width x4) 00:1b:21:3e:ae:2d
igb 0000:0a:00.1: eth2: PBA No: e64750-002
igb 0000:0a:00.1: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)

На первый взгляд все замечательно адаптер определился и работает, но смутили надписи 4 rx queue(s), 1 tx queue(s) на каждом из портов. Все дело в том, что у данного адаптера по 8 rx (прием) очередей на каждый порт.

Проверим версию драйвера

modinfo igb


filename: /lib/modules/2.6.18-164.6.1.el5PAE/kernel/drivers/net/igb/igb.ko
version: 1.3.16-k2
license: GPL
description: Intel(R) Gigabit Ethernet Network Driver
author: Intel Corporation,
srcversion: 78555F0A019E05BADBD95AA
alias: pci:v00008086d000010D6sv*sd*bc*sc*i*
alias: pci:v00008086d000010A9sv*sd*bc*sc*i*
alias: pci:v00008086d000010A7sv*sd*bc*sc*i*
alias: pci:v00008086d000010E8sv*sd*bc*sc*i*
alias: pci:v00008086d000010E7sv*sd*bc*sc*i*
alias: pci:v00008086d000010E6sv*sd*bc*sc*i*
alias: pci:v00008086d0000150Asv*sd*bc*sc*i*
alias: pci:v00008086d000010C9sv*sd*bc*sc*i*
depends: 8021q
vermagic: 2.6.18-164.6.1.1.el5PAE SMP mod_unload 686 REGPARM 4KSTACKS gcc-4.1
parm: max_vfs:Maximum number of virtual functions to allocate per physical function (uint)
module_sig: 883f3504af3fe359a79aca2e69819291121b4409f6ecc47545455cf3b51a9aa99f40859e7bd7931a09f76b4b34dde9013eed67638dee172193713aff51f

Очень напрягает практически полное отсутствие секции parm, т.е. драйвер не знает практически никаких параметров.

Поднимем один порт например eth2 и посмотрим как обстоят дела в /proc/interrupts
cat /proc/interrupts|grep eth2

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
122:          0          0          0          0          0          0          0          0       PCI-MSI-X  eth2-tx-0
130:        182          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-0
138:        182          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-1
146:        182          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-2
154:        182          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-3
162:          2          0          0          0          0          0          0          0       PCI-MSI-X  eth2

Попытаемся подгрузить драйвер igb с нужными нам параметрами, для того чтобы задействовать по 8 очередей на порт.

modprobe igb IntMode=3,3,3,3 RSS=8,8,8,8


FATAL: Error inserting igb (/lib/modules/2.6.18-164.6.1.1.el5PAE/kernel/drivers/net/igb/igb.ko): Unknown symbol in module, or unknown parameter (see dmesg)

в dmesg

igb: Unknown parameter `IntMode'
igb: Unknown parameter `RSS'

опс, драйвер не знает таких параметров 🙁

Надо попробовать свежую версию драйвера igb может с ним нам повезет больше, т. к. собирать драйвер вручную было категорически лень, то вспомнился репозиторий ELREPO в котором данный драйвер присутствует.

Устанавливаем:

rpm -ihv http://elrepo.org/linux/elrepo/el5/i386/RPMS/kmod-igb-PAE-2.0.6-1.el5.elrepo.i686.rpm

Загружается http://elrepo.org/linux/elrepo/el5/i386/RPMS/kmod-igb-PAE-2.0.6-1.el5.elrepo.i686.rpm
предупреждение: /var/tmp/rpm-xfer.cvXEF5: Заголовок V3 DSA signature: NOKEY, key ID baadae52
Подготовка...     ########################################### [100%]
   1:kmod-igb-PAE           ########################################### [100%]
Creating the symbolic links. This may take some time ...
Done.


modinfo igb

filename:       /lib/modules/2.6.18-164.6.1.1.el5PAE/weak-updates/igb/igb.ko
version:        2.0.6
license:        GPL
description:    Intel(R) Gigabit Ethernet Network Driver
author:         Intel Corporation, 
srcversion:     AD1D1A409C0E0945FADD6A2
alias:          pci:v00008086d000010D6sv*sd*bc*sc*i*
alias:          pci:v00008086d000010A9sv*sd*bc*sc*i*
alias:          pci:v00008086d000010A7sv*sd*bc*sc*i*
alias:          pci:v00008086d000010E8sv*sd*bc*sc*i*
alias:          pci:v00008086d0000150Dsv*sd*bc*sc*i*
alias:          pci:v00008086d000010E7sv*sd*bc*sc*i*
alias:          pci:v00008086d000010E6sv*sd*bc*sc*i*
alias:          pci:v00008086d00001518sv*sd*bc*sc*i*
alias:          pci:v00008086d0000150Asv*sd*bc*sc*i*
alias:          pci:v00008086d000010C9sv*sd*bc*sc*i*
depends:
vermagic:       2.6.18-8.el5PAE SMP mod_unload 686 REGPARM 4KSTACKS gcc-4.1
parm:           InterruptThrottleRate:Interrupt Throttling Rate (array of int)
parm:           IntMode:Interrupt Mode (array of int)
parm:           LLIPort:Low Latency Interrupt TCP Port (array of int)
parm:           LLIPush:Low Latency Interrupt on TCP Push flag (array of int)
parm:           LLISize:Low Latency Interrupt on Packet Size (array of int)
parm:           RSS:RSS - multiqueue receive count (array of int)
parm:           VMDQ:VMDQ - VMDq multiqueue receive (array of int)
parm:           QueuePairs:QueuePairs - TX/RX queue pairs for interrupt handling (array of int)
parm:           debug:Debug level (0=none, ..., 16=all) (int)

Подгружаем драйвер с нужными параметрами

modprobe igb IntMode=3,3,3,3 RSS=8,8,8,8

Проверяем

cat /proc/interrupts |grep eth2

 51:          6          0          0          0          0          0          0          0       PCI-MSI-X  eth2
 52:          5          0          0          0          0          0          0          0       PCI-MSI-X  eth2-TxRx-0
 53:         34          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-1
 54:          8          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-2
 59:          8          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-3
 60:          8          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-4
 61:         34          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-5
 62:          8          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-6
 67:          5          0          0          0          0          0          0          0       PCI-MSI-X  eth2-rx-7

Теперь на каждый порт мы имеем по 8 очередей.

Победа.

Попытаемся заNAT-ить несколько сот мегабит трафика при штатном драйвере igb и проверим нагрузку на систему.

Для сравнения вспомним данный обзор, в котором процессоры умирали от si (system interrupts) при трафике в 400 мегабит.

«Дунем» через наш адаптер 400 мегабит входящего трафика, исходящий окажется в пределах 300 мегабит.


vnstat -i eth0 -tr

609740 packets sampled in 5 seconds
Traffic average for eth0

      rx       45864.72 kB/s          61261 packets/s
      tx       29906.86 kB/s          60686 packets/s

Посмотрим количество conntrack соединений

sysctl -a|grep net.ipv4.netfilter.ip_conntrack_count

net.ipv4.netfilter.ip_conntrack_count = 272495

Посмотрим нагрузку на ядра

top

Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 68.4%id,  0.0%wa,  0.7%hi, 30.9%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni, 65.6%id,  0.0%wa,  3.3%hi, 31.1%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni, 66.7%id,  0.0%wa,  0.3%hi, 33.0%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni, 69.7%id,  0.0%wa,  0.3%hi, 30.0%si,  0.0%st
Cpu4  :  0.3%us,  0.0%sy,  0.0%ni, 65.1%id,  0.0%wa,  0.7%hi, 33.9%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni, 63.8%id,  0.0%wa,  0.7%hi, 35.5%si,  0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni, 64.7%id,  0.0%wa,  3.0%hi, 32.3%si,  0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni, 63.7%id,  0.0%wa,  0.7%hi, 35.7%si,  0.0%st
Mem:   4147676k total,   340000k used,  3807676k free,    40524k buffers
Swap:  1052248k total,        0k used,  1052248k free,   123408k cached

Как видим средняя нагрузка на CPU от system interrupts в районе 33% т.е. сервер будет в состоянии занатить 1 гигабит трафика.

http://centos.alt.ru/?p=311

Увеличиваем размер conntrack таблицы в CentOS Linux

Многие системные администраторы сталкивались с проблемой, когда количество сетевых соединений с сервером велико, происходит переполнение conntrack таблицы, из-за чего новые соединения не обрабатываются сервером.

Пример записи в логах указывающих на нехватку количества соединений:

localhost kernel: ip_conntrack: table full, dropping packet.

Увеличить размер conntrack таблицы можно через sysctl.

Размер conntrack таблицы во многих дистрибутивах составляет всего 65536 записей.

В CentOS 5 посмотреть текущее значение можно так:

sysctl -a|grep net.ipv4.netfilter.ip_conntrack_max

Увеличить значение можно через файл /etc/sysctl.conf внеся туда строку:

net.ipv4.netfilter.ip_conntrack_max = НОВОЕ_ЗНАЧЕНИЕ

и заставив систему перечитать изменения:
sysctl -p

Посмотреть сколько в данный момент записей в conntrack таблице можно так:
sysctl -a|grep net.ipv4.netfilter.ip_conntrack_count

В литературе нигде не указываются рекомендованные параметры данной переменной, мои рекомендации следующие:

Если у Вас нет нехватки оперативной памяти на сервере, то установите значение переменой net.ipv4.netfilter.ip_conntrack_max в 1 миллион записей, и отслеживайте значения количества соединений, постепенно уменьшая значение даной переменной до значения (Максимльное количество соединений) +30%.

Будьте осторожны, установив заведомо большое количество соединений при малом количестве оперативной памяти в сервере вы рискуете получить в логах сообщения ядра о нехватке оперативной памяти типа:

localhost kernel: Out of Memory:

и получить в итоге неуправляемую систему.

P.S.В дистрибутивах с новыми ядрами (>2.6.20) параметры задающие максимальное количество записей в conntrack таблице называются
net.netfilter.nf_conntrack_max и net.nf_conntrack_max

Исчерпывающая информация о conntrack находится здесь

http://centos.alt.ru/?p=89