[USRP-users] UHD error: EnvironmentError: IOError - packet parse error

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[USRP-users] UHD error: EnvironmentError: IOError - packet parse error

Martin Braun via USRP-users
Greetings,

I'm experiencing an error in communicating with several X310s over 10 GbE. I noticed this problem in June after first acquiring this system, but didn't have time to look into it until now. This was a clean Ubuntu 16.04 system then, and I used the build-gnuradio script to build the (then) latest versions of GNU Radio and UHD. Since UHD was recently updated, I decided to upgrade using build-gnuradio today (after sudo apt-get update/upgrade and rebooting), and see if the problem still existed. It does, and here are the details:

xxx@yyy:~$ uname -a
Linux yyy 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Two X310s are connected to this system, one at eno1, with IP address 192.168.41.2, and one at eno2 with IP address 192.168.40.2.

xxx@yyy:~$ ifconfig
eno1      Link encap:Ethernet  HWaddr f8:bc:12:0d:c6:80  
          inet addr:192.168.41.1  Bcast:192.168.41.255  Mask:255.255.255.0
          inet6 addr: fe80::fabc:12ff:fe0d:c680/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:83849 errors:0 dropped:0 overruns:0 frame:0
          TX packets:82605 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5450370 (5.4 MB)  TX bytes:21073419 (21.0 MB)

eno2      Link encap:Ethernet  HWaddr f8:bc:12:0d:c6:82  
          inet addr:192.168.40.1  Bcast:192.168.40.255  Mask:255.255.255.0
          inet6 addr: fe80::fabc:12ff:fe0d:c682/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:91971 errors:0 dropped:0 overruns:0 frame:0
          TX packets:90789 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:6182926 (6.1 MB)  TX bytes:21787924 (21.7 MB)

<snip>

eno4      Link encap:Ethernet  HWaddr f8:bc:12:0d:c6:a1  
          inet addr:192.168.10.1  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::fabc:12ff:fe0d:c6a1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2412 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1131 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:144796 (144.7 KB)  TX bytes:69117 (69.1 KB)
          Memory:a8200000-a82fffff 


For the 10 GbE interfaces (eno1 and eno2) the problem can be observed with uhd_usrp_probe:

xxx@yyy:~$ uhd_usrp_probe --args="addr=192.168.40.2"

linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_003.011.000.git-20-gedb363c3

-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- [DMA FIFO] Running BIST for FIFO 0... pass (Throughput: 1184.7MB/s)
-- [DMA FIFO] Running BIST for FIFO 1... pass (Throughput: 1184.0MB/s)
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 335  Received index: 327

I repeated uhd_usrp_probe a dozen times with both the X310 at 40.2 and the X310 at 41.2. The "expected packet index" varies between approximately 335 and 365, and the received index is always 327. 

eno1 and eno2 are an Intel XL710 with i40e driver version 1.4.25-k, which came with Ubuntu 16.04. This is a Dell Precision rackmount 7910; the XL710 is integrated on a daughterboard with 2x 10 GbE i40e interfaces and 2x 1 GbE igb interfaces. 

xxx@yyy:~$ dmesg | grep Intel
<snip>
[    2.255811] i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 1.4.25-k
[    2.255812] i40e: Copyright (c) 2013 - 2014 Intel Corporation.
<snip>

The error is limited to that driver/interface, though. Probing either of the X310s over the 1 GbE interface (eno4) works fine:

xxx@yyy:~$ uhd_usrp_probe --args="addr=192.168.10.2"
linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_003.011.000.git-20-gedb363c3

-- X300 initialization sequence...
-- Determining maximum frame size... 1472 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- [DMA FIFO] Running BIST for FIFO 0... pass (Throughput: 1165.0MB/s)
-- [DMA FIFO] Running BIST for FIFO 1... pass (Throughput: 1181.4MB/s)
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
-- Performing timer loopback test... pass
-- Performing timer loopback test... pass
  _____________________________________________________
 /
|       Device: X-Series Device
|     _____________________________________________________
|    /
|   |       Mboard: X310
|   |   revision: 8
|   |   revision_compat: 7
|   |   product: 30818
|   |   mac-addr0: 00:80:2f:23:52:f6
|   |   mac-addr1: 00:80:2f:23:52:f7
|   |   gateway: 192.168.10.1
|   |   ip-addr0: 192.168.10.2
|   |   subnet0: 255.255.255.0
|   |   ip-addr1: 192.168.20.2
|   |   subnet1: 255.255.255.0
|   |   ip-addr2: 192.168.30.2
|   |   subnet2: 255.255.255.0
|   |   ip-addr3: 192.168.41.2
|   |   subnet3: 255.255.255.0
|   |   serial: 30B9A2D
|   |   FW Version: 4.0
|   |   FPGA Version: 32.0
|   |   RFNoC capable: Yes
|   |   
|   |   Time sources:  internal, external, gpsdo
|   |   Clock sources: internal, external, gpsdo
|   |   Sensors: ref_locked
|   |     _____________________________________________________
|   |    /
|   |   |       RX Dboard: A
|   |   |   ID: UBX-160 v1 (0x007a)
|   |   |   Serial: 30B8D5B
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Frontend: 0
|   |   |   |   Name: UBX RX
|   |   |   |   Antennas: TX/RX, RX2, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: IQ
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Codec: A
|   |   |   |   Name: ads62p48
|   |   |   |   Gain range digital: 0.0 to 6.0 step 0.5 dB
|   |     _____________________________________________________
|   |    /
|   |   |       RX Dboard: B
|   |   |   ID: UBX-160 v1 (0x007a)
|   |   |   Serial: 30B8D71
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Frontend: 0
|   |   |   |   Name: UBX RX
|   |   |   |   Antennas: TX/RX, RX2, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: IQ
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Codec: B
|   |   |   |   Name: ads62p48
|   |   |   |   Gain range digital: 0.0 to 6.0 step 0.5 dB
|   |     _____________________________________________________
|   |    /
|   |   |       TX Dboard: A
|   |   |   ID: UBX-160 v1 (0x0079)
|   |   |   Serial: 30B8D5B
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Frontend: 0
|   |   |   |   Name: UBX TX
|   |   |   |   Antennas: TX/RX, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: QI
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Codec: A
|   |   |   |   Name: ad9146
|   |   |   |   Gain Elements: None
|   |     _____________________________________________________
|   |    /
|   |   |       TX Dboard: B
|   |   |   ID: UBX-160 v1 (0x0079)
|   |   |   Serial: 30B8D71
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Frontend: 0
|   |   |   |   Name: UBX TX
|   |   |   |   Antennas: TX/RX, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: QI
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Codec: B
|   |   |   |   Name: ad9146
|   |   |   |   Gain Elements: None
|   |     _____________________________________________________
|   |    /
|   |   |       RFNoC blocks on this device:
|   |   |   
|   |   |   * DmaFIFO_0
|   |   |   * Radio_0
|   |   |   * Radio_1
|   |   |   * DDC_0
|   |   |   * DDC_1
|   |   |   * DUC_0
|   |   |   * DUC_1

As I said above, this error is not unique to this version of UHD, as it occurred (according to my lab notebook) similarly in June with whatever version of GNU Radio and UHD the build-gnuradio script retrieved from github at that time.

I'd be happy to provide any additional information to assist in diagnosing and fixing this issue.

Charles

_______________________________________________
USRP-users mailing list
[hidden email]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Reply | Threaded
Open this post in threaded view
|

Re: [USRP-users] UHD error: EnvironmentError: IOError - packet parse error

Martin Braun via USRP-users
On 08/15/2016 07:59 PM, Charles Dietlein via USRP-users wrote:
Greetings,

I'm experiencing an error in communicating with several X310s over 10 GbE. I noticed this problem in June after first acquiring this system, but didn't have time to look into it until now. This was a clean Ubuntu 16.04 system then, and I used the build-gnuradio script to build the (then) latest versions of GNU Radio and UHD. Since UHD was recently updated, I decided to upgrade using build-gnuradio today (after sudo apt-get update/upgrade and rebooting), and see if the problem still existed. It does, and here are the details:

xxx@yyy:~$ uname -a
Linux yyy 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Two X310s are connected to this system, one at eno1, with IP address 192.168.41.2, and one at eno2 with IP address 192.168.40.2.

xxx@yyy:~$ ifconfig
eno1      Link encap:Ethernet  HWaddr f8:bc:12:0d:c6:80  
          inet addr:192.168.41.1  Bcast:192.168.41.255  Mask:255.255.255.0
          inet6 addr: fe80::fabc:12ff:fe0d:c680/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:83849 errors:0 dropped:0 overruns:0 frame:0
          TX packets:82605 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5450370 (5.4 MB)  TX bytes:21073419 (21.0 MB)

eno2      Link encap:Ethernet  HWaddr f8:bc:12:0d:c6:82  
          inet addr:192.168.40.1  Bcast:192.168.40.255  Mask:255.255.255.0
          inet6 addr: fe80::fabc:12ff:fe0d:c682/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:91971 errors:0 dropped:0 overruns:0 frame:0
          TX packets:90789 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:6182926 (6.1 MB)  TX bytes:21787924 (21.7 MB)

<snip>

eno4      Link encap:Ethernet  HWaddr f8:bc:12:0d:c6:a1  
          inet addr:192.168.10.1  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::fabc:12ff:fe0d:c6a1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2412 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1131 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:144796 (144.7 KB)  TX bytes:69117 (69.1 KB)
          Memory:a8200000-a82fffff 


For the 10 GbE interfaces (eno1 and eno2) the problem can be observed with uhd_usrp_probe:

xxx@yyy:~$ uhd_usrp_probe --args="addr=192.168.40.2"

linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_003.011.000.git-20-gedb363c3

-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- [DMA FIFO] Running BIST for FIFO 0... pass (Throughput: 1184.7MB/s)
-- [DMA FIFO] Running BIST for FIFO 1... pass (Throughput: 1184.0MB/s)
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 335  Received index: 327

I repeated uhd_usrp_probe a dozen times with both the X310 at 40.2 and the X310 at 41.2. The "expected packet index" varies between approximately 335 and 365, and the received index is always 327. 

eno1 and eno2 are an Intel XL710 with i40e driver version 1.4.25-k, which came with Ubuntu 16.04. This is a Dell Precision rackmount 7910; the XL710 is integrated on a daughterboard with 2x 10 GbE i40e interfaces and 2x 1 GbE igb interfaces. 

xxx@yyy:~$ dmesg | grep Intel
<snip>
[    2.255811] i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 1.4.25-k
[    2.255812] i40e: Copyright (c) 2013 - 2014 Intel Corporation.
<snip>

The error is limited to that driver/interface, though. Probing either of the X310s over the 1 GbE interface (eno4) works fine:

xxx@yyy:~$ uhd_usrp_probe --args="addr=192.168.10.2"
linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_003.011.000.git-20-gedb363c3

-- X300 initialization sequence...
-- Determining maximum frame size... 1472 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- [DMA FIFO] Running BIST for FIFO 0... pass (Throughput: 1165.0MB/s)
-- [DMA FIFO] Running BIST for FIFO 1... pass (Throughput: 1181.4MB/s)
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
-- Performing timer loopback test... pass
-- Performing timer loopback test... pass
  _____________________________________________________
 /
|       Device: X-Series Device
|     _____________________________________________________
|    /
|   |       Mboard: X310
|   |   revision: 8
|   |   revision_compat: 7
|   |   product: 30818
|   |   mac-addr0: 00:80:2f:23:52:f6
|   |   mac-addr1: 00:80:2f:23:52:f7
|   |   gateway: 192.168.10.1
|   |   ip-addr0: 192.168.10.2
|   |   subnet0: 255.255.255.0
|   |   ip-addr1: 192.168.20.2
|   |   subnet1: 255.255.255.0
|   |   ip-addr2: 192.168.30.2
|   |   subnet2: 255.255.255.0
|   |   ip-addr3: 192.168.41.2
|   |   subnet3: 255.255.255.0
|   |   serial: 30B9A2D
|   |   FW Version: 4.0
|   |   FPGA Version: 32.0
|   |   RFNoC capable: Yes
|   |   
|   |   Time sources:  internal, external, gpsdo
|   |   Clock sources: internal, external, gpsdo
|   |   Sensors: ref_locked
|   |     _____________________________________________________
|   |    /
|   |   |       RX Dboard: A
|   |   |   ID: UBX-160 v1 (0x007a)
|   |   |   Serial: 30B8D5B
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Frontend: 0
|   |   |   |   Name: UBX RX
|   |   |   |   Antennas: TX/RX, RX2, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: IQ
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Codec: A
|   |   |   |   Name: ads62p48
|   |   |   |   Gain range digital: 0.0 to 6.0 step 0.5 dB
|   |     _____________________________________________________
|   |    /
|   |   |       RX Dboard: B
|   |   |   ID: UBX-160 v1 (0x007a)
|   |   |   Serial: 30B8D71
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Frontend: 0
|   |   |   |   Name: UBX RX
|   |   |   |   Antennas: TX/RX, RX2, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: IQ
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Codec: B
|   |   |   |   Name: ads62p48
|   |   |   |   Gain range digital: 0.0 to 6.0 step 0.5 dB
|   |     _____________________________________________________
|   |    /
|   |   |       TX Dboard: A
|   |   |   ID: UBX-160 v1 (0x0079)
|   |   |   Serial: 30B8D5B
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Frontend: 0
|   |   |   |   Name: UBX TX
|   |   |   |   Antennas: TX/RX, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: QI
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Codec: A
|   |   |   |   Name: ad9146
|   |   |   |   Gain Elements: None
|   |     _____________________________________________________
|   |    /
|   |   |       TX Dboard: B
|   |   |   ID: UBX-160 v1 (0x0079)
|   |   |   Serial: 30B8D71
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Frontend: 0
|   |   |   |   Name: UBX TX
|   |   |   |   Antennas: TX/RX, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: QI
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Codec: B
|   |   |   |   Name: ad9146
|   |   |   |   Gain Elements: None
|   |     _____________________________________________________
|   |    /
|   |   |       RFNoC blocks on this device:
|   |   |   
|   |   |   * DmaFIFO_0
|   |   |   * Radio_0
|   |   |   * Radio_1
|   |   |   * DDC_0
|   |   |   * DDC_1
|   |   |   * DUC_0
|   |   |   * DUC_1

As I said above, this error is not unique to this version of UHD, as it occurred (according to my lab notebook) similarly in June with whatever version of GNU Radio and UHD the build-gnuradio script retrieved from github at that time.

I'd be happy to provide any additional information to assist in diagnosing and fixing this issue.

Charles
Are you 10G over fiber, or copper?

Have you tried different SFP+ transceiver modules?


_______________________________________________
USRP-users mailing list
[hidden email]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Reply | Threaded
Open this post in threaded view
|

Re: [USRP-users] UHD error: EnvironmentError: IOError - packet parse error

Martin Braun via USRP-users
On Mon, Aug 15, 2016 at 8:12 PM, Marcus D. Leech via USRP-users
<[hidden email]> wrote:
>
> Are you 10G over fiber, or copper?
>
> Have you tried different SFP+ transceiver modules?
>
>

Marcus,

I'm using copper direct-attach cables that work well with an Intel
X710DA2 in a different workstation (also talking with X310s). These
cables also worked with an Intel X520, IIRC. I haven't tried any other
cables/modules yet.

http://www.cablesondemand.com/pcategory/91/category/SFP%2B+CBL/URvars/Catalog/Library/InfoManage/SFP+_CABLES_(DIRECT_ATTACH).htm

Charles

_______________________________________________
USRP-users mailing list
[hidden email]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Reply | Threaded
Open this post in threaded view
|

Re: [USRP-users] UHD error: EnvironmentError: IOError - packet parse error

Martin Braun via USRP-users
In reply to this post by Martin Braun via USRP-users
On 08/15/2016 07:59 PM, Charles Dietlein via USRP-users wrote:
Greetings,

I'm experiencing an error in communicating with several X310s over 10 GbE. I noticed this problem in June after first acquiring this system, but didn't have time to look into it until now. This was a clean Ubuntu 16.04 system then, and I used the build-gnuradio script to build the (then) latest versions of GNU Radio and UHD. Since UHD was recently updated, I decided to upgrade using build-gnuradio today (after sudo apt-get update/upgrade and rebooting), and see if the problem still existed. It does, and here are the details:

xxx@yyy:~$ uname -a
Linux yyy 4.4.0-22-generic #40-Ubuntu SMP Thu May 12 22:03:46 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Two X310s are connected to this system, one at eno1, with IP address 192.168.41.2, and one at eno2 with IP address 192.168.40.2.

xxx@yyy:~$ ifconfig
eno1      Link encap:Ethernet  HWaddr f8:bc:12:0d:c6:80  
          inet addr:192.168.41.1  Bcast:192.168.41.255  Mask:255.255.255.0
          inet6 addr: fe80::fabc:12ff:fe0d:c680/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:83849 errors:0 dropped:0 overruns:0 frame:0
          TX packets:82605 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5450370 (5.4 MB)  TX bytes:21073419 (21.0 MB)

eno2      Link encap:Ethernet  HWaddr f8:bc:12:0d:c6:82  
          inet addr:192.168.40.1  Bcast:192.168.40.255  Mask:255.255.255.0
          inet6 addr: fe80::fabc:12ff:fe0d:c682/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:91971 errors:0 dropped:0 overruns:0 frame:0
          TX packets:90789 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:6182926 (6.1 MB)  TX bytes:21787924 (21.7 MB)

<snip>

eno4      Link encap:Ethernet  HWaddr f8:bc:12:0d:c6:a1  
          inet addr:192.168.10.1  Bcast:192.168.10.255  Mask:255.255.255.0
          inet6 addr: fe80::fabc:12ff:fe0d:c6a1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2412 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1131 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:144796 (144.7 KB)  TX bytes:69117 (69.1 KB)
          Memory:a8200000-a82fffff 


For the 10 GbE interfaces (eno1 and eno2) the problem can be observed with uhd_usrp_probe:

xxx@yyy:~$ uhd_usrp_probe --args="addr=192.168.40.2"

linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_003.011.000.git-20-gedb363c3

-- X300 initialization sequence...
-- Determining maximum frame size... 8000 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- [DMA FIFO] Running BIST for FIFO 0... pass (Throughput: 1184.7MB/s)
-- [DMA FIFO] Running BIST for FIFO 1... pass (Throughput: 1184.0MB/s)
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
Error: EnvironmentError: IOError: [0/Radio_0] user_reg_read32() failed: EnvironmentError: IOError: [0/Radio_0] sr_read32() failed: EnvironmentError: IOError: Block ctrl (CE_01_Port_40) packet parse error - EnvironmentError: IOError: Expected packet index: 335  Received index: 327

I repeated uhd_usrp_probe a dozen times with both the X310 at 40.2 and the X310 at 41.2. The "expected packet index" varies between approximately 335 and 365, and the received index is always 327. 

eno1 and eno2 are an Intel XL710 with i40e driver version 1.4.25-k, which came with Ubuntu 16.04. This is a Dell Precision rackmount 7910; the XL710 is integrated on a daughterboard with 2x 10 GbE i40e interfaces and 2x 1 GbE igb interfaces. 

xxx@yyy:~$ dmesg | grep Intel
<snip>
[    2.255811] i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 1.4.25-k
[    2.255812] i40e: Copyright (c) 2013 - 2014 Intel Corporation.
<snip>

The error is limited to that driver/interface, though. Probing either of the X310s over the 1 GbE interface (eno4) works fine:

xxx@yyy:~$ uhd_usrp_probe --args="addr=192.168.10.2"
linux; GNU C++ version 5.4.0 20160609; Boost_105800; UHD_003.011.000.git-20-gedb363c3

-- X300 initialization sequence...
-- Determining maximum frame size... 1472 bytes.
-- Setup basic communication...
-- Loading values from EEPROM...
-- Setup RF frontend clocking...
-- Radio 1x clock:200
-- [DMA FIFO] Running BIST for FIFO 0... pass (Throughput: 1165.0MB/s)
-- [DMA FIFO] Running BIST for FIFO 1... pass (Throughput: 1181.4MB/s)
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
-- [RFNoC Radio] Performing register loopback test... pass
-- Performing timer loopback test... pass
-- Performing timer loopback test... pass
  _____________________________________________________
 /
|       Device: X-Series Device
|     _____________________________________________________
|    /
|   |       Mboard: X310
|   |   revision: 8
|   |   revision_compat: 7
|   |   product: 30818
|   |   mac-addr0: 00:80:2f:23:52:f6
|   |   mac-addr1: 00:80:2f:23:52:f7
|   |   gateway: 192.168.10.1
|   |   ip-addr0: 192.168.10.2
|   |   subnet0: 255.255.255.0
|   |   ip-addr1: 192.168.20.2
|   |   subnet1: 255.255.255.0
|   |   ip-addr2: 192.168.30.2
|   |   subnet2: 255.255.255.0
|   |   ip-addr3: 192.168.41.2
|   |   subnet3: 255.255.255.0
|   |   serial: 30B9A2D
|   |   FW Version: 4.0
|   |   FPGA Version: 32.0
|   |   RFNoC capable: Yes
|   |   
|   |   Time sources:  internal, external, gpsdo
|   |   Clock sources: internal, external, gpsdo
|   |   Sensors: ref_locked
|   |     _____________________________________________________
|   |    /
|   |   |       RX Dboard: A
|   |   |   ID: UBX-160 v1 (0x007a)
|   |   |   Serial: 30B8D5B
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Frontend: 0
|   |   |   |   Name: UBX RX
|   |   |   |   Antennas: TX/RX, RX2, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: IQ
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Codec: A
|   |   |   |   Name: ads62p48
|   |   |   |   Gain range digital: 0.0 to 6.0 step 0.5 dB
|   |     _____________________________________________________
|   |    /
|   |   |       RX Dboard: B
|   |   |   ID: UBX-160 v1 (0x007a)
|   |   |   Serial: 30B8D71
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Frontend: 0
|   |   |   |   Name: UBX RX
|   |   |   |   Antennas: TX/RX, RX2, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: IQ
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       RX Codec: B
|   |   |   |   Name: ads62p48
|   |   |   |   Gain range digital: 0.0 to 6.0 step 0.5 dB
|   |     _____________________________________________________
|   |    /
|   |   |       TX Dboard: A
|   |   |   ID: UBX-160 v1 (0x0079)
|   |   |   Serial: 30B8D5B
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Frontend: 0
|   |   |   |   Name: UBX TX
|   |   |   |   Antennas: TX/RX, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: QI
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Codec: A
|   |   |   |   Name: ad9146
|   |   |   |   Gain Elements: None
|   |     _____________________________________________________
|   |    /
|   |   |       TX Dboard: B
|   |   |   ID: UBX-160 v1 (0x0079)
|   |   |   Serial: 30B8D71
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Frontend: 0
|   |   |   |   Name: UBX TX
|   |   |   |   Antennas: TX/RX, CAL
|   |   |   |   Sensors: lo_locked
|   |   |   |   Freq range: 10.000 to 6000.000 MHz
|   |   |   |   Gain range PGA0: 0.0 to 31.5 step 0.5 dB
|   |   |   |   Bandwidth range: 160000000.0 to 160000000.0 step 0.0 Hz
|   |   |   |   Connection Type: QI
|   |   |   |   Uses LO offset: No
|   |   |     _____________________________________________________
|   |   |    /
|   |   |   |       TX Codec: B
|   |   |   |   Name: ad9146
|   |   |   |   Gain Elements: None
|   |     _____________________________________________________
|   |    /
|   |   |       RFNoC blocks on this device:
|   |   |   
|   |   |   * DmaFIFO_0
|   |   |   * Radio_0
|   |   |   * Radio_1
|   |   |   * DDC_0
|   |   |   * DDC_1
|   |   |   * DUC_0
|   |   |   * DUC_1

As I said above, this error is not unique to this version of UHD, as it occurred (according to my lab notebook) similarly in June with whatever version of GNU Radio and UHD the build-gnuradio script retrieved from github at that time.

I'd be happy to provide any additional information to assist in diagnosing and fixing this issue.

Charles
Are you running a standard FPGA image, or a customized one?




_______________________________________________
USRP-users mailing list
[hidden email]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Reply | Threaded
Open this post in threaded view
|

Re: [USRP-users] UHD error: EnvironmentError: IOError - packet parse error

Martin Braun via USRP-users
On Mon, Aug 15, 2016 at 8:55 PM, Marcus D. Leech via USRP-users
<[hidden email]> wrote:
>
> Are you running a standard FPGA image, or a customized one?
>

Marcus,

Standard. Immediately after build-gnuradio completed, I used
uhd_image_loader with type=x300.

Charles

_______________________________________________
USRP-users mailing list
[hidden email]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Reply | Threaded
Open this post in threaded view
|

Re: [USRP-users] UHD error: EnvironmentError: IOError - packet parse error

Martin Braun via USRP-users
On 08/15/2016 09:04 PM, Charles Dietlein via USRP-users wrote:

> On Mon, Aug 15, 2016 at 8:55 PM, Marcus D. Leech via USRP-users
> <[hidden email]> wrote:
>> Are you running a standard FPGA image, or a customized one?
>>
> Marcus,
>
> Standard. Immediately after build-gnuradio completed, I used
> uhd_image_loader with type=x300.
>
> Charles
>
Hmm, well this is curious indeed.  The same X3xxs work on other machines
at 10G with the same UHD?  Same Ubuntu?



_______________________________________________
USRP-users mailing list
[hidden email]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Reply | Threaded
Open this post in threaded view
|

Re: [USRP-users] UHD error: EnvironmentError: IOError - packet parse error

Martin Braun via USRP-users
On Mon, Aug 15, 2016 at 9:10 PM, Marcus D. Leech via USRP-users
<[hidden email]> wrote:

> On 08/15/2016 09:04 PM, Charles Dietlein via USRP-users wrote:
>>
>> On Mon, Aug 15, 2016 at 8:55 PM, Marcus D. Leech via USRP-users
>> <[hidden email]> wrote:
>>>
>>> Are you running a standard FPGA image, or a customized one?
>>>
>> Marcus,
>>
>> Standard. Immediately after build-gnuradio completed, I used
>> uhd_image_loader with type=x300.
>>
>> Charles
>>
> Hmm, well this is curious indeed.  The same X3xxs work on other machines at
> 10G with the same UHD?  Same Ubuntu?
>

Marcus,

No, the systems are not quite the same. Let's call this system with
the issue System 2. The other system, System 1, with X310s and an
X710-DA2, is running Ubuntu 14.04 LTS and working fine at 10G.

I'm suspicious of the default i40e driver that comes with Ubuntu
16.04. For 14.04, I (and others before me [1]) had issues with the
default i40e driver that Ubuntu included for the X710 series of NICs.
The solution was to download and use the latest version from Intel
[2]. I did that with System 1 and everything worked (IIRC it is using
i40e version 1.3.49 from December 2015, after which time I had to
freeze that system and thus haven't upgraded its X710 driver or UHD
since). Then again, System 2 is using a *later* version of the driver
than what is working in System 1 - 1.4.25-k - which one would hope
hadn't reverted to behavior observed with version 1.2.2-k [1].

But is it the NIC driver that's having issues, or UHD? I can burn the
FPGA image, *begin* the probe sequence (note in my original email that
this error occurs after connectivity is established and some tests
have completed successfully), ping the X310s, etc., so there's
certainly network functionality. Is there a fundamental difference
between the kernel's built-in drivers (indicated by the 'k' suffix)
and those manually built/installed?

Obviously I need to install the latest i40e driver, version 1.5.16,
and see if the problem disappears. However, I have not been able to
'make install' it successfully on System 2, which prompted me to start
this email inquiry.

Charles

[1] http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2015-September/015731.html
[2] https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCI-E-40-Gigabit-Ethernet-Network-Connections-under-Linux-

_______________________________________________
USRP-users mailing list
[hidden email]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Reply | Threaded
Open this post in threaded view
|

Re: [USRP-users] UHD error: EnvironmentError: IOError - packet parse error

Martin Braun via USRP-users
On 08/15/2016 09:38 PM, Charles Dietlein via USRP-users wrote:

>
>
>> Hmm, well this is curious indeed.  The same X3xxs work on other machines at
>> 10G with the same UHD?  Same Ubuntu?
>>
> Marcus,
>
> No, the systems are not quite the same. Let's call this system with
> the issue System 2. The other system, System 1, with X310s and an
> X710-DA2, is running Ubuntu 14.04 LTS and working fine at 10G.
>
> I'm suspicious of the default i40e driver that comes with Ubuntu
> 16.04. For 14.04, I (and others before me [1]) had issues with the
> default i40e driver that Ubuntu included for the X710 series of NICs.
> The solution was to download and use the latest version from Intel
> [2]. I did that with System 1 and everything worked (IIRC it is using
> i40e version 1.3.49 from December 2015, after which time I had to
> freeze that system and thus haven't upgraded its X710 driver or UHD
> since). Then again, System 2 is using a *later* version of the driver
> than what is working in System 1 - 1.4.25-k - which one would hope
> hadn't reverted to behavior observed with version 1.2.2-k [1].

>
> But is it the NIC driver that's having issues, or UHD? I can burn the
> FPGA image, *begin* the probe sequence (note in my original email that
> this error occurs after connectivity is established and some tests
> have completed successfully), ping the X310s, etc., so there's
> certainly network functionality. Is there a fundamental difference
> between the kernel's built-in drivers (indicated by the 'k' suffix)
> and those manually built/installed?
My guess is that it's a NIC-driver issue.  Since everything works fine
over 1G, and noted that this issue
   crosses multiple UHD versions on System2.


>
> Obviously I need to install the latest i40e driver, version 1.5.16,
> and see if the problem disappears. However, I have not been able to
> 'make install' it successfully on System 2, which prompted me to start
> this email inquiry.
I can't comment on which drivers to use for which OS/NIC combinations.  
it's not like UHD does anything super-special--it's
   just a pretty-ordinary application.

It's conceivable that there's a bug in the X3xx-side code that handles
the 10G link, but then why only on your 16.04 system?



_______________________________________________
USRP-users mailing list
[hidden email]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
Reply | Threaded
Open this post in threaded view
|

Re: [USRP-users] UHD error: EnvironmentError: IOError - packet parse error

Martin Braun via USRP-users
Problem resolved - it was the i40e NIC driver (version 1.4.25-k) that
ships with Ubuntu 16.04. To recap: the latest "official" i40e driver
on Intel's site [1] has been version 1.5.16 since 5/20/2016, but it
does not compile on Ubuntu 16.04 as I mentioned in my previous email.

However, the i40e SourceForge repository notes that version 1.5.19
(released 7/25/2016) fixed "a compilation error occurring on Ubuntu
16.04 and RHEL 6.8". I successfully compiled and installed the latest
version (1.5.22) [2], and now can communicate successfully with the
x310s.

Although I suggested it in my previous email, this confirms that the
i40e drivers that shipped with both Ubuntu 14.04 (1.2.2-k) and 16.04
(1.4.25-k) don't work correctly with Intel's X(L)710 NICs [3] and
Ettus X310s. The solution is still to follow the instructions in [3]
but to download the latest i40e driver from SourceForge rather than
Intel.

[1] https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCI-E-40-Gigabit-Ethernet-Network-Connections-under-Linux-
[2] https://sourceforge.net/projects/e1000/files/i40e%20stable/1.5.22/
[3] http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2015-September/015731.html

On Mon, Aug 15, 2016 at 9:49 PM, Marcus D. Leech via USRP-users
<[hidden email]> wrote:

> On 08/15/2016 09:38 PM, Charles Dietlein via USRP-users wrote:
>>
>>
>>
>>> Hmm, well this is curious indeed.  The same X3xxs work on other machines
>>> at
>>> 10G with the same UHD?  Same Ubuntu?
>>>
>> Marcus,
>>
>> No, the systems are not quite the same. Let's call this system with
>> the issue System 2. The other system, System 1, with X310s and an
>> X710-DA2, is running Ubuntu 14.04 LTS and working fine at 10G.
>>
>> I'm suspicious of the default i40e driver that comes with Ubuntu
>> 16.04. For 14.04, I (and others before me [1]) had issues with the
>> default i40e driver that Ubuntu included for the X710 series of NICs.
>> The solution was to download and use the latest version from Intel
>> [2]. I did that with System 1 and everything worked (IIRC it is using
>> i40e version 1.3.49 from December 2015, after which time I had to
>> freeze that system and thus haven't upgraded its X710 driver or UHD
>> since). Then again, System 2 is using a *later* version of the driver
>> than what is working in System 1 - 1.4.25-k - which one would hope
>> hadn't reverted to behavior observed with version 1.2.2-k [1].
>
>
>>
>> But is it the NIC driver that's having issues, or UHD? I can burn the
>> FPGA image, *begin* the probe sequence (note in my original email that
>> this error occurs after connectivity is established and some tests
>> have completed successfully), ping the X310s, etc., so there's
>> certainly network functionality. Is there a fundamental difference
>> between the kernel's built-in drivers (indicated by the 'k' suffix)
>> and those manually built/installed?
>
> My guess is that it's a NIC-driver issue.  Since everything works fine over
> 1G, and noted that this issue
>   crosses multiple UHD versions on System2.
>
>
>>
>> Obviously I need to install the latest i40e driver, version 1.5.16,
>> and see if the problem disappears. However, I have not been able to
>> 'make install' it successfully on System 2, which prompted me to start
>> this email inquiry.
>
> I can't comment on which drivers to use for which OS/NIC combinations.  it's
> not like UHD does anything super-special--it's
>   just a pretty-ordinary application.
>
> It's conceivable that there's a bug in the X3xx-side code that handles the
> 10G link, but then why only on your 16.04 system?
>
>
>>
>> Charles
>>
>> [1]
>> http://lists.ettus.com/pipermail/usrp-users_lists.ettus.com/2015-September/015731.html
>> [2]
>> https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCI-E-40-Gigabit-Ethernet-Network-Connections-under-Linux-
>>
>> _______________________________________________
>> USRP-users mailing list
>> [hidden email]
>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>
>
>
> _______________________________________________
> USRP-users mailing list
> [hidden email]
> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com

_______________________________________________
USRP-users mailing list
[hidden email]
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com