C2750D4I stability problems

[freex]

Im getting a lot of crashes / kernal panics with this board
They are abit random:
e.g. cpu not syncronized, double fault ...

Im trying to figure out if its a hardware issue or a software issue.
Hope that someone can point me in the right direction.

The biggest pointer is this error in the IPMI, but not sure what it means.
System Reconfigured,OEM System Boot Event,Undetermined System Hardware Failure,Entry Added to Auxiliary Log,PEF Action - Asserted

The setup is the following
Motherboard: Asrock C2750D4I (Firmware 0.14.0 Bios 1.80)
Ram: Kingston HyperX blu DDR3-1600 DC - 16GB (2x8G none ecc)
PSU: 300W Gold (included with Fractal-Design-Array-R2-Mini-ITX)
Kingston SSDNow V300 SSD - 60GB (on intel controller)
Western Digital WD Red - 4TB (on marvell controller)

Using the following software:
Os: Debian testing (stable does not include nic drivers, debian testing is usaly stable)
Samba fileshare.
NFS Fileshare.
Delunge Torrent server.Serviio DLNA server.

In order to find the error i have tryed the following:
I have done an 8 hour memory test without finding any issues.

I have tryed different new linux kernals (currently 3.14.0-rc6)
It seems that new kernals are less unstable but none i have tryed it stable enough for normal use.
It seems i get most kernal panics when running Delunge turrent server.


I rly hope someone can help me with this.

[freex]

Re: C2750D4I stability problems

Here is a littel update

I Have tryed the following to debug it future.
Updated to new linux kernel rc7 (guess not related at all)
Disabled intel speedstep
Moved harddrive from marvell controller to intel.
tryed to disabel AES-NI (some other forum post said it might be related)
Updated nic driver (igb to newest from intel)
did a cpu stress test (no crash)

It seems to run stable as long as its idle.
when ever i start my torrent program or use samba active it crashes

i don't get a kernel panic anymore but it freezes and the only thing i can do is to ctrl + alt + del to reboot

now i get the same error everytime, no more random panics so i guess something has helped.


i get this error

Mar 19 19:46:40 store kernel: [ 4489.921925] ------------[ cut here ]------------
Mar 19 19:46:40 store kernel: [ 4489.921944] WARNING: CPU: 6 PID: 0 at net/sched/sch_generic.c:264 dev_watchdog+0x226/0x230()
Mar 19 19:46:40 store kernel: [ 4489.921947] NETDEV WATCHDOG: eth0 (igb): transmit queue 0 timed out
Mar 19 19:46:40 store kernel: [ 4489.921949] Modules linked in: igb(O) nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc nct6775 hwmon_vid joydev hid_generic usbhid hid coretemp iTCO_wdt kvm iTCO_vendor_support crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel snd_pcm aesni_intel snd_timer aes_x86_64 snd lrw gf128mul soundcore glue_helper ast ablk_helper cryptd evdev lpc_ich pcspkr ttm mfd_core i2c_i801 drm_kms_helper drm i2c_ismt shpchp ipmi_si ipmi_msghandler tpm_tis tpm button processor thermal_sys ext4 crc16 mbcache jbd2 sg sd_mod crc_t10dif crct10dif_common ehci_pci ehci_hcd usbcore usb_common ahci libahci i2c_algo_bit i2c_core dca libata ptp pps_core scsi_mod [last unloaded: igb]
Mar 19 19:46:40 store kernel: [ 4489.922024] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G O 3.14.0-rc7-freex #1
Mar 19 19:46:40 store kernel: [ 4489.922026] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./C2750D4I, BIOS P1.80 01/14/2014
Mar 19 19:46:40 store kernel: [ 4489.922028] 0000000000000009 ffffffff814acc2e ffff88047fd83e20 ffffffff8105eed2
Mar 19 19:46:40 store kernel: [ 4489.922034] 0000000000000000 ffff88047fd83e70 0000000000000010 0000000000000006
Mar 19 19:46:40 store kernel: [ 4489.922038] ffff880467d2c000 ffffffff8105ef37 ffffffff817472a0 ffff880400000030
Mar 19 19:46:40 store kernel: [ 4489.922042] Call Trace:
Mar 19 19:46:40 store kernel: [ 4489.922044] <IRQ> [<ffffffff814acc2e>] ? dump_stack+0x41/0x51
Mar 19 19:46:40 store kernel: [ 4489.922056] [<ffffffff8105eed2>] ? warn_slowpath_common+0x72/0x90
Mar 19 19:46:40 store kernel: [ 4489.922059] [<ffffffff8105ef37>] ? warn_slowpath_fmt+0x47/0x50
Mar 19 19:46:40 store kernel: [ 4489.922064] [<ffffffff813e7866>] ? dev_watchdog+0x226/0x230
Mar 19 19:46:40 store kernel: [ 4489.922067] [<ffffffff813e7640>] ? dev_graft_qdisc+0x70/0x70
Mar 19 19:46:40 store kernel: [ 4489.922071] [<ffffffff81069b2c>] ? call_timer_fn+0x2c/0x100
Mar 19 19:46:40 store kernel: [ 4489.922075] [<ffffffff813e7640>] ? dev_graft_qdisc+0x70/0x70
Mar 19 19:46:40 store kernel: [ 4489.922079] [<ffffffff8106a9f9>] ? run_timer_softirq+0x1f9/0x2b0
Mar 19 19:46:40 store kernel: [ 4489.922083] [<ffffffff81063e3a>] ? __do_softirq+0xfa/0x2a0
Mar 19 19:46:40 store kernel: [ 4489.922087] [<ffffffff81064205>] ? irq_exit+0x95/0xa0
Mar 19 19:46:40 store kernel: [ 4489.922091] [<ffffffff810423eb>] ? smp_apic_timer_interrupt+0x3b/0x50
Mar 19 19:46:40 store kernel: [ 4489.922095] [<ffffffff814bb15d>] ? apic_timer_interrupt+0x6d/0x80
Mar 19 19:46:40 store kernel: [ 4489.922097] <EOI> [<ffffffff813938fd>] ? cpuidle_enter_state+0x4d/0xc0
Mar 19 19:46:40 store kernel: [ 4489.922106] [<ffffffff81393a19>] ? cpuidle_idle_call+0xa9/0x1d0
Mar 19 19:46:40 store kernel: [ 4489.922111] [<ffffffff8101c695>] ? arch_cpu_idle+0x5/0x30
Mar 19 19:46:40 store kernel: [ 4489.922116] [<ffffffff810acf45>] ? cpu_startup_entry+0x95/0x230
Mar 19 19:46:40 store kernel: [ 4489.922119] ---[ end trace 4f40e87b2b5695e5 ]---


anyone got an idea?

[LeWaN]

Re: C2750D4I stability problems

Here is a littel update

I Have tryed the following to debug it future.
Updated to new linux kernel rc7 (guess not related at all)
Disabled intel speedstep
Moved harddrive from marvell controller to intel.
tryed to disabel AES-NI (some other forum post said it might be related)
Updated nic driver (igb to newest from intel)
did a cpu stress test (no crash)

It seems to run stable as long as its idle.
when ever i start my torrent program or use samba active it crashes

i don't get a kernel panic anymore but it freezes and the only thing i can do is to ctrl + alt + del to reboot

now i get the same error everytime, no more random panics so i guess something has helped.


i get this error

Mar 19 19:46:40 store kernel: [ 4489.921925] ------------[ cut here ]------------
Mar 19 19:46:40 store kernel: [ 4489.921944] WARNING: CPU: 6 PID: 0 at net/sched/sch_generic.c:264 dev_watchdog+0x226/0x230()
Mar 19 19:46:40 store kernel: [ 4489.921947] NETDEV WATCHDOG: eth0 (igb): transmit queue 0 timed out
Mar 19 19:46:40 store kernel: [ 4489.921949] Modules linked in: igb(O) nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc nct6775 hwmon_vid joydev hid_generic usbhid hid coretemp iTCO_wdt kvm iTCO_vendor_support crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel snd_pcm aesni_intel snd_timer aes_x86_64 snd lrw gf128mul soundcore glue_helper ast ablk_helper cryptd evdev lpc_ich pcspkr ttm mfd_core i2c_i801 drm_kms_helper drm i2c_ismt shpchp ipmi_si ipmi_msghandler tpm_tis tpm button processor thermal_sys ext4 crc16 mbcache jbd2 sg sd_mod crc_t10dif crct10dif_common ehci_pci ehci_hcd usbcore usb_common ahci libahci i2c_algo_bit i2c_core dca libata ptp pps_core scsi_mod [last unloaded: igb]
Mar 19 19:46:40 store kernel: [ 4489.922024] CPU: 6 PID: 0 Comm: swapper/6 Tainted: G O 3.14.0-rc7-freex #1
Mar 19 19:46:40 store kernel: [ 4489.922026] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./C2750D4I, BIOS P1.80 01/14/2014
Mar 19 19:46:40 store kernel: [ 4489.922028] 0000000000000009 ffffffff814acc2e ffff88047fd83e20 ffffffff8105eed2
Mar 19 19:46:40 store kernel: [ 4489.922034] 0000000000000000 ffff88047fd83e70 0000000000000010 0000000000000006
Mar 19 19:46:40 store kernel: [ 4489.922038] ffff880467d2c000 ffffffff8105ef37 ffffffff817472a0 ffff880400000030
Mar 19 19:46:40 store kernel: [ 4489.922042] Call Trace:
Mar 19 19:46:40 store kernel: [ 4489.922044] <irq> [<ffffffff814acc2e>] ? dump_stack+0x41/0x51
Mar 19 19:46:40 store kernel: [ 4489.922056] [<ffffffff8105eed2>] ? warn_slowpath_common+0x72/0x90
Mar 19 19:46:40 store kernel: [ 4489.922059] [<ffffffff8105ef37>] ? warn_slowpath_fmt+0x47/0x50
Mar 19 19:46:40 store kernel: [ 4489.922064] [<ffffffff813e7866>] ? dev_watchdog+0x226/0x230
Mar 19 19:46:40 store kernel: [ 4489.922067] [<ffffffff813e7640>] ? dev_graft_qdisc+0x70/0x70
Mar 19 19:46:40 store kernel: [ 4489.922071] [<ffffffff81069b2c>] ? call_timer_fn+0x2c/0x100
Mar 19 19:46:40 store kernel: [ 4489.922075] [<ffffffff813e7640>] ? dev_graft_qdisc+0x70/0x70
Mar 19 19:46:40 store kernel: [ 4489.922079] [<ffffffff8106a9f9>] ? run_timer_softirq+0x1f9/0x2b0
Mar 19 19:46:40 store kernel: [ 4489.922083] [<ffffffff81063e3a>] ? __do_softirq+0xfa/0x2a0
Mar 19 19:46:40 store kernel: [ 4489.922087] [<ffffffff81064205>] ? irq_exit+0x95/0xa0
Mar 19 19:46:40 store kernel: [ 4489.922091] [<ffffffff810423eb>] ? smp_apic_timer_interrupt+0x3b/0x50
Mar 19 19:46:40 store kernel: [ 4489.922095] [<ffffffff814bb15d>] ? apic_timer_interrupt+0x6d/0x80
Mar 19 19:46:40 store kernel: [ 4489.922097] <eoi> [<ffffffff813938fd>] ? cpuidle_enter_state+0x4d/0xc0
Mar 19 19:46:40 store kernel: [ 4489.922106] [<ffffffff81393a19>] ? cpuidle_idle_call+0xa9/0x1d0
Mar 19 19:46:40 store kernel: [ 4489.922111] [<ffffffff8101c695>] ? arch_cpu_idle+0x5/0x30
Mar 19 19:46:40 store kernel: [ 4489.922116] [<ffffffff810acf45>] ? cpu_startup_entry+0x95/0x230
Mar 19 19:46:40 store kernel: [ 4489.922119] ---[ end trace 4f40e87b2b5695e5 ]---


anyone got an idea?

No solution to the problem....


But I had also thought about running Debian, but never got it to work.
Had the same error that you have, what I remember.


I have now closed all extra stuff for cpu, turbo etc.


Installed Ubuntu 13.10, and currently has an uptime of 38 days.
Thinking of installing Ubuntu 14.04 LTS when it is released.


But really, I want to run Debian!!! :)
</ffffffff810acf45></ffffffff8101c695></ffffffff81393a19></ffffffff813938fd></eoi></ffffffff814bb15d></ffffffff810423eb></ffffffff81064205></ffffffff81063e3a></ffffffff8106a9f9></ffffffff813e7640></ffffffff81069b2c></ffffffff813e7640></ffffffff813e7866></ffffffff8105ef37></ffffffff8105eed2></ffffffff814acc2e></irq>

[slykens]

Re: C2750D4I stability problems

No solution to the problem....

I've been running Ubuntu 13.10 on a C2550D4I for several months and had *LOTS* of stability problems which I believe are tied back to the BMC.

When I first got it up and running it would crash with Machine Check Exception errors which led me to setting up watchdog and utilizing the feature.

This helped to mitigate the problems but it didn't fix them. I applied the BMC and BIOS updates available from Asrock which, again, helped with stability but still has not solved the problem.

Now I can get uptimes to 5-6 days but this inevitably ends with a crash and reboot or a complete power off for no reason. Tonight I had a reboot and am left with a non-responsive BMC which will likely require a hard power cycle to recover from.

I thought about RMAing the board but I believe it to be a software problem in the BMC at this point and highly doubt a replacement board would make me any happier.

For $280 I was hoping for something more reliable. Fortunately, it is only being used a home media server and I can mitigate it's problems for now. I absolutely would not put this into a business production environment.

[freex]

Re: C2750D4I stability problems

Ubuntu share the same error for me (tcp hang)
so i set out to try windows server 2012 r2
in windows i get a lot of the same warnings/error and random reboots



A corrected hardware error has occurred.


Component: PCI Express Root Port
Error Source: Advanced Error Reporting (PCI Express)


Bus:Device:Function: 0x0:0x3:0x0
Vendor ID:Device ID: 0x8086:0x1F12
Class Code: 0x30400


The details view of this entry contains further information.




Detail


+ System


- Provider


[ Name] Microsoft-Windows-WHEA-Logger
[ Guid] {C26C4F3C-3F66-4E99-8F8A-39405CFED220}

EventID 17

Version 0

Level 3

Task 0

Opcode 0

Keywords 0x8000000000000000

- TimeCreated


[ SystemTime] 2014-03-29T08:10:55.887659700Z

EventRecordID 79562

- Correlation


[ ActivityID] {A3A3A7EB-9258-4237-8FDE-F46E92CC6B71}

- Execution


[ ProcessID] 924
[ ThreadID] 2820

Channel System

Computer WIN-V41U3ASELER

- Security


[ UserID] S-1-5-19



- EventData


ErrorSource 4
FRUId {00000000-0000-0000-0000-000000000000}
FRUText
ValidBits 0xdf
PortType 4
Version 0x101
Command 0x10
Status 0x407
Bus 0x0
Device 0x3
Function 0x0
Segment 0x0
SecondaryBus 0x0
Slot 0x0
VendorID 0x8086
DeviceID 0x1f12
ClassCode 0x30400
DeviceSerialNumber 0x0
BridgeControl 0x0
BridgeStatus 0x0
UncorrectableErrorStatus 0x0
CorrectableErrorStatus 0x1000
HeaderLog 0100004A040000070800000000000000
Length 672
RawData 435045521002FFFFFFFF02000200000002000000A002000036 0A08001D030E14000000000000000000000000000000000000 0000000000000000000000000000BDC407CF89B7184EB3C41F 732CB571311FC093CF161AFC4DB8BC9C4DAF67C104845A3A7F 254BCF01000000004552000000000000000000000000000000 00000010010000D0000000010200000100000054E995D9C1BB 0F43AD91B44DCB3C6F35000000000000000000000000000000 00020000000000000000000000000000000000000000000000 E0010000C00000000102000000000000ADCC7698B447DB4BB6 5E16F193C4F3DB000000000000000000000000000000000300 00000000000000000000000000000000000000000000DF0000 0000000000040000000101000010000704000000008680121F 00040300030000000000000000000000000000000000000010 804201208000000721010042487903010042F000FD1000C003 40000F00000000000000B70300001000000000000000000000 00000000000000000001000100000000000000000010200600 0010000000000000000000000100004A040000070800000000 00000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000 000000000043010000000000000002000000000000D8060400 00000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000 000000000000000E0000000000000000000000000000000000 00000000000000000000000000000000000000000000

[Jenus]

Re: C2750D4I stability problems

I also had trouble with a lot of 0x124 error crasches the first three days with this board. Around 4-5 chrashes a day. I'm running Windows Server 2012 R2.
I did a bit of googling and saw a lot of Haswell users with 0x124 crashes. One of the suggestions was to disable Intel C States in the bios. I did that yesterday morning and it has been running for two days now without a single crash.
So it might be something to try. Please report back to this thread with the results.

EDIT: Just want to update and say that disabling C States definitely worked for me. Have an uptime of one week now and no problems at all.

[freex]

Re: C2750D4I stability problems

#6 Thanks for the hint
i was very optimistic when i read it, cus it was one thing i havent tryed yet.
but im afraid it reacts the same way with cstate disabled in bios.

i do have a case going with the dealer now, they have been very helpfull so far.
i will report my finding when i have news.

[nitecoder]

Re: C2750D4I stability problems

I'm also having stability issues with my new build based on this board.
The behavior is very odd. This is a FreeNAS build. It appears to work just fine for a while. Then at some point both non-IPMI network interfaces and the console (regular VGA) become unresponsive. The console continues to show whatever it was showing, but no new output appears. USB keyboard seems to have no effect.

However, the following things continue to work:
- if FreeNAS was performing a background operation (such as a disk scrub or a file copy) it continues to run and eventually completes
- disk activity light on the case flashes indicating the same
- power button on the case works properly (single press causes an orderly shutdown of FreeNAS with the power going off after the normal shutdown period of time)

IMPI port seems to work, but
(1) only one of my two routers was able to work with it, whereas before I was able to use both routers just fine,
(2) I am unable to login into IPMI interface. I'm pretty sure I know what I set the password to. Yet it is not accepted and I get Invalid Authentication response

This crash seems to be associated with heavy disk activity, but not always.
When I moved one of my drives from the Marvel to the Intel controller it seemed to help. But didn't totally eliminate the problem.

I have a few leads from this thread to try, but I just wanted to say that I'm having what appears to be a related problem.

[phin]

Re: C2750D4I stability problems

Not exactly helpful, but I am having exactly the same lockup issues (IPMI console frozen, actual console frozen as well) running FreeNAS 9.2.1 on a new build. I thought it RAM at first but after a bevy of tests the error is just as random. I managed to disable bonding in the IPMI console and now I cannot access the box in any way. Whoops.

EDIT:

Firmware 0.14.0 Bios 1.80
Crucial 16GB Kit (8GBx2) DDR3L 1600MT/s (PC3-12800) DR x8 ECC UDIMM 240-Pin Server Memory CT2KIT102472BD160B

[nitecoder]

Re: C2750D4I stability problems

Hi phin
Thanks for reply. Currious, what do you use for testing of this box? Esp. ECC memory? One burn-in tool i tried said it didnt have support for ECC memory for this board.

[larch]

Re: C2750D4I stability problems

I also have stability problems with this board.

Same setup as phin

Firmware 0.14.0 Bios 1.80
Crucial 16GB Kit (8GBx2) DDR3L 1600MT/s (PC3-12800) DR x8 ECC UDIMM

So far i have tried ubuntu server 12.04 / 14.0 and Centos 6.5.

When i first got it i setup KvM and started to build some VM's I then spent around 2 weeks messing about with various server core 2012 installs and installing SQL server blahblahblah. No issues.

So now i need to backup my VMDKs. Started to copy the vmdk went off to watch tv etc... 30 mins later still copying. So i booted webmin and it times out. Check the BMC logs and i get this:

FRB2/Hang in POST Faliure - Asserted

Long story short its completely hung. Thus begins my testing of the failures.

I can reproduce this crash within 5-120 seconds every single time its easy.

Mount an nfs share and then simply copy a large file from it to the local hdd.

mount x.x.x.x:/export/nas nas/
cp nas/somefile.ext somefile.ext

Starts copying then just stops.

I have noticed 2 things:

Ubuntu dies on its ass. It is a complete hang, console hangs, keyboard unresponsive etc...

CentOS is a different story on cent I start the copy via SSH and my direct login (i.e. logged into the machine itself) is still responsive i can press keys blinking cursor but... commands do nothing. CTRL+ALT+DEL does work but it hangs mid shutdown with:

unable to umount nas share is busy (i am paraphrasing here cant remember the EXACT wording).

I haven't tried applying the bios again just in case.
I refuse point blank to switch of cstate cpu options etc... because quite frankly i did not purchase partial use of the board and those aren't my problem to solve.
I CAN copy the vmdk's as many times as i want to and from the local HDD. In fact i created 350 gb of these files just for ****s and giggles.

So in my random testings i encountered the following error which it appears is the cause of my problems (i dont suppose it is all problems - it is mine though - and you wont get this error in ubuntu it just folds like a pack of cards).

igb 0000:07:00.0 Detected Tx Unit Hang
igb 0000:07:00.0 Eth0 Adapter Reset

I am convinced the problem is the i210 drivers in the linux kernels.

Next stop is to install server 2012 (non R2) and the drivers from the asrock website. This is tonights mission but since i have to spend time with the wife and the baby probably more likely to be the weekend :)

[larch]

Re: C2750D4I stability problems

So Server 2012 (non r2) installed network cards do not automatically work so you need the drivers from ASRocks website. With these i managed to copy 20gb or so before the network went down, interestingly server 2012 did not hang it just lost the network.

I flashed the bios with 1.8 again.

Same as above.

I updated the drivers to the latest april ones from intels website.

Same as above.

One thing of note from windows i am being spammed in the event log (every second) by:

Component: PCI Express Root Port
Error Source: Advanced Error Reporting (PCI Express)

Bus:Device:Function: 0x0:0x3:0x0
Vendor ID:Device ID: 0x8086:0x1f12
Class Code: 0x30400

[nitecoder]

Re: C2750D4I stability problems

I ended up returning the board because of these problems. You may want to try testing with memtest86. It runs without any OS so it is a much cleaner test. In my case it locked up within a few minutes of running very reliably.

[larch]

Re: C2750D4I stability problems

12 hours memtest 86 no errors.

Caving in and disabling speedstep i was able to copy 350gb worth of vmdk files (the same 50gb file 7 times) to and from my nas. Did this 3 times for over 1 tb and never had an issue.

Reenabled speedstep got the crashing back within 2 gb

[larch]

Re: C2750D4I stability problems

So with speedstep and c-states off this board is 100% stable except...


# dd if=/dev/zero of=nas/downloads/testfile.bin bs=1M count=1000 conv=fsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 15.028 s, 69.8 MB/s

dd if=/dev/zero of=/tmp/testfile.bin bs=1M count=1000 conv=fsync
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 7.42082 s, 141 MB/s

dd if=nas/downloads/testfile.bin of=/dev/null bs=1M
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 108.679 s, 9.6 MB/s



Basically the 'nas' i am copying from is the box i want to replace (old e8400 with some caviar greens - reds were not available 4/5 years ago).

I have tried :

CentOS with and without ElRepo i210 drivers
Ubuntu 12 and 14
Winblows 2012 and r2

All have the except same issue. Incoming network speed is terrible sometimes i can see it climb to 700 even 900 mbits via nload but it kinda bounces around then drops to 80 again.

Outgoing network speed is exactly as i expect 990 mbits and consistent.

I have tried this in my routers gb switch (rt-ac68u) an old dlink web smart gb switch and my current procurve 1810-24g v2 (the v2 is gb version).